Lexical subroutines are a stable feature starting with v5.26
Perl v5.18 allows you to define named subroutines that exist only in the current lexical scope. These act (almost) just like the regular named subroutines that you already know about from Learning Perl, but also like the lexical variables that have limited effect. The problem is that the feature is almost irredeemably broken, which you’ll see at the end of this Item.
How it’s supposed to work
Remember how lexical variables work. You can have package (global) variables and lexical variables with the same name and they do not affect each other.
$cat = 'Buster';
print "Global scope - package cat is $cat\n";
{
my $cat = 'Mimi';
print "Lexical scope - my cat is $cat\n";
}
print "Global scope again - package cat is $cat\n";
Inside the braces, you have a variable with the same name, but it’s a completely different variable. Changing the my variable does not affect the package one:
Global scope - package cat is Buster Lexical scope - my cat is Mimi Global scope again - package cat is Buster
Perl v5.6 added the our keyword, which makes a package variable visible in a lexical scope:
$cat = 'Buster';
print "Global scope - package cat is $cat\n";
OUTER: {
my $cat = 'Mimi';
print "Lexical scope - my cat is $cat\n";
INNER: {
our $cat = 'Roscoe';
print "INNER lexical scope - our cat is $cat\n";
}
print "After INNER scope - my cat is $cat\n";
}
print "Global scope again - package cat is $cat\n";
Inside the INNER scope, you change the value of $cat, which is the package version, and you see the change at the end too. Instead of using the my variable in the enclosing scope, you get the global version back:
Global scope - package cat is Buster Lexical scope - my cat is Mimi INNER lexical scope - our cat is Roscoe After INNER scope - my cat is Mimi Global scope again - package cat is Roscoe
Also remember that the code in a subroutine you define outside of a scope cannot see the my variables:
$cat = 'Buster';
show_cat( '1' );
OUTER: {
my $cat = 'Mimi';
show_cat( '2 (my)' );
INNER: {
our $cat = 'Roscoe';
show_cat( '3 (our)' );
}
show_cat( '4 (my)' );
}
show_cat( '5' );
sub show_cat {
print "$_[0]: Cat is $cat\n";
}
In the calls to show_cat with the my labels, you still see the global version because that’s the only $cat that the subroutine can see (this is not new):
1: Cat is Buster 2 (my): Cat is Buster 3 (our): Cat is Roscoe 4 (my): Cat is Roscoe 5: Cat is Roscoe
This changes if you define show_cats in the same scope as the lexical variable. If you define it after you declare the my variable, that’s the only version the subroutine sees:
$cat = 'Buster';
show_cat( '1' );
OUTER: {
my $cat = 'Mimi';
sub show_cat {
print "$_[0]: Cat is $cat\n";
}
show_cat( '2 (my)' );
INNER: {
our $cat = 'Roscoe';
show_cat( '3 (our)' );
}
show_cat( '4 (my)' );
}
show_cat( '5' );
That show_cat bound to the my variable and stuck with that one. This is a closure that allows that lexical variable to sneak out of its scope. That last call takes place after the my variable is gone, even though the subroutine still has a reference to the data:
1: Cat is 2 (my): Cat is Mimi 3 (our): Cat is Mimi 4 (my): Cat is Mimi 5: Cat is Mimi
You see something different if you define the my variable after you define the named subroutine:
$cat = 'Buster';
show_cat( '1' );
OUTER: {
sub show_cat {
print "$_[0]: Cat is $cat\n";
}
my $cat = 'Mimi';
show_cat( '2 (my)' );
INNER: {
our $cat = 'Roscoe';
show_cat( '3 (our)' );
}
show_cat( '4 (my)' );
}
show_cat( '5' );
That version only sees the global version of the variable and you get the same output as you saw when you defined the subroutine outside of the scope:
1: Cat is Buster 2 (my): Cat is Buster 3 (our): Cat is Roscoe 4 (my): Cat is Roscoe 5: Cat is Roscoe
If you are comfortable with the variable version of this, you shouldn’t have too much trouble with the subroutine version of it. In Perl, a dynamic language, subroutines are actually data and are also variables. You can change their definitions, and I spend quite a bit of space of Mastering Perl writing about that.
Define two versions of show_cat. Prefix the definition inside the OUTER scope with my even though it’s a named subroutine:
use v5.18;
use feature qw(lexical_subs);
no warnings qw(experimental::lexical_subs);
use vars qw($cat); # use v5.18 turns on strict
$cat = 'Buster';
show_cat( '1' );
OUTER: {
my sub show_cat {
print "Inside cat! $_[0] is $cat\n";
}
my $cat = 'Mimi';
show_cat( '2 (my)' );
INNER: {
our $cat = 'Roscoe';
show_cat( '3 (our)' );
}
show_cat( '4 (my)' );
}
show_cat( '5' );
sub show_cat {
print "Outside cat! $_[0] is $cat\n";
}
The output isn’t that different except the middle three calls use the definition of show_cat you defined with my, even though it still looked at the global $cat:
Outside cat! 1 is Buster Inside cat! 2 (my) is Buster Inside cat! 3 (our) is Roscoe Inside cat! 4 (my) is Roscoe Outside cat! 5 is Roscoe
If you move the lexical subroutine definition after the my $cat, it binds to a different variable just as before:
use v5.18;
use feature qw(lexical_subs);
no warnings qw(experimental::lexical_subs);
use vars qw($cat); # use v5.18 turns on strict
$cat = 'Buster';
show_cat( '1' );
OUTER: {
my $cat = 'Mimi';
my sub show_cat {
print "Inside cat! $_[0] is $cat\n";
}
show_cat( '2 (my)' );
INNER: {
our $cat = 'Roscoe';
show_cat( '3 (our)' );
}
show_cat( '4 (my)' );
}
show_cat( '5' );
sub show_cat {
print "Outside cat! $_[0] is $cat\n";
}
Now the inner subroutine know about the my variable, just as it did without the lexical version.
Outside cat! 1 is Buster Inside cat! 2 (my) is Mimi Inside cat! 3 (our) is Mimi Inside cat! 4 (my) is Mimi Outside cat! 5 is Roscoe
That’s how it works. More importantly, though, is how it doesn’t work and if there’s any new value in this feature.
Problems
Lexical subroutines are experimental, which means they might change or even disappear (Update: They are stable features in v5.26). It also means they likely have problems. And, from the problems I found, they won’t work the same way in the future if the feature persists.
First, you can’t use a lexical subroutine with sort. This is broken in v5.18 and v5.20 (so far) but is fixed in v5.22.
my sub my_way { ... }
my @sorted = sort my_way @elements; # looks for main::my_way
This would have been one of the more interesting uses but it doesn’t really add anything that you can’t already do. If you want a private sort subroutine, a coderef works:
my $my_way = sub { ... }
my @sorted = sort $my_way @elements; # works
Next, you might immediately think of private methods. It would be quite useful to have a way to do that, but Perl uses the symbol table to resolve methods. An object will not find a lexical subroutine in the symbol table.
So far, I have found only one interesting case where this new feature might provide something that didn’t exist before. The my sub hides a subroutine. An our sub, however, can expose a subroutine in one package to all the other packages in the same lexical scope (Know what creates a scope; a package isn’t one of them). If you declare a subroutine with our sub, for the rest of the lexical scope, even across packages, that’s the subroutine you get with that name:
use v5.18;
use feature qw(lexical_subs);
no warnings qw(experimental::lexical_subs);
our sub speak { say "Logging $_[0]" }
package Cat {
sub meow {
speak( __PACKAGE__ );
}
}
package Dog {
sub woof {
speak( __PACKAGE__ );
}
}
Cat->meow;
Dog->woof;
The output isn’t surprising:
Logging Cat Logging Dog
But, what if Cat already has a speak method? In that case, it gets really weird:
use v5.18;
use feature qw(lexical_subs);
no warnings qw(experimental::lexical_subs);
our sub speak { say "Logging $_[0]" }
package Cat {
sub meow {
speak( __PACKAGE__ );
}
sub speak {
say "This is a cat speaking!";
}
}
package Dog {
sub woof {
speak( __PACKAGE__ );
}
}
Dog->woof;
speak( "At the end" );
This time, you only call Dog->woof, but you get output from a subroutine that is not in your package nor in your scope:
This is a cat speaking! This is a cat speaking!
How does Dog know anything about that subroutine? You don’t even get a warning for a redefined subroutine (although use warnings would take care of that).
You can do the same thing without the named subroutine and it works:
my $speak = sub { say "Logging $_[0]" };
package Cat {
my $speak = sub { "This is a cat speaking!" };
sub meow {
$speak->( __PACKAGE__ );
}
}
package Dog {
sub woof {
$speak->( __PACKAGE__ );
}
}
Dog->woof;
$speak->( "At the end" );
Besides the incredibly bad idea of confusing almost every programmer with the source of a subroutine, but it appears to be insanely broken before v5.22. You might like reading Is it a design flaw that Perl subs aren’t lexically scoped?.
Things to Remember
- Lexical subroutines are a broken experimental feature you shouldn’t remember.
- You can already just about everything with a coderef already.
Your examples largely seem to be expressing the confusion that lexical can introduce. I don’t think I would’ve ever thought to use private subs in the way you have outlined. It is not unthinkable that a new programmer would be confused by them, and might inadvertently construct classes employing some of the confusing ordering you’ve demonstrated.
Part of the interesting thing to me is you’ve seemed to try and show some polymorphism in your method names, without actually employing it in your object structure. It kind of feels like a mixing of paradigms which would very likely introduce confusion no matter how well something was designed.
Here is an example that I think would show how I would use lexical subs when mixed with objects.
#!/usr/bin/env perl use v5.18; use feature qw(lexical_subs); no warnings qw(experimental::lexical_subs); package Animal { sub speak { my $class = shift; say "This is a $class speaking!"; } } package Cat { use base 'Animal'; sub meow { my $class = shift; $class->speak(); } my sub private_speak { my $class = shift; say "I am a private $class"; } sub speak { my $class = shift; # like brian said, it isn't in the symbol table # and cannot be called with $class->private_speak private_speak($class); $class->SUPER::speak(); } } Cat->speak(); Cat->private_speak();Outputs:
In this view I think the lexical sub is acting very much like a correct private method in that nothing outside of the scope can see that method and can never call it directly.
Thank you for the article and the exploration of odd lexical cases.
This article would have been much different if I hadn’t run into so many problems trying to get the feature to work, which is why the examples look like they want to go in a certain direction. I kept being surprised by the odd behavior.
You don’t actually create a private method in your example. It’s just a subroutine. The method dispatch stuff isn’t invoked, so many other features will go wrong. Your other methods have to know it’s not a regular method and you have to call it differently. I don’t think that’s correct. It’s certainly not an improvement on what you can already do with coderefs.
Yes. I am not happy with the notation of having to call it as if it were a normal subroutine.
That being said, I have never liked using
my $private_speak = sub { ... };as the way of encapsulating private methods either. Usually at that point, I just write my “private” methods as normal methods and move out from there. I still document them in my pod and end up losing little sleep.
So, in bearing with your article’s title, I likely won’t use named lexical subroutines in my normal code. But for me it won’t be because of the ambiguity you very nicely laid out, but because the benefit of privatization isn’t worth the odd code construction.
I would go further and say that you should take it easy on using coderefs… I had to maintain some code by someone who liked the “clever” trick of using coderefs to get lexically scoped subs inside of subs. This was used entirely as a structuring device, and it was complete overkill– it makes it nearly impossible to write tests that run the individual coderefs, and it’s awkward to use OOP inheritence with code like this (there’s no way to override just the bit that needs to be changed, you’re stuck with the one big publically visible sub).
There are reasons the Inside Out object fad fizzled so quickly:
bullet-proof encapsulation is really overrated…
If you have private methods, a subclass isn’t supposed to know about them. You can’t override them on purpose. It’s not all about encapsulation either. But, since I haven’t seen the code you reference, I don’t know if they were overkill or not.
They never worked as private methods, but even if I ignore all that, the feature is still broken in so many other regards that my advice stands.
It seems that after defining speak as a my sub that any later sub definitions for speak are then treated lexically. That seems like the wrong behaviour to me
In the wild since external modules have a separate lexical scope I’m assuming this issue can’t manifest there. The following code works around this
use v5.18; use feature qw(lexical_subs); no warnings qw(experimental::lexical_subs); my sub speak { say "Logging $_[0]" } package Cat { our sub speak { say "This is a cat speaking"; } sub meow { __PACKAGE__->speak; } } package Dog { sub woof { speak( __PACKAGE__ ); } } Cat->meow; Dog->woof; speak("At the end");As for your last-but-one example:
> “You don’t even get a warning for a redefined subroutine.”
If you “use warnings”, you get your warning (checked in perl 5.22 and 5.26):
Subroutine main::speak redefined at cat.pl line 12.