develooper Front page | perl.perl5.porters | Postings from March 2023

Re: strictness and "use 5.whatever"

Thread Previous | Thread Next
Zefram via perl5-porters
March 11, 2023 07:32
Re: strictness and "use 5.whatever"
Message ID:
Paul "LeoNerd" Evans wrote:
>  I can see (and overall like) the central change around op.c's lines
>  7902-7906; that seems to be prettymuch what I had in mind to do, yes.
>  I'd be happy to extract that little part out and apply it.

I'd better explain the relationship between the four patches.  The patches
perform four thematically related but logically independent changes,
of which any subset could be meaningfully accepted.  However there are
textual dependencies between the patches.  If you accept a subset that
amounts to an initial subsequence of the train of four patches then
no dependency issue arises; to accept a subset other than that would
require a bit of editing, particularly of the documentation.

There are other things that I'd be interested in doing in the same
vicinity, such as replacing the feature-bundle bits in $^H with pure
use of the individual feature bits (which semantically live in %^H
but are rapidly accessible via cop_features).  But that's an invisible
implementation change, which can wait.  The changes in this patch set are
specifically the user-visible changes that it would be advantageous to get
into 5.38.  (With contentious changes freeze already past, I appreciate
that you contending most of the changes pretty much rules that out.)

On the subject of freeze dates: the first patch, which changes the meaning
of "use v5.37" and the prospective meaning of "use v5.38", and the third
patch, reversing the deprecation, are "user-visible changes to correctly
functioning programs", so are subject to that freeze that's coming up
in a few days.  The other two patches, one purely documentation and the
other adding a new facility to, do not fall under that rubric.

>  consider making the cut-off version 5.35, rather than 5.37, so that
>  perl release 5.36.0 continues to have its current behaviour.

The cutoff being 5.37 is precisely what achieves "use v5.36" retaining
its current meaning.  Try it out on an actual 5.36.0, actual 5.37.9, and
blead with my proposed patches: on all three, "use v5.36" performs soft
stricture enablement.  Setting the cutoff to 5.35 would mean changing
the effect of "use v5.36" from the soft stricture enablement that it
currently has to the same hard stricture enablement that this patch
gives to "use v5.37".

>  particular I don't see why any of the changes in are needed
>  here.

That's the implementation of what I said upthread about the explainability
of the current behaviour of "use v5.16" et al.  The motivation for that
change is to provide invocations that are precisely equivalent
to the stricture aspect of version declarations lower than v5.37.
The change goes a bit further, to provide full explicit control over
the lexical state that already exists, including the ability to cancel
the effects of all prior stricture declarations (which can be useful
around eval).

>  Additionally, I see that PL_prevailing_version has been removed,
>  because it was only being used to support the version downgrade
>  ratchet warning.

Not only because that was its only use, but also because it's
intrinsically a bad idea.  I quite agreed with your comment at
<> that it's a slippery slope
that provides the temptation to make semantics depend directly on it.

The `prevailing version' is additional lexical state, and therefore
an additional thing that programmers need to know exists and need to
manipulate.  It isn't documented it in its full form; only the effect of
changing it from >=v5.11 to <v5.11 is documented, which could have been
achieved with just a one-bit flag.  So the full PL_prevailing_version
is sitting there, available to XS and providing obvious temptation,
but unknown to programmers who don't read the source.  Even if properly
documented, though, it's a problem.  Being available, it will inevitably
get used, and all sorts of version thresholds will proliferate.

You express concern that a version declaration should be equivalent
to some series of more explicit pragmata.  Well, PL_prevailing_version
directly prevents there being such an equivalence.  "use v5.34" doesn't
just do the version check and the feature thing and the stricture thing,
it also sets PL_prevailing_version to v5.34.  If one wants to replicate
its lexical effects, the feature and stricture pragmata aren't enough
(even with a pragma that has the matching effect on strictures), one
must also set PL_prevailing_version itself, and the only supported way
to do that is "use v5.34".

>                   We'd still verymuch like to keep that warning in
>  place

Any form of the downgrade ratchet is a bad idea.  There's nothing
intrinsically wrong with having an island of old-style code in a
generally new-style environment.  It's no more wrong than having
an island of new-style code in a generally old-style environment.
(It would be more coherent, though still a bad idea, to prohibit *any*
shadowing of version declaration by version declaration.)

>where "reset a bunch of stuff" means turning back off all the
>strict/warnings/feature bundles,

The resetting from a version declaration has never had any effect of
turning off warnings.  The turning-off-stricture effect (for versions
<v5.11) has also never been a full reset that would override explicit
stricture enablement.  It has only gone as far as soft disablement,
cancelling the soft enablement from a higher version declaration.

>                 These resets all work because

What does it mean for these resets to "work"?  I find this to be a
slippery concept.  The broad goal seems to be to enable a chunk of code
written for one version bundle to be embedded in code written for a
different version bundle, the intent being that the version declaration
should be sufficient for the inner code to be interpreted correctly.
But the details are tricky.  Because the state that can be manipulated
by version declarations can mostly also be manipulated by other pragmata,
there are many interactions to consider.  And because we want some kinds
of lexical state to persist through a version declaration, the reset
can't be entirely to a fixed lexical state.

>                                       because strict flags, warning
>flags, and feature flags are all implemented as dedicated bits in some
>special unique lexically-scoped set of storage in the interpreter,

A relevant criterion, which I think you're hunting for here, is whether
version declarations always set these bits of lexical state.  If every
version declaration sets the same state then you get a simple mental
model that avoids surprises.  What you seem to be getting at with
"special unique lexically-scoped set of storage" is the idea that it's
a well-defined portion of state that everyone understands belongs to
version declarations.

A big complication is that, for this mental model to work for the
programmer, "always" has to apply not just to every version declaration
as implemented by the current version, but also to version declarations
as historically implemented.  This doesn't require that all the state
in question exist as a settable thing in all historical Perl versions.
What it rquires is that, as soon as a new bit of state became settable
at all, every version declaration had the effect of setting it.
(Also, for basic stability, all version declarations for versions
that predate it being settable must set it to the state matching the
historical behaviour.)  So an additional criterion for your "special
unique lexically-scoped set of storage" is that it's new enough that
it's always been under the control of version declarations.

(The two complicating factors that I pointed out above, regarding how
to interpret "these resets all work", correspond to two ways that the
criterion I've just developed could in theory have been easily satisfied.
If the state set by version declarations couldn't be manipulated in any
other way then that would obviously make it a well-defined bit of state
that's easy to have version declarations always set.  And if we didn't
want any lexical state to be preserved across a version declaration then
the state being reset would be well-defined by being everything.)

The aspects of lexical state that you list don't really achieve this
ideal that "the reset works", precisely because they are not always
set by version declarations.  They're all more complicated than that.
Warning state has only ever been set by version declarations >=5.35.
Stricture is more complicated: on current Perls version declarations can
be viewed as always setting a hidden `soft' stricture state, but taking a
view that encompasses historical Perl versions it's not that consistent.
Prior to Perl 5.15.6 some version declarations set the hard stricture
state and some didn't touch stricture at all.  Feature flags are the
closest to achiving the ideal, the newer ones being always effectively
set, but version declarations <5.9.5 didn't always clear the "say"
feature, and version declarations <5.11 didn't always clear the
"unicode_strings" feature.  (Both of those also changed in Perl 5.15.6.)

The proposed change of meaning for the stricture aspects of historical
version declarations can be examined in terms of this criterion.
Apart from its goal of freeing up $^H bits, the aim of the proposal seems
to be to arrange for all version declarations to set the same range of
lexical state, given that declarations of sufficiently new versions are
going to set the hard stricture state.  If that's a given then, to keep
all version declarations setting the same state, it requires that they
all set the hard stricture state, which is precisely what is proposed.
Thus it's a proposal to help version declarations cleanly shadow each
other, including downgrades, and thus undermines the deprecation of
downgrading (across the stricture behaviour threshold) that was done in
its name.

But this analysis of the proposal only holds when looking only at the
behaviour of version declarations on new Perls.  When looking across
Perl versions, the proposal obviously fails to keep the affected set of
stricture state consistent, by making it inconsistent whether version
declarations affect the hard stricture state.  (For version declarations
>=5.16 this would be a totally new inconsistency; for lower version
declarations there was already some inconsistency when considering
running on Perls <5.15.6.)

There's some tension here between getting consistency between different
version declarations running on one Perl and consistency of a single
version declaration running on different Perl versions.  My view is
that the former hasn't historically been the case and is essentially
unachievable, whereas the latter is an important backcompat issue.
You seem to be pursuing the former at the expense of the latter.

>As part of our umbrella of "making use VERSION simpler to explain", we
>thought that if we forbid downgrading from a post-5.11 to a pre-5.11
>version (the time at which strict happens), then it becomes a bit
>easier to explain without needing to explain this reset.

I reckon 5.11 is the wrong threshold to achieve easy explanation.  Your
"explaining the reset" thing amounts to establishing the range of lexical
state that is set by a version declaration: it's easy to explain if the
range of state is the same for all involved versions.  More generally
it's easy if the inner version declaration sets a superset of the state
set by the outer version, and you're presuming that the range of state
being set will monotonically increase with increasing declared version.
But 5.11 doesn't bring in any change to the range of state being set.
Under the current semantics, both "use v5.10" and "use v5.12" set the
soft stricture state, so they fully shadow each other, so that's easy
to explain.

If declaring a new version (say 5.40) sets the hard stricture state and
declaring historical versions still only sets the soft stricture state,
then that does create a threshold at which there is a change in the
range of state being set, and that threshold is somewhere around 5.39.
It's downgrading across *that* threshold that one might want to forbid.
It wouldn't require a deprecation, because you'd only be forbidding
something that was impossible before anyway.  The proposal to change
historical version declarations to hard stricture setting is a way to
avoid creating this threshold, and thus to avoid the perceived need to
forbid downgrades (at any threshold) due to stricture behaviour.

However, any proposed change with respect to historical version
declarations fails at "making use VERSION simpler to explain".  You can't
make it simpler by making the behaviour more complicated, especially
if the new explanation needs to include a full explanation of the old
behaviour.  We don't get to fully replace the behaviour of historical
version declarations; we only get to add another Perl version's treatment
of them.  The minimum overall complexity is almost always achieved by
adding behaviour that's the same as in the preceding version.

>             At that point it makes sense for, like
>, to have version bundles. At which point, it would feel
>quite natural for `use VERSION` to activate those things too.

That's certainly not a crazy thing to do.

>                     They're just more functions in the lexical pad.

Yes.  Now you'd have a version declaration setting some bits of lexical
namespace.  That's preexisting lexical state that version declarations
have never touched before, and you're not setting all of the lexical
namespace or some delimited portion of it but a handful of arbitrary
names within it.  This certainly does pose some difficulty for deciding
the semantics of shadowing version declarations.

As a motivation for a downgrade ratchet, it has some merit with respect to
the future versions to which these bundles are attached.  The downgrades
that are a problem are those that decrease the range of lexical state
being set by a version declaration.  This logic would lead one to forbid
downgrades that decrease the set of implicit imports from builtin.
This does not rationally motivate forbidding downgrades across 5.11, let
alone all downgrades.  The downgrades that are already possible are not a
problem, and have no reason to be deprecated, with one exception: the 5.35
threshold, which changes whether a version declaration affects warnings.

However, prohibiting downgrades seems an overreaction.

The bits of lexical state that are at issue, the referents of specific
lexical subroutine names, is a type of state that is already familiar to
code written for the older Perl versions.  This same feature that makes it
able to avoid the reset and persist past a later version declaration also
makes it tractable to the inner code.  I noted above that we generally
want some kinds of lexical state to persist through a version declaration.
The bindings of names are an excellent example of state that we do want
to persist (except as specifically overridden by imports).  Whereas v5.14
code could well be surprised if it were parsed with "fc" being a keyword,
and v5.26 code may be surprised if it's parsed with "^" being a strictly
numeric operator, we don't have anything like those problems from "&true"
being bound to a particular sub.

I think the semantics should be that a version declaration leaves in
place bindings that it's not specifically overriding as part of its
bundle, even bindings that result from a prior version declaration.
This seems totally workable and consistent with the principle of least
astonishment.  Another workable possibility, less good in my opinion,
would be to try to determine which bindings came from prior version
declarations, and shadow them with some kind of null pad entry that's
equivalent to them having never been declared.  We should pick one of
these semantics, implement it, document it, and let it be.

We could, however, do with supplying some kind of pragma or other
declaration to explicitly evict bindings from the pad.  My Lexical::Var
module supplies such a facility (on the Perl versions where it works --
I'm working on an update).  This would have some value in combination
with downgrading version declarations, but more generally have value
where there's a tricky relationship between nested lexical scopes.

The same logic that says that new lexical bindings aren't a problem
for version declaration shadowing also suggest that the existing 5.35
threshold, for whether warnings are set by a version declaration,
isn't a problem.  Lexical warning state has long existed, and was
always previously preserved through version declarations.  So it's not
a problem for "use v5.34" that warnings might already be turned on, and
specifically that they might have been turned on by a prior "use v5.36".
Similarly, in Perl 5.14 it wasn't a problem that strictures persisted
through "use v5.10" and that those strictures might have been enabled
by a prior "use v5.12".  The feature enablements that used to persist
through low version declarations, though, were a bit of an issue.

>                                          To put it another way, we want
>to prevent islands of lower versions from being allowed inside a sea of
>a later version.

You're stating this as if it's the motivation, when it's actually the
conclusion that you've reached.  The problem isn't islands of lower
versions per se.  The problem is the semantics of version-implied imports
around version declaration shadowing, and you've concluded that we should
avoid having to decide on semantics by prohibiting the shadowing that
raises that issue.

>Of course, there's nothing wrong with having them the other way around.
>It's perfectly fine to bump upwards to a later version inside an island
>within a sea of a lesser version.

You're assuming here, as I brought out earlier, that higher version
declarations will always set all of the lexical state that lower version
declarations set.  In the context of implicit imports, you're assuming
that higher version declarations will always import everything that was
imported by lower version declarations.  But things like the "indirect"
feature being removed from recent feature bundles show that this won't
necessarily be the case.

What happens when we decide that some implicitly-imported builtin
sub is no longer good style, and remove it from new import bundles?
Upgrading version declarations across such a threshold raises the same
issues as downgrading across a threshold where imports were added.
Do you forbid the upgrade?  Do you have all future bundles import some
unusable dummy sub under the name of the sub that's no longer approved?

>The other reason I originally added it is because we don't want to keep
>growing more feature bits for all of time.

We have a pretty good generic treatment of feature bits.  It doesn't
seem like a problem to maintain a gradually-growing list of them.

>                                                    It may be useful at
>that point to stop tracking individual feature bits and instead just
>say that those features are always enabled if the prevailing version is
>greater than some boundary version when it first showed up.

Eww.  That would be gratuitous breakage of existing correct code.


Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About