Tcl Source Code

Package managing code might in certain cases not notice the availability of a more recent version of a package X that was made available by modifying auto_path at runtime if package manager have already seen any version of package X at the scanning stage.

How to trigger:
1) We have a package named, say, "foo" somewhere accessible via $auto_path;
2) Then by some means (usually by calling [package require something]) we cause the package manager to make its run through the $auto_path scanning the packages. After that the package manager will know that the package "foo" is available.
3) Then we add another direcroty to auto_path which makes some more recent version of the same package "foo" available to the package manager. (Use case: we want to locally override a certain package in the standard package hierarchy).
4) Then [package require foo] will fetch the original version (not the latest) since the package manager won't rescan the directories listed on auto_path.

From the Tclers' chatroom:
[00:14]<jenglish> kostix - the default [package unknown] handler does keep track of auto_path adjustments -- if any pkgIndex.tcl diddles with auto_path (which, really, they shouldn't, but some do anyway...) it will scan the new directories too.

[00:18]<jenglish> ... but IIRC the default [package unknown] handler has some short-circuiting logic; it might stop scanning for pkgIndex.tcl files once it finds a package that satisfies the requirements.

The ways to overcome the problem are:
1) Add [package forget foo] before calling [package require foo];
2) Specify the exact version in [package require] to make [package unknown] be not happy with the version of "foo" it already known about.

My poposition is to make the package manager rescan the hierarchy on any change of auto_path (may be it's worth remembering the sorted list of normalized paths from auto_path and compare it with the similar list aquired from the changed auto_path to track changes which don't really change things). Changes to auto_path are rare and probably such change in behaviour won't lead to slowdowns.

P.S.
This issue is also confirmed to exist in 8.4.14 and .15

Logged In: YES 
user_id=80530
Originator: NO

since I'm short on time, here's what
I had to say on this in the chat:

dgpre 1733117 my intent is to follow up in detail in the Tracker...
[12:02]dgpbut briefly the answer is "not a bug"

[12:03]kbkHmm, "not a bug" perhaps but at least surprising.

[12:04]dgpprobably the most surprising thing...
[12:04]dgp...is that [package require $p $requirement] does *not* make any guarantee to provide the latest version available
[12:05]dgpit only promises to satisfy the requirements passed in as arguments

[12:05]dgpIf some documentation somewhere says otherwise, then *that* is a bug
[12:06]kbkHmmm. OK so far.  Would an explicit requirement of the version that was on the new auto_path have found it?
[12:06]dgpso, if an app really, really, really won't be satisfies with anything other than the very latest available, they'll have to use the lower level [package] subcommands to accomplish that for themselves
[12:06]dgpI think all the tools are there.
[12:06]kbk(That militates in favour of aku's practice of advancing patch levels in tcllib modules with every bugfix)
[12:08]dgpif the modified auto_path makes findable a new version of a package...
[12:08]dgpthat was not findable before...
[12:08]dgp...then a requirement that is only satisfied by that new version, and was not satisfied by the older ones available
[12:08]dgpwill find the new one, yes.

[12:10]kbkok, I'm almost satisfied... although "latest available version that satisfies requirements" would make a lot of sense.

[12:10]kbkOtherwise, consider an app that does [package require foo 1.0; package require bar 1.0]

[12:11]kbkwhere (properly unknown to the app writer), foo does [package require grill 1.1] and bar does [package require grill 1.2].
[12:11]dgpWithout thinking it fully through, I think a custom [package unknown] handler can do this

[12:17]dgpso quickly, how a cusom [package unknown] can solve the posed problem....
[12:17]dgpstart with [::tclPkgUnknown] which searches according to the value of ::auto_path...
[12:18]dgpaugment it with a variable trace on ::auto_path that calls [package forget] when ::auto_path is written to.
[12:18]dgpthen register this modified package finder with [package unknown]
[12:18]dgpand voila, your app has the feature requested.

jenglish-- I think all you need is a way to say "scan this directory for available packages _now_".

[12:21]jenglish([lappend auto_path ...] just says "scan this directory for packages the next time somebody asks for a package you don't already know about", which is error-prone and order-dependent)

[12:22]dgp[lappend auto_path ...; eval [package unknown] Tcl [package provide Tcl]]

[12:22]dgpThat's not 100%, but it works with [tclPkgUnknown]
[12:23]dgpcan't be 100% because a cusom [package unknown] handler is free to provide anything that satisfies requirements even if that means ignoring later available versions.

[12:24]jenglishwait, I thought the package [package unknown] handler was only responsible for providing ifneeded scripts - not for providing the packages themselves.

[12:25]dgp[package unknown] is for finding packages.
[12:25]dgpwhen [package require] determines it can't satisfy requirement it knows about.

[12:25]dgpThe default [package unknown], [tclPkgUnknown] goes searching for every version of every package it can find.
[12:26]dgp...but other custom ones don't have to do that.

[12:26]dgpThey get passed the package and requirements being sought.
[12:26]jenglishSo the contract for [package unknown] handlers is: either provide the package, or register an ifneeded script that will provide the package?
[12:26]dgpand are free to search less broadly with the aim to satisfy the stated requirement.
[12:26]dgpFor example, the Tcl Module [package unknown] routine
[12:27]dgpno [package unknown] never [package provide]s a package
[12:27]dgp(assuming proper, nonabusive, usage)

jenglishTFM says: "For example, if [the package-unknown handler] is 'foo bar' [...] then Tcl will execute the command 'foo bar test 2.4' *to load the package*"  (emphasis mine)


[12:32]dgpthat would probably be better rephrased.
[12:32]dgpthe task for [foo bar test 2.4] to accomplish is to run at least one [package ifneeded foo $x] command
[12:32]dgpwhere [package vsatisfies 2.4 $x] is true
[12:33]jenglishRelated: calling [package forget] when auto_path changes might not be a good idea -- you don't want to forget previous [package provide]s, just the [package ifneeded]s.
[12:33]dgpor to run no [package ifneeded] command at all, in the cases where no satisfaction can be had.
[12:33]dgpyou would pass the package name to [package forget]
[12:34]dgpthere are details to get right, you are correct.
[12:34]dgpI think all the tools are available to accomplish that.
[12:35]dgpwhen I get back to full time, perhaps I can offer a complet example

Leaving this open since there's doc improvement likely to be useful.

Logged In: YES 
user_id=1350198
Originator: YES

Correction: I don't mean "rescan packages when auto_path changes", I mean "Invalidate the cache of known packages when auto_path changes" so that any subsequent [package require] (or whatever) make the package manager to rescan the list of packages.

File Added - 232193: pkgtest.zip