Tcl Source Code

This relatively simple patch to Tcl 8.4.4 adds a new
"lexpand" command to the core for expanding lists "in
place" during evaluation.  It is suggested as an
alternative to the proposed {} prefix syntax convention
described by TIP #144.  "lexpand" was chosen instead of
"expand" because Unix systems tend to have an "expand"
command, which might be more likely to cause
silent-but-erroneous behavior runing scripts using the
new command in older versions of Tcl (this way it's
more likely to give an "unknown command" error than do
something incorrect with the arguments).  Usage:

lexpand ?list ...?

The patch adds a new Tcl_Obj type that is just a
renamed list object for purposes of flagging the object
as requiring expansion. The command takes zero or more
arguments and returns a list object consisting of those
arguments but with the new object type.

TclEvalObjvInternal() is modified to recognize this
type and reconstruct the objv array, expanding any
elements of the new type into separate arguments before
looking up and calling the command.

lexpand accepts multiple arguments so that it can be
nested for multiple levels of expansion.  e.g.

command [lexpand [lexpand $args]]

It can also be used to execute list variables directly
(similar to eval):

[expand $script]

It cannot, of course, be used with "set" unless the
expansion results in a single element result:

set x [list a b c d e]
set y [expand $x]

...will result in an error because it is equivalent to...

set y a b c d e

Likewise, you cannot return [expand $list], unless
$list matches the allowed options, return codes, etc.
for return.  In other words, I guess you could say that
the new Tcl_Obj type is not a first-class object, which
is essentially the intent of expansion in place.

One caveat: no assumption is made in this version of
the patch as to whether empty list elements should be
discarded.  That is...

[expand $script]

..is not the same as...

eval $script

... if $script contains empty strings or lists.  e.g.

set script [list puts ""]
[expand $script]  =>  puts ""
eval $script => puts

However, it could be with very minor changes (just
build the result element-by-element in
Tcl_LexpandObjCmd() rather than calling
Tcl_NewListObj() with the full set of arguments). 
Perhaps the behavior could be made an option.  But I
wasn't certain which behavior people expect/desire.

My hope is that this will make the {} syntax
unnecessary, but I believe that it would be extremely
easy to support both forms if desired or use this code
as the basis for {} if (unfortunately) that syntax is
chosen over [lexpand].

Oh, and please check to make sure there aren't any
reference counting issues! :)

Logged In: YES 
user_id=79902

Closing this as invalid because there was a large consensus
(in KBK's poll on tcl-core) that the [lexpand] approach to
this problem was not the right way to do it.

File Added - 58284: lexpandv2.diff

Logged In: YES 
user_id=498198

lexpandv2.diff replaces lexpand.diff, correcting problems
handling multiple arguments (due to an incomplete merge from
my 8.4.3 original to 8.4.4) and with handling of various
invocations involving empty lists that could crash due to
Tcl's creating empty list objects as string objects instead.
 ie., the following test cases are fixed:

cmd [lexpand]
cmd [lexpand [list]]
[lexpand]
[lexpand [list]]
[lexpand [list [list]]]
cmd [lexpand a b c d e]

This does not address full BCC compatibility yet.

Logged In: YES 
user_id=148712

The only point which is relevant here is: this is not a Tcl
command in the current sense, it implies a syntax change. 
Whether that is good or bad, better or worse than other
alternatives should be discussed elsewhere.
Sorry, I started the discussion in these terms, didn't I? My
bad: TCLCORE, chat and wiki is already too many places ...

The problem with the [set] example is that
   set x [whatever arg1 $argVal ...]
would never raise a "too many args" error in [set], *except*
if "whatever" happens to be "lexpand". This illustrates
nicely that [lexpand] does not behave like a Tcl command.
Note that you cannot write a proc or a C-command that raises
that error. 

Adopting this patch implies modifying the Endekalogue, hence
its approval requires a TIP. Therefore, I will wait till the
dust settles to evaluate the patch - I can't do that
properly know, as the decision as to what is acceptable is
not in my hands. An evaluation today would have to be
"rejected, violates Tcl's syntax".

Fair enough?

Logged In: YES 
user_id=498198

> Making that example work within the current syntax would
> require [set] to expand its arguments, thus causing errors
> in thing like
>     set x [list 1 2 3]
> Alternatively, [set] would need to know more about how each
> word argument was generated (is it the resut of an
> [lexpand]?)

But it is by design that "set x [lexpand ...]" will cause an
error.  Do you expect if TIP #144's syntax is adopted that
any of these will succeed?

set x {}[...]
set x {}$y

How about these, under current circumstances?

eval set x [...]
eval set x $y

I very much doubt it!  What do these all have in common?  In
every case you're telling Tcl to expand either the value of
y or the result of [...] into multiple arguments before
passing them to the set command.  All four of the above
behave exactly the same (modulo needing to fix the bytecode
issue) as:

set x [lexpand [...]]
set x [lexpand $y]

(As for whether you want to say that alters Tcl's syntax, or
as I see it that it alters the substitution rules a little
but does not introduce new syntax, is largely matter of
semantics.  I do not see it as altering the syntax because
the parser remains unchanged, unlike the {} prefix proposal.
 [lexpand] is just [list] with a different object type, so
it *is* just a command; it's just that this new object type
is treated specially during evaluation.  Whether {} prefix
or [lexpand] is adopted, however, either way the Endekalogue
will have to be updated, so that's not in its self a logical
reason to choose one over the other.)

I think it would be a really bad idea for *either* the {}
prefix or the [lexpand] command to cause that set to
succeed, because that would amount to *two* changes to the
Endekalogue, or at least inconsistent behavior: "if ... is
used, the value is expanded into multiple arguments before
passing to the command.  oh, unless that command is set." 
Why would you want that, especially when that's not the
behavior expected of eval set ...?

Logged In: YES 
user_id=148712

This discussion is being actively pursued in other forums,
but I'll register my caveats here for the record and defer
any resolution until the syntax discussion is settled.
It is definitely not true that this command does not alter
Tcl's syntax; it does, and IMHO in a very perverse manner.
This is shown by the author's remark that
   set y [lexpand $x]
results in an error, a violation of rules 6, 10 and/or 11 in
the endekalogue, which mandate that the resultof a [..]
substitution is a single word and that substitution is
single-pass.
Making that example work within the current syntax would
require [set] to expand its arguments, thus causing errors
in thing like
    set x [list 1 2 3]
Alternatively, [set] would need to know more about how each
word argument was generated (is it the resut of an
[lexpand]?) to treat it accordingly; this violates rule 2.
Or else there is a second substitution pass just for the
results of [lexpand], violating rule 10

In any case, this proposal does introduce new syntax;
[lexpand] is not a Tcl command in the usual sense.

Logged In: YES 
user_id=498198

This appears to just be due to the fact that certain core
bytecode compiled commands don't go through
TclEvalObjvInternal() when executed.  Probably not difficult
to fix (at least not significantly different than what would
need to be done for the {} prefix to work).  I'll look into
it soon, though I'm not sure the TCT is even considering it
(don't see anything on the list - just new-syntax versus
eval-like-replacement a la TIP#103, where this is neither.).

Logged In: YES 
user_id=68433

This works fine when invoked at top-level:

% set l [list x y z] 
% set x [list a b c [lexpand $l] d e f]
==> a b c x y z d e f

but not inside a proc:

proc test {} {
    set l [list x y z] 
    set x [list a b c [lexpand $l] d e f]
    return $x
}
test
==> a b c {{x y z}} d e f

Logged In: YES 
user_id=498198

Oops!  My caveat is wrong.  These are actually equivalent
after all:

[expand $script]
eval $script

File Added - 57623: lexpand.diffs