Tcl Source Code

want to enable the rewriting of tclhttpd's templating system using coroutines, but this in turn requires [subst] to be NRE-enabled

allow_comments - 1

Fixed in the HEAD.

There are still internals cleanup,
and new interface exposure tasks
to do, but in terms of the bug reported,
it is now fixed.

Still in the pipeline is an NR-enabled version
of Tcl_SubstObj().  Will TIP when ready.

Appears to work in informal testing; added a (very simple) test to coroutine.test but leaving open for more tests. Dropping prio, as we appear now to have enough to support the key templating use-case.

committed a few additional fixes to the HEAD.
Thanks to aku for the rapid testing.

Patch committed.  Ticket still open since the patch
is only a partial solution.

New patch for HEAD includes tests and fixes
for all the TIP 280 issues I could find.

File Added - 341785: 2314561.patch

File Deleted - 338805:

File Deleted - 341382:

rebased patch to the HEAD.

File Added - 341382: 2314561.patch

File Deleted - 338799:

Thanks for the tips on TIP 280 matters. Will
look into them.

If the additional instructions are a large problem,
[subst] bytecode can be achieved using the existing
INST_JUMP_TABLE instruction, I believe, but
not nearly as well.  I also expect the new
INST_RETURN_CODE_BRANCH instruction
will be useful creating bytecode for the [try]
command.

Attached some notes on the patch.

File Added - 338805: notes_2314561.patch.txt

Attached patch implements a bytecode compiler
for [subst], which solves this problem for the
cases covered by it since bytecode execution
is already NR-enabled.  This is a partial solution.

File Added - 338799: 2314561.patch

Yes, that's along the lines of what I am thinking.

That's why I'm working on 2823282 first which
has a similar nature.

This does sound a bit like the template could be compiled to bytecode. Just a sequence of
  push literal
  push variable value
  run a command
and a 'concat' of all the pieces. Either a big one at the end, or incrementally, i.e. interspersed with all the push ops.

I.e. instead of bytecoding [subst] in itself have the subst implementation use the compiler to process the template (if needed), then just run it.

A bit deeper explanation about why this matters.

Tclhttpd is used a fair bit to handle custom application-on-the-www scenarios, and the most common way of using it that way is to write the pages in TMML, which involves the use of [subst]. Something that's often desired is a way to stop the processing of a page for a while until some other processing (e.g., running a subprocess) is finished; right now, with tclhttpd they're doing this by using [vwait] recursively (inside another procedure call). If we could [yield] during [subst]-processing, then it would be possible to change the template processing library so that tclhttpd/tmml doesn't have this nasty recursive-vwait case without having to rewrite lots of existing pages that are already in production.

I've thought a bit about what it would take to make [subst] more efficient in the past. Possibilities included directly bytecoding it, but almost all substantive (and interesting) cases use a non-constant template. Hence it's probably good enough to convert the template to something with an internal rep that has three sub-components: ordinary string, variable substitution, command substitution (all modulo the options). It would then be possible to just concatenate these bits appropriately (and quickly) to make the result, and doing so in an NRE-aware way would be straight-forward too (only affects command substitutions as var traces aren't NRE-aware).

The parser is sufficiently complex that the conversion to the above-mentioned sequence form is beyond what I can do in a small amount of effort, hence dropping it on your lap, Don. :-)

Also noting that I'm specifically talking about doing [subst], and not Tcl_SubstObj

Colin's been hitting this; upping the prio...

I've found that with many other things you have to add a parallel internal NRE-enabled version of a function. Doing so for Tcl_SubstObj is not surprising.

ok, I've been confused.

NRE-enabling [subst] seems feasible.

Doing so while having it continue
to call Tcl_SubstObj() is, of course,
impossible.

At a higher level, what I want is to be able to [yield] inside the processing of a command-substitution of a [subst] (assuming the outer code is in a coroutine). That would allow templates to suspend their processing to allow other events to be processed without having problems with reentrancy (classically, tclhttpd templates just use [vwait]!).

One way to tackle this would be to take the parsed token sequence and to issue bytecode for it, just as if it was a "quote"-delimited word. Another would be to make some kind of NRE-enabled "accumulate the substitutions for these tokens" function (no, I don't know which).

Argh! Tcl_SubstObj is now rather more complex than it used to be (well, on the surface anyway) so it isn't clear how to restructure it to be NRE enabled. Hence it's a job for either the master of parsing or the master of NRE...