Tcl Source Code

When I run the following test using tcl8.5 on Debian Squeeze (8.5.8-2) it works just fine, but when I try the same test using 8.6.0~b2-1 on the same PC I get the following error.

couldn't compile regular expression pattern: out of memory

I've looked to see if this bug had already been reported but I cound't find it, so apologies if I have raised a duplicate,
I've checked whether a newer version of 8.6 is available in Sid but it doens't look like it to me.

jmills@jo:~$ tclsh8.5
tclsh8.5 [~]source jotest.tcl 
tclsh8.5 [~]jotest [list TETRA_MODE_CMD ETS_2_2 CONTINUOUS 32 1 FALSE FALSE  512 8192 BOTH UNKNOWN TRUE 300 NORMAL NONE 10 1 0 25  -125 -33 720 -100 0 3 1111111111111111 110101110000  IMMEDIATE 5 15 0 0 0 FALSE 2 FALSE DISABLED]
1
tclsh8.5 [~]exit


jmills@jo:~$ tclsh8.6
tclsh8.6 [~]source jotest.tcl
tclsh8.6 [~]jotest [list TETRA_MODE_CMD ETS_2_2 CONTINUOUS 32 1 FALSE FALSE  512 8192 BOTH UNKNOWN TRUE 300 NORMAL NONE 10 1 0 25  -125 -33 720 -100 0 3 1111111111111111 110101110000  IMMEDIATE 5 15 0 0 0 FALSE 2 FALSE DISABLED]
couldn't compile regular expression pattern: out of memory
while evaluating {jotest [list TETRA_MODE_CMD ETS_2_2 CONTINUOUS 32 1 FALSE FALSE  512 8192 BOTH UNKNOWN TRUE 300 NORMAL NONE 10 1 0 25  -125 -33 720 -100 0 3 1111111111111111 110101110000  IMMEDIATE 5 15 0 0 0 FALSE 2 FALSE DISABLED]}
tclsh8.6 [~]exit

allow_comments - 1

Thanks. Now verified fixed (with a limit of 700, compile-time overridable) on trunk.

Myself? I'd not try to verify such a large string by a single regular expression; the issue is that in a failure, reporting what failed to match is difficult (the RE just says "didn't match" and not why). But that's by-the-by.

>
> Can we use that RE as part of our test suite?
>

Please feel free, strangely enough the code is from a test harness.  And thanks for the tip about using (?:[[:blank:]]+), I'll make this change.

Also, I'll expose the limit a bit better so that it can be boosted in future by defining a symbol in the makefile. We shouldn't remove the depth check entirely — that'd be an invitation to a stack smash — but we can make it tunable in 8.6.

Confirming that raising the limit to 1000 gives enough room.

Can we use that RE as part of our test suite? It might be not very nice (indeed, you'd find it more efficient to use (?:[[:blank:]]+) instead of ([[:blank:]]+) for the padding, so avoiding capturing it at all) but it does apparently stress part of the code that we didn't expect to get hit that way; it's worth something *as part of a test*. Testing of edge cases *isn't* nice in the first place!

Won't commit anything until I've got a confirmed test case in place; we want to ensure that we don't hit unexpected problems in the future too. ;-)

>
> but *dude*, the input is still morally wrong.
>
Apologies, I've started to read up on how I should be parsing such messages.

That RE is 8163 characters long. With 68 capturing sub-REs.

...
...
...

I guess we can fix this by raising the limit, especially given that Tcl no longer makes the stack bloat up hugely, but *dude*, the input is still morally wrong.

2008-07-01 14:29:11 1ab85d8c7df2ba97 BAD CURRENT
2008-07-01 13:24:06 a8de9c95aa018165 GOOD

$ fossil status
...
comment:      Add focussed stack limiting to the RE compiler. Tuning might not
              yet be right but it passes everything normally checked in the
              test suite. [Bug 1905562] (user: dkf)

File Added - 460201: jotest.tcl