Tcl Source Code

The long list below names 202 tests out of the tcl core test suite, that pass, if they run in the context of a full test suite run with e.g make test, but fail, if they run standalone, selected by the tcltest option -match, e.g.

./tcltest ../tests/all.tcl -match <unique_test_name>

(In practice, one would restrict the sourced test files with the help of the tcltest -file option to the one test file, that includes the test of interest, but that aside.)

In other words, the below named tests doesn't run independed from other tests in their test file.

The usual advise is: tests should be written in a way, that they run independed from other tests. This is not only a matter of taste. 

The - admitted small - impact comes into the picture, if someone tries to track down a memory leak, reported by a mem-checker (like valgrind) after a test suite run to a single test, that triggers the mem leak. 

One simple brute-force attempt for that would be, to bisec the tests in an affected test file with an appropriate -match option, look which part of the tests still show the leak, repeate the bisec on this part and so on until a single test is found, that demonstrates the problem. (Either 'by hand' on the cmd line or with a helper script.)

Having 'false' test failure and (at least in the case of tests with script error) other executed code paths depending on how you range the tests to run makes this process even more tedious, as it already is and harder to write a reliable script, to automate this task.

The following list was generaded and checked on a linux system, using 8.6.3, with just standard ./configure 

The tests not running sucessfully (either wrong result or script error in the body) when run independ from others are: 

chan-io-33.5
chan-io-52.11
cmdAH-2.6.3
cmdMZ-return-3.0
cmdMZ-return-3.1
cmdMZ-return-3.3
cmdMZ-return-3.8
cmdMZ-return-3.9
cmdMZ-return-3.10
cmdMZ-return-3.11
cmdMZ-return-3.12
cmdMZ-return-3.13
cmdMZ-return-3.14
env-2.3
env-2.4
env-3.1
env-4.1
env-4.2
env-4.3
env-4.4
env-4.5
history-8.3
interp-2.4
interp-2.10
interp-3.7
interp-3.9
interp-3.10
interp-6.4
interp-6.6
interp-7.3
interp-7.4
interp-7.5
io-33.5
io-52.11
load-4.2
load-7.4
load-8.1
load-8.3
load-8.4
namespace-old-1.10
namespace-old-1.11
namespace-old-1.12
namespace-old-1.14
namespace-old-1.15
namespace-old-1.16
namespace-old-1.18
namespace-old-1.19
namespace-old-1.21
namespace-old-1.22
namespace-old-1.23
namespace-old-1.24
namespace-old-1.25
namespace-old-1.26
namespace-old-1.27
namespace-old-2.1
namespace-old-2.2
namespace-old-2.3
namespace-old-2.4
namespace-old-2.5
namespace-old-2.6
namespace-old-2.7
namespace-old-2.8
namespace-old-2.9
namespace-old-4.4
namespace-old-5.4
namespace-old-5.5
namespace-old-5.6
namespace-old-5.11
namespace-old-5.12
namespace-old-5.13
namespace-old-5.14
namespace-old-5.15
namespace-old-5.16
namespace-old-5.17
namespace-old-5.21
namespace-old-6.2
namespace-old-6.3
namespace-old-6.4
namespace-old-6.6
namespace-old-6.7
namespace-old-6.8
namespace-old-6.9
namespace-old-6.10
namespace-old-6.11
namespace-old-6.13
namespace-old-6.14
namespace-old-6.15
namespace-old-6.16
namespace-old-6.19
namespace-old-6.21
namespace-old-7.2
namespace-old-7.3
namespace-old-7.4
namespace-old-7.5
namespace-old-7.6
namespace-old-7.7
namespace-old-7.9
namespace-old-7.10
namespace-old-7.11
namespace-old-7.12
namespace-old-9.4
namespace-old-9.5
namespace-old-9.8
namespace-old-9.9
namespace-old-9.10
namespace-old-9.11
namespace-old-9.13
namespace-old-9.14
namespace-old-9.15
namespace-old-9.16
namespace-old-9.17
namespace-old-9.18
namespace-old-9.19
namespace-old-10.7
namespace-old-10.8
namespace-5.2
namespace-9.5
namespace-9.6
namespace-10.3
namespace-11.2
namespace-11.3
namespace-13.1
namespace-14.3
namespace-14.7
namespace-14.8
namespace-14.10
namespace-14.11
namespace-16.2
namespace-16.8
namespace-17.5
namespace-17.6
namespace-17.7
namespace-19.2
namespace-19.4
namespace-21.2
namespace-21.3
namespace-21.4
namespace-21.7
namespace-25.4
namespace-25.5
namespace-26.5
namespace-26.6
namespace-26.7
namespace-29.5
namespace-29.6
namespace-34.6
namespace-34.7
parseOld-9.3
parseOld-11.8
resolver-1.5
safe-8.9
safe-8.10
safe-11.7.1
safe-11.8.1
safe-14.1
set-old-10.14
set-2.5
set-2.6
set-4.5
set-4.6
tcltest-21.4
tcltest-21.5
tcltest-21.6
tcltest-21.10
tcltest-21.11
tcltest-21.12
tcltest-24.9
tcltest-24.10
tcltest-24.13
tcltest-24.15
tcltest-24.17
tcltest-24.18
tcltest-24.20
tcltest-25.3
unload-2.3
unload-2.4
unload-2.5
unload-2.6
unload-3.3
unload-3.4
unload-3.5
unload-3.6
unload-3.7
unload-4.1
unload-4.2
unload-4.4
unload-4.5
unload-4.6
var-1.3
var-1.4
var-1.5
var-1.6
var-3.7
var-3.8
var-3.10
var-5.3
var-6.3
var-7.3
var-7.4
var-7.10
var-7.11
var-18.1

This is the current list of not independend tests as of 8931dc163c0e 2015-08-06 04:25:50 UTC. Compared to the original report: A few tests are fixed (mostly in cmdMZ.test), a few got added, mostly the status is unchanged.

chan-io-33.5
chan-io-52.11
cmdAH-2.6.3
env-2.3
env-2.4
env-3.1
env-4.1
env-4.2
env-4.3
env-4.4
env-4.5
history-8.3
http-4.15
http11-2.4
http11-2.11
http11-4.3
interp-2.4
interp-2.10
interp-3.7
interp-3.9
interp-3.10
interp-6.4
interp-6.6
interp-7.3
interp-7.4
interp-7.5
interp-14.1
interp-36.7
io-33.5
io-52.11
load-4.2
load-7.4
load-8.1
load-8.3
load-8.4
msgcat-14.1
msgcat-14.2
msgcat-14.3
msgcat-14.4
msgcat-14.5
msgcat-14.6
namespace-old-1.10
namespace-old-1.11
namespace-old-1.12
namespace-old-1.14
namespace-old-1.15
namespace-old-1.16
namespace-old-1.18
namespace-old-1.19
namespace-old-1.21
namespace-old-1.22
namespace-old-1.23
namespace-old-1.24
namespace-old-1.25
namespace-old-1.26
namespace-old-1.27
namespace-old-2.1
namespace-old-2.2
namespace-old-2.3
namespace-old-2.4
namespace-old-2.5
namespace-old-2.6
namespace-old-2.7
namespace-old-2.8
namespace-old-2.9
namespace-old-4.4
namespace-old-5.4
namespace-old-5.5
namespace-old-5.6
namespace-old-5.11
namespace-old-5.12
namespace-old-5.13
namespace-old-5.14
namespace-old-5.15
namespace-old-5.16
namespace-old-5.17
namespace-old-5.21
namespace-old-6.2
namespace-old-6.3
namespace-old-6.4
namespace-old-6.6
namespace-old-6.7
namespace-old-6.8
namespace-old-6.9
namespace-old-6.10
namespace-old-6.11
namespace-old-6.13
namespace-old-6.14
namespace-old-6.15
namespace-old-6.16
namespace-old-6.19
namespace-old-6.21
namespace-old-7.2
namespace-old-7.3
namespace-old-7.4
namespace-old-7.5
namespace-old-7.6
namespace-old-7.7
namespace-old-7.9
namespace-old-7.10
namespace-old-7.11
namespace-old-7.12
namespace-old-9.4
namespace-old-9.5
namespace-old-9.8
namespace-old-9.9
namespace-old-9.10
namespace-old-9.11
namespace-old-9.13
namespace-old-9.14
namespace-old-9.15
namespace-old-9.16
namespace-old-9.17
namespace-old-9.18
namespace-old-9.19
namespace-old-10.7
namespace-old-10.8
namespace-5.2
namespace-9.5
namespace-9.6
namespace-10.3
namespace-11.2
namespace-11.3
namespace-13.1
namespace-14.3
namespace-14.7
namespace-14.8
namespace-14.10
namespace-14.11
namespace-16.2
namespace-16.8
namespace-17.5
namespace-17.6
namespace-17.7
namespace-19.2
namespace-19.4
namespace-21.2
namespace-21.3
namespace-21.4
namespace-21.7
namespace-25.4
namespace-25.5
namespace-26.5
namespace-26.6
namespace-26.7
namespace-29.5
namespace-29.6
namespace-34.6
namespace-34.7
parseOld-9.3
parseOld-11.8
resolver-1.5
safe-8.9
safe-8.10
safe-11.7.1
safe-11.8.1
set-old-10.14
set-2.5
set-2.6
set-4.5
set-4.6
tcltest-21.4
tcltest-21.5
tcltest-21.6
tcltest-21.10
tcltest-21.11
tcltest-21.12
tcltest-24.9
tcltest-24.10
tcltest-24.13
tcltest-24.15
tcltest-24.17
tcltest-24.18
tcltest-24.20
tcltest-25.3
unload-2.3
unload-2.4
unload-2.5
unload-2.6
unload-3.3
unload-3.4
unload-3.5
unload-3.6
unload-3.7
unload-4.1
unload-4.2
unload-4.4
unload-4.5
unload-4.6
var-1.3
var-1.4
var-1.5
var-1.6
var-3.7
var-3.8
var-3.10
var-5.3
var-6.3
var-7.3
var-7.4
var-7.10
var-7.11
var-18.1

This results are collected by one of a few test suite tool scripts of mine. You'll find them hosted at

http://46.163.78.80/cgi-bin/repros/test-suite-tools/dir?ci=tip

You'll need singletests.tcl, utils.tcl and test-names.tcl. Since that url (although stable since years) may not last forever, I'll attatch them in the current state.

Or what other way of sharing around did you have in mind for such exotic stuff?

A great contribution.  Thank you!

I'm sure the "tool" to build this list is simple,
but it still might be worth sharing around.
This test suite hygiene task is one many
projects could benefit from.