Tcl Source Code

Some test constraints are handled incorrectly, letting tests being run when they should not and conversely. This behaviour appears to depend on certain characters being present in the name of the constraint. The attached Tcl script "testconstraints.tcl" exercises the incorrect behaviour for two cases.

Expected behaviour: test foo is skipped and test bar is run.
Observed behaviour: test foo is run and test bar is skipped.

> cat testconstraints.tcl
#! /bin/sh
# If executed as a shell script, the next line replaces the shell with wish \
exec tclsh "$0" ${1+"$@"}

puts "tcltest version: [package require tcltest]"
namespace import tcltest::*
tcltest::configure -verbose {pass error}

testConstraint tcl>=11.1 [expr {$tcl_version >= 11.1}]
testConstraint abc-def-ghi 1

# This test ought be skipped
test foo {foo} -constraints {tcl>=11.1} -body {
        # no need to do anything here
} -result ""

# This test ought be run
test bar {bar} -constraints {abc-def-ghi} -body {
        # no need to do anything here
} -result ""

tcltest::cleanupTests
> tclsh8.5 testconstraints.tcl
tcltest version: 2.4.1
++++ foo PASSED
testconstraints.tcl:    Total   2       Passed  1       Skipped 1       Failed  0
Number of tests skipped for each constraint:
        1       abc-def-ghi
> 


The attached patch corrects the behaviour:

> tclsh8.5 testconstraints.tcl
tcltest version: 2.4.1
++++ bar PASSED
testconstraints.tcl:    Total   2       Passed  1       Skipped 1       Failed  0
Number of tests skipped for each constraint:
        1       tcl>=11.1
> 


Sincerely,
Erik Leunissen
--

B.t.w. can anything be said about the planning for tcltest 3? On what does it depend? Any 2.5 or 2.6 on the horizon?

I'd be happy to propvide a patch that removes the "support for constraint expressions" entirely, when the moment/opportunity is there.

I agree with removing "expression support" completely.

In fact, I considered removing it completely in my patch. The reason I didn't is that support for expressions is mentioned in the docs. However, the part that I stripped in the patch is not what I'd call "expression support" because the targetted constraint names are not proper expressions: [expr] will error out on them. I'd rather call it "support for combinations of constraints", which is not documented as such. Therefore, I considered that specific misfeature a bug.

Regardless, both misfeatures (expressions and combinations of constraints) interfere with the regular handling of constraint names, and I'd be very happy to see them go. The functionality that these misfeatures target, can easily (ought?) be implemented outside the tcltest framework.

Re-opened as a feature request.  I see your patch is just to rip out half of the expression support.  My preferences is when they go, they all go.

I'd expect the patch to conflict most often with test scripts that use both 'foo' and '!foo' as constraint expressions.

The constraint names you are trying to use conflict with the support for constraint expressions

https://www.tcl.tk/man/tcl/TclCmd/tcltest.htm#M50

Constraint expressions are an over-ambitious, fragile, bad idea, but they are there.

I'm happy to see them not continue in tcltest 3.

That said, I'll take a look at your patch idea.