Check-in [10c01a7f93]

Login
Bounty program for improvements to Tcl and certain Tcl packages.
Tcl 2019 Conference, Houston/TX, US, Nov 4-8
Send your abstracts to tclconference@googlegroups.com
or submit via the online form by Sep 9.

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:New TIP on small change to tcltest for use in CI environments
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256:10c01a7f934067107f8838244be546f05cf5238715ff2858796de0d8d63f7d81
User & Date: dkf 2018-10-24 12:50:21
Context
2018-10-24
12:52
Fix my DERP check-in: ea7c9bc6e4 user: dkf tags: trunk
12:50
New TIP on small change to tcltest for use in CI environments check-in: 10c01a7f93 user: dkf tags: trunk
2018-10-23
20:07
Explain the flags TCL_TYPE_? check-in: 9d1225715d user: jan.nijtmans tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to index.json.

1

2
3
4
5
6
7
8
...
521
522
523
524
525
526
527
528
529
{"tip": {

	"524":{"url":"./tip/524.md","keywords":"Tcl, object orientation, customization","created":"23-Oct-2018","post-history":"","state":"Draft","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 524: Custom Definition Dialects for TclOO","author":["Donal K. Fellows <donal.k.fellows@manchester.ac.uk>"],"is-jest":false},
	"523":{"url":"./tip/523.md","keywords":"queue, stack, pop","created":"22-Oct-2018","post-history":"","state":"Draft","tcl-branch":"tip-523","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 523: New lpop command","author":["Peter Spjuth <peter.spjuth@gmail.com>"],"is-jest":false},
	"522":{"url":"./tip/522.md","keywords":"tcltest, error codes","created":"22-Oct-2018","post-history":"","state":"Draft","tcl-branch":"tip-522","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 522: Test error codes with Tcltest","author":["Peter Spjuth <peter.spjuth@gmail.com>"],"is-jest":false},
	"521":{"url":"./tip/521.md","keywords":"Tcl, floating point, NaN, not a number","created":"21 October 2018","post-history":"","state":"Draft","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 521: Floating Point Classification Functions","author":["Kevin B. Kenny <kevin.b.kenny@gmail.com>"],"is-jest":false},
	"520":{"url":"./tip/520.md","keywords":"Tcl, floating point, NaN, not a number","created":"18 October 2018","post-history":"","state":"Draft","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 520: Make NaN Quiet","author":["Kevin B. Kenny <kevin.b.kenny@gmail.com>"],"is-jest":false},
	"519":{"url":"./tip/519.md","keywords":"TclOO","":"","state":"Draft","this tip proposes to extend the `method` subcommand of the `oo":":define` command","type":"Project","author":["Pietro Cerutti <gahr@gahr.ch>"],"follows":"","oo":":define RPCClient {","created":"18-Oct-2018","title":"# TIP 519: Inline export/unexport option to TclOO method definition","the class definition in the previous example would then be as follows":"","post-history":"","vote":"Pending","tcl-version":"8.7","tcl-branch":"tip-519","is-jest":false},
	"518":{"url":"./tip/518.md","keywords":"Tk","created":"22-Sep-2018","post-history":"","state":"Draft","tk-branch":"tip518-event-last-child-unmanaged","tcl-version":"8.7","vote":"Done","type":"Project","title":"# TIP 518: Virtual Event when Last Child is not Managed any more","author":["Harald Oehlmann <oehhar@users.sourceforge.net>"],"is-jest":false},
................................................................................
	"5":{"url":"./tip/5.md","created":"17-Oct-2000","post-history":"","state":"Final","vote":"Done","tcl-version":"8.4","type":"Project","title":"# TIP 5: Make TkClassProcs and TkSetClassProcs Public and Extensible","author":["Eric Melski <ericm@ajubasolutions.com>"],"is-jest":false},
	"4":{"url":"./tip/4.md","created":"26-Oct-2000","post-history":"","state":"Draft","vote":"Pending","type":"Informative","title":"# TIP 4: Tcl Release and Distribution Philosophy","discussions-to":"news:comp.lang.tcl","author":["Brent Welch <welch@acm.org>","Donal K. Fellows <fellowsd@cs.man.ac.uk>","Larry W. Virden <lvirden@cas.org>","Larry W. Virden <lvirden@yahoo.com>"],"is-jest":false},
	"3":{"url":"./tip/3.md","created":"14-Sep-2000","post-history":"","state":"Accepted","vote":"Done","type":"Process","title":"# TIP 3: TIP Format","author":["Andreas Kupries <a.kupries@westend.com>","Donal K. Fellows <fellowsd@cs.man.ac.uk>"],"is-jest":false},
	"2":{"url":"./tip/2.md","created":"12-Sep-2000","post-history":"","state":"Draft","vote":"Pending","type":"Process","title":"# TIP 2: TIP Guidelines","author":["Andreas Kupries <a.kupries@westend.com>","Donal K. Fellows <fellowsd@cs.man.ac.uk>","Don Porter <dgp@users.sourceforge.net>","Mo DeJong <no@spam.com>","Larry W. Virden <lvirden@yahoo.com>","Kevin Kenny <kennykb@acm.org>"],"is-jest":false},
	"1":{"url":"./tip/1.md","created":"14-Sep-2000","post-history":"","state":"Active","vote":"No voting","type":"Informational","title":"# TIP 1: TIP Index","author":["TIP Editor <donal.fellows@cs.man.ac.uk>"],"is-jest":false},
	"0":{"url":"./tip/0.md","created":"11-Dec-2000","post-history":"","state":"Final","vote":"Done","type":"Process","title":"# TIP 0: Tcl Core Team Basic Rules","author":["John Ousterhout <ouster@pacbell.net>"],"is-jest":false},
	"@min": 0,
	"@max": 524
}, "@timestamp": 1540309425}

>







 







|
|
1
2
3
4
5
6
7
8
9
...
522
523
524
525
526
527
528
529
530
{"tip": {
	"525":{"url":"./tip/525.md","keywords":"Tcl, testing, continuous integration","created":"24-Oct-2018","post-history":"","state":"Draft","tcl-version":"8.5","vote":"Pending","type":"Project","title":"# TIP 524: Make Tcltest Report Overall Success in a Machine-Readable Way","author":["Donal K. Fellows <donal.k.fellows@manchester.ac.uk>"],"is-jest":false},
	"524":{"url":"./tip/524.md","keywords":"Tcl, object orientation, customization","created":"23-Oct-2018","post-history":"","state":"Draft","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 524: Custom Definition Dialects for TclOO","author":["Donal K. Fellows <donal.k.fellows@manchester.ac.uk>"],"is-jest":false},
	"523":{"url":"./tip/523.md","keywords":"queue, stack, pop","created":"22-Oct-2018","post-history":"","state":"Draft","tcl-branch":"tip-523","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 523: New lpop command","author":["Peter Spjuth <peter.spjuth@gmail.com>"],"is-jest":false},
	"522":{"url":"./tip/522.md","keywords":"tcltest, error codes","created":"22-Oct-2018","post-history":"","state":"Draft","tcl-branch":"tip-522","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 522: Test error codes with Tcltest","author":["Peter Spjuth <peter.spjuth@gmail.com>"],"is-jest":false},
	"521":{"url":"./tip/521.md","keywords":"Tcl, floating point, NaN, not a number","created":"21 October 2018","post-history":"","state":"Draft","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 521: Floating Point Classification Functions","author":["Kevin B. Kenny <kevin.b.kenny@gmail.com>"],"is-jest":false},
	"520":{"url":"./tip/520.md","keywords":"Tcl, floating point, NaN, not a number","created":"18 October 2018","post-history":"","state":"Draft","tcl-version":"8.7","vote":"Pending","type":"Project","title":"# TIP 520: Make NaN Quiet","author":["Kevin B. Kenny <kevin.b.kenny@gmail.com>"],"is-jest":false},
	"519":{"url":"./tip/519.md","keywords":"TclOO","":"","state":"Draft","this tip proposes to extend the `method` subcommand of the `oo":":define` command","type":"Project","author":["Pietro Cerutti <gahr@gahr.ch>"],"follows":"","oo":":define RPCClient {","created":"18-Oct-2018","title":"# TIP 519: Inline export/unexport option to TclOO method definition","the class definition in the previous example would then be as follows":"","post-history":"","vote":"Pending","tcl-version":"8.7","tcl-branch":"tip-519","is-jest":false},
	"518":{"url":"./tip/518.md","keywords":"Tk","created":"22-Sep-2018","post-history":"","state":"Draft","tk-branch":"tip518-event-last-child-unmanaged","tcl-version":"8.7","vote":"Done","type":"Project","title":"# TIP 518: Virtual Event when Last Child is not Managed any more","author":["Harald Oehlmann <oehhar@users.sourceforge.net>"],"is-jest":false},
................................................................................
	"5":{"url":"./tip/5.md","created":"17-Oct-2000","post-history":"","state":"Final","vote":"Done","tcl-version":"8.4","type":"Project","title":"# TIP 5: Make TkClassProcs and TkSetClassProcs Public and Extensible","author":["Eric Melski <ericm@ajubasolutions.com>"],"is-jest":false},
	"4":{"url":"./tip/4.md","created":"26-Oct-2000","post-history":"","state":"Draft","vote":"Pending","type":"Informative","title":"# TIP 4: Tcl Release and Distribution Philosophy","discussions-to":"news:comp.lang.tcl","author":["Brent Welch <welch@acm.org>","Donal K. Fellows <fellowsd@cs.man.ac.uk>","Larry W. Virden <lvirden@cas.org>","Larry W. Virden <lvirden@yahoo.com>"],"is-jest":false},
	"3":{"url":"./tip/3.md","created":"14-Sep-2000","post-history":"","state":"Accepted","vote":"Done","type":"Process","title":"# TIP 3: TIP Format","author":["Andreas Kupries <a.kupries@westend.com>","Donal K. Fellows <fellowsd@cs.man.ac.uk>"],"is-jest":false},
	"2":{"url":"./tip/2.md","created":"12-Sep-2000","post-history":"","state":"Draft","vote":"Pending","type":"Process","title":"# TIP 2: TIP Guidelines","author":["Andreas Kupries <a.kupries@westend.com>","Donal K. Fellows <fellowsd@cs.man.ac.uk>","Don Porter <dgp@users.sourceforge.net>","Mo DeJong <no@spam.com>","Larry W. Virden <lvirden@yahoo.com>","Kevin Kenny <kennykb@acm.org>"],"is-jest":false},
	"1":{"url":"./tip/1.md","created":"14-Sep-2000","post-history":"","state":"Active","vote":"No voting","type":"Informational","title":"# TIP 1: TIP Index","author":["TIP Editor <donal.fellows@cs.man.ac.uk>"],"is-jest":false},
	"0":{"url":"./tip/0.md","created":"11-Dec-2000","post-history":"","state":"Final","vote":"Done","type":"Process","title":"# TIP 0: Tcl Core Team Basic Rules","author":["John Ousterhout <ouster@pacbell.net>"],"is-jest":false},
	"@min": 0,
	"@max": 525
}, "@timestamp": 1540385393}

Changes to index.md.

116
117
118
119
120
121
122








123
124
125
126
127
128
129
<th>Type</th>
<th>Tcl Version</th>
<th>Status</th>
<th>Title</th>
<th>Impl.</th>
</tr></thead><tbody>









<tr class='project projectdraft projectdraft87 project87'>
<td valign='top'><a href='./tip/524.md'>524</a></td>
<td valign='top'>Project</td>
<td valign='top'>8.7</td>
<td valign='top'>Draft</td>
<td valign='top'># TIP 524: Custom Definition Dialects for TclOO</td>
<td></td>







>
>
>
>
>
>
>
>







116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
<th>Type</th>
<th>Tcl Version</th>
<th>Status</th>
<th>Title</th>
<th>Impl.</th>
</tr></thead><tbody>

<tr class='project projectdraft projectdraft85 project85'>
<td valign='top'><a href='./tip/525.md'>525</a></td>
<td valign='top'>Project</td>
<td valign='top'>8.5</td>
<td valign='top'>Draft</td>
<td valign='top'># TIP 524: Make Tcltest Report Overall Success in a Machine-Readable Way</td>
<td></td>
</tr>
<tr class='project projectdraft projectdraft87 project87'>
<td valign='top'><a href='./tip/524.md'>524</a></td>
<td valign='top'>Project</td>
<td valign='top'>8.7</td>
<td valign='top'>Draft</td>
<td valign='top'># TIP 524: Custom Definition Dialects for TclOO</td>
<td></td>

Added tip/525.md.























































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
# TIP 524: Make Tcltest Report Overall Success in a Machine-Readable Way
	Author:         Donal K. Fellows <donal.k.fellows@manchester.ac.uk>
	State:          Draft
	Type:           Project
	Vote:           Pending
	Created:        24-Oct-2018
	Post-History:
	Keywords:       Tcl, testing, continuous integration
	Tcl-Version:    8.5
-----

# Abstract

This TIP makes it easier to use tcltest in a continuous integration
environment (such as [Travis](https://travis-ci.org/) or
[Appveyor](https://www.appveyor.com/)) by making `tcltest::runAllTests` return
a boolean indicator of success or failure.

# Rationale and Specification

The Tcl test suite is great, but when a failure happens it reports success to
its environment (by effectively doing an `exit 0`) which mans that there's no
way to tell at the wider level whether the tests passed or failed except by
manually inspecting the logs, which is a manual process that prevents the Tcl
test suite from being used in an automated environment such as a continuous
integration system.

This TIP changes this by making `tcltest::runAllTests` return a boolean
(instead of the empty string) that reports whether the executed tests all
passed or not; the boolean is a true value if any test file fails, and false
if all tests pass.

Note that because we want to use this in continuous integration environments
with all main branches for which releases could be done in the future, the
change must be applied to Tcl 8.5 and 8.6 as well as our main development
targets.

## Usage

This is sufficient to allow the execution environment to report an overall
success or failure. This is done inside Tcl's test suite by changing:

    # require packages and configure...
    tcltest::runAllTests
	exit

to:

    # require packages and configure...
	set ErrorOnFailures [info exists env(ERROR_ON_FAILURES)]
	unset -nocomplain env(ERROR_ON_FAILURES)
	if {[tcltest::runAllTests] && $ErrorOnFailures} {
	    exit 1
	}

so that existing code and tests are not affected by the change. (A system has
to set the `ERROR_ON_FAILURES` environment variable in order to get this
additional behaviour at the outer level, so only an environment that is
prepared to deal with the consequences will notice.)

# Implementation

See the branches `travis-8.5`, `travis-8.6`, `travis-8.7` and `travis-9.0`;
this TIP is specifically altering the procdure `runAllTests` to include
exactly one more line at the end:

    return [expr {[info exists testFileFailures] || [llength $failFiles]}]

This is exactly the condition for Tcltest to report a failing test file (the
existence of `testFileFailures`) or a file with a failing test (`$failFiles`
being non-empty).

# Copyright

This document has been placed in the public domain.