Tk Source Code

Test files sourced into current interpreter
Running tests that match:  textDisp-27.7.*
Skipping test files that match:  l.*.test
Only running test files that match:  textDisp.test
Tests began at Mon Mar 20 23:45:40 CET 2017
textDisp.test
++++ textDisp-27.7.1 SKIPPED: textfonts
---- textDisp-27.7.2 start
/bin/sh: line 1: 55039 Segmentation fault: 11  ./tktest /Users/fvogel/Documents/tcltk/fossil/tk/unix/../tests/all.tcl -geometry +0+0 -file textDisp.test -match textDisp-27.7.* -verbose bepst
make[2]: *** [test-classic] Error 139
make[1]: *** [test-tk] Error 2
make: *** [test-develop] Error 2
bll-mac:tk fvogel$

I confirm that the bug is fixed and that it compiles OK on MSVC now. Thanks!

I committed [6597e03e73], I think this should compile again.

With MSVC:

d:\tcltk-fossil\tk\generic\tkBool.h(22) : error C2059: syntax error : 'string'

d:\tcltk-fossil\tk\generic\tkBitField.h(100) : error C2061: syntax error : identifier 'TkBitIsEmpty'
d:\tcltk-fossil\tk\generic\tkBitField.h(100) : error C2059: syntax error : ';'
d:\tcltk-fossil\tk\generic\tkBitField.h(100) : error C2059: syntax error : 'type'
d:\tcltk-fossil\tk\generic\tkBitField.h(104) : error C2061: syntax error : identifier 'TkBitTest'
d:\tcltk-fossil\tk\generic\tkBitField.h(104) : error C2059: syntax error : ';'
d:\tcltk-fossil\tk\generic\tkBitField.h(104) : error C2059: syntax error : 'type'
d:\tcltk-fossil\tk\generic\tkBitField.h(105) : error C2061: syntax error : identifier 'TkBitNone'
d:\tcltk-fossil\tk\generic\tkBitField.h(105) : error C2059: syntax error : ';'
d:\tcltk-fossil\tk\generic\tkBitField.h(105) : error C2059: syntax error : 'type'
d:\tcltk-fossil\tk\generic\tkBitField.h(106) : error C2061: syntax error : identifier 'TkBitAny'
d:\tcltk-fossil\tk\generic\tkBitField.h(106) : error C2059: syntax error : ';'
d:\tcltk-fossil\tk\generic\tkBitField.h(106) : error C2059: syntax error : 'type'
d:\tcltk-fossil\tk\generic\tkBitField.h(107) : error C2061: syntax error : identifier 'TkBitComplete'
[... many many more...]

Many thanks for your test. This bug was difficult to solve, congratulations.

No warnings other than the usual, and the test passes.

Based on your report I've changed the definition to _Bool (stdbool.h), this should compile without warnings, and no Mac-specific work-around is required. All places which need an int are changed from bool to int. I've committed [d349ffb168].

Nasty warnings.
However, 27.7.2 does pass now.

/Users/bll/tcl/tk/unix/../generic/tkText.h:36:9: warning: 'bool' macro redefined
      [-Wmacro-redefined]
#define bool tkbool_t /* this ugly work-around is needed for Mac */
        ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.1.0/include/stdbool.h:31:9: note: 
      previous definition is here
#define bool _Bool

So I have done the next fix [ebd71a27c4]. Now it is named "tkbool_t", and the quasi keyword "bool" is not visible anymore outside. If the pre-processor works, then sizeof(bool)==4 now must be satisfied, because it's only a macro definition for tkbool_t. The only thing is that this compiler may print a warning when used as a pointer type. If so, then I have to use a macro definition instead of a type definition for tkbool_t.

That didn't work...

'bool' is entirely too common of a name.
boolean_t is also in use in the mac header files.

tclbool_t ?

/Users/bll/tcl/tk/unix/../generic/tkTextTag.c:1966:64: warning: incompatible
      pointer types passing 'bool *' to parameter of type 'int *'
      [-Wincompatible-pointer-types]
        hPtr = Tcl_CreateHashEntry(&sharedTextPtr->tagTable, tagName, &isNew);

Assertion failed: (sizeof(bool) == sizeof(int)), function TkTextCreateTag, file /Users/bll/tcl/tk/unix/../generic/tkTextTag.c, line 1951.

Many, many thanks for your investigations. Finally it's a compiler bug: Apple's clang compiler (not the official release under Linux) is converting the following type:

typedef int bool;

into a different type with byte size one. This should not happen. As a bugfix I changed this into:

#define bool int

I've committed this change with [84b6de278f]. If this does also not work - I don't know in which way Apple has modified clang - then I will do a different fix. If Mac is printing warnings while compiling then it's likely that this #define is also not working on Mac.

Same declaration/usage problem in:
txTextWind.c, tkTextMark.c, tkTextDisp.c

Crashes with clean checkout, (tcl:core-8-6-7-rc tk:revised_text) compiled with
-g -O2 -fstack-protector-strong -fstack-protector-all

Changed declaration of isNew in txTextTag.c:1947 to an int, and it no longer crashes.

Using as the test:

text .t
.t tag configure big -font {Helvetica -24}
.t insert  end "Hello\n" big
.t insert  end "Hello\n"
pack .t

This line casts 'isNew', declared here:
        bool isNew, isSelTag;
as an int.

hPtr = Tcl_CreateHashEntry(&sharedTextPtr->tagTable, tagName, (int *) &isNew);

sizeof(bool) = 1
sizeof(int) = 4

I found a suggestion to use 'canary' values before and after the local variable declarations and that worked out well to narrow down the problem.

Please verify.  I need to try and figure out how to get a clean fossil checkout and then I will test again.

Yes, now objc magically switches from 2 to 0 when returning from TkTextCreateTag() in tktextTag.c:1103

Still the same problem, the Mac specific code (tk/macosx/) is corrupting the memory. But the option table has changed, so the corruption in this option table has changed. Still this problem needs debugging, probably now it's easier?!

Maybe Kevin Walzer can help, he is the maintainer of the Mac code.

As of current state of revised_text branch, it does no longer crash, but it spits :

bll-mac:tk fvogel$ /Users/fvogel/Documents/tcltk/fossil/build/tk/Deployment/wish
% text .t
.t
% .t tag configure big -font {Helvetica -24}
{-background {} {} {} {}} {-bgstipple {} {} {} {}} {-borderwidth {} {} 0 {}} {-elide {} {} 0 {}} {-eolcolor {} {} {} {}} {-fgstipple {} {} {} {}} {-font {} {} {} {}} {-foreground {} {} {} {}} {-hyphencolor {} {} {} {}} {-hyphenrules {} {} {} {}} {-inactivebackground {} {} {} {}} {-inactiveforeground {} {} {} {}} {-inactiveselectbackground {} {} {} {}} {-inactiveselectforeground {} {} {} {}} {-indentbackground {} {} 0 {}} {-justify {} {} {} {}} {-lang {} {} {} {}} {-lmargin1 {} {} {} {}} {-lmargin2 {} {} {} {}} {-lmargincolor {} {} {} {}} {-offset {} {} {} {}} {-overstrike {} {} {} {}} {-overstrikecolor {} {} {} {}} {-overstrikefg -overstrikecolor} {-relief {} {} {} {}} {-rmargin {} {} {} {}} {-rmargincolor {} {} {} {}} {-selectbackground {} {} {} {}} {-selectforeground {} {} {} {}} {-spacing1 {} {} {} {}} {-spacing2 {} {} {} {}} {-spacing3 {} {} {} {}} {-tabs {} {} {} {}} {-tabstyle {} {} {} {}} {-underline {} {} {} {}} {-underlinecolor {} {} {} {}} {-underlinefg -underlinecolor} {-undo {} {} 1 1} {-wrap {} {} {} {}}

That is: it apparently spits what would be the output of the configure command for a newly created tag, e.g.:

  .t tag configure foobar

I was using gcc6 from the MacPorts distribution.

The compile error is in the core foundation headers (objective C / C mix), nothing that can be done.  A known problem with Mac OS X 10.12.x and gcc.  (e.g. https://github.com/LLNL/spack/issues/1847 )

> clang/X11 does not crash (-g -O2).

This is indeed an important result, so it's quite sure that we have an Aqua/Carbon only problem, although the reason is not yet 100% clear.

> gcc6 will not compile the core foundation headers, so gcc/X11 is the best we can do for testing with gcc rather than clang.

I know that gcc on Mac is normally nothing else than a modified (by Apple) clang compiler - or do you use a real gcc, on Mac this is a bit unclear. So it's unbelievable that it won't compiler the core foundation headers, if it is a modified clang. Probably it's only a clash with the Tk header files? Please attach a compilation log, with a bit luck I can see a solution, I'm very familiar with the Tk stuff.

clang/X11 does not crash (-g -O2).

gcc6 will not compile the core foundation headers, so gcc/X11 is the best we can do for testing with gcc rather than clang.

Halfway managed to get it to build with gcc.
CFLAGS="-g -O2"
gcc with X11 (not aqua) has no crash.
Now if I can figure out why aqua is disabled.

OK I understand now what you meant.

1. I have:

- compiled Tcl with "-g -00 -fstack-protector-all"

- compiled Tk with "-g -01 -fstack-protector-all"

What I said below:

Stepping further, everything seems to be OK until the call to: 
	name = Tcl_GetHashKey(&sharedTextPtr->tagTable, hPtr);
in TkTextCreateTag(). 

Making further experiments with lldb, the segfault happens when returning from TkTextCreateTag(), on line 2010 just after the final return. Confirms stack corruption I guess since at this point we get a __stack_chk_fail.

Looking at sharedTextPtr->tagTable, and comparing what's in there on OS X and on Windows, I didn't detect any issue. Same values (except specific pointer values, of course), from the first call when creating the text widget until the return point when creating the "big" tag, including before and after each call to Tk_CreateOptionTable().

2. I also have:

- compiled Tcl with "-g -00 -fstack-protector-all"

- compiled Tk with "-g -01"

And making further tests without this stack protector for Tk, the backtrace in that case shows (this is from the attached "TkConfigureTag crash report 4.txt"):

0   Tk                                  0x000000010d74ae8a Tk_AllocFontFromObj + 26 (tkFont.c:1092)
1   Tk                                  0x000000010d7451ab DoObjConfig + 571 (tkConfig.c:716)
2   Tk                                  0x000000010d7457da Tk_SetOptions + 314 (tkConfig.c:1299)
3   Tk                                  0x000000010d7f2a24 TkConfigureTag + 164 (tkTextTag.c:1094)
4   Tk                                  0x000000010d7f169b TkTextTagCmd + 2475 (tkTextTag.c:472)
5   Tk                                  0x000000010d7b6c27 TextWidgetObjCmd + 2631 (tkText.c:2632)
...

The EXC_BAD_ACCESS happens at the very beginning of Tk_AllocFontFromObj(), on line 1092:

(lldb) n
Process 88869 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x50)
    frame #0: 0x00000001000aae8a Tk`Tk_AllocFontFromObj(interp=0x000000010101b010, tkwin=0x0000000000000000, objPtr=0x0000000100839f10) at tkFont.c:1092 [opt]
   1089     Tcl_Obj *objPtr)            /* Object describing font, as: named font,
   1090                                  * native format, or parseable string. */
   1091 {
-> 1092     TkFontInfo *fiPtr = ((TkWindow *) tkwin)->mainPtr->fontInfoPtr;
   1093     Tcl_HashEntry *cacheHashPtr, *namedHashPtr;
   1094     TkFont *fontPtr, *firstFontPtr, *oldFontPtr;
   1095     int isNew, descent;

At this point:

(lldb) p (TkWindow *) tkwin
(TkWindow *) $15 = 0x0000000000000000

Aha! tkwin is NULL when Tk_AllocFontFromObj() is entered.

It is already NULL when entering DoObjConfig(), and when calling Tk_SetOptions() in TkConfigureTag().

Actually, textPtr->tkwin becomes NULL exactly when returning from TkTextCreateTag() on tkTextTag.c:1076

Watching textPtr->tkwin all along TkTextCreateTag(), it appears that up to the very end (the return point), it still has the correct non NULL value (the value does not change from the beginning to the end of TkTextCreateTag()). But when we're back in TkConfigureTag() then it's gone (it has switched to NULL):

(lldb)
Process 88918 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x0000000100152303 Tk`TkTextCreateTag(textPtr=0x0000000100880810, tagName=<unavailable>, newTag=<unavailable>) at tkTextTag.c:2003 [opt]
   2000         TkBitSet(sharedTextPtr->selectionTags, index);
   2001         TkBitSet(sharedTextPtr->dontUndoTags, index);
   2002     } else {
-> 2003         tagPtr->relief = TK_RELIEF_FLAT;
   2004         assert(hPtr);
   2005         Tcl_SetHashValue(hPtr, tagPtr);
   2006     }
(lldb) p textPtr->tkwin
(Tk_Window) $33 = 0x00000001011bf610
(lldb) n
Process 88918 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x000000010015230e Tk`TkTextCreateTag(textPtr=0x0000000100880810, tagName=<unavailable>, newTag=<unavailable>) at tkTextTag.c:2005 [opt]
   2002     } else {
   2003         tagPtr->relief = TK_RELIEF_FLAT;
   2004         assert(hPtr);
-> 2005         Tcl_SetHashValue(hPtr, tagPtr);
   2006     }
   2007
   2008     MarkIndex(sharedTextPtr, tagPtr, true);
(lldb) p textPtr->tkwin
(Tk_Window) $34 = 0x00000001011bf610
(lldb) n
Process 88918 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x0000000100152381 Tk`TkTextCreateTag(textPtr=0x0000000100880810, tagName=<unavailable>, newTag=<unavailable>) at tkTextTag.c:2008 [opt]
   2005         Tcl_SetHashValue(hPtr, tagPtr);
   2006     }
   2007
-> 2008     MarkIndex(sharedTextPtr, tagPtr, true);
   2009     return tagPtr;
   2010 }
   2011

(lldb) p textPtr->tkwin
(Tk_Window) $35 = 0x00000001011bf610
(lldb) n
Process 88918 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x0000000100152391 Tk`TkTextCreateTag(textPtr=0x0000000100880810, tagName=<unavailable>, newTag=<unavailable>) at tkTextTag.c:2010 [opt]
   2007
   2008     MarkIndex(sharedTextPtr, tagPtr, true);
   2009     return tagPtr;
-> 2010 }
   2011

   2012 /*
   2013  *----------------------------------------------------------------------
(lldb) p textPtr->tkwin
(Tk_Window) $36 = 0x00000001011bf610
(lldb) n
Process 88918 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x00000001001529bd Tk`TkConfigureTag(interp=0x0000000101024610, textPtr=0x0000000100000000, tagName=<unavailable>, objc=2, objv=0x00000001010286d0) at tkTextTag.c:1076 [opt]
   1073     int mask = 0;
   1074     bool newTag;
   1075     TkSharedText *sharedTextPtr = textPtr->sharedTextPtr;
-> 1076     TkTextTag *tagPtr = TkTextCreateTag(textPtr, tagName, &newTag);
   1077     const char *elideString = tagPtr->elideString;
   1078     bool elide = tagPtr->elide;
   1079     bool undo = tagPtr->undo;
(lldb) p textPtr->tkwin
(Tk_Window) $37 = 0x0000000000000000
(lldb)

I'm sorry, I was not clear enough. Tk_CreateOptionTable() (line 1981) will be called during the creation of the text widget (for "sel" tag). Then Tcl_CreateHashEntry() (line 1924) will be called during "tag configure big -font {Helvetica -24}", after Tk_CreateOptionTable() has been called the first time.

BTW, probably it helps if you link with a Tcl library which is compiled with stack protector (-fstack-protector-all).

I don't understand this:

> the font initialization while performing 
>
>   tagPtr->optionTable = Tk_CreateOptionTable(textPtr->interp, tagOptionSpecs);
>
>needs investigation. Later, when the lines tkTextTag.c:1924-1931 will be >executed, one of the functions Tcl_CreateHashEntry(), or more likely >Tcl_GetHashKey(), seems to corrupt the stack

In the code I'm reading, Tk_CreateOptionTable() is line 1980, and this comes after Tcl_CreateHashEntry() and Tcl_GetHashKey() which are lines 1924-1931 in tkTextTag.c

> Hmmm, it seems the lldb debugger cannot be trusted with -01

It can be trusted, but debugging optimized code is uncomfortable, because the optimization is reorganizing the code a bit, and some variables will be replaced by registers, or simply removed.

If I look onto the results then I guess that the font initialization of the Mac code has any severe problem, and is probably destroying the hash table (sharedTextPtr->tagTable). So I think that the font initialization while performing

   tagPtr->optionTable = Tk_CreateOptionTable(textPtr->interp, tagOptionSpecs);

needs investigation. Later, wen the lines tkTextTag.c:1924-1931 will be executed, one of the functions Tcl_CreateHashEntry(), or more likely Tcl_GetHashKey(), seems to corrupt the stack, as a side-effect.

Hmmm, it seems the lldb debugger cannot be trusted with -01. Look at this:

(lldb) n
Process 18845 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x0000000100167be7 Tk`TkTextCreateTag(textPtr=0x00000001020c5610, tagName=<unavailable>, newTag=<unavailable>) at tkTextTag.c:1984 [opt]
   1981     assert(!tagPtr->reliefPtr);
   1982
   1983     textPtr->sharedTextPtr->numTags += 1;
-> 1984     textPtr->sharedTextPtr->numEnabledTags += 1;
   1985
   1986     if (isSelTag) {
   1987         tagPtr->textPtr = textPtr;
(lldb) n
Process 18845 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x0000000100167bed Tk`TkTextCreateTag(textPtr=0x00000001020c5610, tagName=<unavailable>, newTag=<unavailable>) at tkTextTag.c:1973 [opt]
   1970     tagPtr->justify = TK_TEXT_JUSTIFY_LEFT;
   1971     tagPtr->tabStyle = TK_TEXT_TABSTYLE_NONE;
   1972     tagPtr->wrapMode = TEXT_WRAPMODE_NULL;
-> 1973     tagPtr->undo = !isSelTag;
   1974     tagPtr->sharedTextPtr = sharedTextPtr;
   1975     tagPtr->undoTagListIndex = -1;
   1976     tagPtr->refCount = 1;
(lldb) n
Process 18845 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x0000000100167bf1 Tk`TkTextCreateTag(textPtr=0x00000001020c5610, tagName=<unavailable>, newTag=<unavailable>) at tkTextTag.c:2004 [opt]
   2001         TkBitSet(sharedTextPtr->selectionTags, index);
   2002         TkBitSet(sharedTextPtr->dontUndoTags, index);
   2003     } else {
-> 2004         tagPtr->relief = TK_RELIEF_FLAT;
   2005         assert(hPtr);
   2006         Tcl_SetHashValue(hPtr, tagPtr);
   2007     }

See how it unexpectedly jumps from line 1984 to line 1973 for no reason, and then directly to line 2004, for no better reason?

We were warned:

(lldb)  breakpoint set --file tkTextTag.c --line 1925
Breakpoint 1: where = Tk`TkTextCreateTag + 68 at tkTextTag.c:1925, address = 0x0000000100167964
(lldb) r
There is a running process, kill it and restart?: [Y/n]
Process 18840 exited with status = 9 (0x00000009)
Process 18845 launched: '/Users/fvogel/Documents/tcltk/fossil/build/tk/Deployment/wish' (x86_64)
% text .t
.t
% .t tag configure big -font {Helvetica -24}

Process 18845 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000100167964 Tk`TkTextCreateTag(textPtr=0x00000001020c5610, tagName="big", newTag=0x00007fff5fbfedeb) at tkTextTag.c:1925 [opt]
   1922         }
   1923         name = "sel";
   1924     } else {
-> 1925         hPtr = Tcl_CreateHashEntry(&sharedTextPtr->tagTable, tagName, (int *) &isNew);
   1926         if (newTag) {
   1927             *newTag = isNew;
   1928         }

Stepping further, everything seems to be OK until the call to:

	name = Tcl_GetHashKey(&sharedTextPtr->tagTable, hPtr);

in TkTextCreateTag().

Look at the following:

(lldb)
Process 18845 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x0000000100167984 Tk`TkTextCreateTag(textPtr=0x00000001020c5610, tagName="big", newTag=0x00007fff5fbfedeb) at tkTextTag.c:1932 [opt]
   1929         if (!isNew) {
   1930             return Tcl_GetHashValue(hPtr);
   1931         }
-> 1932         name = Tcl_GetHashKey(&sharedTextPtr->tagTable, hPtr);
   1933     }
   1934
   1935     if ((index = TkBitFindFirstNot(sharedTextPtr->usedTags)) == TK_BIT_NPOS) {
(lldb) p *hPtr
(Tcl_HashEntry) $21 = {
  nextPtr = 0x0000000000000000
  tablePtr = 0x0000000100819020
  hash = 0x000000000000231a
  clientData = 0x0000000000000000
  key = {
    oneWordValue = 0x00007fff00676962 <no value available>
    objPtr = 0x00007fff00676962
    words = ([0] = 6777186)
    string = {
      [0] = 'b'
    }
  }
}
(lldb) p sharedTextPtr->tagTable
(Tcl_HashTable) $22 = {
  buckets = 0x0000000100819028
  staticBuckets = {
    [0] = 0x0000000000000000
    [1] = 0x0000000000000000
    [2] = 0x00000001020bc690
    [3] = 0x0000000000000000
  }
  numBuckets = 4
  numEntries = 1
  rebuildSize = 12
  downShift = 28
  mask = 3
  keyType = 0
  findProc = 0x00000001003dc960 (Tcl`FindHashEntry at tclHash.c:242)
  createProc = 0x00000001003dc9c0 (Tcl`CreateHashEntry at tclHash.c:286)
  typePtr = 0x0000000000000000
}
(lldb) n
Process 18845 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x00000001001679df Tk`TkTextCreateTag(textPtr=0x00000001020c5610, tagName="big", newTag=0x00007fff5fbfedeb) at tkTextTag.c:1935 [opt]
   1932         name = Tcl_GetHashKey(&sharedTextPtr->tagTable, hPtr);
   1933     }
   1934
-> 1935     if ((index = TkBitFindFirstNot(sharedTextPtr->usedTags)) == TK_BIT_NPOS) {
   1936         unsigned oldSize = TkBitSize(sharedTextPtr->usedTags);
   1937         unsigned newSize = TkBitAdjustSize((index = oldSize) + 1);
   1938
(lldb) p name
error: Couldn't materialize: couldn't get the value of variable name: variable not available
error: errored out in DoExecute, couldn't PrepareToExecuteJITExpression
(lldb) p *name
error: Couldn't materialize: couldn't get the value of variable name: variable not available
error: errored out in DoExecute, couldn't PrepareToExecuteJITExpression
(lldb) p *hPtr
error: Couldn't materialize: couldn't get the value of variable hPtr: variable not available
error: errored out in DoExecute, couldn't PrepareToExecuteJITExpression

> So this looks similar to what you show me below on Linux.

So far so good, but why don't you step further until the segfault happens, we are still searching for the reason.

> Since there is no frames from inside Tcl

But there should be frames, because you did compile with "-g -00".

> this again points to TkTextCreateTag(), isn't it?

No, otherwise Linux/Windows would also crash, and if I look on the backtrace then it is crashing inside Tk_AllocFontFromObj(), I think that the font initialization did not work properly, and I cannot see that TkTextCreateTag() is influencing this.

I have:

- compiled the linked Tcl library with "-g -00"

- compiled Tk with "-g -01 -fstack-protector-all"

Didn't do anything about stripping. unix/Makefile is not created/used when compiling on OS X through make -C ./macosx

Then, breakpointing tktextTag.c:1925, stepping in and inspecting:

(lldb) p *tablePtr
(Tcl_HashTable) $0 = {
  buckets = 0x000000010084c628
  staticBuckets = {
    [0] = 0x0000000000000000
    [1] = 0x0000000000000000
    [2] = 0x0000000000000000
    [3] = 0x0000000000000000
  }
  numBuckets = 4
  numEntries = 0
  rebuildSize = 12
  downShift = 28
  mask = 3
  keyType = 0
  findProc = 0x00000001003cef40 (Tcl`FindHashEntry at tclHash.c:242)
  createProc = 0x00000001003cef70 (Tcl`CreateHashEntry at tclHash.c:286)
  typePtr = 0x0000000000000000
}
(lldb) p key
(const char *) $1 = 0x00000001010a27b0 "big"

So this looks similar to what you show me below on Linux.

In another test I did the same but added the -fstack-protector-all flag to Tcl. The crash report looks identical to "TkConfigureTag crash report 3.txt". Since there is no frames from inside Tcl, this again points to TkTextCreateTag(), isn't it?

We know the following:
-O0 works, -O1 is crashing, independently from -g.
Probably it's really a clang optimization bug?

Now we have a first entry point for debugging, but we need a Tcl library with debug symbols, so please compile the linked Tcl library w/ -g, and w/o optimization. Furthermore please compile wish (or tktest) w/ -g -O1. This works only if we build the executable w/o stripping, this means that in unix/Makefile the strip flags has to be removed. In file unix/Makefile, change the lines 220/221 into:

INSTALL_STRIP_PROGRAM   =
INSTALL_STRIP_LIBRARY   =

Now we can set a breakpoint at tkTextTag.c:1925. Please step into function Tcl_CreateHashEntry, after this step you will be inside function CreateHashEntry (the implementation of Tcl_CreateHashEntry).

Please inspect the arguments. If I debug under Linux I have the following (when entering function CreateHashEntry):

(gdb) p *tablePtr

$2 = {buckets = 0x819b8d4, staticBuckets = {0x0, 0x0, 0x0, 0x0}, numBuckets = 4, numEntries = 0, rebuildSize = 12, 
  downShift = 28, mask = 3, keyType = 0, findProc = 0xb7cf951d <FindHashEntry>, 
  createProc = 0xb7cf9563 <CreateHashEntry>, typePtr = 0x0}

(gdb) p key

$3 = 0x81a39f8 "big"

Of course, under Mac you will see different memory addresses.

If you continue debugging line by line probably you will see any problem. But it is possible that the hash table is already corrupted when entering the function CreateHashEntry, in this case we have to use a different debug strategy.

Result of this trimming down:

This is crashing because of stack overflow:

	hPtr = Tcl_CreateHashEntry(&sharedTextPtr->tagTable, tagName, (int *) &isNew);
#if 0
...

And this is not crashing:

#if 0
	hPtr = Tcl_CreateHashEntry(&sharedTextPtr->tagTable, tagName, (int *) &isNew);
...

(the latter reports "unknown option -font", which is normal I think since the code coming after this line is not executed.)

This line calling Tcl_CreateHashEntry() is line 1925 in tkTextTag.c

> The application still terminates with a regular exit if I insert "if (!isSelTag) exit(0);" just before the final return in TkTextCreateTag().

I think you mean that now it terminates with a regular exit, because w/o exit(0) it is crashing because of a stack overflow. So this method does not help to find the causing line of code, it seems that stack checking only occurs when leaving the function. In this case we need to remove code from this function, until the relevant line is the last in this function. This means: insertion of #if 0 ... #endif until it's sure that the last executable line in this function is causing the stack overflow. Example:

int foo()
{
  <line:1>
  <line:2>
  ...
  <line:k-1>
  <line:k>
#if 0
  <line:k+1>
  ...
#endif
}

is not crashing, but

int foo()
{
  <line-1>
  <line-2>
  ...
  <line:k-1>
#if 0
  <line:k>
  <line:k+1>
  ...
#endif
}

is crashing because of stack overflow. We want to know number line number k.

All results here with -fstack-protector-all.

1.
Deployment version with -DTK_MAC_DEBUG=1 and without -DTCL_CFG_OPTIMIZED=1: running the test script reports a __stack_chk_fail same as with -DTCL_CFG_OPTIMIZED=1 ("TkConfigureTag crash report 3.txt").

This is completely as per my expectation since TCL_CFG_OPTIMIZED is totally unused in Tk (except in tkConfig.h.in where it is #undefined - belt and suspenders...). In Tcl this #define is only used to record inside the build that this build was compiled with optimizations (check out [tcl::pkgconfig list]). Reference on this is TIP #59, but this TIP did not cover Tk.

2.
The application still terminates with a regular exit if I insert "if (!isSelTag) exit(0);" just before the final return in TkTextCreateTag().

I did a run under Linux with a version compiled with -fstack-protector-all, and I couldn't encounter any stack problem. Nevertheless with your result we are one step closer to a solution.

The next steps are (please use -fstack-protector-all anytime):

1. Compile the deployment version w/ -DTK_MAC_DEBUG=1 and w/o -DTCL_CFG_OPTIMIZED=1, does this work? Probably this difference it causing a broken build with stack errors, I don't know.

2. If (1) is not successful: Insert one "if (!isSelTag) exit(0);" statement into function TkTextCreateTag(), starting at the end of the function, and then moving this statement backwards, line by line, at which line does the application terminate with a regular exit, and not with stack overflow? Please consider that this test can be shortened with a "binary insert" methodology. Use the two-liner for testing. I hope that my interpretation of the crash report is right, it reports a stack overflow inside function TkTextCreateTag(), otherwise this test would be useless.

The two builds use the same options except the following differences:

Development (debug) version: -g -DTK_MAC_DEBUG=1

Deployment (release) version: -Os -DNDEBUG=1 -DTCL_CFG_OPTIMIZED=1

By modifying unix/configure:5467, I have built a Deployment-like version, that is the same as a Deployment version, but with optimization option different from -Os (this version does not have -g as well)

With -O1 or with -O2 it segfaults.
With -O0 or nothing (no -O option, which is the same as -O0) it does NOT segfault.

Now, with a vanilla Deployment version (i.e. without -g and with -Os), with an added -fstack-protector-all, it now says "Abort trap: 6" instead of a segfault, and it indeed reports a __stack_chk_fail, see the attached "TkConfigureTag crash report 3.txt".

The whole thing looks like a build problem, causing stack errors, see my second submission: "Another possibility is a built failure, of this kind". Please test whether the release build is using the same compiler options as the debug build, of course except of the difference with -g and -O2.

Is clang supporting -fstack-protector-all? I think it is, compiling with this option might help to detect stack problems.

> I have not yet looked at how to tweak the makefiles to obtain this (I'm using the macosx makefiles, not the Linux ones)
 
So far as I remember right the maxosx makefiles are including the UNIX makefiles, so modify unix/Makefile.

About "build with -g and without -O2": I've meant w/o -g and w/o -O2. Without -g disables the default initialization with zero (null), and if it works w/o -O2 (and w/o -g) then optimization problems maybe a reason, provided that the version w/o -g and w/o -O2 works (under Linux this version works based on my tests). And I must confess that I cannot believe that an optimization bug happens here.

More investigation shows a different failure mode. This is an output of printf at different locations of the code:

TkTextTagCmd: textPtr->tkwin is 0x7ff54e8abe10
TkTextTagCmd: objv[4] is 0x7ff54f0ffc40
TkTextTagCmd: objv[5] is 0x7ff54f06efb0
TkConfigureTag: 1: textPtr->tkwin is 0x7ff54e8abe10
TkConfigureTag: 1: objc is 2
TkConfigureTag: 1: objv[0] is 0x0
TkConfigureTag: 1: objv[1] is 0x0
TkConfigureTag: tagPtr->tkfont is 0x0

Shouldn't objv[4] in TkTextTagCmd be the same as objv[0] in TkConfigureTag? In a release build it's the case.

It's very strange. If I add more traces I can get a different failure mode. Now tkwin is no longer NULL for instance.

More notes:

- .t debug on: no difference I could see

- build with -g and without -O2: I have not yet looked at how to tweak the makefiles to obtain this (I'm using the macosx makefiles, not the Linux ones)

> So it crashes in Tk_AllocFontFromObj(), right?

You're right, I have not expressed myself clearly, I'm sorry.

The point is: it seems to be a waste of time to find a coding bug in text widget. The mac build has any problem, and the revised implementation is exploiting this problem in any way.

Another possibility for the crash is an optimization bug in compiler (with -O2). This can be excluded if a build w/ -g but w/o -O2 is working.

You've pointed out that textPtr->tkwin is NULL, but this seems to be impossible, this variable will be set when creating the widget. So it seems that this variable will be overwritten. But why does this not happen with debug version? So I see only one possibility: to find out at which place textPtr->tkwin becomes NULL, with the help of traces, starting with line 474 in tkTextTag.c. And if the variable is already NULL at this line, then the problem seems to be caused with the creation of the text widget, so tracing should start at tkText.c:1053. In such cases something like a "binary search" might accelerate the search.

By the way: it might help if the release version will be build w/o -DNDEBUG. And it might help to use a three-liner instead of a two-liner:

text .t
.t debug on
.t tag configure big -font {Helvetica -24}

The problem is on the very first line of Tk_AllocFontFromObj():

    TkFontInfo *fiPtr = ((TkWindow *) tkwin)->mainPtr->fontInfoPtr;

tkwin is NULL, and it's NULL through the entire calling chain, from TkConfigureTag() to Tk_AllocFontFromObj().

So the problem is that on OS X textPtr->tkwin is NULL in TkConfigureTag() at the time Tk_SetOptions() is called.

On Windows this is not the case, textPtr->tkwin has a valid value.

Why are you saying it crashes in Tk_SetOptions()? The call stack is:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   Tk                                  0x0000000106da9b93 Tk_AllocFontFromObj + 27
1   Tk                                  0x0000000106da4425 DoObjConfig + 558
2   Tk                                  0x0000000106da4a1e Tk_SetOptions + 302
3   Tk                                  0x0000000106e4fd15 TkConfigureTag + 146
4   Tk                                  0x0000000106e4e370 TkTextTagCmd + 3736

So it crashes in Tk_AllocFontFromObj(), right?

And, again, I only see the probem in a release build, I don't have symbols. Stepping does not happen line by line, but assembly instruction by assembly instruction.

However, I have confirmed through printf that tagPtr->tkfont is NULL just before entering Tk_SetOptions.

Obviously it is crashing in Tk_SetOptions, why do you not step into this function until you reach the concerned line of code? And please check tagPtr->tkfont before entering Tk_SetOptions, this value should be NULL.

A two liner producing the problem:

text .t
.t tag configure big -font {Helvetica -24}

(lldb) r
Process 23884 launched: '/Users/fvogel/Documents/tcltk/fossil/build/tk/Development/wish' (x86_64)
% text .t
.t tag configure big -font {Helvetica -24}

.t
% Process 23884 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001000a9b6b Tk`Tk_AllocFontFromObj + 27
Tk`Tk_AllocFontFromObj:
->  0x1000a9b6b <+27>: movq   0x50(%rbx), %rax
    0x1000a9b6f <+31>: movq   0x88(%rax), %rax
    0x1000a9b76 <+38>: movq   %rax, -0x50(%rbp)
    0x1000a9b7a <+42>: leaq   0x11891f(%rip), %r15      ; tkFontObjType
(lldb)

I have now succeeded in sourcing the textDisp.test file in widh launched by lldb.

The problem is that it crashes in the 'Deployment' (release) folder, not in the 'Development' (debug) folder, so I only see the assembly code.

Nevertheless, setting a breakpoint in TkConfigureTag() and running step by step from there it's OK until and including the call to Tk_SetOptions() in tkTextTag.c:1096

Then the next step is the offending one:

(lldb) n
Process 23486 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = instruction step over
    frame #0: 0x000000010014fc94 Tk`TkConfigureTag + 141
Tk`TkConfigureTag:
->  0x10014fc94 <+141>: callq  0x1000a48c8               ; Tk_SetOptions
    0x10014fc99 <+146>: addq   $0x10, %rsp
    0x10014fc9d <+150>: movl   $0x1, %ecx
    0x10014fca2 <+155>: testl  %eax, %eax
(lldb) n
Process 23486 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x50)
    frame #0: 0x00000001000a9b6b Tk`Tk_AllocFontFromObj + 27
Tk`Tk_AllocFontFromObj:
->  0x1000a9b6b <+27>: movq   0x50(%rbx), %rax
    0x1000a9b6f <+31>: movq   0x88(%rax), %rax
    0x1000a9b76 <+38>: movq   %rax, -0x50(%rbp)
    0x1000a9b7a <+42>: leaq   0x11891f(%rip), %r15      ; tkFontObjType
(lldb)

But this does not provide much light on the problem.

Then lldb is the only way I can see.

Sorry, same result when commenting that line:

//# define SUPPORT_DEPRECATED_TAG_OPTIONS 1

Indeed, the bug seems to be caused in function TkConfigureTag. Now I have an idea: please do a change in tkText.h, out-comment the line

  /*# define SUPPORT_DEPRECATED_TAG_OPTIONS 1*/

and recompile (after cleaning). Does this version work?

I'm sharing color attributes in different options (for the support of deprecated options). Probably this is causing problems on Mac.

I have attached a new crash report based on today's content of the revised_text branch. It is not exactly the same stack trace as the first one I have posted:

0   Tk                                  0x0000000106da9b93 Tk_AllocFontFromObj + 27
1   Tk                                  0x0000000106da4425 DoObjConfig + 558
2   Tk                                  0x0000000106da4a1e Tk_SetOptions + 302
3   Tk                                  0x0000000106e4fd15 TkConfigureTag + 146
4   Tk                                  0x0000000106e4e370 TkTextTagCmd + 3736
5   Tk                                  0x0000000106e12ae3 TextWidgetObjCmd + 7391
6   Tcl                                 0x0000000106f34163 Dispatch + 263
7   Tcl                                 0x0000000106f3117e TclNRRunCallbacks + 80
8   Tcl                                 0x0000000106f3234e TclEvalEx + 1802
9   Tcl                                 0x0000000106fd1ba2 Tcl_FSEvalFileEx + 642
10  Tk                                  0x0000000106db78c5 Tk_MainEx + 964
11  tktest                              0x0000000106d87feb main + 47

I'm realizing now that it segfaults even before running any test. Look at the following output created by make -C ./macosx test TESTFLAGS="-file textDisp.test -match textDisp-27.7.* -verbose bepst":

bll-mac:tk fvogel$ make -C ./macosx test TESTFLAGS="-file textDisp.test -match textDisp-27.7.* -verbose bepst"
/Applications/Xcode.app/Contents/Developer/usr/bin/make test-tk \
        BUILD_STYLE=Development CONFIGURE_ARGS=--enable-symbols
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C "/Users/fvogel/Documents/tcltk/fossil/tk/macosx/../../build/tk/Development" all tktest INSTALL_ROOT='' INSTALL_TARGETS='install-binaries install-libraries install-private-headers install-headers install-demos' VERSION='8.7'
make[2]: Nothing to be done for `all'.
make[2]: `tktest' is up to date.
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C "/Users/fvogel/Documents/tcltk/fossil/tk/macosx/../../build/tk/Development" test INSTALL_ROOT='' INSTALL_TARGETS='install-binaries install-libraries install-private-headers install-headers install-demos' VERSION='8.7'
DYLD_FRAMEWORK_PATH="`pwd`:/Users/fvogel/Documents/tcltk/fossil/build/tcl/Development:${DYLD_FRAMEWORK_PATH}"; export DYLD_FRAMEWORK_PATH; TCL_LIBRARY=/Users/fvogel/Documents/tcltk/fossil/tcl/library; export TCL_LIBRARY; TK_LIBRARY=/Users/fvogel/Documents/tcltk/fossil/tk/library; export TK_LIBRARY; ./tktest /Users/fvogel/Documents/tcltk/fossil/tk/unix/../tests/all.tcl -geometry +0+0 -file textDisp.test -match textDisp-27.7.* -verbose bepst
2017-04-19 00:14:58.204 tktest[76039:1532641] tkMacOSXInit.c:266: TkpInit(): Tcl_MacOSXOpenVersionedBundleResources failed
Tests running in interp:  /Users/fvogel/Documents/tcltk/fossil/build/tk/Development/tktest
Tests located in:  /Users/fvogel/Documents/tcltk/fossil/tk/tests
Tests running in:  /Users/fvogel/Documents/tcltk/fossil/build/tk/Development
Temporary files stored in /Users/fvogel/Documents/tcltk/fossil/build/tk/Development
Test files sourced into current interpreter
Running tests that match:  textDisp-27.7.*
Skipping test files that match:  l.*.test
Only running test files that match:  textDisp.test
Tests began at Wed Apr 19 00:15:00 CEST 2017
textDisp.test
++++ textDisp-27.7.1 SKIPPED: textfonts
---- textDisp-27.7.2 start
++++ textDisp-27.7.2 PASSED

Tests ended at Wed Apr 19 00:15:08 CEST 2017
all.tcl:        Total   392     Passed  1       Skipped 391     Failed  0
Sourced 1 Test Files.
Number of tests skipped for each constraint:
        1       textfonts
PERFORMANCE -------------------
Calls to DisplayText:        29
Calls to DisplayDLine:      164
Calls to LayoutDLine:      6857
Calls to XCopyArea:           0
Re-used display lines:        0
Cached display lines:         0
Found in cache:               0
Line metric calculation:      0
Break info computation:       0
DYLD_FRAMEWORK_PATH="`pwd`:/Users/fvogel/Documents/tcltk/fossil/build/tcl/Development:${DYLD_FRAMEWORK_PATH}"; export DYLD_FRAMEWORK_PATH; TCL_LIBRARY=/Users/fvogel/Documents/tcltk/fossil/tcl/library; export TCL_LIBRARY; TK_LIBRARY=/Users/fvogel/Documents/tcltk/fossil/tk/library; export TK_LIBRARY; ./tktest /Users/fvogel/Documents/tcltk/fossil/tk/unix/../tests/ttk/all.tcl -geometry +0+0 \
        -file textDisp.test -match textDisp-27.7.* -verbose bepst
2017-04-19 00:15:08.722 tktest[76099:1533291] tkMacOSXInit.c:266: TkpInit(): Tcl_MacOSXOpenVersionedBundleResources failed
Tests running in interp:  /Users/fvogel/Documents/tcltk/fossil/build/tk/Development/tktest
Tests located in:  /Users/fvogel/Documents/tcltk/fossil/tk/tests/ttk
Tests running in:  /Users/fvogel/Documents/tcltk/fossil/build/tk/Development
Temporary files stored in /Users/fvogel/Documents/tcltk/fossil/build/tk/Development
Test files sourced into current interpreter
Running tests that match:  textDisp-27.7.*
Skipping test files that match:  l.*.test
Only running test files that match:  textDisp.test
Tests began at Wed Apr 19 00:15:08 CEST 2017
Error:  No test files remain after applying your match and skip patterns!

Tests ended at Wed Apr 19 00:15:08 CEST 2017
all.tcl:        Total   0       Passed  0       Skipped 0       Failed  0
Sourced 0 Test Files.
/Applications/Xcode.app/Contents/Developer/usr/bin/make test-tk \
         BUILD_STYLE=Deployment INSTALL_TARGET=install-strip
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C "/Users/fvogel/Documents/tcltk/fossil/tk/macosx/../../build/tk/Deployment" all tktest INSTALL_ROOT='' INSTALL_TARGETS='install-binaries install-libraries install-private-headers install-headers install-demos' VERSION='8.7'
make[2]: Nothing to be done for `all'.
make[2]: `tktest' is up to date.
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C "/Users/fvogel/Documents/tcltk/fossil/tk/macosx/../../build/tk/Deployment" test INSTALL_ROOT='' INSTALL_TARGETS='install-binaries install-libraries install-private-headers install-headers install-demos' VERSION='8.7'
DYLD_FRAMEWORK_PATH="`pwd`:/Users/fvogel/Documents/tcltk/fossil/build/tcl/Deployment:${DYLD_FRAMEWORK_PATH}"; export DYLD_FRAMEWORK_PATH; TCL_LIBRARY=/Users/fvogel/Documents/tcltk/fossil/tcl/library; export TCL_LIBRARY; TK_LIBRARY=/Users/fvogel/Documents/tcltk/fossil/tk/library; export TK_LIBRARY; ./tktest /Users/fvogel/Documents/tcltk/fossil/tk/unix/../tests/all.tcl -geometry +0+0 -file textDisp.test -match textDisp-27.7.* -verbose bepst
Tests running in interp:  /Users/fvogel/Documents/tcltk/fossil/build/tk/Deployment/tktest
Tests located in:  /Users/fvogel/Documents/tcltk/fossil/tk/tests
Tests running in:  /Users/fvogel/Documents/tcltk/fossil/build/tk/Deployment
Temporary files stored in /Users/fvogel/Documents/tcltk/fossil/build/tk/Deployment
Test files sourced into current interpreter
Running tests that match:  textDisp-27.7.*
Skipping test files that match:  l.*.test
Only running test files that match:  textDisp.test
Tests began at Wed Apr 19 00:15:09 CEST 2017
textDisp.test
/bin/sh: line 1: 76122 Segmentation fault: 11  ./tktest /Users/fvogel/Documents/tcltk/fossil/tk/unix/../tests/all.tcl -geometry +0+0 -file textDisp.test -match textDisp-27.7.* -verbose bepst
make[2]: *** [test-classic] Error 139
make[1]: *** [test-tk] Error 2
make: *** [test-deploy] Error 2

This single "make test" command launched two runs of the tests, the first one with BUILD_STYLE=Development CONFIGURE_ARGS=--enable-symbols and the second one with BUILD_STYLE=Deployment INSTALL_TARGET=install-strip, i.e. the second one is a debug build and the second one is a release build.

The debug build runs the tests OK: there is the output:

Tests began at Wed Apr 19 00:15:00 CEST 2017
textDisp.test
++++ textDisp-27.7.1 SKIPPED: textfonts
---- textDisp-27.7.2 start
++++ textDisp-27.7.2 PASSED

But the release build does not show the above lines, meaning it crashes before reaching the first test, not inside the test. This is therefore now slightly different from what I originally reported (look at the orginial output), which is confirmed by the now different stack trace.

Finally: no, running make -C ./macosx test TESTFLAGS="-file textDisp.test -match textDisp-27.7.* -verbose bepst" does not crash with the legacy code from trunk.

In most cases, if --enable-symbols=yes works, but --enable-symbols=no is crashing, uninitialized variables are the causer. Valgrind is detecting uninitialized variables, but my test with valgrind is reporting nothing.

What's about the legacy code in trunk, is it also crashing?

What if you remove line by line from test case (starting at the end), which is the relevant line?

Yes, I should try to find time to study lldb.

In the meantime and additional data point: it does no longer crash when compiled with CONFIGURE_ARGS=--enable-symbols

It's very unlikely that TkConfigureTag has a bug, because valgrind would detect it. So it seems that this crash is caused by any memory problem which occurs at any other place.

Another possibility is a built failure, of this kind:

--------- FILE foo.h ---------------------
struct A
{
   int value;
#ifdef TEST
   int value2;
#endif
}
--------- FILE foo.c ---------------------
#define TEST
#include "foo.h"

void foo() {
   struct A a;
   ...
   baz(&a);
}
--------- FILE baz.c ---------------------
#undef TEST
#include "foo.h"

void baz(A *a) {
   ...
}

When building an executable it would crash, because struct A has different sizes in different modules.

Probably debugging with lldb provides more clues.

valgrind still not working on this OS X 10.12.4 machine.

However I have found a crash report (submitted as an attachment to this ticket), that points to TkConfigureTag().

IMO the first choice for such problems is the execution with valgrind.