Tcl Source Code

The Big Var Reform of 2007 (partial implementation)
_________________
Present Situation

Up to 8.5a6 a variable requires (all numbers on 32 bit platforms, assuming use of the threaded allocator ''zippy''):

* Compiled local variables
  - 32 bytes on the Tcl stack

* Hashtable variables (namespace vars, array elements, runtime defined locals): 
  - 32 bytes for the Var struct: zippy allocates 64 bytes
  - Tcl_HashEntry 
    . varnames of up to 7 chars: 20/24 bytes, zippy allocates 32 bytes
    . varnames of up to 79 chars: zippy allocates 64 bytes

Most variable accesses read first the flags (field #8), then the value (field #1).
__________
This patch

* Compiled local variables
  - 8 bytes on the Tcl stack; this has also an impact on proc initialisation and cache friendliness of local variable access (2 words/var instead of 8) 

* Hashtable variables (namespace vars, array elements, runtime defined locals): single alloc for Var (12 bytes) and Tcl_HashEntry (20+ bytes)
  - 64 bytes allocated by zippy for varnames of up to 119 chars

Most variable accesses read first the flags (field #1), then the value (field #2).

The patch passes the testsuite, valgrind soon.
___________
Forthcoming

By moving to Tcl_Obj keys we can reduce the hashTable variables to 24 bytes independent of name length (zippy -> 32), plus a possibly shared Tcl_Obj storing the name. If it is stored in the literal table we can also
insure faster key comparison, also in many "fail" cases. A further advantage is that no string is copied on variable creation.
_________________________________________
Impact on extensions that include tclInt.h 

Extensions that allocate or access Var structs directly may lose binary compatibility. It should be relatively simple to restore source compatibility. Special attention should be paid if they allocate or init
variable tables, as these are *not* plain string-keyed Tcl anymore.

Extensions that alloc or access CallFrame or ByteCode structs may lose binary compat, as the structs are grown by one field that must be initialised to NULL.

Extensions that alloc or access Interp structs may lose binary compat, as the structs are grown by one field that must be initialised properly.

File Added - 239336: VarReform.diff.9.5.gz

Logged In: YES 
user_id=148712
Originator: YES

New patch, to be committed. Still missing optimisations in var lookup and name caching enabled by the Tcl_Obj hash indices.
File Added: VarReform.diff.9.5.gz

File Deleted - 238226: 



File Added - 238228: VarReform.diff.9.3

Logged In: YES 
user_id=148712
Originator: YES

(missed a patch to the tests)
File Added: VarReform.diff.9.3

File Added - 238226: VarReform.diff.9.3

Logged In: YES 
user_id=148712
Originator: YES

New patch with additional features:
  - variable tables now have Tcl_Obj keys, VarInHash struct now has a fixed size (still using standard Tcl hash tables).
  - localCache now storing Tcl_Obj names, refering to interp's literal table
  - all variable handling in tclVar.c is now Tcl_Obj based
  - new Tcl_Obj interface to var traces (the traces themselves still string based)

Not completely optimised yet: the varName caching is clearly suboptimal for the new code, TBD
File Added: VarReform.diff.9.3

File Added - 237097: VarReform.diff.8.9

Logged In: YES 
user_id=148712
Originator: YES

update patch, with better usage of the new approach. Resolves most performance issues (I think), but not all: remaining "mysterious" slowdown in proc init sequence.
File Added: VarReform.diff.8.9

File Deleted - 236342: 



File Added - 236408: VarReform.diff.8.5

Logged In: YES 
user_id=148712
Originator: YES

Fixed newbie error in alloc that was causing a buffer overrun on 64bit platforms.
File Added: VarReform.diff.8.5

File Added - 236342: VarReform.diff.8.5

Logged In: YES 
user_id=148712
Originator: YES

Patch updated, cleaner struct and cast handling.
File Added: VarReform.diff.8.5

Logged In: YES 
user_id=1463011
Originator: NO

Ran the following script on todays HEAD with and without the patch (compiler gcc 3.4.2 (mingw) on WinXP SP2)

interp recursionlimit {} 10000
proc rec {} {puts [info level] ; rec}
catch {rec}
puts [time {set ::a([incr ::i]) 1} 1000000]
puts "check mem usage"
gets stdin mem
puts "mem usage was: $mem"
      
This resulted in:
 
HEAD
recurselimit: 2001
creating a 10e6 elemenst array 3.133378 microseconds per element
mem usage was: 77576K

HEAD+VAR patch
recursionlimit: 1940
creating a 10e6 elemenst array 2.549667 microseconds per element
mem usage was: 54024K

File Added - 236309: VarReform.diff.8.4

Logged In: YES 
user_id=148712
Originator: YES

Bugfix (thanks mjanssen!)
File Added: VarReform.diff.8.4

File Deleted - 236176:

File Added - 236249: VarReform.diff.8.4

Logged In: YES 
user_id=148712
Originator: YES

Fix for valgrind-detected bug. Incidentally, the 8.4 has nothing to do with Tcl's version, this is a patch for HEAD.
File Added: VarReform.diff.8.4

Logged In: YES 
user_id=148712
Originator: YES

Counting is obviously not my strength :}

Today's Tcl_HashTable entry takes 32 bytes for names of up to 7 chars, 64 bytes for names of up to 39. In this patch, the single alloc fits in 64 bytes for names of up to 27 chars.

File Added - 236176: VarReform.diff.8.4