TIP 494: More use of size_t in Tcl 9

Login
FlightAware bounty program for improvements to Tcl and certain Tcl packages.
Author:         Jan Nijtmans <jan.nijtmans@gmail.com>
State:          Draft
Type:           Project
Vote:           Pending
Created:        29-Dec-2017
Post-History:
Keywords:       tcl
Tcl-Version:    9.0

Abstract

This TIP describes the non-controversial part of the Tcl 9 changes: Make Tcl 9 ready for the 64-bit era.

Rationale

Many Tcl API functions and struct fields have int parameters, which don't provide sufficient room on 64-bit platforms.

Proposal

  • Enhance the hash functions, such that the hash value is stored in a size_t in stead of unsigned int. This allows hash tables to grow beyond 4Gb on 64-bit platforms

  • Enhance all struct fields representing refCounts or epochs, make them of type size_t

  • All memory-related functions, such as Tcl_Alloc(), will change its size argument from int to size_t, and its "char *" arguments to "void *"

  • Many functions, which have size parameters of type int (but NOT int *) will change to type size_t

  • All ClientData type arguments will be changed to void * arguments. This is actually the same type, but it prevents the need for type-casts in some situations.

  • Provide a compilation option -DTCL_8_COMPAT, which provides fully source compatibility. Using this option, extensions compile and run fine using either Tcl 8 or Tcl 9 headers. More explanation below.

  • Provide 2 new macro's TCL_IO_FAILURE and TCL_AUTO_LENGTH, both equal to ((size_t)-1). They can help making extensions use the full 64-bit range with Tcl 9, which still compile with Tcl 8 as well (see below). Tcl 8.7 will get those macro's too, but the value there is (-1)

  • The functions Tcl_Alloc(), Tcl_Free() (and friends) change to redirect to their debugging variants if the TCL_MEM_DEBUG is defined. So, Tcl_Alloc() becomes the same as ckalloc(), ending the general confusion regarding the difference between those two groups of functions: Starting with Tcl 9.0 there is no difference anymore. ckalloc() and friends are deprecated starting with 9.0, but there are no plans to actually remove those and no deprecation warning will be given if extensions use it.

On 32-bit platforms, this is all 100% upwards binary compatible, provided no internal API is used (since some internal structs might have incompatible but externally invisible changes)

On 64-bit platforms, those changes cause binary incompatibility. Therefore the TCL_STUB_MAGIC value needs to change, so extensions compiled using Tcl 9 headers will not load in Tcl 8 and reverse.

Implications

Although those changes are binary compatible on 32-bit platforms, there is a small potential source incompatibility. There are 10 functions which previously had an int return argument, which is changed to size_t. This signed/unsigned change might lead to subtle difference in behavior.

The 10 functions are:

  • Tcl_Gets

  • Tcl_GetsObj

  • Tcl_Read

  • Tcl_ReadChars

  • Tcl_ReadRaw

  • Tcl_Write

  • Tcl_WriteChars

  • Tcl_WriteObj

  • Tcl_WriteRaw

  • Tcl_Ungets

If the return value of such function is directly used in a compare, this could lead to the use of an unsigned compare in stead of a signed compare. If you compile your extension with -DTCL_8_COMPAT, those 10 functions will be changed to wrapper macro's which makes everything behave as if those functions return Tcl_WideInt. That's the easiest way to resolve this potential problem.

There are 2 other ways to make this change, which can do it without the use of the TCL_8_COMPAT macro:

  • Add 1 to the left and right hand side of the comparison. E.g. as here

  • Don't compare to -1 using < or >, but always to TCL_IO_FAILURE using == or != . E.g. as here

Tk and sqlite are already converted to make full use of TIP #494. Other extensions included with Tcl (e.g. tdbc) are unaffected.

Implementation

The implementation of this TIP can be found in the memory-API branch.

Copyright

This document has been placed in the public domain.