TIP 477: Modernize the nmake build system

Bounty program for improvements to Tcl and certain Tcl packages.
Tcl 2017 Conference, Houston/TX, US, Oct 16-20
Send your abstracts to tclconference@googlegroups.com
by Aug 21.
Author:         Ashok P. Nadkarni <apnmbx-wits@yahoo.com>
State:          WIP
Type:           Process
Vote:           Pending
Created:        30-Sep-2017
Keywords:       Windows nmake build


Tcl, Tk and extensions are currently built using one of two build systems - the Tcl Extension Architecture (TEA) and the Windows/Visual C++ specific nmake based environment. This TIP addresses only the latter with a view to modernize it to remove obsolete features, non-optimal build settings, and simplify writing and maintenance of extensions.

This TIP also serves as the documentation of the reworked nmake build system.

Background and Rationale

The current nmake-based build system is based on three files in the win directory of a project.

As currently implemented, the system has several drawbacks including duplication of code making maintenance difficult, unnecessarily verbose extension makefiles, and inconsistent application of compiler options.

Simplifying extension makefiles

In addition, creating a makefile.vc for even a simple extension involves a lot of unnecessary boilerplate leading to copy-n-paste-itis and attendant problems. This TIP aims to simplify the writing of extension makefiles to a minimalist form that reduces the boilerplate to extension-specific configuration.

The following makefile should suffice for the majority of extensions.

!include "rules-ext.vc"
PRJ_OBJS = $(TMP_DIR)\sample.obj \
!include "targets.vc"
pkgindex: default-pkgindex

(In contrast, the current makefile for the sample extension is of the order of 300 lines, excluding comments.)

Outside of the makefile itself, the TIP also proposes removing the burden from the extension author of having to write their own Windows version resource file, pkgIndex.tcl and other common boilerplate.

For more complex extensions, the build system should be incrementally customizable using standard "building block" macros.

Ensuring consistency

In principle, rules.mk and nmakehlp.c should be shared across Tcl, Tk and all extensions. In practice, each has its own copy leading to divergence between the various copies and the resulting maintenance headaches. Currently, in the author found every single extension, including Tk, had diverged from Tcl. This means updates for new compilers, changes in Tcl build configuration (e.g. -DUNICODE) do not make their way into extensions. Moreover, making fixes involves individually fixing extensions, some of which are orphaned.

The TIP proposes distribution of the nmake support files so that each extension, including Tk, is built off Tcl's master copy (either installed or from source) without having to maintain its own.

This not only ensures consistency but also reduces the burden on extension authors to maintain their makefiles to keep up with Tcl and Microsoft compiler changes.

Auditing compiler configuration

There are some bugs, inconsistencies and misconfiguration of compiler options in the various incarnations of the extension makefiles. Examples include unoptimized release builds in some important extensions like Tk and Sqlite, differing floating point conformance options in debug and release builds even within Tcl and so on.

Therefore, a standard set of documented nmake macros are defined to ensure consistency across build configurations.


It is not the intent of this TIP to look at alternatives to nmake. There is already the TEA based system for those who prefer to use it. Sean Woods is also working on the practcl build tools which may supplant both TEA and nmake in the future.

The nmake build system is not intended to be "compatible" with the TEA based system. For example, the generated libraries may not link with those generated from the other system. Of course, the built extensions should run with Tcl compiled from any compiler if stubs are enabled. At the same time, the nmake-ng system does make some minimal integration with TEA so that configuration values such as version numbers, pkgIndex.tcl.in etc. only need to be defined in one place.

Certain limitations in the current nmake system, in particular the requirement that there be no spaces in the source directories path are not addressed.

Command line interface

There will be no changes to the command line interface used to build Tcl and extensions using nmake except for removal of some obsolete features.

nmake /f makefile.vc ?targets? ?macros? 

The targets are as always defined by the makefile. However, the system also has standard predefined targets that makes it unnecessary for the extension makefile to define targets in many common cases. These are detailed later.

Build configuration macros

The macros passed on the command line are for the most part the same as in the current system. They specify the build configuration such as whether an extension is built as a static or shared library, instrumentation, generation of debug information and so on. The most commonly useful macros and their possible values are listed below. For a full list, see the comments in rules.vc file.

The OPTS macro controls the build configuration for the compiler and linker. Its value is a comma-separated list of option values, the more common options being listed in the table below.

Option Effect
none Nullifies other options even if they are specified.
static Builds the module as a static library instead of the default shared library
pdbs Generates PDB files with symbol information even for release builds
symbols Builds a debug version (no optimization, debug C runtime, PDB's)
staticpkg Specifies registry and dde extensions should be statically bound (tclsh and wish only)

The STATS macro, also specified as a comma-separated list, controls generation of instrumentation code as shown below. This is relevant only for building Tcl itself, not extensions.

Option Effect
none Turns off all instrumentation irrespective of other options being specified.
memdbg Enables instrumentation of memory allocation.
compdbg Enables byte compiler logging for debugging purposes.

The CHECKS macro configure additional compiler checks and warnings.

Option Effect
none Turns off other CHECKS options even if specified.
nodep Disables support for deprecated functions.
fullwarn Cranks up the compiler warnings level.
64bit Enables 64-bit portability warnings.

The following command will generate a static library with PDB debug information, memory instrumentation, full warnings and disabling of deprecated functions.

nmake /f makefile.vc OPTS=static,pdbs STATS=memdbg CHECKS=nodep,fullwarn

Basic makefiles

Makefile for a basic Tcl extension

NOTE: By convention, makefiles using the nmake system are named makefile.vc. Here we refer to them simply as makefile.

In the simplest case, a Tcl extension follows the common convention where sources are stored in the subdirectories generic, win and compat (not necessarily all). For such an extension, the following serves as a complete makefile.

!include "rules-ext.vc"
PRJ_OBJS = $(TMP_DIR)\sample.obj \
               $(TMP_DIR)\util.obj \
!include "targets.vc"
pkgindex: default-pkgindex

The lines must be in the order shown with PROJECT defined before inclusion of rules-ext.vc and PRJ_OBJS after. The standard targets, defined in targets.vc, are included last.

Given the above, the commands

nmake /f makefile.vc INSTALLDIR=/path/to/tcl
nmake /f makefile.vc INSTALLDIR=/path/to/tcl OPTS=static
nmake /f makefile.vc INSTALLDIR=/path/to/tcl OPTS=debug
nmake /f makefile.vc INSTALLDIR=/path/to/tcl OPTS=static,debug


Note the extension author need not write pkgIndex.tcl or Windows resource definition file unless there is some custom need.

In addition, standard targets for installation and clean up are also included so for example

nmake /f makefile.vc INSTALLDIR=/path/to/tcl install

will install the extension while

nmake /f makefile.vc INSTALLDIR=/path/to/tcl clean
nmake /f makefile.vc INSTALLDIR=/path/to/tcl realclean

will do various levels of cleaning.

The two macros that need to be defined for a basic extension are shown below.

Macro Description
PROJECT Name of the package. Must be defined before including rules-ext.vc.
PRJ_OBJS List of object and resource files for building the extension. The object files must be prefixed with $(TMP_DIR)\ exactly as shown above.

Makefile for a basic Tk extension

In case of a Tk extension, the only change required is to set the PROJECT_REQUIRES_TK macro to 1 before including rules-ext.vc.

!include "rules-ext.vc"
PRJ_OBJS = $(TMP_DIR)\sample.obj \
               $(TMP_DIR)\util.obj \
!include "targets.vc"
pkgindex: default-pkgindex

All else remains the same as for a Tcl extension.

The nmake build environment

The nmake build environment described above is implemented through four files rules-ext.vc, rules.vc, targets.vc and nmakehlp.c. The role of each is described in this section.

The rules-ext.vc file

The rules-ext.vc file is intended to be included by the extension's makefile to locate and load the latest compatible rules.vc file. It checks if the installed Tcl has copies of rules.vc and nmakehlp.c that are newer versions than the ones in the extension sources, and if so uses them instead of the extension's copies. In the case of extensions that build against the Tcl source (as opposed to a Tcl installation), it checks the versions in the Tcl source directory in a similar manner.

The compilation rules are versioned via the RULES_VERSION_MAJOR and RULES_VERSION_MINOR macros defined in rules.vc. Versioning is similar to Tcl's in how major and minor versions are treated. When comparing versions, the files in the Tcl installation are used if they have the same major version as that in the extension's rules file and their minor version is the equal or greater.

The nmakehlp.c program

The nmakehlp.c program has the same purpose and functionality as in the current system. It is unchanged and not detailed here. It is compiled and invoked on the fly from nmake for some utility purposes such as extracting versions, searching for strings etc.

The rules.vc file

This is the heart of the current nmake system and remains so, with enhancements to include as much of project-independent functionality as possible. The file is responsible for

It is intended that there will be only one "master" rules.vc file, the one in the Tcl repository where all changes are made. Extensions will have unmodified copies of this if they need to be build against older versions of Tcl. Otherwise, they will use the one installed with Tcl or from the Tcl sources if building against the latter.

The targets.vc file

This file, optionally included by the extension's master makefile, defines some standard targets that relieves the extension from having to define its own. It is separated from rules.vc so as to permit master makefile to modify macros set by rules.vc before they are expanded in the target rules.

Inclusion of this file is optional in the sense that more complex makefiles may choose to define their own standard targets.

Distributing the nmake build system

To eliminate the issue of divergence between the nmake support files as well as the need for continual maintenance and update, the files rules.vc, nmakehlp.c and targets.vc will be installed as part of a Tcl install in a similar fashion to tclconfig.sh, tclstub86.lib etc. except that they will be placed in the lib\nmake subdirectory under the Tcl installation's root directory.

These files will also be copied to each extension's source repository as is (supposed to be) done today. However, this is only a one-time copy and unlike the current system, it is not required to be done every time Tcl's version of these files change. This also allows the extension to be built against older versions of Tcl that do not include these files in their installation.

Locating headers and libraries

Building an extension requires Tcl, and optionally Tk, header files and libraries.

Locating Tcl

Tcl extensions need to locate either Tcl installed headers and libraries or the Tcl source directory. The nmake build system locates the Tcl directory containing these by trying directories in the following order:

The installation directory is specified via the INSTALLDIR macro on the command line. If unspecified, this defaults to C:\Tcl. NOTE: this is a change from the current default of C:\Program Files\Tcl because the latter may not have access permissions for the user.

The location of the Tcl header files and libraries is specified with the TCLDIR macro.

The macros _INSTALLDIR and _TCLDIR are generated from INSTALLDIR and TCLDIR respectively and contain the native form of the path (using backslashes as directory separators).

In addition, the following macros are defined for use by the extension makefile if desired.

Macro Description
TCLINSTALL In the case of extensions, the macro TCLINSTALL is set to 1 if the extension is being built against an installed Tcl and 0 if it is built against Tcl sources.
TCL_MAJOR_VERSION The major version of the Tcl against which an extension is being built.
TCL_MINOR_VERSION The minor version of the Tcl against which an extension is being built.
TCL_PATCH_LEVEL The patch level of the Tcl against which an extension is being built.
_TCL_H The path to the tcl.h header file.

Locating Tk

A similar method as above is used for extensions that build against Tk. The nmake build system locates directories in the following order:

The installation directory is specified via the INSTALLDIR macro on the command line. If unspecified, this defaults to C:\Tcl. NOTE: this is a change from the current default of C:\Program Files\Tcl because the latter may not have access permissions for the user.

The location of the Tk header files and libraries is specified with the TKDIR macro.

The macro _TKDIR contains the native form of $(TKDIR).

In addition, the following macros are defined for use by the extension makefile if desired.

Macro Description
TKINSTALL In the case of extensions, the macro TKINSTALL is set to 1 if the extension is being built against an installed Tk and 0 if it is built against Tk sources.
TK_MAJOR_VERSION The major version of the Tk against which an extension is being built.
TK_MINOR_VERSION The minor version of the Tk against which an extension is being built.
TK_PATCH_LEVEL The patch level of the Tk against which an extension is being built.
_TK_H The path to the tk.h header file.

Customizing and extending the build environment

It is hoped that the basic makefile structure described earlier will suffice for the majority of extensions which follow standard conventions of source directory location etc. However, extensions may diverge from convention for several reasons.

These all require different levels of customization and extension and the nmake system tries to accomodate these with minimal effort as described in the following sections.

Location of extension sources and implicit rules

The macro ROOT is set by the rules.vc to point to the root of the extension source tree. This must be exactly one level above the directory containing the extension makefile.

The basic makefile shown earlier assumes the following directory structure for the extension sources under $(ROOT):

Subdirectory Macro Description
generic GENERICDIR Directory containing platform-neutral source files.
win WINDIR Directory containing Windows specific source files.
compat COMPATDIR Directory with additional source files implementing functions not present on all platforms.
doc DOCDIR Directory containing documentation files.
demos DEMODIR Directory containing any demo files.
tools TOOLSDIR Directory containing any build tools.
tests TESTDIR Directory containing the test suite

The macros shown in the table above are intialized to the corresponding subdirectory names if not already defined by the parent makefile. Implicit rules are defined to generate the object files corresponding to sources in any of these directories. If the source directories are named differently, the corresponding macro can be defined appropriately before including rules-ext.vc.

So for example, if your generic sources were in directory src, the makefile would be modified as

PROJECT = sample
!include "rules-ext.vc"
...rest remains same...

In case there are additional directories containing sources, you will have to define an additional implicit rule for each such directory. For example, if the directory extrasrc contained the additional sources, you would add the lines below after including rules-ext.vc.

EXTRADIR = $(ROOT)\extrasrc
        $(cc32) $(pkgcflags) -Fo$(TMP_DIR)\ @<<

Here cc32 and pkgcflags are standard macros defined within rules.vc. These are described later.

Output directories and file names

Based on the options specified on the command line, compiler version and target architecture, the following macros are defined for the output directories:

Macro Description
OUT_DIR Directory to place output executables and libraries
TMP_DIR Directory to place all the object files

The output directory path placed in OUT_DIR has the form Release_COMPILERVERSION for x86 release builds and Release_AMD64_COMPILERVERSION for x64 release builds (the discrepancy in inclusion of architecture is historical). For debug builds (when symbols is included in OPTS on the command line), Release_ is replaced by Debug_.

The object files directory path placed in TMP_DIR has the form $(PROJECT)_ThreadedDynamic or $(PROJECT)_ThreadedStatic depending on whether the build options specify dynamic or static linking.

In addition, rules.vc defines the macros shown below for the output files themselves. These are only generally needed by the extension makefile if it is defining custom targets and rules instead of using the default built-in ones.

Macro Description Example
PRJLIBNAME File name of the generated binary extension sample12t.dll
PRJLIB Path to the generated binary extension .\Release\sample12t.dll
PRJIMPLIB Path to the link library for the generated extension .\Release\sample12t.lib
PRJSTUBLIBNAME File name of the stub library for the extension samplestub12.dll
PRJSTUBLIB Path to the stub library .\Release\samplestub12.dll

The generated file names follow a convention based on the project name (sample), the version (1.2) and a suffix composed of one or more of the letters shown below.

Suffix Meaning
t Threaded build. Always set as Tcl requires a threaded build on Windows.
s Static build. Not present for dynamic builds.
g Debug build with symbols.
x Static build but linking to dynamic C runtime.

For most extension makefiles, these file names are not very relevant. However, they are needed when the default rules are not adequate, and the extension makefile needs to define its own.

Implicit rules


Compiler configuration

The compiler is defined via the cc32 macro.

Compiler flags and switches

The flags and switches passed to the compiler are dependent on the build configuration such as whether a shared library is being built, debug options being in effect and so on. The rules.vc file defines these appropriately for a build configuration through the macros shown in the table below.

Macro Description
pkgcflags Compiler switches required to compile a file for an extension that will be linked against the Tcl with or without stubs depending on whether the nostubs option is in effect.
pkgcflags_nostubs Compiler switches required to compile a file for an extension that will be linked against the Tcl library without using stubs.
appcflags Compiler switches required to compile a file for an application program that will be linked against the Tcl with or without stubs depending on whether the nostubs option is in effect.
appcflags_nostubs Compiler switches required to compile a file for an application program that will be linked against the Tcl library without using stubs.
stubscflags Compiler switches that will be used to compile objects for a extension stubs library.

The appcflags and appcflags_nostubs macro includes all switches to be passed to the compiler to compile a source file to an object file to be linked into an application program. These include switches related to optimizations, debug information, warning levels, C runtime selection, C header include paths and C preprocessor definitions.

The pkgcflags and pkgcflags_nostubs are similar except that they include additional C preprocessor definitions required by convention to build a binary extension. These definitions are PACKAGE_NAME, defined as the name of the package, PACKAGE_VERSION, set to the version of the pacakge, MODULE_SCOPE and $(PROJECT)_build. These last two are used in the C source to mark DLL visibility of functions and to distinguish between definition and use of declaration within a C header. A further explanation is out of the scope of the nmake build system.

The last entry in the table above, stubscflags is used to compile an extension's stubs library, if it provides one (most do not).

It is strongly recommended that if extension makefiles need to define their own implicit or explicit rules, they make use of the above macros. If they need additional flags, they can be added as described in the next section.

Passing additional compiler switches

By default, the constructed compiler flags include compiler include paths and preprocessor definitions common to all extensions. An extension may wish to extend these without having to define its own rules. This can be accomplished by defining the following macros before including rules-ext.vc.

Macro Description
PRJ_DEFINES Additional preprocessor defines
PRJ_INCLUDES Additional directories for locating C header files

The contents of these macros are included in the appcflags, pkgcflags compiler switch macros.

Linker flags

Similar to compiler switches, rules.vc defines standard macros, shown in the table below, for linking object files.

Macro Description
link32 Linker application name
dlllflags Switches to pass to the linker for compiling a shared library extension.
conlflags Switches to pass to the linker for compiling a console application.
guilflags Switches to pass to the linker for compiling a gui application.

These flags include a standard set of libraries for linking Tcl extensions. If additional libraries are required, they can be specified by defining the PRJ_LIBS macro before including rules-ext.vc.

Windows resource files

Windows programs and DLL's are expected to have a version resource embedded in them though this is not mandatory. To free the extension author from having to write a resource definition file, the nmake build system will automatically generate one and embed it into the built DLL extension.

If the standard template is not adequate for whatever reason, a custom resource definition file may be specified by defining the RCFILE macro before including rules-ext.vc in the extension makefile.

RCFILE = custom.rc

By default, the nmake system will look in the win and win\rc directories for this file. If located elsewhere, define the RCDIR as the path to the directory where the file is located. Again, this must be defined before inclusion of rules-ext.vc. For example,

RCDIR = $(ROOT)\rc
!include "rules-ext.vc"

(ROOT is automatically defined as the root of the extension source tree.)

All of the above will make use of implicit rules defined in rules.vc. If more control is desired, for example to pass additional flags to the resource compiler, an explicit rule needs to be added to the extension makefile after including rules-ext.vc. For example,

$(TMP_DIR)\custom.res: $(RCDIR)\custom.rc

Note the use of TMP_DIR and RCDIR in the dependency. The MAKERESCMD macro defines the command and associated standard flags to compile a resource definition file.

Generating pkgIndex.tcl

The standard default install target expects a pkgIndex.tcl file to have been generated and placed in the $(OUT_DIR) output directory. It then copies this into the extension's installation directory as part of the install. To generate this file, the top level target that builds the project has a dependency on the pkgindex target which is responsible for generating this file.

For a pure binary extension (without any additional Tcl script files), the following line added after including targets.vc

pkgindex: default-pkgindex

will generate this file with no additional work required on the author's part.

Alternately, if there is a pkgIndex.tcl.in file in the extension root directory that is used with the TEA build environment, the default-pkgindex-tea target can be used instead.

pkgindex: default-pkgindex-tea

In this case, the pkgIndex.tcl is generated from pkgIndex.tcl.in by replacing all occurences of @PACKAGE_VERSION@, @PACKAGE_NAME@ and @PKG_LIB_FILE@ with the values of macros $(DOTVERSION), $(PROJECT) and $(PRJLIBNAME) respectively.

For all other cases, the extension author needs to write their own target commands for pkgindex that results in a pkgIndex.tcl file being placed in the $(OUT_DIR) directory.


The default target

If unspecified on the command line, the default target is assumed to be $(PROJECT). To change this, define the macro DEFAULT_BUILD_TARGET before including rules-ext.vc. For example,


Note you have to define the target yourself unless it is predefined or standard.

Predefined targets

Since most extensions build commands in a similar fashion, the rules.vc file provides some predefined targets that encapsulate the requisite commands. These are shown in the table below.

Target Description
default-pkgindex Generates a pkgIndex.tcl file for a pure binary extension in the output directory.
default-pkgindex-tea Generates a pkgIndex.tcl file in the output directory using a TEA pkgIndex.tcl.in file as a template.
default-install Combines default-install-binaries and default-install-libraries.
default-install-binaries Copies the generated binary extension to the installation directory.
default-install-libraries Copies all *.tcl files from $(LIBDIR), if it exists, to the installation directory.
default-clean Cleans up $(TMP_DIR) but not $(OUT_DIR)
default-hose Cleans up $(OUT_DIR) which normally also includes $(TMP_DIR)
default-distclean Maps to default-hose plus cleans up nmakehlp helpers.
default-setup Does common build initialization like creation of output directories.

These pre-defined targets are used as building blocks by the standard targets described in the next section as well as potentially by additional custom targets defined by a makefile.

Standard targets

Inclusion of the file targets.vc defines "standard" targets commonly defined in makefiles and shown in the table below. These generally map to the predefined targets described earlier but can be extended as discussed in the next section.

Target Description
$(PROJECT) The name of the project builds the binary extension.
install Maps to default-install.
clean Maps to default-clean.
hose Maps to default-hose.
realclean Same as hose.
distclean Maps to default-distclean.
test Runs the extension test suite assuming the script all.tcl in $(TESTDIR)
shell Starts a Tcl shell with the built extension's path added to auto_path.

Extending targets

In some cases, the standard targets "almost" suffice but are missing some additional actions. In such cases, instead of not including the targets.vc file and defining your standard targets, you can extend the standard targets by adding dependencies. For example, the default rules do not generate and install documentation as Tcl extensions do not have a common format and structure for the same. In simple cases like this, the appropriate target can be extended with additional commands:

install: install-docs
    $(CPY) $(DOCDIR)\*.html "$(DOC_INSTALL_DIR)"

Note CPY, DOCDIR and DOC_INSTALL_DIR are all predefined or computed macros within rules.vc.

Ignoring standard targets

In the extreme case, the standard targets can be completely ignored by the extension makefile by not including targets.vc. Note the built-in targets can still be accessed as default-install etc. as these are defined within rules.vc, not targets.vc. This is useful if for example, you wish to override the targets for building but use the default cleanup. In such a case, instead of including targets.vc the following targets can be defined in the extension makefile.

    ...Your custom build command to build the extension...
install: default-install
clean: default-clean
hose: default-hose

As an aside, when defining your own commands for building targets, you are strongly encouraged to make use of the predefined compiler and linker flags such as $(pkgcflags) etc. described elsewhere.


TBD - using stubs

TBD - generating stubs


By default, the nmake build determines the version of the extension by looking for a TEA configure.in or configure.ac file in extension's source root directory $(ROOT). The extension's version is extracted from the AC_INIT macro in that file. This allows the version to be maintained in a single place for the two build environments.

If the version cannot be determined as above or if the extension needs to have a different version on Windows for whatever reason, the macro DOTVERSION must be defined before including rules-ext.vc in the extension's makefile. This must be the major and minor version of the extension in dotted form. For example,

!include "rules-ext.vc"

After including rules-ext.vc the DOTVERSION and VERSION macros are guaranteed to be defined. The latter is the DOTVERSION value with all periods removed.

TBD - Tcl and Tk version macros


Installation macros (SCRIPTINSTALLDIR etc.)



Documentation and demos

TBD - default-install-html default-install-man defailt-install-demos

Utility macros


Sanity checking build configuration


Reference Implementation

The vc-reform branch in the core.tcl.tk repositories contains work-in-progress.


This document has been placed in the public domain.