TIP 100: Add Support for Unloading Dynamic Libraries Loaded with [load]

Login
Author:         George Petasis <petasis@iit.demokritos.gr>
State:          Final
Type:           Project
Vote:           Done
Created:        11-Jun-2002
Post-History:   
Discussions-To: news:comp.lang.tcl
Keywords:       load,unload,dynamic library
Tcl-Version:    8.5
Tcl-Ticket:     823486

Abstract

Tcl already provides facilities for loading dynamic libraries, through the load command. However no facilities are currently offered in order to unload dynamic libraries already loaded with the load command. This TIP tries to add support for unloading libraries, by introducing a new Tcl command (unload) and the guidelines that dynamic libraries must follow, in order to be unloadable. Note that the unload command will operate only on libraries that are designed to be unloadable by their developers. This way backward compatibility with older extensions is maintained, as unload will never try to unload libraries unaware of this new functionality.

Rationale

Tcl is an ideal language for component-based applications. Usually these applications offer a framework in which components developed by the users of the application can be embedded, in order to extend the functionality of the framework. Usually, these extensions are implemented as C/C++ dynamic libraries that are loaded through the load Tcl command.

However the development of such components can be a time-consuming process, as developers have to restart the framework application in order to be able to reload the library into it and test its altered functionality. And this can be quite annoying (depending on the application of course), as usually processing within the application is required in order to bring it into a proper state before testing the library.

The development cycle can be significantly shortened if Tcl provides a mechanism for unloading a dynamic library. A new version of the library can be created, as its object file can now be written, and the updated library can be re-loaded.

However, this is not the only application of unload. Services running for long periods of time and want to unload no longer needed functionality, replacing parts of an applications (i.e. from an automatic update procedure) or functionality temporarily needed (i.e. a web browser that loads a plugin to display a file of a particular file type) are some additional fields of application.

Introduction

Unload functionality has been left out of the Tcl core, mainly because library unloading was poorly implemented in many operating systems. But almost all operating systems have been improved in the meantime, and as a result most modern operating systems now support library unloading.

The main idea of this TIP is to enable dynamic library unloading at the Tcl level, in the same sense load provides dynamic library loading. However, library unloading will be provided only when the underlying operating system support this feature (as is also the case for load) and only when the library to be unloaded provides a set of functions that can "undo" the changes the library has made to the interpreter. In all other cases, unloading a library will result in an error.

This TIP proposes the insertion of a new Tcl command named unload and two functions pkg_Unload and pkg_SafeUnload, modelled after pkg_Init and pkg_SafeInit, that libraries which can be unloaded should implement.

Unloadable Libraries

A main concern is related to when a shared library can be "unloadable". An unloadable library is a library characterised as such by its developer. The developer of the library must export a function from the library, similar to the library's initialisation function. The unload command will never try to unload a library that does not provide such a function. This makes old libraries (before the introduction of the unload functionality) safe.

There is a category of libraries that can never be unloaded (i.e. libraries that register new tcl object types). However, the choice is upon the developer: the developer knows if the library can be unloadable. The simpler case, libraries that only register new commands are the most probable libraries to be unloadable. Libraries that export functions through a stub mechanism cannot be unloaded, as the were meant for having dependencies with other libraries that use the exported API. There is no way for the provider library to know wether it is used or not.

Specification

Actually, all the facilities for unloading dynamic libraries already exist in the Tcl core and simply they are not yet exposed at the Tcl level. As a result, the implementation of the unload command should be fairly easy.

load as it is currently implemented loads a dynamic library only the first time this library is loaded. It keeps an internal cache of all loaded libraries and if an already loaded library is requested again, only its initialisation function is called. This cache should be extended to keep some additional information:

  1. Two reference counts, counting how many times a specific library has been loaded. This reference count should be increased by each load and decreased for each unload. When it reaches 0, the library can be unloaded. Safe interpreters and normal interpreters should have different reference counts. Both should be 0 in order for a library to be unloaded.

  2. The addresses of the pkg_Unload and pkg_SafeUnload functions, if these are implemented by the library. If both of these functions are missing, the library will never be unloaded. If only pkg_Unload is implemented, the library can be unloaded if it never has been loaded in a safe interpreter. Finally, if pkg_SafeUnload is implemented, the library can be unloaded if it has never been loaded in a normal interpreter.

The unload command will always return an error, if the operating system does not support library unloading. In case the operating system supports library unloading:

  1. unload will examine the cache of load to locate the entry for the library to be unloaded. It is an error to unload a library that has not been loaded with load.

  2. If the entry in the cache is found, unload checks whether the corresponding for the interpreter type unload function pointer is NULL or not. If it is NULL, the library cannot be unloaded under this interpreter and again an error is returned.

  3. If the unload function pointer is not NULL, it is executed. If an error is returned by this function, unload also returns an error.

  4. If the unload function finishes without errors, the reference count corresponding to the interpreter type is decreased. If both reference counts reach 0, the library is unloaded.

Responsibilities of the Unload Functions

Its up to the developer of the library to decide if its library can be unloaded or not. A library can be unloaded if the function pkg_Unload is implemented and exported as a symbol from the library, and the library will never be loaded in a safe interpreter. A library that can be also loaded in safe interpreters is unloadable if the function pkg_SafeUnload is also available. These two functions will accept two arguments, the interpreter under which the library is unloaded and an integer, holding various flags. The flags argument can be either TCL_UNLOAD_DETACH_FROM_INTERPRETER or TCL_UNLOAD_DETACH_FROM_PROCESS. In case the library will remain attached to the process after the unload procedure returns (i.e. because the library is used by other interpreters), TCL_UNLOAD_DETACH_FROM_INTERPRETER will be defined. However, if the library is used only by the target interpreter and the library will be detached from the application as soon as the unload procedure returns, the flags argument will be set to TCL_UNLOAD_DETACH_FROM_PROCESS.

The main responsibility of these functions is to remove from the interpreter they are unloaded under any reference to a function residing inside the library. For example, such a function must:

  1. Unregister any commands that have been registered by the Init() function to the interpreter given by the first argument. In order to do this, the library should keep internally the tokens returned by each Tcl_Create*Command, as command may have been renamed.

  2. Unregister any other commands that may have been registered to the interpreter during the use of the library (usually used to represent special special data structures).

If the flag TCL_UNLOAD_DETACH_FROM_PROCESS is defined, the developer must do additional task, that are not normally required when the library gets unloaded from an interpreter:

  1. Free any memory occupied by the internal structures of the library.

  2. In general, try to remove any references Tcl may have to functions provided by the library.

If the developer cannot remove all reference to functions to the library, its better to not provide at all these two functions, so as unload to never attempt to unload the library.

Dependencies Among Libraries

It is possible that a library A has been loaded that exports some symbols. Then a library B is loaded, that has dependencies (i.e. uses some exported symbols) on A. What if A gets unloaded?

Actually, most modern operating systems seem to provide a solution to this problem, as reference counts are hold internally by the operating system for each library. Newer Windows, Solaris and Linux seem to provide similar solutions and in reality they don't unload the library if such symbols remain, even if the unload system call has been made for the library. Both libraries A and B have to be unloaded in order for A to be really removed from the address space.

Reference Implementation

A reference implementation can be found at: http://sf.net/tracker/?func=detail&aid=823486&group\_id=10894&atid=310894

Copyright

This document has been placed in the public domain.

Appendix: The unload man page.

     NAME
     unload - Unload machine code.
     SYNOPSIS
     unload ?switches? fileName
     unload ?switches? fileName packageName
     unload ?switches? fileName packageName interp
     DESCRIPTION
     -nocomplain
     -keeplibrary
     - -
     PORTABILITY ISSUES
     Unix
     Macintosh
     BUGS
     SEE ALSO
     KEYWORDS

NAME

unload - Unload machine code.

SYNOPSIS

unload ?switches? fileName

unload ?switches? fileName packageName

unload ?switches? fileName packageName interp

DESCRIPTION

This command tries to unload shared libraries previously
loaded with load from the application's address space.
fileName is the name of the file containing the library
file to be unload; it must be the same as the filename
provided to load for loading the library. packageName is
the name of the package, and is used to compute the name
of the unload procedure. interp is the path name of the
interpreter from which to unload the package \(see the
interp manual entry for details\); if interp is omitted,
it defaults to the interpreter in which the unload
command was invoked.

If the initial arguments to unload start with - then they
are treated as switches. The following switches are
currently supported:

-nocomplain
     Supresses all error messages. If this switch is
     given unload will never report an error.

-keeplibrary
     This switch will prevent unload from issuing the
     operating system call that will unload the library
     from the process.

--
     Marks the end of switches. The argument following
     this one will be treated as a fileName even if it
     starts with a -.

When a file containing a shared library is loaded through
the load command, Tcl associates two reference counts to
the library file. The first counter shows how many times
the library has been loaded into normal \(trusted\)
interpreters while the second describes how many times
the library has been loaded into safe interpreters. As a
file containing a shared library can be loaded only once
by Tcl \(with the first load call on the file\), these
counters track how many interpreters use the library.
Each subsequent call to load after the first, simply
increaments the proper reference count.

unload works in the opposite direction. As a first step,
unload will check whether the library is unloadable: an
unloadable library exports a special unload procedure.
The name of the unload procedure is determined by
packageName and whether or not the target interpreter is
a safe one. For normal interpreters the name of the
initialization procedure will have the form pkg\_Unload,
where pkg is the same as packageName except that the
first letter is converted to upper case and all other
letters are converted to lower case. For example, if
packageName is foo or FOo, the initialization procedure's
name will be Foo\_Unload. If the target interpreter is a
safe interpreter, then the name of the initialization
procedure will be pkg\_SafeUnload instead of pkg\_Unload.

If unload determines that a library is not unloadable \(or
unload functionality has been disabled during
compilation\), an error will be returned. If the library
is unloadable, then unload will call the unload
procedure. If the unload procedure returns TCL\_OK, unload
will proceed and decrease the proper reference count
\(depending on the target interpreter type\). When both
reference counts have reached 0, the library will be
detached from the process.

The unload procedure must match the following prototype:
typedef int Tcl\_PackageUnloadProc\(Tcl\_Interp \*interp, int
flags\);

The interp argument identifies the interpreter from which
the library is to be unloaded. The unload procedure must
return TCL\_OK or TCL\_ERROR to indicate whether or not it
completed successfully; in the event of an error it
should set the interpreter's result to point to an error
message. In this case, the result of the unload command
will be the result returned by the unload procedure. The
flags argument can be either
TCL\_UNLOAD\_DETACH\_FROM\_INTERPRETER or
TCL\_UNLOAD\_DETACH\_FROM\_PROCESS. In case the library will
remain attached to the process after the unload procedure
returns \(i.e. because the library is used by other
interpreters\), TCL\_UNLOAD\_DETACH\_FROM\_INTERPRETER will be
defined. However, if the library is used only by the
target interpreter and the library will be detached from
the application as soon as the unload procedure returns,
the flags argument will be set to
TCL\_UNLOAD\_DETACH\_FROM\_PROCESS.

The unload command cannot unload libraries that are
statically linked with the application. If fileName is an
empty string, then packageName must be specified.
If packageName is omitted or specified as an empty
string, Tcl tries to guess the name of the package. This
may be done differently on different platforms. The
default guess, which is used on most UNIX platforms, is
to take the last element of fileName, strip off the first
three characters if they are lib, and use any following
alphabetic and underline characters as the module name.
For example, the command unload libxyz4.2.so uses the
module name xyz and the command unload bin/last.so \{\}
uses the module name last.

PORTABILITY ISSUES

Unix
     Not all unix operating systems support library
     unloading. Under such an operating system unload
     returns an error \(unless -nocomplain has been
     specified\).

Macintosh
     

BUGS

If the same file is loaded by different fileNames, it
will be loaded into the process's address space multiple
times. The behavior of this varies from system to system
\(some systems may detect the redundant loads, others may
not\). In case a library has been silently detached by the
operating system \(and as a result Tcl thinks the library
is still loaded\), it may be dangerous to use unload on
such a library \(as the library will be completely
detached from the application while some interpreters
will continue to use it\).

SEE ALSO

info sharedlibextension, load, safe

KEYWORDS

binary code, unloading, safe interpreter, shared library