TIP 500: Private Methods and Variables in TclOO

FlightAware bounty program for improvements to Tcl and certain Tcl packages.
Author:         Donal K. Fellows <donal.k.fellows@manchester.ac.uk>
State:          Draft
Type:           Project
Vote:           Pending
Created:        10-Feb-2018
Keywords:       Tcl, object orientation, visibility
Tcl-Version:    8.7


This TIP proposes a mechanism for (somewhat) private variables in TclOO.


One of the principles of object oriented programming is that classes should be isolated from each other. This particularly includes the isolation of a superclass (which might be in one package) from its subclasses (in other packages) other than for its published API. The TclOO object system in Tcl 8.6 does not provide a way for user code to get such isolation: all classes that are collaborating in a hierarchy to implement a particular instance object see the same set of methods and variables. In the case of methods, it is because the classes end up placing their methods into the same general resolution scope (formed effectively from the union of all their methods), and in the case of variables, it is because all variables are actually just normal Tcl variables in the instance object's namespace.

A mechanism is desired to increase the isolation of classes from each other, even where they are in a superclass-subclass relationship. Note that any such mechanism does not need to be totally proof against prying (and introspection mechanisms are desirable) as TclOO deliberately does not provide security mechanisms; such mechanisms are the purpose of Tcl's interpreters. All that is desired is a way of ensuring that code does not inadvertently trample other code, allowing a superclass to evolve its implementation without needing to take into account every detail of its subclasses.

Private Declarations: Syntax and Immediate Semantics

This TIP introduces a new definition command, private, that is present when doing both oo::define and oo::objdefine. Much as with the self definition command (or the oo::define and oo::objdefine commands themselves), it takes either single arguments or a list of arguments (that it then builds a script of by turning them into a list) and evaluates the result in a special context. The special context is the same as the outer definition context, except that some definition commands instead create private definitions in that context.

The following commands are private-definition aware:

  • forward — the forwarded method it creates is a private method.
  • method — the procedure-like method it creates is a private method.
  • self (in oo::class) — the definitions on the class instance inside will be private definitions (if the concept applies).
  • variable — the variable resolution rules created or manipulated (as this is a slot) will be for private variables.

If provided with no arguments, the private definition command will return whether it was invoked from a context that should create private definitions. Otherwise it returns the result of evaluating its script.

The meaning of a private method and a private variable is defined below.

Detailed Proposal: Methods

For methods, as they actually exist in storage that is totally controlled by TclOO (and which is not addressable other than via TclOO-created commands or introspectors) the main complexity is working out what lookups are possible in a particular case while ensuring that method lookups are fast. Determining what the context class (or instance) is easy enough, as that's present pretty much directly in the Tcl_ObjectContext, but rapidly determining whether the object that is invoked is able to be seen in that way is the non-trivial part. (As with most things in TclOO, appropriate caching is the key to accelerating this sort of thing with a reasonable memory overhead.)

A key objective is that if one invokes $otherobj ClassPrivateMethod from inside a method that is part of the class but a different instance or possibly even a different subclass of that class, then that method will be found. That makes this a useful mechanism that is not provided by the TclOO system in Tcl 8.6.

Method Invoke Semantics

When a method is invoked, a public (or unexported, if via the my command) method is usually preferred. However, if the class of the object on which the method is called is also the class that defined the currently executing method (specifically, is the creator of the method that defines the current stack frame at the point of the call), bearing in mind that the determination of this also respects the inheritance and mixin hierarchies, then private methods on that context class will also be found. Similarly, if the currently executing method was defined by an object and that object is the object to which the method invoke is directed, private methods on that object are also found (the author expects these to be a rarely used feature). Searches for private methods precede the equivalent public and unexported methods in the method resolution order, but share a common naming space (i.e., they're stored in the same hash tables) so that there are no ambiguities.

Since every method has at most one declaring class or object (and not both simultaneously) there is at most one private method that can be on any call chain. External calls (e.g., from the top level of Tcl, from a namespace eval, from a procedure) will never have a private method on thir call chains.

Once the call chain is created, the execution is handled as prior to this TIP; the implementation of the first element on the call chain is executed, whatever that is, and that can dispatch to later items on the call chain using next and nextto.

A private method on a class (or object) may not have the same name as another method on that class; all methods of a class (or object) are part of a single space of names. When resolving a method call of a subclass of the class that has the private method, even if the method names match, the private method does not participate in the call chain; only an exact match from exactly the right context counts.

In particular:

oo::class create Top {
    method Foo {} {
        puts "This is Top::Foo for [self]"
oo::class create Super {
    superclass Top
    private method Foo {} {
        puts "This is Super::Foo for [self]"
    method bar {other} {
        puts "This is Super::bar for [self]"
        my Foo
        [self] Foo
        $other Foo
oo::class create Sub {
    superclass Super
    method Foo {} {
        puts "This is Sub::Foo for [self]"
    private method bar {other} {
        error "This should be unreachable"
    method grill {} {
        puts "This is Sub::grill for [self]"
        my Foo
Sub create obj1
Sub create obj2
obj1 bar obj2
obj1 grill

is expected to print out:

This is Super::bar for ::obj1
This is Super::Foo for ::obj1
This is Super::Foo for ::obj1
This is Super::Foo for ::obj2
This is Sub::grill for ::obj1
This is Sub::Foo for ::obj1
This is Top::Foo for ::obj1

Note that the calls via my, self and through the public handle of another object all pick up the private method from Super, and the call to next from Sub does not pick up the method on Super but instead goes straight to Top.

Creation and Introspection

Private methods are created calling oo::define (and oo::objdefine, and via the constructor of oo::class) like this:

oo::define TheClass {
    private method foo {...} {
    private forward bar  SomeOtherCommand

At the C API level, private methods can be directly created by setting the flags parameter (called isPublic in older versions of the API) of Tcl_NewMethod and Tcl_NewInstanceMethod to the constant TCL_OO_METHOD_PRIVATE. This lets custom types of methods be declared private by their C code directly.

Introspection is done via appropriate options to info class methods and info object methods. In particular, they are never returned when the -all option is given, are found when -private option is given (unless the -all option is also given), and can be found when the new option -scope is given, depending on what scope is chosen from the options below:

-scope public
Reports the public (public) methods only.
-scope unexported
Reports the unexported (non-public, created via naming or unexport) methods only.
-scope private
Reports the truly private methods only.

The -scope option causes the -all and -private options to be ignored.

Detailed Proposal: Variables

Variables are more complex, as they necessarily can be named by other commands (e.g., so that they can be vwaited upon). This requires them to be named by compounding the user-supplied name (to reduce confusion) and a unique identifier that identifies the particular context of the name. Now, one of the trickier things about TclOO is that it does not provide fixed names for objects; the rename command is supported. Because of that, the fully-qualified name of an object or class, while unique, is not a stable identifier. The full name of the backing namespace is stably-named (as namespaces cannot be renamed), but only has a unique name when the fully-qualified name is used; single components of the name are not guaranteed to be unique as users can control what those namespace names actually are. However, there is also another unique identifier inside TclOO, the object creation ID, which is based on a simple interpreter-level counter that is incremented for each time an object is manufactured. (The creation ID is used to allow very fast comparison between cached Tcl_Obj internal representations, as well as being the seed for the automatically-created object names.) This creation ID has a number of advantages, but its key ones are that it is guaranteed to be a simple token (an integer!) and it is guaranteed to be unique within an interpreter.

In particular, once a variable has been identified as being private by declaration, it is visible to the methods of the class (or instance) that defined it to be private with its given name, but it exists with a different actual name in the namespace. That actual name (which is theoretically globally visible once discovered) consists of the creation ID, a separator (a string consisting of a space, a single colon, and another space) chosen to be unlikely in any user script while also being disjoint from the syntax of arrays, and the user-supplied portion of the name. String substitution using these global names is not recommended. The variables concerned are not created by declaration; that's up to an appropriate method to actually do (and the variable can be created as normal scalar, array or link). When a method runs in the relevant class, the class's variable resolver will map the private variable in with the user-expected name.

The variable and varname methods of oo::object will be aware of the mechanism, respecting what names are visible in the context that invokes them.

Creation and Introspection

A private variable mapping is created like this:

oo::define Foo {
    private variables x y
    method foo {} {
        set x 1
        set y(z) 2

That is, the private declaration command described above (for methods) also impacts the variables slot declaration in oo::define and oo::objdefine, causing it to introduce this more complex mapping pattern. Note that the variable resolver already is class-scoped.

In the above example, a subclass of Foo that refers to variables x and y will not see the variables defined by Foo, but will instead see either the current standard variable scheme or their own x and y. (If the creation ID of Foo is 123, then Foo's x and y will really be called 123 : x and 123 : y, respectively.)

Introspection is via adding an extra option to info class variables and info object variables, -private, which causes these commands to list the private variables. Without the option, the non-private (i.e., direct mapped) variables are listed.

A supporting introspector is also added, info object creationid, which returns the creation ID of any existing object. It also applies to classes. Again, note that creation IDs are always system-allocated and are never guaranteed to be unique between interpreters, either in multiple processes or in the same thread or process; they are only ever locally unique.


oo::class create LabelEqual {
    constructor {label} {
        set [my varname label] $label
    private {
        # Two private declarations in the same block
        variable label
        method getLabel {} {
            return $label
    method equals {other} {
        expr {$label eq [$other getLabel]}

oo::class create Evaluated {
    superclass LabelEqual
    # Poorly chosen variable name! Happens too easily in real life
    variable label
    constructor {expression} {
        next $expression
        set label [expr $expression]
    method value {} {
        return $label

set expr1 [Evaluated new {1 + 2 + 3}]
set expr2 [Evaluated new {3 + 2 + 1}]
puts "one is two? [$expr1 equals $expr2]"
# Prints: one is two? 0
puts "one=[$expr1 value] two=[$expr2 value]"
# Prints: one=6 two=6
puts [info vars [info object namespace $expr1]::*]
# Prints something like: {::oo::Obj13::11 : label} ::oo::Obj13::label
catch {$expr2 getLabel} msg
puts $msg
# Prints: unknown method "getLabel": must be destroy, equals or value


See the tip-500 branch.


This document has been placed in the public domain.