HOWTO A. Kupries HOWTO Andreas Computer Laboratories (Me, myself and I) July 1999 How to write a transformation (channel) Abstract This document describes the issued involved in writing a transformation channel using the new ability to stack channels upon each other introduced in Tcl 8.2 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Facilities in the core . . . . . . . . . . . . . . . . . . . 3 3. Writing a transformation . . . . . . . . . . . . . . . . . . 4 3.1 InstanceData . . . . . . . . . . . . . . . . . . . . . . . . 4 3.2 Creation of a transformation . . . . . . . . . . . . . . . . 4 3.3 Destruction of a transformation . . . . . . . . . . . . . . 5 3.4 Accessing the channel downstream . . . . . . . . . . . . . . 6 3.5 The driver in detail . . . . . . . . . . . . . . . . . . . . 7 3.5.1 GetHandleProc . . . . . . . . . . . . . . . . . . . . . . . 7 3.5.2 SetOptionProc . . . . . . . . . . . . . . . . . . . . . . . 7 3.5.3 GetOptionProc . . . . . . . . . . . . . . . . . . . . . . . 8 3.5.4 SeekProc . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.5.5 BlockModeProc . . . . . . . . . . . . . . . . . . . . . . . 10 3.5.6 CloseProc . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.5.7 InputProc . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.5.8 OutputProc . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.5.9 WatchProc . . . . . . . . . . . . . . . . . . . . . . . . . 13 References . . . . . . . . . . . . . . . . . . . . . . . . . 17 Author's Address . . . . . . . . . . . . . . . . . . . . . . 17 A. Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . 18 B. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 Kupries [Page 1] HOWTO How to write a transformation (channel) July 1999 1. Introduction Transformation channels are based on the new feature/ability of the Tcl core (since version 8.2) to stack a channel upon any other arbitrary channel, i.e. allowing a channel to take over the identity of the other. Doing so causes the transformation channel to intercept all read/write calls directed at the original channel and to modify the data in them at will. Sometimes they are called 'Filter channels' too. As transformations are just channels and not something new it is not necessary to learn new data structures to write them, only basic knowledge on how to write a new channel driver. The information needed to write a new channel driver is explained here. Kupries [Page 2] HOWTO How to write a transformation (channel) July 1999 2. Facilities in the core The core provides 3 new procedures in the C-API, o Tcl_StackChannel, o Tcl_UnstackChannel and o Tcl_GetStackedChannel which will explained later in more detail. For now it is important to know that the following guarantees are made by the core with respect to stacked channels: 1. Stacking a transformation on a channel given through a 'Tcl_Channel' token (a reference) will neither invalidate this nor all other references held by some C code, like other interpreters, threads or libraries. 2. Only the topmost channel in a stack will do EOL-translation, UTF <-> XX encoding and buffering. All channels below will neither buffer, nor translate EOL, nor encode UTF. 3. Events occurring at the bottom channel are delivered to all channels in the stack, with the topmost channel the first to be notified. 4. A transformation channel automatically starts out with the same blocking mode as the channel it replaces. Kupries [Page 3] HOWTO How to write a transformation (channel) July 1999 3. Writing a transformation 3.1 InstanceData Whenever a channel is created the core will not only get a reference to the structure containing the references to the driver procedures, i.e. the channel type, but a reference to a structure allocated by the caller as well. This reference is given to all driver procedures when called for that particular channel. The internals of this structure are known only to the channel driver; the core will just pass the reference around. This allows us to associate the specific state of the driver with the channel. In the case of a transformation the following information should be present, not necessarily under the name I gave them, you are free to choose your own identifiers: Tcl_Channel channel; is a backlink from the instanceData to the channel. Without this link the transformation will be unable to access the downstream channel, which really makes no sense for it. Please note that due to the guarantee in Section 2 this reference will always point to the top of the stack and not necessarily to the channel structure referring to this instanceData. This means that anytime the driver wants to access the downstream channel of the transformation it has to walk the chain of channels to find its own channel. Not doing this will cause an infinite recursion later on. This is best done in a utility function employing 'Tcl_GetStackedChannel' (Section 3.4). See for example 'generic/giot.c:DownChannel'[1]. Tcl_TimerToken timer; This timer is necessary if the transformation is able to buffer processed data the generic I/O layer has no knowledge of. It will be used to flush out such data. See Section 3.5.9 for more explanations. int flags; The transformation has to remember the current blocking mode to handle EOF on input right. See Section 3.5.7 for more explanations. int mask; The transformation has to remember the current interest in events; see Section 3.5.9 for more explanations. 3.2 Creation of a transformation As with every channel a transformation has to have a creation command at the tcl level and an equivalent procedure at the C-level. In contrast to other channel types this procedure must not use 'Tcl_CreateChannel' as that would create a new and separate channel, Kupries [Page 4] HOWTO How to write a transformation (channel) July 1999 but 'Tcl_StackChannel' instead. This procedure takes as one of its parameters a reference to an existing channel and creates a new channel structure which is used to save the current state of the channel we are stacking the transformation onto. The old structure is then reinitialized for the transformation. The returned value refers to the saved state of the old channel. The advantage of this design is that all references to the channel we are stacking onto stay valid without opening backdoors into the whole stack. Other things, like the creation and initialization of the necessary clientData for the transformation, have to be done as usual. Please note that the backlink to the channel of the transformation must be initialized before the call to 'Tcl_StackChannel' as it is required for the WatchProc (Section 3.5.9) implementation and that one can be called from the inside of 'Tcl_StackChannel'. This restriction is no problem, as we know from Section 2 that the channel structure of the transformation is the same as the old channel. Creation procedure skeleton int XX_CreateTransformation (interp, objc, objv, cd) { old_channel = find (handle (objv [1])); clientData = Tcl_Alloc (...); initialize clientData... clientData->channel = old_channel; save_area = Tcl_StackChannel (interp, &trans_type, clientData, old_channel); interp->SetResult (old_channel->name); return TCL_OK; } The 'trans_type' in the code above is the structure containing the references to the driver procedures for the transformation. 3.3 Destruction of a transformation The two ways of destroying a transformation are 'Tcl_CloseChannel' and 'Tcl_UnstackChannel'. Kupries [Page 5] HOWTO How to write a transformation (channel) July 1999 The difference between them ? 'Tcl_CloseChannel' will not only destroy the transformation, but the underlying channel too, an action which then ripples through the whole stack until all channels in it are destroyed. 'Tcl_UnstackChannel' on the other hand will only destroy the topmost transformation in the stack. Of course, if the stack consisted only of one element both procedures are equivalent. As 'Tcl_CloseChannel' knows that the whole stack of channels is in destruction it does not deal with events anymore, except for destroying the internal data structures supposed to deal with them. But it does ask the various channels in the stack to flush buffered information down the stack so that nothing which is stuck is lost. This is not possible for information in the upward/read buffers, as there is no ultimate receiver for them, so these bytes are lost. 'Tcl_UnstackChannel' does the same as above, but also takes action to keep the event-system up and running (by redirecting existing scripts to the uncovered channel). In all cases the system will call the 'CloseProc' of the transformation to cleanup the data structures of the destroyed channels. A procedure to make 'Tcl_UnstackChannel' available at the script level is rather simple: Unstack skeleton int XX_UnstackTransformation (interp, objc, objv, cd) { channel = Tcl_GetChannel (interp, Tcl_GetString (objv [1]), &mode); Tcl_UnstackChannel (channel); return TCL_OK; } 3.4 Accessing the channel downstream It was said in Section 3.1 that the transformation has to walk the stack to find its own channel structure, and that of the channel downstream, but not how this is done. The functionality for this is provided by the third new function in the C-API, 'Tcl_GetStackedChannel'. It takes a channel token and returns a token for the channel immediately downstream. A (Tcl_Channel) NULL indicates that the channel used as argument was Kupries [Page 6] HOWTO How to write a transformation (channel) July 1999 at the bottom of the stack. 3.5 The driver in detail Now that the environment of the driver is a little more known we can explain the operations of the various driver procedures in detail. Every description will start with the general condition under which the procedure is called by the generic I/O layer of the tcl core and proceeds to the specialities a transformation has to take care of. 3.5.1 GetHandleProc This procedure is called by the C-API function 'Tcl_GetChannelHandle' to retrieve the OS specific file handle associated to the queried channel. Transformations are not associated with OS specific handles, making this call a bit meaningless. So one possible implementation simply returns a NULL handle (erroring out is not possible). On the other hand the downstream channel could be OS specific and have such a handle. So the other implementation is to delegate that request to the underlying channel, via 'Tcl_GetChannelHandle'. If all transformations cooperate in this manner, the request will fall down to the channel at the bottom, like a file or socket, which is able to handle it and return something meaningful. To me the second implementation is the better policy of the two described here. 3.5.2 SetOptionProc This procedure is called by the generic I/O layer whenever 'Tcl_SetChannelOption' is used (f.e. by 'fconfigure') and a non-standard option was specified as argument. The interesting case here is how to handle options unknown to the transformation. One could discard them, or delegate the call to the downstream channel. I recommend delegating the call. Kupries [Page 7] HOWTO How to write a transformation (channel) July 1999 SetOptionProc skeleton static int SetOptionProc (clientData, interp, optionName, value) { ... handle your own options /* delegate unknown options downstream */ return Tcl_SetChannelOption (interp, clientData->channel->supercedes, optionName, value); } 3.5.3 GetOptionProc This procedure is called by the generic I/O layer whenever 'Tcl_GetChannelOption' is used (f.e. by 'fconfigure') to query the value of a non-standard (or all) option(s). Implement everything from the Section 3.5.2, probably more (a read-only option makes sense, write-only doesn't). Again it is possible to delegate unknown options to the underlying channel. In the case of a query for all options, delegation will generate a mighty long result. Pruning the unnecessary options values from the result of the underlying channel (-encoding, -buffering, -translation) is possible, but tedious (We have a DString, not Tcl_Obj'ects, especially no ListObj'ects). The named options have fixed and known values, due to the behaviour of 'Tcl_StackChannel' as described in Section 3.2. GetOptionProc skeleton static int GetOptionProc (clientData, interp, optionName, dsPtr) { ... handle your own options. /* delegate unknown options and full queries downstream */ return Tcl_GetChannelOption (interp, clientData->channel->supercedes optionName, dsPtr); } 3.5.4 SeekProc This procedure is called by the generic I/O layer whenever the user asks the channel to move the 'file access point'. This is a hard problem for transformations. Should they seek using Kupries [Page 8] HOWTO How to write a transformation (channel) July 1999 their own notion of access point ? Or should they use the notion of the underlying channel and then try to adapt their own state for fine-positioning? Depending on the transformation both can be impossible. A nice example are compressors (like zlib), with their completely non-linear and position-dependent relationship between the number of bytes coming in from the downstream channel and going out to its caller. Another reason could be that the transformation state is not reversible, i.e. cannot be rolled back in a simple way, without hogging memory. An example for this would be an encryption transformation using a cryptographically strong hash-function to go from the current state to the state for the encryption of the next byte (or block). This is not reversible. We can go forward from state to state, but not back to the old state, except for saving them all. Because of this my personal policy to deal with seeking is to propagate the request unchanged to the underlying channel and to discard all information in the internal buffers of the transformation. Data waiting to be written is converted as if they are the last block, in other words the special end of information processing is applied, and then flushed. The current state is abandoned too. The next call to InputProc (Section 3.5.7) or OutputProc (Section 3.5.8) will be handled as if it were the first call to the transformation. This is basically a 'The user knows best, is able to compute a place making sense and not creating garbage during recover'. In the end this means that the user of a certain transformation has to understand its properties and whether a seek on it makes sense at all. Oh, it is possible to deal even with non-reversible state, by recording all read/write calls and maintaining an exact image of the information read/written so far, but this is, ah, memory-extensive, to understate this a little. Kupries [Page 9] HOWTO How to write a transformation (channel) July 1999 SeekProc skeleton static int SeekProc (clientdata, offset, mode, errorCodePtr) { ... flush waiting output ... flush waiting input, if possible (f.e. into a configured variable!) /* Chain the call */ result = Tcl_Seek ( clientdata->channel->supercedes, offset, mode ); *errorCodePtr = (result == -1) ? Tcl_GetErrno ():0; return result; } 3.5.5 BlockModeProc This procedure is called by generic the I/O layer whenever the user (re)configures the value of option '-blocking'. For a normal channel this procedure has to take the necessary actions at OS level to switch the managed channel into (non-)blocking behaviour. For a transformation however the behaviour is pretty much fixed. It has to o remember the current blocking-mode in the transformation state o and to notify the downstream channel of the change. None of these can be left out because o remembering the current mode is necessary for InputProc (Section 3.5.7) to deal correctly with empty reads on the downstream channel o and the notification is required to synchronize all channels in the stack with respect to blocking. Failing to do this will result in mysterious lockups while trying to read from the stack. Kupries [Page 10] HOWTO How to write a transformation (channel) July 1999 BlockModeProc skeleton static int BlockModeProc (clientdata, mode) { if (mode == TCL_MODE_NONBLOCKING) { clientdata->flags |= ASYNC; } else { clientdata->flags &= ~ASYNC; } return Tcl_SetChannelOption( NULL /* ignore interp arg */, clientdata->channel->supercedes, "-blocking", (mode == TCL_MODE_NONBLOCKING) ? "0" : "1"); } 3.5.6 CloseProc This procedure is called by generic the I/O layer to tell a channel that it is about to be destroyed. It is the responsibility of the procedure to clean up any data structures held by the channel. A transformation has the additional responsibility to complete the conversion of all incomplete information sitting in its internal write buffers and to write the result into the downstream channel. Another task is the removal of all event related things, like ChannelHandlers and Timers, although this could be billeted under 'clean up of any data structures held by the channel' too. CloseProc skeleton int CloseProc (clientdata, interp) { ... delete channelhandler at underlying channel, if any. ... delete timer, if any. See 'WatchProc' too. ... do last minute conversions on r/w/ buffers and try to flush their results to the underlying channel. ... Tcl_Write (clientdata->channel); /* (x) */ ... free data structures on the heap. return TCL_OK } Remark (x): At the time of the call the backlink to the transformation channel actually already refers to the downstream Kupries [Page 11] HOWTO How to write a transformation (channel) July 1999 channel instead, which is now at the top. 3.5.7 InputProc This procedure is called by the generic I/O layer whenever some input is required. Instead of going to the OS for this a transformation has to ask the channel downstream for data to convert. Use 'Tcl_Read' to do this, not 'Tcl_Gets'. The channel downstream does no EOL-translation (See Section 2) and so has basically has no notion of lines. 'Tcl_Gets' might trip on that, unnecessarily waiting for the end of a line, or EOF, although there is data which could be converted. Instead of a skeleton which would be overwhelming despite trimming down I list the rules my input procedures are based upon. See 'generic/giot.c:TransformInputProc'[2]for a real-life example. 1. If the request can be satisfied by the information in the internal read buffers of the transformation, use that. 2. In case of not having enough data, use everything available in the internal buffers and then ask the underlying channel for more. * In blocking mode this will wait until we get some data, or EOF. Convert the incoming bytes and check whether the result is enough to satisfy the initial request. See 1). Loop, if not. In case of EOF do all the necessary special handling of the incomplete data in the internal buffers, then return (a possibly partial result). Signal EOF upward only if the internal buffers are empty. * In non-blocking mode we either get nothing, some data or EOF. Handle EOF and some data as in the previous item. In case of nothing retrieved return the partial (or even empty) result. If there is nothing in the internal buffers signal error EWOULDBLOCK. Other things to consider: o The calling procedure may have an unannounced reference to the result of the interpreter we are working with. So take care to use 'Tcl_SaveResult' and 'Tcl_RestoreResult' (available since Tcl 8.1) to protect that reference if you are calling back into the tcl level from within the transformation. Not doing this may crash the interpreter with a defect list of free objects. o Writing to the underlying channel is allowed! An example using Kupries [Page 12] HOWTO How to write a transformation (channel) July 1999 this is the SSL/TLS transformation[3] created by Matt Newman[4] to handle the initial handshake between the two partners before going into transparent encryption mode. As long as the handshake is not complete nothing can be read from/written to the channel. o The InputProc can have any type and number of side effects. Examples: * Identity transformation collecting statistics (frequency of bytes, byte-pairs, triplets, etc.) * Splitter: Identity transformation piping the information flowing through the channel to a second channel (different from the underlying one). o Recursively reading from/writing to the transformation itself (maybe indirectly, see splitter) is not a good idea, it may lead to infinite looping. 3.5.8 OutputProc This procedure is called by the generic I/O layer whenever something is written to the transformation and an I/O buffer is flushed. The transformation has to convert as much as possible of the data it got, and the result must be written to the channel downstream (Well, not really, but not writing it does not make much sense). Data which cannot be converted at once has to be buffered and converted later, together with the data written by the next calls to the transformation. The API to use here is 'Tcl_Write'. As with InputProc (Section 3.5.7) this procedure is free to read from the underlying channel too, or from some other channel, or ... 3.5.9 WatchProc This procedure is called by the generic I/O layer whenever the user (or the system) announces its (dis)interest in events on the channel. The correct behaviour of the transformation here is essential for a smooth interoperation with the notifier and for working fileevents. Whenever it is called it has to add or delete a Channelhandler in the channel downstream, using the given mask. This propagates the interest in events down the stack until the channel at the bottom Kupries [Page 13] HOWTO How to write a transformation (channel) July 1999 takes the necessary steps to actually generate these events. From Section 2 we know that events generated by the channel at the bottom are delivered to all channels in the stack, with the topmost channel the first to be notified. This will call all of the installed ChannelHandlers. They have to check the internal read buffer for pending information and to set up a timer if there is some. The reason for this will be explained later in this section. To complete the action, the Eventhandler for the timer has to call 'Tcl_NotifyChannel' with the mask for 'READABLE', in other words it has to generate an artificial 'readable' event. This in turn will invoke any 'fileevent' script and all the ChannelHandlers, causing another check for the necessity of setting up more timers. WatchProc skeleton static void WatchProc (clientdata, mask) { if (mask == clientdata->mask) { return; /* no changes */ } if (clientdata->mask) { /* remove old channel handler */ Tcl_DeleteChannelHandler ( clientdata->channel->supercedes, ChannelHandler, clientdata ); } clientdata->mask = mask; if (clientdata->mask) { /* recreate channel handler */ Tcl_CreateChannelHandler ( clientdata->channel->supercedes, clientdata->mask, ChannelHandler, clientdata ); } } Kupries [Page 14] HOWTO How to write a transformation (channel) July 1999 ChannelHandler skeleton static void ChannelHandler (clientdata, mask) { if (clientdata->timer != NULL) { /* cancel outstanding timer */ Tcl_DeleteTimerHandler (clientdata->timer); clientdata->timer = NULL; } if ((mask & TCL_READABLE) && clientdata->ReadBufferContainsDeliverableInformation()) { clientdata->timer = Tcl_CreateTimerHandler (DELAY, ChannelHandlerTimer, clientdata ); } } Timer handler skeleton static void ChannelHandlerTimer (clientData) { /* Timer event, reset token */ clientdata->timer = NULL; /* Handle like normal events */ Tcl_NotifyChannel (clientData->channel, TCL_READABLE); } Now the promised explanation about the necessity of timers. Consider this scenario: 1. A transformation stacked upon a socket, its internal read buffer is empty. The transformation does not merge lines. 2. A fileevent script is set up and waiting for calls. 3. The socket has data available, say 400 bytes, in several lines (more than one), they are the last on the channel, i.e. followed by EOF. The notifier generates the appropriate 'readable' event. 4. This event triggers the execution of the fileevent script in the top channel. 5. The executed script uses 'gets' to read a single line. Kupries [Page 15] HOWTO How to write a transformation (channel) July 1999 6. As the buffers are empty (s.a.) the transformation asks the socket for data to convert, using 'Tcl_Read' and a standard buffer of 4K size. Thus it gets all waiting bytes from the socket. These are converted, resulting in several lines (no merge). Some of them are delivered up into the generic layer and its buffering, but not all (small buffersize). At least one line remains in the buffers of the transformation. 7. The script processes the one line it got and then goes back to sleep. 8. Now what ? 9. The generic I/O layer finds that its buffers are not empty and uses a timer to generate additional events to clear them. 10. The generic I/O buffers are empty. Now what, again ? 11. Nothing. No events, no processing of the remaining line(s) stored in the transformation. Why ? The socket has an EOF pending and will not generate events anymore. The generic layer has empty buffers and ceases to generate events too. It has no knowledge about the buffers inside the driver, i.e the transformation. So the script will not wake up again, neither ask for the line, nor detect the pending EOF. We are hung. The solution is the same one used by the generic layer, but from the inside of the transformation this time: The transformation has to check itself for data waiting to be read and then use a timer to generate the necessary 'readable' events. And that is what the construction of channel handlers and timers described above will do. Kupries [Page 16] HOWTO How to write a transformation (channel) July 1999 References [1] http://www.oche.de/~akupries/soft/giot/ [2] http://www.oche.de/~akupries/soft/giot/ [3] http://www.sensus.org/tcl/ [4] mailto:matt@novadigm.com [5] mailto:mrose@dbc.mtview.ca.us [6] http://www.rfc-editor.org/rfc/rfc2629.txt [7] http://memory.palace.org/authoring/xml2rfc.tar.gz [8] mailto:Jan.Nijtmans@wxs.nl [9] mailto:matt@novadigm.com [10] http://www.sensus.org/tcl/ [11] http://www.scriptics.com/tclhttpd/ [12] mailto:lvirden@cas.org Author's Address Andreas Kupries Andreas Computer Laboratories (Me, myself and I) Kongress-Str. 23/15 Aachen, NRW 52070 DE Phone: +49 241 514 998 EMail: a.kupries@westend.com Kupries [Page 17] HOWTO How to write a transformation (channel) July 1999 Appendix A. Glossary A little glossary of terms used in the paper, but so far without much of an explanation (or none). Tcl_Channel: An opaque token for channels, and used by all interfaces accessing channels. Internally it is a pointer to the relevant data structures (Channel*). stack: If one or more transformations are stacked upon an arbitrary other channel I use this word to refer to the whole group of channels. (un)cover: Placing a transformation on a channel C "covers" C, removing the transformation "uncovers" it again. Kupries [Page 18] HOWTO How to write a transformation (channel) July 1999 Appendix B. Acknowledgements This HOWTO was written in XML using the DTD developed by Marshall T. Rose[5] for writing RFC's and I-D's, see RFC 2629[6], and converted to text and HTML with his tool, 'xml2rfc'[7]. The implementation of a stacked transformation was heavily influenced by Jan Nijtmans[8] who suggested the basic idea of using a plain channel to encapsulate them instead of going deep into the buffering system of the generic I/O layer. Matt Newman[9]based his SSL/TLS binding[10] on transformations, a valuable addition whose use by tclHTTPd[11] made the inclusion into the core possible. My thanks to Larry Virden[12] for proof-reading this document. Kupries [Page 19]