TIP 409: UDP in Tcl

Login
Bounty program for improvements to Tcl and certain Tcl packages.
Tcl 2017 Conference, Houston/TX, US, Oct 16-20
Send your abstracts to tclconference@googlegroups.com
by Aug 21.
Author:         Alexandre Ferrieux <alexandre.ferrieux@gmail.com>
Author:         Colin McCormack <colin@chinix.com>
State:          Draft
Type:           Project
Vote:           Pending
Created:        17-Aug-2012
Post-History:   
Keywords:       udp,datagram,message
Obsoletes:      391
Tcl-Version:    8.7

Abstract

This TIP adds support for UDP in Tcl, with a pure event approach to packet reception (as opposed to hijacking the existing channel infrastructure).

Rationale

UDP support is a long-awaited feature in Tcl. Some extensions have traditionally supported it through the channel abstraction. This is of course doable, but there is a non-negligible cost in both complexity and performance due to the impedance mismatch between the "fluid" model behind channels and the "granular" world of datagrams (where boundaries are significant). Another discrepancy is the existence of (per-packet) metadata, like the source address and port of an incoming UDP packet, which do not fit nicely in the (per-connection) options of the channel API (via fconfigure).

Once this mismatch is acknowledged, it is easy to identify a better home for UDP in Tcl: let it be a direct event source (for the receive side), just like Tk or Snack.

Indeed, hooking a callback for packet reception is a natural fit with Tcl's event-driven tradition, while preserving packet boundaries and easily exposing per-packet metadata.

Sending is trivially handled by a direct per-packet call (but not disguised as a puts). Again, this naturally allows for boundary preservation and metadata specification.

This TIP asks for UDP-in-the core for two strong reasons:

We then believe that the ubiquity of the need and the lack of external dependencies justify for UDP the first-class status that TCP enjoys (regardless of the channel vs event models). Note that an acceptable compromise would be a bundled extension, ie distributed-with-the-core (in "pkgs/").

Specification

The new udp ensemble hosts three subcommands: new, create, and names

udp new ?options?

udp create name ?options?

both create an UDP endpoint ;

udp names

returns the list of existing such endpoints.

Lifecycle

Following the traditions of Tk and Snack, udp new and udp create return a Tcl command, which takes subcommands implementing the needed verbs. By analogy with the corresponding TclOO constructors,

An UDP endpoint is thus created by:

	   set u [udp new ...]

or

   udp create foo ...

Once created, its configuration can be tweaked by:

$u configure -option value

and retrieved with

set value [$u configure -option]

or

set fullconf [$u configure]

To destroy the endpoint, use:

$u destroy

or, as in Tk, destroy the command:

rename $u {}

or the whole namespace.

Status of options

Among options, a few may only be specified on creation:

  • local interface+port

  • (optional) connect() target's address+port

  • reuse flag (described in Advanced Features)

while all the other options may be specified both on creation or through [$u configure].

Addresses and Ports

The first two creation-only options have the following syntax:

set u [udp new ?-local addressport_? ]

set u [udp new ?-remote addressport_? ]

where address_port pairs are represented as 2-element Tcl lists.

set address_port [list $address $port]

Addresses can be given either as numeric IP (v4 or v6) addresses, or as hostnames; in the latter case, a synchronous DNS resolution is performed before actual use, just like for socket.

In the case of -local, $address can be specified as "*", meaning INADDR_ANY, and $port can be 0, asking for the OS to select a free port. Thus a dynamic port on all interfaces can be requested with

   set u [udp new -local [list * 0]]

In case of port 0, after creation of the endpoint, the actual port chosen by the OS can be retrieved with [$u configure]:

   puts "Local port is: [lindex [$u configure -local] 1]"

In the case of -remote, both address and port must be fully specified. The semantics, as is well known, is to tell the kernel (a) to forbid sending to any other destination, and (b) to discard all incoming packets sent by another source.

Sending a Message

Sending is done, unsurprisingly, with the send verb:

$u send payload ?dest_addr_port?

where dest_addr_port is a pair as above.

Its blocking semantics is that of the underlying send()/sendto() system calls: it typically returns immediately, though the hardware may buffer the data for some time, and delivery is not guaranteed. The payload is interpreted as a byte array that holds the entire content of the UDP message to send.

The destination parameter can be omitted if the endpoint has been created with the -remote option (connected mode).

Receiving a message asynchronously

Asynchronous (i.e. event-driven) message reception is done by specifying a listener callback to new/create or configure:

$u configure -listener command_prefix

Subsequently, when an incoming packet hits Tcl in the event loop, the command_prefix is called with the endpoint identifier, payload and metadata:

{*}command_prefix $u payload metadata_dict

where payload is the byte array of the UDP payload and metadata_dict is a dict containing at least two options:

-remote address_port

-local address_port

When read from options or metadata, all address components apart from "*" are returned in numeric form; no reverse DNS is ever performed.

Note that -local may carry more information than a configured -local where the address part is "*", by identifying which of the system's several interfaces was targeted.

Also note that command_prefix is a single command, possibly with arguments, that will be expanded on invocation (hence it must be a proper list); it is *not* an arbitrary script as in Tk's **bind_ tradition.

When command_prefix is the empty list (which is the default), the notifier gives up interest in the underlying UDP socket; this allows to keep the port bound while letting the OS buffer any incoming packets (up to a configurable limit) without any script-level handling, while leaving the event loop active. This is similar to setting a fileevent on a channel to the empty string.

Receiving a message synchronously

Synchronous message reception is done with the recv verb:

set payload [$u recv metadatadictvar]

The payload is returned as a byte array, and the variable passed as argument receives the metadata dict just described.

Advanced features

The following few verbs and options control extra IP features that are typically useful in popular UDP applications like media streaming:

$u configure -sendbuffer buffersize

$u configure -receivebuffer buffersize

Set the OS's send (resp. receive) buffer sizes to the given values (in bytes).

$u join multiaddr ?-source sourceaddr?

Joins the multicast group multiaddr, optionally using IGMPv3's source membership target sourceaddr.

$u leave multiaddr ?-source sourceaddr?

Leaves the same multicast group with or without source membership.

set u [udp new -reuse 1]

Sets the SO_REUSEADDR socket option, so that multiple processes may bind to the same local port and interface, and receive the same packets. Creation-time only.

$u configure -ttl ttl

Sets the TTL of subsequent outgoing packets. Some operating systems typically reduce the default TTL to one for multicast packets; this override may thus come in handy.

Rejected Alternatives

Sources of Inspiration

This TIP builds on experience with previous attempts at UDP extensions: TclUDP, ceptcl, Duft, along with intensive practice with unpublished work, and borrows ideas from all of them.

Copyright

This document has been placed in the public domain.

History