Tcl Source Code

When a CR-LF in an incoming stream is split across packets, and the LF-containing packet is not received when the CR-containing packet is read by the channel, the application receives an spurious empty line on gets invocations. To illustrate, run the attached server.tcl and client.tcl files. Seen output is

d:\src\tcltk\console\tests>tclsh client.tcl
<abc>
<defghi>
<>
<jkl>

Note the extra <> line which should not be present.

Note that this is not specific to socket streams although that's where it is easiest to demonstrate. Also happens on Linux (at least) so not Windows specific. The channel needs to be non-blocking and in auto translation mode.

Marking as severe because although the conditions to trigger are extremely rare, any data corruption in transit is a Bad Thing.

Test illustrating potentially undefined behavior (delay dependent):

proc accept {so args} {
  fconfigure $so -translation binary
  puts -nonewline $so "1 line\r"
  puts -nonewline $so "\n2 li"
  flush $so
  #Now force separate packets
  puts -nonewline $so "ne\r"
  flush $so
  if {[incr ::cnt] & 1} {
    after 20 {set ::cnt $::cnt}; vwait ::cnt; # simulate short delay (and client can process events)
  } else {
    after 20; # small delay without processing events (just to avoid large output), client would receive the buffer in single chunk
  }
  puts -nonewline $so "\n3 line"
  flush $so
  close $so
}
set so [socket -server accept 10000]
proc test {} {
  if {1} {
    set chan [socket 127.0.0.1 10000]
  } else {
    set chan stdin
  }
  fileevent $chan readable [list apply {chan {
    if {[gets $chan line] >= 0} {
      puts <$line>
    } elseif {[eof $chan]} {
      close $chan
      set ::done 1
    }
  }} $chan]
  fconfigure $chan -blocking 0 -buffering line
  vwait ::done
  puts "----$::cnt---"
  update
}
timerate {test} 100

The result will show an extra NL by each odd attempt (due to immediate processing of every buffer) and no extra NL by each even attempt (buffers coming as 1 piece):

<1 line>
<2 line>
<>
<3 line>
----1---
<1 line>
<2 line>
<3 line>
----2---
<1 line>
<2 line>
<>
<3 line>
----3---
<1 line>
<2 line>
<3 line>
----4---
<1 line>
<2 line>
<>
<3 line>
----5---

Here is my analysis of what is happening (most action is inside Tcl_GetObj)

When a CR is read from the channel buffer but the accompanying LF has not arrived, the channel code passes up the line as the gets result and also remembers that a CR has been seen by setting a flag

When subsequently the LF is received, the channel subsystem correctly skips it by bumping the pointer into the receive buffer past the LF.

However, later in the code, the pointer offset is reset to its old value which means the next gets sees it as a new line and in auto translation mode returns an empty line as a result.

This last reset of value is the root of the problem. I do not understand all the manipulations of the buffers and offsets and why that is necessary so hesitant to suggest a fix.