Tcl Source Code

On the msys/mingw build on Windows, 'make test' gives:



==== chan-io-28.7 Tcl_CloseEx (half-close) socket FAILED
==== Contents of test case:

    set echo [makeFile {
        proc accept {s args} {set ::sok $s}
        set s [socket -server accept 0]
        puts [lindex [fconfigure $s -sockname] 2]
        flush stdout
        vwait ::sok
        fconfigure $::sok -buffering line
        while {[gets $::sok line]>=0} {puts $::sok $line}
        puts $::sok DONE
        exit 0
    } echo.tcl]
    set ::ff [open "|[list [interpreter] $echo]" r]
    gets $::ff port
    set ::s [socket 127.0.0.1 $port]
    puts $::s Hey
    close $::s w
    set timer [after 1000 {set ::done Failed}]
    set ::acc {}
    fileevent $::s readable {
        if {[gets $::s line]<0} {
            set ::done Succeeded
        } else {
            lappend ::acc $line
        }
    }
    vwait ::done
    after cancel $timer
    close $::s r
    close $::ff
    list $::done $::acc

---- Result was:
Succeeded {}
---- Result should have been (exact matching):
Succeeded {Hey DONE}
==== chan-io-28.7 FAILED

Backported to 8.[45] branches. Attached patch for reference.

File Added - 324684: rst.patch

allow_comments - 1

Just committed the Win ECONNRESET->EOF change; indeed extending the error domain is not an option in a bugfix (it requires a TIP).
But David, please do file a TIP for the -explicit-rst fconfigure flag.

Thanks.  Yes, 5 and 11 are extra fluff, but I'm hoping I might find a difference with the local state.  8 does an implied 7 anyways but nothing is wrong with being explicit.  12 might be needed on windows, though.  I'll have to run some experiments to verify that TIME_WAIT is skipped.

Correct, though 5,7,11,12 are unneeded, in the sense that exactly the same exchanges occur on the wire without them.

Thinking about proper the graceful close method, I made this chart

    local (host, initiator)    peer (peer, active closer)
(1) send() last data
(2) shutdown(SD_SEND)
(3) recv() last data..
(4) recv() with zero bytes
(5) shutdown(SD_RECV)
(6) send() last info
(7) shutdown(SD_SEND)
(8) closesocket()
port is locked with TIME_WAIT
(9) recv() last data..
(10) recv() with zero bytes
(11) shutdown(SD_RECV)
(12) setsockopt(SO_LINGER, l_onoff=any, l_linger=0)
(13) closesocket()
port is not locked with TIME_WAIT

Is the above accurate and correct?

> (b) be frequently used as a diagnosis tool by server writers.

There is a big difference between a normal close and an aborted one.  Let's not hide ECONRESET anymore.

Do silent resets exist because half-close didn't?

> (2) Perusing the [fconfigure -blocking] bit to decide the behavior just
> like for pipes (which does make sense ex nihilo) would pose a compatibility
> issue with sockets (imagine apps talking to rude servers that don't react
> to a FIN ... they would block forever in [close])

Not exactly..  Reviewing the close chart @ http://msdn.microsoft.com/en-us/library/ms738547(VS.85).aspx and http://msdn.microsoft.com/en-us/library/ms737582(VS.85).aspx I see the following behavior as possible:

    client     |     server

(1)close $s w
                   (2)if {[eof $s]} { finishSending $s; fconfigure -blocking 1; close $s }
(3)if {[eof $s]} {fconfigure -blocking 1; close $s}

As the client was the initiator, TIME_WAIT doesn't have to collect on the client as the EOF indicator describes the closure as complete.  Just let shutdown return immediately and collect the ACK later with setsockopt(SO_DONTLINGER) prior to closesocket().  Which happens to be the default anyways.

Now if you would prefer an abortive close:

    client     |     server

(1)fconfigure -blocking 0; close $s
                 (2) don't care

Same as above, but setsockopt(SO_LINGER) would be set with l_onoff=1,l_linger=0 prior to closesocket() for an immediate abortive close.  closesocket() _will_ _not_ _block_ with that setting.

This unfortunately appears to be in reverse order and might render old code possibly broken.  But given the new ability to do a half-close to initiate a graceful and that resets are already silent EOFs, all might be well.

Re blocking close: afraid the analogy is a bit exaggerated because:
(1) A child process "naturally" communicates success/failure via the exit status, while the RST on TCP is a much less meaningful indication. Indeed, SO_LINGER is rarely used by servers (like inetd, httpd) to fine-tune connection end (this would imply setting to hardclose  at the beginning of processing, and going back to normal close on completion, so that a kill or segfault would automatically send an RST...). So on the client side it would seem hard to depend on it.
(2) Perusing the [fconfigure -blocking] bit to decide the behavior just like for pipes (which does make sense ex nihilo) would pose a compatibility issue with sockets (imagine apps talking to rude servers that don't react to a FIN ... they would block forever in [close])

In summary, you should have suggested that idea back 20yrs ago when SO_LINGER still stood a chance to (a) be uniformly implemented by TCP stacks and (b) be frequently used as a diagnosis tool by server writers...

note to self:  verify that a correct graceful close at the peer doesn't produce a useless TIME_WAIT condition.

According to Tcl convention with pipes, a blocking pipe must wait for a graceful close to collect the exitcode, yet a non-blocking pipe, should be dumped.  Would the same concept of a blocking close imply a graceful close procedure with the non-graceful to create an RST, ala setsockopt(SO_LINGER,...) with l_onoff=nonzero and l_linger=zero for a hardclose?

Just random ponderings..

That's a good idea, though I'm not sure it is widely applied over the test suite ;-)
Note, though, that the patch I submitted in 1028264 solves the "server" side issue (i.e. prevents Tcl from sending RST on exit).
So if the patch is committed, the current bug will be *much* harder to reproduce (let alone in a want-to-fail test).

Thinking a bit about this bug, I seem to remember that (sometime!) someone mentioned that they had a "fix" by changing exactly how the server side worked at the Tcl level. If that's so, perhaps a good thing to do is to duplicate the test and apply the "fix" to one of them so that we check both the tidy and the messy cases.

Of course, if this has already been done then ignore this comment. :-)

See my comment of 2009-01-03 12:53 below: in the presence of an incoming RST, any data in the local OS's queue (i.e. not yet pulled out by a recv()) SHOULD be flushed. I know it is a SHOULD and not a MUST, but at least Winsock cannot be blamed for that: so it is valid and even recommended for the stack to "fast-track" the RST...

That said, let's understand the race condition in detail (your message on FD_CLOSE is a strong hint) and iron out the ECONNRESET :)

I meant doesn't behave as a queue

> Have some confidence and just fix it.

I like that way of thinking.  We first need to find it, though.  The event thread might be our enemy in this as it does behave as a queue and appears to put errors ahead of valid data.  IMO, but not verified.

> Andreas, when you're back, do you know why he said that "the higher level
> of the IO mechanism" would be upset by seeing an EOF instead of an
> ECONNRESET ?

No, I do not know. Sorry. Asking JL (aka Jacob Levy) may still be our best bet, despite it being 12yrs since he worked on the IO parts.

> Do you see an objection to re-establishing symmetry (unix behavior) ?

No, I don't. Mainly because I do not know the Win32 code that well.
IMHO I can see two possibilities for the comment being true, without touching the generic IO layer. Which does handle EOF instead of ECONNRESET well, as evidenced by the unix driver. These possibilities are:
(a)  The win32 code/driver may itself be in two layers and the higher of the two doesn't like EOF over ECONNRESET.
(b) The win32 driver has some mismatch with the generic layer which is exposed when you use EOF, and keeping to use ECONNRESET pastes over it.

This is all speculation on my part however.

I do agree that unix and win32 behaving differently is a Bad Thing (tm) and should be fixed.
As such I would also go with transforming ECONNRESET to EOF. Then we can see if any test cases break.

If there's no API change (including no change to the list of socket [fconfigure] options) then why would you even consider a TIP? Have some confidence and just fix it.

Yes but do we need a TIP to change an Heisenbuggy behavior into a stable one that has been in place since the beginning on unix, with no API change neither C nor Tcl ?

Just a quick note to mention that a TIP to help make platforms consistent is about the only sort of TIP that stands a chance of making it to a vote for 8.6 now, and we'll be looking for minimality. (The more complex the new feature, the harder it makes it to bring 8.6.0 to life...)

Alexandre,  There might be a timing issue in tclWinSock.c:TcpInputProc() here:

/*
 * If an error occurs after the FD_CLOSE has arrived, then ignore the
 * error and report an EOF.
 */

if (infoPtr->readyEvents & FD_CLOSE) {
    infoPtr->flags |= SOCKET_EOF;
    bytesRead = 0;
    break;
}


readyEvents is set by the event thread.  Timing maybe such that it had arrived behind this FD_READ (that is being serviced now) with a WSAECONNRESET or WSAEDISCON ahead of us but not yet read upon, thus a silent reset, maybe?  And truncated data?  I don't know if winsock can't wind itself into such a state.  No available debugging tools to play with at the moment..  sorry.  I would dip in if I could, but can't at the moment.

agreed.  Make win consistent with unix.  TIP a new fconfigure.

Yes but the way it is now on Win is at best half-reliable (since we see sometimes EOF instead of ECONNRESET).
So I agree with -softreset 0 by default, in the sense that we implement the default in 8.6 (symmetry with unix), and TIP for the option in 8.7.

I guess -softreset would be a better name.  defaults to off.

a new fconfigure called -ignorereset?

Now this is getting interesting.  A reset is an error.  It shows an inappropriate closure that might have data loss.  Is it correct that it is hidden?

Andreas, when you're back, do you know why he said that "the higher level of the IO mechanism" would be upset by seeing an EOF instead of an ECONNRESET ?
Do you see an objection to re-establishing symmetry (unix behavior) ?

JL = Jacob Levy, one of the Scriptics staffers. The last I heard he was at Yahoo, but I doubt he'd remember the details of this. After all, it's over 12 years ago...

Some progress on the ECONNRESET/EOF issue:
Detailed analysis on Linux, shows that on receiving an RST, the [gets] yields -1 (EOF) in all cases (i.e whether the recv() is initiated before reception of RST or not, while at the syscall level (strace) we can see that the recv() returned ECONNRESET in both cases. 

In summary: on unix, ECONNRESET is purely an internal matter, purposefully turned into an EOF, as witnessed by  the following code in tclUnixChan.c:

    if (errno == ECONNRESET) {
/*
 * Turn ECONNRESET into a soft EOF condition.
 */
return 0;

while on Windows, it is exposed as an error, as commented in this comment from 1996 in the "changes" file:

   8/20/96 (bug fix) No longer masking ECONNRESET on Windows sockets so
   that the higher level of the IO mechanism sees the error instead of
   entering an infinite loop. (JL)

Very old indeed... and beyond 8.x CVS history ! Do you know who JL is ?
Anyway, I don't see why turning ECONNRESET into EOF (just like on unix) would create an infinite loop...

Yet again, more proof that a bi-directional channel is really two independent simplex connections ;)  Bug 1028264 is very interesting.

OK I'll do some more hunting to bring the ECONNRESET/EOF balance close to 50-50 :)

Regarding SF, starting from https://sourceforge.net/tracker/?atid=110894&group_id=10894&func=browse
if you go into the upper-right box tagged "Trackers" near the Search button and Advanced link, and just paste a bug ID there, it brings you a single solution which has the proper link...

Should be ECONNRESET only, I'd assume.  I don't see that problem here.  Though I haven't stared at packets.

What SF box?  the browse page just searches on keywords.  At least mozilla's BugDB software knows the string 'Bug 12345' is meant to be a link.  I asked the SF folks ages ago to add cross-referencing features such as 'depends on' and 'similar to'..  That was like 7 years ago.

Sorry, I'm being unclear :)
The WSACleanup bug explains the *generation* of the RST (in the server); the race condition is on *reception* of that RST on the other side.

The other URL is https://sourceforge.net/tracker/index.php?func=detail&aid=1028264&group_id=10894&atid=110894
(I gave only the bug number assuming it was easy to paste it in SF's box)

OK to fix the tear-down first (other bug)
But when we're done and back here, what I called "Problem 2" is that on reception of an RST it seems unpredictable whether the outcome will be an EOF or an ECONNRESET.

What race condition?  If WSACleanUp() hit, expect everything open to be screwed.  What's the URL to that other bug as the search functions are hideous on SF?  Please define "problem 2" in more detail.  No [close] bad, with [close] all good, therefore fix the tear-down first and come back to this.

Yes, please read my other comments ;-)
I prefer to leave the bug open as it is not normal to generate an RST on exit.
And I really want to grok Problem 2 !

With 'close $::sok' in the echo server, I don't see that problem at all.  There is no bug anymore.

Yes, please go to the other bug, and see my last comments about the sequence of events. Basically the process-wide SocketExitHandler, which is called much too soon in the sequence, does all the harm. See my proposed solution there and please amend it (there). In the current entry, let's only talk about "problem 2": the race condition on incoming RST that makes them sometimes just make an EOF, sometimes raise "connection reset by peer". Any idea ?

Yup, you nailed it alright.  I feel silly.  I should have seen it.

If WSACleanup hits first, all bets off.  Shouldn't Tcl close open channels as part of the tear-down?  Is this one of those ExitHandler hits, but the per-thread didn't things?  Call the per-thread from the ExitHandler when TheadId is the same?  Just reading I get a bit blurry from the thread transfer stuff.

By the way, this socket finalization problem was already apparent in 1028264.
This is interesting because the analysis there shows that the culprit is the mess in multithreaded finalization; so the problem might not be Winsock, but the fact that the Windows build is threaded.
I'm not closing any of the two as Dup yet; I'd like to know about problem 2 first...

Got it !!! ... At least for the first problem, "why is the server sending an RST ?".

This seems to be linked to the absence of [close] before [exit].
Just adding a close before exit in the server makes the test succeed reliably.
I'm *not* updating the test because this is a bug. On unix for example, a close-less exit() doesn't result in an RST.
David, any idea why the Winsock teardown does this ?
*And* we still need an explanation for problem 2 ;-)

Further analysis of the Heisenbug part: it is not completely abnormal !
Indeed, RFC793 says on http://www.freesoft.org/CIE/RFC/793/34.htm

     ESTABLISHED
      FIN-WAIT-1
      FIN-WAIT-2
      CLOSE-WAIT

        If the RST bit is set then, any outstanding RECEIVEs and SEND
        should receive "reset" responses.  All segment queues should be
        flushed.


This means that two situations may occur:

  - either we manage to drink all data up to the Tcl level before the RST occurs
  - or we lag somewhere (introduce an [after] to see this) and go back to doing recv() while the TCP stack has already received the RST

These two perfectly normal situations lead to different sizes of data on the receive side.
What is *not* normal though is that the Tcl-level outcome has *three* (not two) cases:

  (a) full data
  (b) connection reset by peer
  (c) truncated data

In particular, adding an [after 1000] before the reading loop in my above synchronous client deterministically yields (c), while without the delay we get (b) with prob==0.1 and (a) with prob==0.9 (and also (c) with prob==epsilon)

So the bug hunt can now be summarized as:
  - find out why the Tcl server is sending RST
  - iron out the Tcl-level outcomes (b vs. c)

Even more interesting, the sequence of packets is exactly the same regardless of the outcome of the probability-0.1 test !
That is, there are at least two issues:

 - the server sending RST
 - the client Heisenbugging on that RST

TBC...

Some progress at last !
For some reason one trick that had not worked before, works :)
Basically add a route to the host itself (the LAN address, not 127.0.0.1) forcing it to go through the home router:

   route add 192.168.1.11 mask 255.255.255.255 192.168.1.1

This has the delicious property of going through the non-loopback while remaining on the machine (provided that address is used instead of 127.0.0.1). The only ill effect is that every packet is seen twice, and Wireshark is not too smart about them (complains about out-of-sequence or dup'ed acks), but that doesn't prevent human analysis !

Now the good news is that the bug shows up all the same with this "instrumented loopback" !
I'm happy to have been wrong about loopback-specific issues ;-)

Will continue investigations, but I have already found the following:

 - the FIN+DATA, when it occurs (not with a Tcl server), is handled without bugs.

 - what occurs in Tcl-server tests, is rather "DATA ; RST" (the ';' means 2 separate packets)

So it seems all this is due to an RST from the server where a FIN would have been in order.
Have to dig to understand why that RST is sent...

http://www.freesoft.org/CIE/RFC/793/23.htm

Just for reference.

Beyond ethereal, I never explored much.  I did try -myaddr [info hostname] but had the same effect.  I see the failures and successes, too, at the same percentage.

Does this work across two machines where the other hosts the echo server?  I haven't tried that yet.  Is this purely a localhost issue?

Sure. However, as mentioned earlier the problem only occurs on the loopback interface. Moreover, the code I last posted below has a probability of failure of ~ 0.1 , so you have to run it many times to see it. But when you see it, it's definitely there.

David, do you know of a better tool than CommView, that would both see the loopback *and* display FINs ?

Alexandre, I've been looking at this issue a bit.  Apparently, Google is rude.  I guess they just consider a client initiated graceful close to be a full close.  There was always some weird stuff around http/1.0 anyways.  I guess they are being defensive.  By doing so, they have to collect the TIME_WAITs on their side.

I tested some other services, and got an appropriate closure without loss.

proc go1 {} {
    set s [socket mail.pobox.com smtp]
    fconfigure $s -buffering line -blocking 0
    fileevent $s readable [list gotreadable $s]
    puts $s "ehlo [lindex [fconfigure $s -sockname] 1]"
    puts $s "quit"
    close $s w
}

proc go2 {} {
    set s [socket www.google.com http]
    fconfigure $s -buffering line -blocking 0
    fileevent $s readable [list gotreadable $s]
    puts $s "GET / HTTP/1.1\n"
    close $s w
}

proc gotreadable {chan} {
    if {[string length [set err [fconfigure $chan -error]]]} {
        puts $err
        close $chan
        return
    }
    if {[gets $chan data] != -1} {
        puts $data
    }
    if {[eof $chan]} {
        puts "got eof"
        close $chan
    }
}

Hmm, withdrawing that last remark about separate DATA and FIN: I've just noticed that Tcl sockets don't use TCL_NODELAY, hence the Nagle method is at work. That makes a plausible link with timing: if the data and FIN are within the Nagle delay (200ms), the TCP stack may glue them together !

Update: we can release one constraint !
Indeed, I get spurious "connection reset by peer" and EOFs even with code devoid of any half-close (see below).
So only one FIN from B is needed. Moreover, the code below clearly separates the last data write from the close (flush in the server, distinct from exit), so the FIN packet itself is separate from the data. It seems only the *timing* plays a role in that nasty bug.

#client
set ::s [socket $host $port]
puts $::s Hey1
puts $::s Hey2
puts $::s Hey3
flush $::s
while {[gets $::s line]>=0} {
    lappend ::acc $line
}
puts RESULT:$::acc

#server
proc accept {s args} {set ::sok $s}
set s [socket -server accept $port]
vwait ::sok
set cnt 0
while {[gets $::sok line]>=0} {puts $::sok $line;puts stderr XMIT:$line;incr cnt;if {$cnt>=3} break}
puts $::sok DONE
flush $::sok
puts stderr WRITTEN
exit 0

David, thanks for looking into this (though anonymously ;-)

However, this google example, as indicated previously, is a red herring for the bug under scrutiny, because if you take a network snoop of the interaction you'll see that when that line is uncommented, google refrains from sending the HTTP reply altogether. I assume this is an anti-attack strategy of sorts...

According to my tests, the bug is witness only when the following conditions are met:

 - On loopback interface
 - A sends a FIN
 - B writes and sends a FIN all in one packet
 - A receives the FIN ignoring the data
 - A is Tcl

I like this TIP very much.  I was pondering what to do about a graceful close for my iocp work, too.  close2proc is the right way, and this is just proof of it.

Yes something is wrong..  the following script bit shows it for me:

set s [socket www.google.com 80]
fconfigure $s -buffering line
puts $s "GET / HTTP/1.0\n"
#close $s w
read $s
close $s

Uncommenting the line that causes shutdown() to be called breaks it, but shouldn't.  I can't look at it much further at the moment, but I'll do some debugging soon after the holiday madness.

Reporting progress: using CommView I was able to trace the loopback interface. However, something seems to prevent the normal use of the FIN TCP flag on that faked interface (Windows is definitely weird), since a normal socket closes without a single FIN being observed (while they are normally seen on a non-faked interface).
Anyway, I wrote standalone C executables (variants of "nc") for both sides and they don't display the Tcl bug (ie only the Tcl-Tcl combination displays it), so there's hope ;-)
One key thing seems to be the timing of the write-back and close by the server: an [after] between the child-side [flush] and [exit] removes the problem. In TCP terms this would seem to confirm Andreas' hypothesis that a data-loaded packet bearing the second FIN induces loss of the data by "fast-tracking" the interpretation of the FIN.
Now to find why in tclWinSock.c...

Could well be, though the google example is no proof (here the HTTP server must be doing some optimization to resist attacks).
Still trying to pinpoint it. Windows' unability to wireshark on loopback interface is a nightmare... Do you know of an alternative ?
Or a public echo server ?

Oh, could this be where Windows, the OS, is fast-tracking the close operation, discarding buffered data ? IIRC this was trouble for something else as well. I do not recollect details however :(

Getting closer I think :-)
The following code:

set s [socket www.google.fr 80];puts $s "GET / HTTP/1.0\n";close $s w;after 1000;puts [read $s];close $s

consistently displays en empty line. I have the impression that it shouldn't. (Am I right ?)
Wireshark says that a FIN is sent after the GET, and that google responds by another FIN with no data.
Have to check the RFC again to check why the Windows shutdown() does that bad variant of FIN.

Oh, now I can again with HEAD. Heisenbug.. race condition ?

Kevin, are you still seeing it ? Cannot seem to repro.

2444274 is a gob of multiple problems.

My changes fixed the underlying flush handling in the generic IO layer.

dgp's changes, as I understood it, prevent the test cases from writing errorInfo contents to stderr, causing the chanio.test reported as exiting with error, for -singleproc 0.

For what it's worth, this happened *after* dgp committed changes purporting to fix 2444274

Yes, see the latest comments at
https://sourceforge.net/tracker2/index.php?func=detail&aid=2444274&group_id=10894&atid=110894