Tcl Source Code

 http::geturl returns "data error" with TCL >8.6 and RIPE's REST API. 

With TCL 8.5 it works: 

% puts $tcl_version 
8.5 
% package require http 
2.7.5 
% http::geturl http://www.hp.com/index.html 
::http::1 
% http::geturl http://rest.db.ripe.net/search 
::http::2 
% 


With 8.6 it doesn't: 

% puts $tcl_version 
8.6 
% info patchlevel 
8.6.1 
% package require http 
2.8.7 
% http::geturl http://www.hp.com/index.html 
::http::1 
% http::geturl http://rest.db.ripe.net/search 
data error 
% 


People reported same problems with TCL 8.6.2 & 8.6.3 in this thread:
https://groups.google.com/forum/#!topic/comp.lang.tcl/NiIKhFgEUBY

Fixed; we now prefer the gzip transfer encoding because it's widely known that alternatives are wrong on some servers. It's really not our fault, but we can't help that. The default can be overridden by setting the Accept-Encoding header explicitly (using the normal API mechanism for that); there are examples of what to do in Tcl's test suite (http11.test).

I looked into putting in code that would try to guess from the first few bytes of the stream. We won't be trying to do that! It gets really complicated (in terms of just what to read and when) and just switching to gzip works as well (and it's very widely implemented too). I guess we could have done it with a "do what I mean" decompression method, but that's really horrible.

The fix will ship with the next 8.6 patch release. The workaround is to specify Accept-Encoding to be something like identity, which disables compression and works on 8.5 and 8.6.

BTW, the response you're looking for is:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><whois-resources xmlns:xlink="http://www.w3.org/1999/xlink"><link xlink:type="locator" xlink:href="http://rest.db.ripe.net/search/"/><errormessages><errormessage severity="Error" text="Query param 'query-string' cannot be empty"/></errormessages><terms-and-conditions xlink:type="locator" xlink:href="http://www.ripe.net/db/support/db-terms-conditions.pdf"/></whois-resources>

For more details about what's going on, see this Stack Overflow question and its answers.

I vaguely remember seeing somewhere that compressed HTTP transfers should always be done as gzip because of different interpretations of other names in weird ways.

That said, your test is completely bogus. You need to strip the headers first!

If I do that and then use zlib decompress on the rest (not zlib inflate) then I get what appears to be reasonable content. Which agrees with what my first paragraph says; there's breakage in the way that this part of the HTTP specification is interpreted. Interoperable code uses gzip.

anyone know what the correct inflated result is?

The error is definitely in zlib inflate:

======

set s [socket rest.db.ripe.net 80]
fconfigure $s -translation crlf -buffering line
puts $s "GET /search HTTP/1.1"
puts $s "Host: rest.db.ripe.net"
puts $s "Accept-Encoding: deflate"
puts $s "Connection: close"
puts $s ""
puts $s ""

fconfigure $s -encoding binary -translation binary

set bits [read $s]


puts "got [string length $bits] bytes"
puts [zlib inflate $bits]
======

got 556 bytes
data error
    while executing
"zlib inflate $bits"

http://core.tcl.tk/tcl/info/e861cc28b065288e

is the checkin that breaks things.  Still possible
that [zlib] troubles are at the root, but seems more
likely this patch just gets some things wrong.

Is this really a [zlib] bug then?

As described in the linked c.l.t thread, the problem appears to be due to the response being compressed. Tcl appears unable to decompress it correctly. A workaround is to disable the Accept-Encoding header:

% package require http
2.8.8
% http::geturl http://rest.db.ripe.net/search \
    -headers [list Accept-Encoding ""]
::http::1