Tcl Source Code

2013-09-17  Donal Fellows  <[email protected]>

	* generic/tclBinary.c (BinaryEncodeUu, BinaryDecodeUu): [Bug 2152292]:
	Corrected implementation of the core of uuencode handling so that the
	line length processing is correctly applied.
	***POTENTIAL INCOMPATIBILITY***
	Existing code that was using the old versions and working around the
	limitations will now need to do far less. The -maxlen option now has
	strict limits on the range of supported lengths; this is a limitation
	of the format itself.

2013-09-09  Donal Fellows  <[email protected]>

	* generic/tclOOMethod.c (CloneProcedureMethod): [Bug 3609693]: Strip
	the internal representation of method bodies during cloning in order
	to ensure that any bound references to instance variables are removed.

2013-09-01  Donal Fellows  <[email protected]>

\fBbinary scan\fR, does the opposite: it extracts data
from a binary string and returns it as ordinary Tcl string values.
.VS 8.6
The \fBbinary encode\fR and \fBbinary decode\fR subcommands convert
binary data to or from string encodings such as base64 (used in MIME
messages for example).
.VE 8.6
.PP
Note that other operations on binary data, such as taking a subsequence of it,
getting its length, or reinterpreting it as a string in some encoding, are
done by other Tcl commands (respectively \fBstring range\fR,
\fBstring length\fR and \fBencoding convertfrom\fR in the example cases).  A
binary string in Tcl is merely one where all the characters it contains are in
the range \eu0000\-\eu00FF.
.SH "BINARY ENCODE AND DECODE"
.VS 8.6
.PP
When encoding binary data as a readable string, the starting binary data is
passed to the \fBbinary encode\fR command, together with the name of the
encoding to use and any encoding-specific options desired. Data which has been
encoded can be converted back to binary form using \fBbinary decode\fR. The

\fBuuencode\fR
.
The \fBuuencode\fR binary encoding used to be common for transfer of data
between Unix systems and on USENET, but is less common these days, having been
largely superseded by the \fBbase64\fR binary encoding.
.RS
.PP
During encoding, the following options are supported (though changing them may
produce files that other implementations of decoders cannot process):
.TP
\fB\-maxlen \fIlength\fR
.
Indicates that the output should be split into lines of no more than
\fIlength\fR characters. By default, lines are split every 61 characters, and
this must be in the range 3 to 85 due to limitations in the encoding.
.TP
\fB\-wrapchar \fIcharacter\fR
.
Indicates that, when lines are split because of the \fB\-maxlen\fR option,
\fIcharacter\fR should be used to separate lines. By default, this is a
newline character,
.QW \en .
.PP
During decoding, the following options are supported:
.TP
\fB\-strict\fR
.
Instructs the decoder to throw an error if it encounters unexpected whitespace
characters. Otherwise it ignores them.
.PP
Note that neither the encoder nor the decoder handle the header and footer of
the uuencode format.
.RE
.VE 8.6
.SH "BINARY FORMAT"
.PP
The \fBbinary format\fR command generates a binary string whose layout
is specified by the \fIformatString\fR and whose contents come from
the additional arguments.  The resulting binary value is returned.

universal.  To transfer floating-point numbers portably between all
architectures, use their textual representation (as produced by
\fBformat\fR) instead.
.SH EXAMPLES
.PP
This is a procedure to write a Tcl string to a binary-encoded channel as
UTF-8 data preceded by a length word:
.PP
.CS
proc \fIwriteString\fR {channel string} {
    set data [encoding convertto utf-8 $string]
    puts -nonewline [\fBbinary format\fR Ia* \e
            [string length $data] $data]
}
.CE
.PP
This procedure reads a string from a channel that was written by the
previously presented \fIwriteString\fR procedure:
.PP
.CS
proc \fIreadString\fR {channel} {
    if {![\fBbinary scan\fR [read $channel 4] I length]} {
        error "missing length"
    }
    set data [read $channel $length]
    return [encoding convertfrom utf-8 $data]
}
.CE
.PP
This converts the contents of a file (named in the variable \fIfilename\fR) to
base64 and prints them:
.PP
.CS
set f [open $filename rb]
set data [read $f]
close $f
puts [\fBbinary encode\fR base64 \-maxlen 64 $data]
.CE
.SH "SEE ALSO"
encoding(n), format(n), scan(n), string(n), tcl_platform(n)
.SH KEYWORDS
binary, format, scan
'\" Local Variables:
'\" mode: nroff
'\" fill-column: 78
'\" End:

			    int objc, Tcl_Obj *const objv[]);
static int		BinaryDecodeHex(ClientData clientData,
			    Tcl_Interp *interp,
			    int objc, Tcl_Obj *const objv[]);
static int		BinaryEncode64(ClientData clientData,
			    Tcl_Interp *interp,
			    int objc, Tcl_Obj *const objv[]);
static int		BinaryDecode64(ClientData clientData,
			    Tcl_Interp *interp,
			    int objc, Tcl_Obj *const objv[]);
static int		BinaryEncodeUu(ClientData clientData,
			    Tcl_Interp *interp, int objc,
			    Tcl_Obj *const objv[]);
static int		BinaryDecodeUu(ClientData clientData,
			    Tcl_Interp *interp,
			    int objc, Tcl_Obj *const objv[]);

/*
 * The following tables are used by the binary encoders
 */

    { "format", BinaryFormatCmd, TclCompileBasicMin1ArgCmd, NULL, NULL, 0 },
    { "scan",   BinaryScanCmd, TclCompileBasicMin2ArgCmd, NULL, NULL, 0 },
    { "encode", NULL, NULL, NULL, NULL, 0 },
    { "decode", NULL, NULL, NULL, NULL, 0 },
    { NULL, NULL, NULL, NULL, NULL, 0 }
};
static const EnsembleImplMap encodeMap[] = {
    { "hex",      BinaryEncodeHex, TclCompileBasic1ArgCmd, NULL, NULL, 0 },
    { "uuencode", BinaryEncodeUu,  NULL, NULL, NULL, 0 },
    { "base64",   BinaryEncode64,  NULL, NULL, NULL, 0 },
    { NULL, NULL, NULL, NULL, NULL, 0 }
};
static const EnsembleImplMap decodeMap[] = {
    { "hex",      BinaryDecodeHex, TclCompileBasic1Or2ArgCmd, NULL, NULL, 0 },
    { "uuencode", BinaryDecodeUu,  TclCompileBasic1Or2ArgCmd, NULL, NULL, 0 },
    { "base64",   BinaryDecode64,  TclCompileBasic1Or2ArgCmd, NULL, NULL, 0 },
    { NULL, NULL, NULL, NULL, NULL, 0 }

    Tcl_Interp *interp,
    int objc,
    Tcl_Obj *const objv[])
{
    Tcl_Obj *resultObj = NULL;
    unsigned char *data = NULL;
    unsigned char *cursor = NULL;

    int offset = 0, count = 0;

    if (objc != 2) {
	Tcl_WrongNumArgs(interp, 1, objv, "data");
	return TCL_ERROR;
    }

    TclNewObj(resultObj);
    data = Tcl_GetByteArrayFromObj(objv[1], &count);
    cursor = Tcl_SetByteArrayLength(resultObj, count * 2);
    for (offset = 0; offset < count; ++offset) {
	*cursor++ = HexDigits[((data[offset] >> 4) & 0x0f)];
	*cursor++ = HexDigits[(data[offset] & 0x0f)];
    }
    Tcl_SetObjResult(interp, resultObj);
    return TCL_OK;
}

/*
 *----------------------------------------------------------------------

 *----------------------------------------------------------------------
 *
 * BinaryEncode64 --
 *
 *	This implements a generic 6 bit binary encoding. Input is broken into
 *	6 bit chunks and a lookup table passed in via clientData is used to
 *	turn these values into output characters. This is used to implement
 *	base64 binary encodings.
 *
 * Results:
 *	Interp result set to an encoded byte array object
 *
 * Side effects:
 *	None
 *

    ClientData clientData,
    Tcl_Interp *interp,
    int objc,
    Tcl_Obj *const objv[])
{
    Tcl_Obj *resultObj;
    unsigned char *data, *cursor, *limit;

    int maxlen = 0;
    const char *wrapchar = "\n";
    int wrapcharlen = 1;
    int offset, i, index, size, outindex = 0, count = 0;
    enum {OPT_MAXLEN, OPT_WRAPCHAR };
    static const char *const optStrings[] = { "-maxlen", "-wrapchar", NULL };

		TCL_EXACT, &index) != TCL_OK) {
	    return TCL_ERROR;
	}
	switch (index) {
	case OPT_MAXLEN:
	    if (Tcl_GetIntFromObj(interp, objv[i+1], &maxlen) != TCL_OK) {
		return TCL_ERROR;
	    }
	    if (maxlen < 0) {
		Tcl_SetResult(interp, "line length out of range", TCL_STATIC);
		Tcl_SetErrorCode(interp, "TCL", "BINARY", "ENCODE",
			"LINE_LENGTH", NULL);
		return TCL_ERROR;
	    }
	    break;
	case OPT_WRAPCHAR:
	    wrapchar = Tcl_GetStringFromObj(objv[i+1], &wrapcharlen);
	    if (wrapcharlen == 0) {
		maxlen = 0;
	    }

	limit = cursor + size;
	for (offset = 0; offset < count; offset+=3) {
	    unsigned char d[3] = {0, 0, 0};

	    for (i = 0; i < 3 && offset+i < count; ++i) {
		d[i] = data[offset + i];
	    }
	    OUTPUT(B64Digits[d[0] >> 2]);
	    OUTPUT(B64Digits[((d[0] & 0x03) << 4) | (d[1] >> 4)]);
	    if (offset+1 < count) {
		OUTPUT(B64Digits[((d[1] & 0x0f) << 2) | (d[2] >> 6)]);
	    } else {
		OUTPUT(B64Digits[64]);
	    }
	    if (offset+2 < count) {
		OUTPUT(B64Digits[d[2] & 0x3f]);
	    } else {
		OUTPUT(B64Digits[64]);
	    }
	}
    }
    Tcl_SetObjResult(interp, resultObj);
    return TCL_OK;
}
#undef OUTPUT

/*
 *----------------------------------------------------------------------
 *
 * BinaryEncodeUu --
 *
 *	This implements the uuencode binary encoding. Input is broken into 6
 *	bit chunks and a lookup table is used to turn these values into output
 *	characters. This differs from the generic code above in that line
 *	lengths are also encoded.
 *
 * Results:
 *	Interp result set to an encoded byte array object
 *
 * Side effects:
 *	None
 *
 *----------------------------------------------------------------------
 */

static int
BinaryEncodeUu(
    ClientData clientData,
    Tcl_Interp *interp,
    int objc,
    Tcl_Obj *const objv[])
{
    Tcl_Obj *resultObj;
    unsigned char *data, *start, *cursor;
    int offset, count, rawLength, n, i, j, bits, index;
    int lineLength = 61;
    const unsigned char SingleNewline[] = { (unsigned char) '\n' };
    const unsigned char *wrapchar = SingleNewline;
    int wrapcharlen = sizeof(SingleNewline);
    enum { OPT_MAXLEN, OPT_WRAPCHAR };
    static const char *const optStrings[] = { "-maxlen", "-wrapchar", NULL };

    if (objc < 2 || objc%2 != 0) {
	Tcl_WrongNumArgs(interp, 1, objv,
		"?-maxlen len? ?-wrapchar char? data");
	return TCL_ERROR;
    }
    for (i = 1; i < objc-1; i += 2) {
	if (Tcl_GetIndexFromObj(interp, objv[i], optStrings, "option",
		TCL_EXACT, &index) != TCL_OK) {
	    return TCL_ERROR;
	}
	switch (index) {
	case OPT_MAXLEN:
	    if (Tcl_GetIntFromObj(interp, objv[i+1], &lineLength) != TCL_OK) {
		return TCL_ERROR;
	    }
	    if (lineLength < 3 || lineLength > 85) {
		Tcl_SetResult(interp, "line length out of range", TCL_STATIC);
		Tcl_SetErrorCode(interp, "TCL", "BINARY", "ENCODE",
			"LINE_LENGTH", NULL);
		return TCL_ERROR;
	    }
	    break;
	case OPT_WRAPCHAR:
	    wrapchar = Tcl_GetByteArrayFromObj(objv[i+1], &wrapcharlen);
	    break;
	}
    }

    /*
     * Allocate the buffer. This is a little bit too long, but is "good
     * enough".
     */

    resultObj = Tcl_NewObj();
    offset = 0;
    data = Tcl_GetByteArrayFromObj(objv[objc-1], &count);
    rawLength = (lineLength - 1) * 3 / 4;
    start = cursor = Tcl_SetByteArrayLength(resultObj,
	    (lineLength + wrapcharlen) *
	    ((count + (rawLength - 1)) / rawLength));
    n = bits = 0;

    /*
     * Encode the data. Each output line first has the length of raw data
     * encoded by the output line described in it by one encoded byte, then
     * the encoded data follows (encoding each 6 bits as one character).
     * Encoded lines are always terminated by a newline.
     */

    while (offset < count) {
	int lineLen = count - offset;

	if (lineLen > rawLength) {
	    lineLen = rawLength;
	}
	*cursor++ = UueDigits[lineLen];
	for (i=0 ; i<lineLen ; i++) {
	    n <<= 8;
	    n |= data[offset++];
	    for (bits += 8; bits > 6 ; bits -= 6) {
		*cursor++ = UueDigits[(n >> (bits-6)) & 0x3f];
	    }
	}
	if (bits > 0) {
	    n <<= 8;
	    *cursor++ = UueDigits[(n >> (bits + 2)) & 0x3f];
	    bits = 0;
	}
	for (j=0 ; j<wrapcharlen ; ++j) {
	    *cursor++ = wrapchar[j];
	}
    }

    /*
     * Fix the length of the output bytearray.
     */

    Tcl_SetByteArrayLength(resultObj, cursor-start);
    Tcl_SetObjResult(interp, resultObj);
    return TCL_OK;
}

/*
 *----------------------------------------------------------------------
 *
 * BinaryDecodeUu --
 *
 *	Decode a uuencoded string.

    Tcl_Interp *interp,
    int objc,
    Tcl_Obj *const objv[])
{
    Tcl_Obj *resultObj = NULL;
    unsigned char *data, *datastart, *dataend;
    unsigned char *begin, *cursor;
    int i, index, size, count = 0, strict = 0, lineLen;
    unsigned char c;
    enum {OPT_STRICT };
    static const char *const optStrings[] = { "-strict", NULL };

    if (objc < 2 || objc > 3) {
	Tcl_WrongNumArgs(interp, 1, objv, "?options? data");
	return TCL_ERROR;
    }

    TclNewObj(resultObj);
    datastart = data = (unsigned char *)
	    TclGetStringFromObj(objv[objc-1], &count);
    dataend = data + count;
    size = ((count + 3) & ~3) * 3 / 4;
    begin = cursor = Tcl_SetByteArrayLength(resultObj, size);
    lineLen = -1;

    /*
     * The decoding loop. First, we get the length of line (strictly, the
     * number of data bytes we expect to generate from the line) we're
     * processing this time round if it is not already known (i.e., when the
     * lineLen variable is set to the magic value, -1).
     */

    while (data < dataend) {
	char d[4] = {0, 0, 0, 0};

	if (lineLen < 0) {
	    c = *data++;
	    if (c < 32 || c > 96) {
		if (strict || !isspace(c)) {
		    goto badUu;
		}
		i--;
		continue;
	    }
	    lineLen = (c - 32) & 0x3f;
	}

	/*
	 * Now we read a four-character grouping.
	 */

	for (i=0 ; i<4 ; i++) {
	    if (data < dataend) {
		d[i] = c = *data++;
		if (c < 32 || c > 96) {
		    if (strict) {
			if (!isspace(c)) {
			    goto badUu;
			} else if (c == '\n') {
			    goto shortUu;
			}
		    }
		    i--;
		    continue;
		}


	    }
	}


	/*
	 * Translate that grouping into (up to) three binary bytes output.
	 */

	if (lineLen > 0) {
	    *cursor++ = (((d[0] - 0x20) & 0x3f) << 2)
		    | (((d[1] - 0x20) & 0x3f) >> 4);
	    if (--lineLen > 0) {
		*cursor++ = (((d[1] - 0x20) & 0x3f) << 4)
			| (((d[2] - 0x20) & 0x3f) >> 2);
		if (--lineLen > 0) {
		    *cursor++ = (((d[2] - 0x20) & 0x3f) << 6)
			    | (((d[3] - 0x20) & 0x3f));
		    lineLen--;
		}
	    }
	}

	/*
	 * If we've reached the end of the line, skip until we process a
	 * newline.
	 */

	if (lineLen == 0 && data < dataend) {
	    lineLen = -1;
	    do {
		c = *data++;
		if (c == '\n') {
		    break;
		} else if (c >= 32 && c <= 96) {
		    data--;
		    break;
		} else if (strict || !isspace(c)) {
		    goto badUu;
		}
	    } while (data < dataend);
	}
    }

    /*
     * Sanity check, clean up and finish.
     */

    if (lineLen > 0 && strict) {
	goto shortUu;
    }
    Tcl_SetByteArrayLength(resultObj, cursor - begin);
    Tcl_SetObjResult(interp, resultObj);
    return TCL_OK;

  shortUu:
    Tcl_SetObjResult(interp, Tcl_ObjPrintf("short uuencode data"));
    Tcl_SetErrorCode(interp, "TCL", "BINARY", "DECODE", "SHORT", NULL);
    TclDecrRefCount(resultObj);
    return TCL_ERROR;

  badUu:
    Tcl_SetObjResult(interp, Tcl_ObjPrintf(
	    "invalid uuencode character \"%c\" at position %d",
	    c, (int) (data - datastart - 1)));
    Tcl_SetErrorCode(interp, "TCL", "BINARY", "DECODE", "INVALID", NULL);
    TclDecrRefCount(resultObj);
    return TCL_ERROR;
}

/*
 *----------------------------------------------------------------------
 *

} -returnCodes error -match glob -result {invalid base64 character *}

test binary-74.1 {binary encode uuencode} -body {
    binary encode uuencode
} -returnCodes error -match glob -result "wrong # args: *"
test binary-74.2 {binary encode uuencode} -body {
    binary encode uuencode abc
} -result {#86)C
}
test binary-74.3 {binary encode uuencode} -body {
    binary encode uuencode {}
} -result {}
test binary-74.4 {binary encode uuencode} -body {
    binary encode uuencode [string repeat abc 20]
} -result "M[string repeat 86)C 15]\n/[string repeat 86)C 5]\n"
test binary-74.5 {binary encode uuencode} -body {
    binary encode uuencode \0\1\2\3\4\0\1\2\3
} -result ")``\$\"`P0``0(#\n"
test binary-74.6 {binary encode uuencode} -body {
    binary encode uuencode \0
} -result {!``
}
test binary-74.7 {binary encode uuencode} -body {
    binary encode uuencode \0\0
} -result "\"```
"
test binary-74.8 {binary encode uuencode} -body {
    binary encode uuencode \0\0\0
} -result {#````
}
test binary-74.9 {binary encode uuencode} -body {
    binary encode uuencode \0\0\0\0
} -result {$``````
}
test binary-74.10 {binary encode uuencode} -returnCodes error -body {
    binary encode uuencode -foo 30 abcabcabc
} -result {bad option "-foo": must be -maxlen or -wrapchar}
test binary-74.11 {binary encode uuencode} -returnCodes error -body {
    binary encode uuencode -maxlen 1 abcabcabc
} -result {line length out of range}
test binary-74.12 {binary encode uuencode} -body {
    binary encode uuencode -maxlen 3 -wrapchar | abcabcabc
} -result {!80|!8@|!8P|!80|!8@|!8P|!80|!8@|!8P|}

test binary-75.1 {binary decode uuencode} -body {
    binary decode uuencode
} -returnCodes error -match glob -result "wrong # args: *"
test binary-75.2 {binary decode uuencode} -body {
    binary decode uuencode "#86)C\n"
} -result {abc}
test binary-75.3 {binary decode uuencode} -body {
    binary decode uuencode {}
} -result {}
test binary-75.3.1 {binary decode uuencode} -body {
    binary decode uuencode `\n
} -result {}
test binary-75.4 {binary decode uuencode} -body {
    binary decode uuencode "M[string repeat 86)C 15]\n/[string repeat 86)C 5]\n"
} -result [string repeat abc 20]
test binary-75.5 {binary decode uuencode} -body {
    binary decode uuencode ")``\$\"`P0``0(#"
} -result "\0\1\2\3\4\0\1\2\3"
test binary-75.6 {binary decode uuencode} -body {
    string length [binary decode uuencode "`\n"]
} -result 0
test binary-75.7 {binary decode uuencode} -body {
    string length [binary decode uuencode "!`\n"]
} -result 1
test binary-75.8 {binary decode uuencode} -body {
    string length [binary decode uuencode "\"``\n"]
} -result 2
test binary-75.9 {binary decode uuencode} -body {
    string length [binary decode uuencode "#```\n"]
} -result 3
test binary-75.10 {binary decode uuencode} -body {
    set s ">[string repeat 86)C 10]\n>[string repeat 86)C 10]"
    binary decode uuencode $s
} -result [string repeat abc 20]
test binary-75.11 {binary decode uuencode} -body {
    set s ">[string repeat 86)C 10]\n\t>\t[string repeat 86)C 10]\r"
    binary decode uuencode $s
} -result [string repeat abc 20]
test binary-75.12 {binary decode uuencode} -body {
    binary decode uuencode -strict "|86)C"
} -returnCodes error -match glob -result {invalid uuencode character "|" at position 0}
test binary-75.13 {binary decode uuencode} -body {
    set s ">[string repeat 86)C 10]|[string repeat 86)C 10]"
    binary decode uuencode -strict $s
} -returnCodes error -match glob -result {invalid uuencode character "|" at position 41}
test binary-75.14 {binary decode uuencode} -body {
    set s ">[string repeat 86)C 10]\na[string repeat 86)C 10]"
    binary decode uuencode -strict $s
} -returnCodes error -match glob -result {invalid uuencode character *}
test binary-75.20 {binary decode uuencode} -body {
    set r [binary decode uuencode " 8"]
    list [string length $r] $r
} -result {0 {}}
test binary-75.21 {binary decode uuencode} -body {
    set r [binary decode uuencode "!86"]
    list [string length $r] $r
} -result {1 a}
test binary-75.22 {binary decode uuencode} -body {
    set r [binary decode uuencode "\"86)"]
    list [string length $r] $r
} -result {2 ab}
test binary-75.23 {binary decode uuencode} -body {
    set r [binary decode uuencode "#86)C"]
    list [string length $r] $r
} -result {3 abc}
test binary-75.24 {binary decode uuencode} -body {
    set s "#04)\# "
    binary decode uuencode $s
} -result ABC
test binary-75.25 {binary decode uuencode} -body {
    set s "#04)\#z"
    binary decode uuencode $s
} -returnCodes error -match glob -result {invalid uuencode character "z" at position 5}
test binary-75.26 {binary decode uuencode} -body {
    string length [binary decode uuencode " "]
} -result 0

test binary-76.1 {binary string appending growth algorithm} unix {
    # Create zero-length byte array first
    set f [open /dev/null rb]