Tcl Source Code

.SH NAME
encoding \- Manipulate encodings
.SH SYNOPSIS
\fBencoding \fIoption\fR ?\fIarg arg ...\fR?
.BE
.SH INTRODUCTION
.PP
Strings in Tcl are logically a sequence of 16-bit Unicode characters.
These strings are represented in memory as a sequence of bytes that
may be in one of several encodings: modified UTF\-8 (which uses 1 to 3
bytes per character), 16-bit
.QW Unicode
(which uses 2 bytes per character, with an endianness that is
dependent on the host architecture), and binary (which uses a single
byte per character but only handles a restricted range of characters).
Tcl does not guarantee to always use the same encoding for the same
string.
.PP
Different operating system interfaces or applications may generate
strings in other encodings such as Shift\-JIS.  The \fBencoding\fR
command helps to bridge the gap between Unicode and these other
formats.
.SH DESCRIPTION
.PP
Performs one of several encoding related operations, depending on
\fIoption\fR.  The legal \fIoption\fRs are:
.TP
\fBencoding convertfrom\fR ?\fIencoding\fR? \fIdata\fR
.
Convert \fIdata\fR to Unicode from the specified \fIencoding\fR.  The
characters in \fIdata\fR are treated as binary data where the lower
8-bits of each character is taken as a single byte.  The resulting
sequence of bytes is treated as a string in the specified
\fIencoding\fR.  If \fIencoding\fR is not specified, the current
system encoding is used.
.TP
\fBencoding convertto\fR ?\fIencoding\fR? \fIstring\fR
.
Convert \fIstring\fR from Unicode to the specified \fIencoding\fR.
The result is a sequence of bytes that represents the converted
string.  Each byte is stored in the lower 8-bits of a Unicode
character (indeed, the resulting string is a binary string as far as
Tcl is concerned, at least initially).  If \fIencoding\fR is not
specified, the current system encoding is used.
.TP
\fBencoding dirs\fR ?\fIdirectoryList\fR?
.
Tcl can load encoding data files from the file system that describe
additional encodings for it to work with. This command sets the search
path for \fB*.enc\fR encoding data files to the list of directories
\fIdirectoryList\fR. If \fIdirectoryList\fR is omitted then the
command returns the current list of directories that make up the
search path. It is an error for \fIdirectoryList\fR to not be a valid
list. If, when a search for an encoding data file is happening, an
element in \fIdirectoryList\fR does not refer to a readable,
searchable directory, that element is ignored.
.TP
\fBencoding names\fR
.
Returns a list containing the names of all of the encodings that are
currently available. 
The encodings
.QW utf-8
and
.QW iso8859-1
are guaranteed to be present in the list.
.TP
\fBencoding system\fR ?\fIencoding\fR?
.
Set the system encoding to \fIencoding\fR. If \fIencoding\fR is
omitted then the command returns the current system encoding.  The
system encoding is used whenever Tcl passes strings to system calls.
.SH EXAMPLE
.PP
It is common practice to write script files using a text editor that
produces output in the euc-jp encoding, which represents the ASCII
characters as singe bytes and Japanese characters as two bytes.  This
makes it easy to embed literal strings that correspond to non-ASCII
characters by simply typing the strings in place in the script.
However, because the \fBsource\fR command always reads files using the
current system encoding, Tcl will only source such files correctly
when the encoding used to write the file is the same.  This tends not
to be true in an internationalized setting.  For example, if such a
file was sourced in North America (where the ISO8859\-1 is normally
used), each byte in the file would be treated as a separate character
that maps to the 00 page in Unicode.  The resulting Tcl strings will
not contain the expected Japanese characters.  Instead, they will
contain a sequence of Latin-1 characters that correspond to the bytes
of the original string.  The \fBencoding\fR command can be used to
convert this string to the expected Japanese Unicode characters.  For
example,
.PP
.CS
set s [\fBencoding convertfrom\fR euc-jp "\exA4\exCF"]
.CE
.PP
would return the Unicode string
.QW "\eu306F" ,
which is the Hiragana letter HA.
.SH "SEE ALSO"
Tcl_GetEncoding(3)
.SH KEYWORDS
encoding, unicode
.\" Local Variables:
.\" mode: nroff
.\" End:

These subcommands are currently supported, but are likely to go away in a
future release as their functionality is either virtually never used or highly
misleading.
.TP
\fBstring bytelength \fIstring\fR
.
Returns a decimal string giving the number of bytes used to represent
\fIstring\fR in memory when encoded as Tcl's internal modified UTF\-8;
Tcl may use other encodings for \fIstring\fR as well, and does not
guarantee to only use a single encoding for a particular \fIstring\fR.
Because UTF\-8 uses a variable number of bytes to represent Unicode
characters, the byte length will not be the same as the character
length in general.  The cases where a script cares about the byte
length are rare.
.RS
.PP
In almost all cases, you should use the
\fBstring length\fR operation (including determining the length of a
Tcl byte array value).  Refer to the \fBTcl_NumUtfChars\fR manual
entry for more details on the UTF\-8 representation.
.PP
Formally, the \fBstring bytelength\fR operation returns the content of
the \fIlength\fR field of the \fBTcl_Obj\fR structure, after calling
\fBTcl_GetString\fR to ensure that the \fIbytes\fR field is populated.
This is highly unlikely to be useful to Tcl scripts, as Tcl's internal
encoding is not strict UTF\-8, but rather a modified CESU\-8 with a
denormalized NUL (identical to that used in a number of places by
Java's serialization mechanism) to enable basic processing with
non-Unicode-aware C functions.  As this representation should only
ever be used by Tcl's implementation, the number of bytes used to
store the representation is of very low value (except to C extension
code, which has direct access for the purpose of memory management,
etc.)
.PP
\fICompatibility note:\fR it is likely that this subcommand will be
withdrawn in a future version of Tcl. It is better to use the
\fBencoding convertto\fR command to convert a string to a known
encoding and then apply \fBstring length\fR to that.
.PP
.CS
\fBstring length\fR [encoding convertto utf-8 $theString]
.CE
.RE
.TP
\fBstring wordend \fIstring charIndex\fR
.
Returns the index of the character just after the last one in the word
containing character \fIcharIndex\fR of \fIstring\fR.  \fIcharIndex\fR
may be specified using the forms in \fBSTRING INDICES\fR.  A word is

static void		FileSetupProc(ClientData clientData, int flags);
static void		FileWatchProc(ClientData instanceData, int mask);
static void		FileThreadActionProc(ClientData instanceData,
			    int action);
static int		FileTruncateProc(ClientData instanceData,
			    Tcl_WideInt length);
static DWORD		FileGetType(HANDLE handle);
static int		NativeIsComPort(CONST TCHAR *nativeName);
/*
 * This structure describes the channel type structure for file based IO.
 */

static const Tcl_ChannelType fileChannelType = {
    "file",			/* Type name. */
    TCL_CHANNEL_VERSION_5,	/* v5 channel */

	createMode = TRUNCATE_EXISTING;
	break;
    default:
	createMode = OPEN_EXISTING;
	break;
    }

    /*
     * [2413550] Avoid double-open of serial ports on Windows
     * Special handling for Windows serial ports by a "name-hint"
     * to directly open it with the OVERLAPPED flag set.
     */

    if( NativeIsComPort(nativeName) ) {

	handle = TclWinSerialOpen(INVALID_HANDLE_VALUE, nativeName, accessMode);
	if (handle == INVALID_HANDLE_VALUE) {
	    TclWinConvertError(GetLastError());
	    if (interp != (Tcl_Interp *) NULL) {
		Tcl_AppendResult(interp, "couldn't open serial \"",
			TclGetString(pathPtr), "\": ",
			Tcl_PosixError(interp), NULL);
	    }
	    return NULL;
	}

	/*
	* For natively named Windows serial ports we are done.
	*/
	channel = TclWinOpenSerialChannel(handle, channelName,
		channelPermissions);

	return channel;
    }
    /*
     * If the file is being created, get the file attributes from the
     * permissions argument, else use the existing file attributes.
     */

    if (mode & O_CREAT) {
	if (permissions & S_IWRITE) {

    }

    channel = NULL;

    switch (FileGetType(handle)) {
    case FILE_TYPE_SERIAL:
	/*
	 * Natively named serial ports "com1-9", "\\\\.\\comXX" are 
	 * already done with the code above.
	 * Here we handle all other serial port names.
	 *
	 * Reopen channel for OVERLAPPED operation. Normally this shouldn't
	 * fail, because the channel exists.
	 */

	handle = TclWinSerialOpen(handle, nativeName, accessMode);
	if (handle == INVALID_HANDLE_VALUE) {
	    TclWinConvertError(GetLastError());
	    if (interp != (Tcl_Interp *) NULL) {
		Tcl_SetObjResult(interp, Tcl_ObjPrintf(
			"couldn't reopen serial \"%s\": %s",
			TclGetString(pathPtr), Tcl_PosixError(interp)));
	    }

		type = FILE_TYPE_SERIAL;
	    }
	}
    }

    return type;
}

 /*
 *----------------------------------------------------------------------
 *
 * NativeIsComPort --
 *
 *	Determines if a path refers to a Windows serial port.
 *	A simple and efficient solution is to use a "name hint" to detect 
 *      COM ports by their filename instead of resorting to a syscall 
 *	to detect serialness after the fact.
 *	The following patterns cover common serial port names:
 *	    COM[1-9]:?
 *	    //./COM[0-9]+
 *	    \\.\COM[0-9]+
 *
 * Results:
 *	1 = serial port, 0 = not.
 *
 *----------------------------------------------------------------------
 */

static int
NativeIsComPort(
    const TCHAR *nativePath)	/* Path of file to access, native encoding. */
{
    const WCHAR *p = (const WCHAR *) nativePath;
    int i, len = wcslen(p);

    /*
     * 1. Look for com[1-9]:?
     */

    if ( (len >= 4) && (len <= 5) 
	    && (_wcsnicmp(p, L"com", 3) == 0) ) {
	/*
	* The 4th character must be a digit 1..9 optionally followed by a ":"
	*/
	
	if ( (p[3] < L'1') || (p[3] > L'9') ) {
	    return 0;
	}
	if ( (len == 5) && (p[4] != L':') ) {
	    return 0;
	}
	return 1;
    }
    
    /*
     * 2. Look for //./com[0-9]+ or \\.\com[0-9]+
     */
    
    if ( (len >= 8) && ( 
	       (_wcsnicmp(p, L"//./com", 7) == 0)
	    || (_wcsnicmp(p, L"\\\\.\\com", 7) == 0) ) )
    {
	/*
	* Charaters 8..end must be a digits 0..9
	*/
    
	for ( i=7; i<len; i++ ) {
	    if ( (p[i] < '0') || (p[i] > '9') ) {
		return 0;
	    }
	}
	return 1;
    }
    return 0;
}

/*
 * Local Variables:
 * mode: c
 * c-basic-offset: 4
 * fill-column: 78
 * End:
 */

MODULE_SCOPE TclFile	TclWinMakeFile(HANDLE handle);
MODULE_SCOPE Tcl_Channel TclWinOpenConsoleChannel(HANDLE handle,
			    char *channelName, int permissions);
MODULE_SCOPE Tcl_Channel TclWinOpenFileChannel(HANDLE handle, char *channelName,
			    int permissions, int appendMode);
MODULE_SCOPE Tcl_Channel TclWinOpenSerialChannel(HANDLE handle,
			    char *channelName, int permissions);
MODULE_SCOPE HANDLE	TclWinSerialOpen(HANDLE handle, const TCHAR *name,
			    DWORD access);
MODULE_SCOPE int	TclWinSymLinkCopyDirectory(const TCHAR *LinkOriginal,
			    const TCHAR *LinkCopy);
MODULE_SCOPE int	TclWinSymLinkDelete(const TCHAR *LinkOriginal,
			    int linkOnly);
#if defined(TCL_THREADS) && defined(USE_THREAD_ALLOC)
MODULE_SCOPE void	TclWinFreeAllocCache(void);

    return 0;
}

/*
 *----------------------------------------------------------------------
 *
 * TclWinSerialOpen --
 *
 *	Opens or Reopens the serial port with the OVERLAPPED FLAG set
 *
 * Results:
 *	Returns the new handle, or INVALID_HANDLE_VALUE. 

 *	If an existing channel is specified it is closed and reopened.
 *
 * Side effects:
 *	May close/reopen the original handle
 *
 *----------------------------------------------------------------------
 */

HANDLE
TclWinSerialOpen(
    HANDLE handle,
    const TCHAR *name,
    DWORD access)
{
    SerialInit();

    /*
     * If an open channel is specified, close it
     */

    if ( handle != INVALID_HANDLE_VALUE && CloseHandle(handle) == FALSE) {
	return INVALID_HANDLE_VALUE;
    }

    /*
     * Multithreaded I/O needs the overlapped flag set otherwise
     * ClearCommError blocks under Windows NT/2000 until serial output is
     * finished
     */




    handle = CreateFile(name, access, 0, 0, OPEN_EXISTING,
	    FILE_FLAG_OVERLAPPED, 0);

    return handle;
}

/*
 *----------------------------------------------------------------------
 *
 * TclWinOpenSerialChannel --