Tcl Source Code

So far as I understood "encoding convertfrom $enc $text" is the right command to convert from a string with known encoding to unicode. This works fine, except $enc is utf-8. Please try the following script (I hope this tracker is not corrupting the Greek):

----------------------------------------------------------------------
#!/usr/bin/wish

# 1. ensure you have the right encoding to read the string
# 2. we want conversion utf-8 -> unicode
encoding system utf-8

set src "Η Σάμος είναι"

# the result of this operation is broken!
set uni [encoding convertfrom utf-8 $src]
set utf [encoding convertto utf-8 $uni]

# this works fine, it's a iso8859-7 encoded string
set grk [encoding convertfrom iso8859-7 "Ç ÓÜìïò åßíáé"]

pack [label .l1 -text $src]
pack [label .l2 -text $uni]
pack [label .l3 -text $utf]
pack [label .l4 -text $grk]
----------------------------------------------------------------------

You will see that the conversion from utf-8 is broken. Converting back is still broken. Conversion from iso8859-7 is fine.

My application is only working if I exclude utf-8 explicitly (my system encoding is utf-8):

if {$encoding eq "utf-8"} {
  set dst $src ;# because convertfrom doesn't work
} else {
  set dst [encoding convertfrom $encoding $src]
}

But I don't think that this is intended, the generic command "convertfrom" should work with all encodings due to the manual.

PS: In my tests I've figured out that sometimes the conversion to utf-8 gives results with overlong sequences, such results are invalid due to the utf-8 specification.

Since no-one disputed dkf's excellent explanation (neither do I), this can be closed.

You're doing this backwards.

encoding convertto takes a string and produces a byte-sequence that is the encoded form of the string using the specified encoding (or the system encoding). encoding convertfrom takes a byte-sequence and interprets that as an encoded string, returning the string that it thinks it is the encoded form of (where the encoding is the specified one or the system encoding). The process of converting a string to a particular encoding can result in information loss if the target encoding can't handle a character, and the reverse can result in information loss if the source contains a byte sequence that is not valid in the encoding.

If we take all that and try the reverse operations:

% set src "Η Σάμος είναι"
Η Σάμος είναι
% encoding convertto utf-8 $src
Η Σάμος είναι
% encoding convertfrom utf-8 $src
— £¬¼¿Â µ¯½±¹
% encoding convertto utf-8 [encoding convertfrom utf-8 $src]
— £¬¼¿Â µ¯½±¹
% encoding convertfrom utf-8 [encoding convertto utf-8 $src]
Η Σάμος είναι

Changing encoding system is probably a very bad idea; the main thing that is used for is actually controlling how Tcl and the OS understand bytes used in filenames…