Tcl Source Code

There are some problems about Japanese related encodings 
which is discussed on <[email protected]>. 
Detailed description is available (but in Japanese though:-<) at:

http://lists.sourceforge.jp/pipermail/tcltkjp-develop/2002-December/000028.html

In short, current (tcl8.4.1) encoding still breaks Japanese Industrial
Standard(JIS).  Following JIS chars could not be converted correctly.

212D
212F
2141
2142
215D
215E
215F
2160
216B
2171
2172
2178
2240
2241
224C
225C
225D
2261
2262
2265
2268
2269
2279

This patch fixes this problem.

Logged In: YES 
user_id=72656

This is in for 8.4.2.  Please get CVS and test.  It passes all 
old tests (which included some iso2022 ones) and new 
tests.  Ref original tcl-core post 
http://aspn.activestate.com/ASPN/Mail/Message/tcl-
core/1542413

This also significantly increases the tclbench iso2022-jp 
encoding tests.  I hope this means that we are still correct.  :)

File Added - 43002: encoding-jajp.test

Logged In: YES 
user_id=148473

Please try this test suite.

10/16 will fail on prepatch.
None will fail on postpatch.

Here is failure list.

==== encoding-jajp-25.1 jisx0208 cp932 => cp932 FAILED
---- Result was:
{2240 81be 879c} {2241 81bf 879b} {224C 81ca fa54} {225C
81da 8797} {225D 81db 8796} {2261 81df 8791} {2262 81e0
8790} {2265 81e3 8795} {2268 81e6 fa5b} {2269 81e7 8792}
---- Result should have been (exact matching):

==== encoding-jajp-25.1 FAILED


==== encoding-jajp-25.2 jisx0208 cp932 => shiftjis FAILED
---- Result was:
{2141 8160 96ff} {2142 8161 3f} {215D 817c 3f} {2171 8191
3f} {2172 8192 3f} {224C 81ca 3f}
---- Result should have been (exact matching):

==== encoding-jajp-25.2 FAILED


==== encoding-jajp-25.3 jisx0208 cp932 => euc-jp FAILED
---- Result was:
{2141 a1c1 3f} {2142 a1c2 3f} {215D a1dd 3f} {2171 a1f1 3f}
{2172 a1f2 3f} {224C a2cc 3f}
---- Result should have been (exact matching):

==== encoding-jajp-25.3 FAILED


==== encoding-jajp-25.4 jisx0208 cp932 => iso2022-jp FAILED
---- Result was:
{212D 1b2442212d1b2842 b4} {212F 1b2442212f1b2842 a8} {2141
1b244221411b2842 1b2441212b1b2842} {2142 1b244221421b2842
1b2441214e1b2842} {215D 1b2442215d1b2842 1b2441232d1b2842}
{215E 1b2442215e1b2842 b1} {215F 1b2442215f1b2842 d7} {2160
1b244221601b2842 f7} {216B 1b2442216b1b2842 b0} {2171
1b244221711b2842 1b244121691b2842} {2172 1b244221721b2842
1b2441216a1b2842} {2178 1b244221781b2842 a7} {224C
1b2442224c1b2842 1b242843217e1b2842} {2279 1b244222791b2842 b6}
---- Result should have been (exact matching):

==== encoding-jajp-25.4 FAILED


==== encoding-jajp-25.5 jisx0208 shiftjis => cp932 FAILED
---- Result was:
{2141 8160 3f} {2142 8161 3f} {215D 817c 3f} {2171 8191 3f}
{2172 8192 3f} {2240 81be 879c} {2241 81bf 879b} {224C 81ca
3f} {225C 81da 8797} {225D 81db 8796} {2261 81df 8791} {2262
81e0 8790} {2265 81e3 8795} {2268 81e6 fa5b} {2269 81e7 8792}
---- Result should have been (exact matching):

==== encoding-jajp-25.5 FAILED


==== encoding-jajp-25.8 jisx0208 shiftjis => iso2022-jp FAILED
---- Result was:
{212D 1b2442212d1b2842 b4} {212F 1b2442212f1b2842 a8} {215E
1b2442215e1b2842 b1} {215F 1b2442215f1b2842 d7} {2160
1b244221601b2842 f7} {216B 1b2442216b1b2842 b0} {2171
1b244221711b2842 a2} {2172 1b244221721b2842 a3} {2178
1b244221781b2842 a7} {224C 1b2442224c1b2842 ac} {2279
1b244222791b2842 b6}
---- Result should have been (exact matching):

==== encoding-jajp-25.8 FAILED


==== encoding-jajp-25.9 jisx0208 euc-jp => cp932 FAILED
---- Result was:
{2141 8160 3f} {2142 8161 3f} {215D 817c 3f} {2171 8191 3f}
{2172 8192 3f} {2240 81be 879c} {2241 81bf 879b} {224C 81ca
3f} {225C 81da 8797} {225D 81db 8796} {2261 81df 8791} {2262
81e0 8790} {2265 81e3 8795} {2268 81e6 fa5b} {2269 81e7 8792}
---- Result should have been (exact matching):

==== encoding-jajp-25.9 FAILED


==== encoding-jajp-25.12 jisx0208 euc-jp => iso2022-jp FAILED
---- Result was:
{212D 1b2442212d1b2842 b4} {212F 1b2442212f1b2842 a8} {215E
1b2442215e1b2842 b1} {215F 1b2442215f1b2842 d7} {2160
1b244221601b2842 f7} {216B 1b2442216b1b2842 b0} {2171
1b244221711b2842 a2} {2172 1b244221721b2842 a3} {2178
1b244221781b2842 a7} {224C 1b2442224c1b2842 ac} {2279
1b244222791b2842 b6}
---- Result should have been (exact matching):

==== encoding-jajp-25.12 FAILED


==== encoding-jajp-25.13 jisx0208 iso2022-jp => cp932 FAILED
---- Result was:
{2141 8160 3f} {2142 8161 3f} {215D 817c 3f} {2171 8191 3f}
{2172 8192 3f} {2240 81be 879c} {2241 81bf 879b} {224C 81ca
3f} {225C 81da 8797} {225D 81db 8796} {2261 81df 8791} {2262
81e0 8790} {2265 81e3 8795} {2268 81e6 fa5b} {2269 81e7 8792}
---- Result should have been (exact matching):

==== encoding-jajp-25.13 FAILED


==== encoding-jajp-25.16 jisx0208 iso2022-jp => iso2022-jp
FAILED
---- Result was:
{212D 1b2442212d1b2842 b4} {212F 1b2442212f1b2842 a8} {215E
1b2442215e1b2842 b1} {215F 1b2442215f1b2842 d7} {2160
1b244221601b2842 f7} {216B 1b2442216b1b2842 b0} {2171
1b244221711b2842 a2} {2172 1b244221721b2842 a3} {2178
1b244221781b2842 a7} {224C 1b2442224c1b2842 ac} {2279
1b244222791b2842 b6}
---- Result should have been (exact matching):

==== encoding-jajp-25.16 FAILED

encoding-jajp.test:     Total   16      Passed  6      
Skipped 0       Failed 10

Logged In: YES 
user_id=148473

> It is best not to rely on external tools since this will
be x-platform.

Good point. I agreed. 
I will try to adapt my test to tcl test framework. I will
post it here.

Logged In: YES 
user_id=72656

If you look at the core tcl/tests/encoding.test you will see 
some existing iso2022 tests that could be extended.  It 
doesn't have to cover every char (although that is good).  It is 
best not to rely on external tools since this will be x-platform.
thanks.

Logged In: YES 
user_id=148473

> Is it possible to include some tests that do not work pre-
patch and do work post-patch to ensure we don't break this in 
the future?

Yes, I have. I will translate Japanese comments into english
and post here.
It uses {GNU make, perl, zsh, nkf}.  Do you have nkf?
(Network Kanji Filter)
If not, I will include input tables of all JIS char in 4
encodings.

Logged In: YES 
user_id=72656

Is it possible to include some tests that do not work pre-
patch and do work post-patch to ensure we don't break this in 
the future?

File Added - 42864: tcl-ja-encoding.patch3