Tcl Source Code

Paul, I allowed me to change the title of the ticket.

I hope this is ok. Take care, Harald

Nathan, sorry. The process for fixing was done and accepted by the author of the ticket.

The reason for this are many voices fearing incompatibilites and code breakage due to errors thrown, were there were no errors before. The drivers for this are people maintaining a lot of code within large organizations like Don Porter.

It was also made clear, that exec has limitations and open could be used, if the codepage profile is important.

I can understand your agument and I also was convinced of "strict anywhere". But this has changed as a community view.

As a consequence, I close the ticket and set the status to "fixed".

Thank you for your comprehension, Harald

..... and Rolf, of course ....

> I propose that exec retain its newfound strict powers

I'll discuss this with Harald, Brian, Steve and Ashok. My guess is that your proposal will not be accepted for 9.0b2.

I've reopened this issue and changed the resolution to "wont fix" in order to further discuss this. "replace" is a minefield, and saving unsuspecting scripts from stepping on a mine when using exec was a big benefit of the decision to use "strict" by default. glob was only excluded from TIP 657 because no one had the wherewithal to fix it at the time of that TIP. Bad data due to bad encoding is a big problem. Bad data cause problems down the line after it's too late to do anything about it, especially when they get into persistent storage and are processed much later. It is quite common for naive scripts to use exec to obtain binary data. In this case the earlier there is a notable error the better, as it allows the author to correct the script. This case is much more common than the mixed-encoding problem raised in the current issue. For programs like the diff program mentioned in this issue that are just doing their job in some encoding-agnostic way, open can always be used instead. Therefore, there is no real need to strip exec of its strict behaviour.

I propose that exec retain its newfound strict powers, and that Tcl developers focus on adding further knobs to exec if that is deemed worthwhile.

I want to thank Ashok for this work.

We had that open for TCL 9.0 in the weekly wrap-up and Ashok volunteered to look into this.

We should always remember, that behind each commit, there are hours of work.

Thank you all, guys, Harald

Fixed in [43d35b1d8b].

Works fine with the test script and inside tkdiff.

Paul

% catch {eval ::exec $cmd} ret
1
% puts "ret=$ret"
ret=2c2
< ����
---
> äöüß
child process exited abnormally

WIP in [apn-bug-0f1ddc0df7]. This intentionally fixes only exec to use the replace profile. In particular, open is unchanged (strict default) as the -profile option can be used there to if necessary.

Still WIP because stderr output behaves as if strict profile is in use.

Hi Jan, thanks, that you jump into the boat. Things move, if this happens.

My opinion:

• In TCL9, any encoding profile default should change to replace, if it is tcl8. So, I agree with the change.
• I don't think, this handles the initial issue. The exec profile must be set somewhere else to strict.

Take care, Harald

If we decide that the best profile for env/glob/argv handling is "replace", we could also make one step further and change the default flag for Tcl_UtfToExternalDString/Tcl_ExternalToUtfDString. That's easier than replace all such calls with the *Ex() alternative, specifying TCL_ENCODING_PROFILE_REPLACE separately everywhere.

How do we think about: [0d84761f9fecedd4|this]?

> This was already the case in 657 for glob, env as Rob mentions

glob, env currently still uses the "tcl8" profile, since no-one decided to change that. Command line arguments to tclsh/wish still use "tcl8" profile as well.

I also share Rolf's opinion that commands where there is no option to configure the encoding profile used by the command should use the replace profile. This was already the case in 657 for glob, env as Rob mentions, and I would argue should be the case for exec as well.

As an aside, I would think the same would apply to *arguments* passed to exec as well as command line passed to tclsh (argv, argv0) though that may be a more minor issue and not worth bothering over.

  set f [open |$cmd rb]
  # Binary output can be any byte sequence; no encoding problems possible
  set output [read $f]
  catch {close $f} err