Tcl Source Code

On MS Windows, [file normalize C:\\Test\\Dir1\\Dir2] will return only C:/Test/Dir1/ if the "List Folder Contents" ACL property is missing from Dir1.
Likewise [file normalize C:\\Test\\Dir1\\Dir2\\Dir3] returns C:/Test/Dir1//Dir3

So even if a user has permission to access Dir2, using the normalized path of Dir2 is lethal. This particularly happens with "junction points".

See http://groups.google.com/group/comp.lang.tcl/browse_thread/thread/c52f1b16699139b7/e4b19f32bdcf0f9b?lnk=gst&q=file+normalize#e4b19f32bdcf0f9b

For documentation purposes, here is the description of Jan of the final solution:

Am 08.01.2013 09:45, schrieb Jan Nijtmans:
> 2013/1/8 Harald Oehlmann <[email protected]>:
>> - accept the patch as helpful
> I was considering that, but after I wrote that I ran core-8-5-branch tests
> many times without using junctions. Sometimes the tests fail, but
> significantly less that with the patch. Therefore, I am still not sure.
> 
> The patch is certainly a step in the right direction. But still I want to
> explore more possibilities before merging anything. for example, I
> wonder why there is a NativeReadReparse function (which assumes
> the junction is readable) and a NativeWriteReparse function (
> when creating a new directory). I would expect there to be
> a NativeAccessReparse function as well, which assumes that
> the junction is just accessable, nothing more than that.
> Hm, this can be done re-using NativeReadReparse, with an
> additional parameter: requiredAccess.
> 
> Done now in the branch. Seems to work better. Can you confirm
> that? It fixes the [file normalize] bug, currently testing.....

the new patch:
- has cleaner code, no trial and error any more.
- solves the startup issue
- solves the "file normalize" issue
- passed a testsuite run without additional errors as administrator
- passed a testsuite run without additional errors as normal user

Here are the usual test suite errors:
http://wiki.tcl.tk/37529

For me, anything is solved with this patch.

Fixed now on core-8-4-branch, core-8-5-branch and trunk

Thanks! I created the junction as described. But I have Windows 7, not Vista.

The result: running the test suite, most of the time everything passes
(except the well-known filesystem-1.3/1.4). But running them
multiple times, or just fast after each other, I get random failures, like:
==== fCmd-4.7 TclFileMakeDirsCmd: multi levels deep FAILED
==== Contents of test case:

    cleanup
    file mkdir [file join td1 td2 td3 td4]
    glob td1 [file join td1 td2]

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: can't create directory "td1": permission denied
    while executing
"file mkdir [file join td1 td2 td3 td4]"
    ("uplevel" body line 3)
    invoked from within
"uplevel 1 $script"
---- errorCode: POSIX EACCES {permission denied}
==== fCmd-4.7 FAILED

or
==== fCmd-7.2 FileForceOption: -force FAILED
==== Contents of test case:

    cleanup
    file mkdir [file join tf1 tf2]
    file delete -force tf1

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: error deleting "tf1": file already exists
    while executing
"file delete -force tf1"
    ("uplevel" body line 4)
    invoked from within
"uplevel 1 $script"
---- errorCode: POSIX EEXIST {file already exists}
==== fCmd-7.2 FAILED

or
==== fCmd-23.1 TclMacRmdir: trying to remove a nonempty directory FAILED
==== Contents of test case:

    file mkdir [file join tfad dir]

    list [catch {file delete tfad}] [file delete -force tfad]

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: error deleting "tfad": file already exists
    while executing
"file delete -force tfad"
    ("uplevel" body line 4)
    invoked from within
"uplevel 1 $script"
---- errorCode: POSIX EEXIST {file already exists}
==== fCmd-23.1 FAILED

Doing the same from the original directory, the same
happens. So I think that those failures are not related
at all to the bug handled here. They just happen,
at random, with both the patched and unpatched
Tcl 8.5.13 versions. I don't see any difference
between them in the non-junction case.

Here is the documentation of a E-Mail conversation with Jan:

Jan Nijtmans wrote:
> At this moment, I'm simply puzzled why io-39.8 and io-39.9 fail to
> open file $path(test1) for writing. io-39.6 and io-39.7 (and earlier)
> do exactly the same, so why do the others pass.  ???????

No idea neither...

> I think that question needs to be answered before being able
> to go on on this one. I tried to create a junction too, as you did,
> trying to reproduce it, but I don't seem to be able to create it
> with the 'right' permissions to get your result ......

Do you also have Vista 32 bit  Professional ?
AFAIK, the home edition does lack some property controls in the Windows
explorer.

I have retried my instructions on
http://sourceforge.net/tracker/?func=detail&atid=110894&aid=3587096&group_id=10894

Here is the protocol:

- create c:\test2\tcl8513 with the whole tcl distribution and the exe
path: c:\test2\tcl8513\bin\wish85.exe
- Start a dos box with administrator privileges
- in the dos box:
C:\Windows\system32>cd \

C:\>mklink /j test2_junction2 test2
Verbindung erstellt für test2_junction2 <<===>> test2

C:\>test2_junction2\tcl8513\bin\wish85.exe
<Dos box end>
Wish starts as expected.

- in Windows Explorer
  - go to c:\
  - Right click on "c:\test2_junction2"
  - Choose "Properties"
  - Card "Security"
  - Button "Change" (with Admin sign)
    Window "Permissions for test2_junction2" opens
  - click "any" in user name pane
  - Click the deny checkbox for "Show Folder"
    (See attached screenshot)
  - Press ok - a warning appears - press ok

- Go to the dos box and type (again):
C:\>test2_junction2\tcl8513\bin\wish85.exe

Now, the tcl error box appears for me...

> So, any idea is welcome now. For example, what happens if you
> put tests  io-39.8 and io-39.9 before io-39.6 and io-39.7? Do
> they still fail then?

How may I reorder the tests ? Do I have to rename them ?
Or just run some tests by a test pattern ?

I have rerun the tests in an administrator dos box two times.
Failed tests:
filesystem-1.3: first and second run
filesystem-1.4: first and second run
io-39.9: only first run !!!

==== io-39.9 Tcl_SetChannelOption, blocking mode FAILED
==== Contents of test case:

    file delete $path(test1)
    set f1 [open $path(test1) w]
    close $f1
    set f1 [open $path(test1) r]
    set x ""
    lappend x [fconfigure $f1 -blocking]
    fconfigure $f1 -blocking off
    lappend x [fconfigure $f1 -blocking]
    lappend x [gets $f1]
    lappend x [read $f1 1000]
    lappend x [fblocked $f1]
    lappend x [eof $f1]
    close $f1
    set x

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: couldn't open "C:/test/tcl-bug-3092089/win/test1":
permission de
nied
    while executing
"open $path(test1) w"
    ("uplevel" body line 3)
    invoked from within
"uplevel 1 $script"
---- errorCode: POSIX EACCES {permission denied}
==== io-39.9 FAILED

Thank you, Jan, for the patch.

I have build and tested it, together with tk8.5.13 on Win Vista 32bit GER and msvc 6++.

Startup of tcl and tk over a restricted junction is ok.
Practical test: (junction setup as described in former post:
 - tcl in c:\test2\tcl-bug-3092089
 - Junction c:\test2_junction to c:\test2 )
Start wish in c:\test2_junction\tcl8513-bug-3092089\bin\wish85t.exe
---wish console---
% file normalize c:/test2_junction
C:/test2_junction
% info nameofexecutable
c:/test2_junction/tcl8513-bug-3092089/bin/wish85t.exe
% file normalize [info nameofexecutable]
C:/test2/tcl8513-bug-3092089/bin/wish85t.exe
% file normalize c:/test2_junction/nonexisting_folder
C:/test2/nonexisting_folder
% open c:/test2_junction/tcl8513-bug-3092089/include/tcl.h
file2aa5ad8
---END---
Here is a comparison to tcl8.5.13 (not started over the junction):
---wish console---
% file normalize c:/test2_junction
C:/test2_junction
% file normalize c:/test2_junction/tcl8513-bug-3092089/bin/wish85t.exe
C:/test2_junction//bin/wish85t.exe
% file normalize c:/test2_junction/nonexisting_folder
C:/test2_junction/nonexisting_folder
open c:/test2_junction/tcl8513-bug-3092089/include/tcl.h
file2aa7318
---END---
All bugs disapeared. To resolve "file normalize c:/test2_junction" to "c:/test2" instead
"C:/test2_junction" is IMHO a "Nice to have".

Make test results of the patch and the regular 8.5.13 release are below.
The following tests fail in the different versions:
Test # : bugfix : 8.5.13
------------------------------
filesystem-1.3 : fail : fail
filesystem-1.4 : fail : fail
io-39.8 : fail : ok
io-39.9 : fail : ok
winFCmd-6.1 : fail : fail
winFCmd-6.9 : fail : fail
winFCmd-6.13 : fail : fail
winFCmd-9.3 : fail : fail
winFCmd-12.6 : fail : fail

Thus, there are the two io tests failing in the bug fix release which did succeed in the 8.5.13 release.
I have no idea why.

Thank you,
Harald
---------------------------------------------------------------------------

--- Make test of the bugfix release --

nmake -f Makefile.vc release OPTS=threads
nmake -f Makefile.vc test OPTS=threads

==== filesystem-1.3 link normalisation FAILED
==== Contents of test case:

    testPathEqual [file normalize [file join dir.dir foo]]  [file normalize [fil
e join dir.link foo]]

---- Result was:
not equal: C:/test/tcl-bug-3092089/win/dir.dir/foo C:/test/tcl-bug-3092089/win/d
ir.link/foo
---- Result should have been (exact matching):
1
==== filesystem-1.3 FAILED



==== filesystem-1.4 link normalisation FAILED
==== Contents of test case:

    testPathEqual [file normalize [file join dir.dir inside.file]]  [file normal
ize [file join dir.link inside.file]]

---- Result was:
not equal: C:/test/tcl-bug-3092089/win/dir.dir/inside.file C:/test/tcl-bug-30920
89/win/dir.link/inside.file
---- Result should have been (exact matching):
1
==== filesystem-1.4 FAILED

==== io-39.8 Tcl_SetChannelOption, different buffering options FAILED
==== Contents of test case:

    file delete $path(test1)
    set f1 [open $path(test1) w]
    set l ""
    fconfigure $f1 -translation lf -buffering none -eofchar {}
    puts -nonewline $f1 hello
    lappend l [file size $path(test1)]
    puts -nonewline $f1 hello
    lappend l [file size $path(test1)]
    fconfigure $f1 -buffering full
    puts -nonewline $f1 hello
    lappend l [file size $path(test1)]
    fconfigure $f1 -buffering none
    lappend l [file size $path(test1)]
    puts -nonewline $f1 hello
    lappend l [file size $path(test1)]
    close $f1
    lappend l [file size $path(test1)]
    set l

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: couldn't open "C:/test/tcl-bug-3092089/win/test1": permission de
nied
    while executing
"open $path(test1) w"
    ("uplevel" body line 3)
    invoked from within
"uplevel 1 $script"
---- errorCode: POSIX EACCES {permission denied}
==== io-39.8 FAILED

==== io-39.9 Tcl_SetChannelOption, blocking mode FAILED
==== Contents of test case:

    file delete $path(test1)
    set f1 [open $path(test1) w]
    close $f1
    set f1 [open $path(test1) r]
    set x ""
    lappend x [fconfigure $f1 -blocking]
    fconfigure $f1 -blocking off
    lappend x [fconfigure $f1 -blocking]
    lappend x [gets $f1]
    lappend x [read $f1 1000]
    lappend x [fblocked $f1]
    lappend x [eof $f1]
    close $f1
    set x

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: couldn't open "C:/test/tcl-bug-3092089/win/test1": permission de
nied
    while executing
"open $path(test1) w"
    ("uplevel" body line 3)
    invoked from within
"uplevel 1 $script"
---- errorCode: POSIX EACCES {permission denied}
==== io-39.9 FAILED

==== winFCmd-6.1 TclpRemoveDirectory: errno: EACCES FAILED
==== Contents of test case:

    cleanup
    file mkdir td1
    testchmod 000 td1
    catch {
        testfile rmdir td1
        file exists td1
    } r
    catch {
        testchmod 777 td1
        cleanup
    }
    set r

---- Result was:
td1 EACCES
---- Result should have been (exact matching):
0
==== winFCmd-6.1 FAILED



==== winFCmd-6.9 TclpRemoveDirectory: errno == EACCES FAILED
==== Contents of test case:

    cleanup
    file mkdir td1
    testchmod 000 td1
    catch {
        testfile rmdir td1
        file exists td1
    } r
    catch {
        testchmod 777 td1
        cleanup
    }
    set r

---- Result was:
td1 EACCES
---- Result should have been (exact matching):
0
==== winFCmd-6.9 FAILED

==== winFCmd-6.13 TclpRemoveDirectory: write-protected FAILED
==== Contents of test case:

    cleanup
    file mkdir td1
    testchmod 000 td1
    catch {
        testfile rmdir td1
        file exists td1
    } r
    catch {
        testchmod 777 td1
        cleanup
    }
    set r

---- Result was:
td1 EACCES
---- Result should have been (exact matching):
0
==== winFCmd-6.13 FAILED



==== winFCmd-9.3 TraversalDelete: DOTREE_PRED FAILED
==== Contents of test case:

    cleanup
    file mkdir td1/td2
    testchmod 000 td1
    catch {
        testfile rmdir -force td1
        file exists td1
    } r
    catch {
        testchmod 777 td1
        cleanup
    }
    set r

---- Result was:
td1 EACCES
---- Result should have been (exact matching):
0
==== winFCmd-9.3 FAILED

==== winFCmd-12.6 ConvertFileNameFormat: absolute path with drive FAILED
==== Contents of test case:

    catch {file delete -force -- c:/td1}
    close [open c:/td1 w]
    list [catch {string tolower [file attributes c:/td1 -longname]} msg] $msg [f
ile delete -force -- c:/td1]

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: couldn't open "c:/td1": invalid argument
    while executing
"open c:/td1 w"
    ("uplevel" body line 3)
    invoked from within
"uplevel 1 $script"
---- errorCode: POSIX EINVAL {invalid argument}
==== winFCmd-12.6 FAILED

-- Make test of tcl8.5.13 --

==== filesystem-1.3 link normalisation FAILED
==== Contents of test case:

    testPathEqual [file normalize [file join dir.dir foo]]  [file normalize [fil
e join dir.link foo]]

---- Result was:
not equal: C:/test/tcl8.5.13/win/dir.dir/foo C:/test/tcl8.5.13/win/dir.link/foo
---- Result should have been (exact matching):
1
==== filesystem-1.3 FAILED



==== filesystem-1.4 link normalisation FAILED
==== Contents of test case:

    testPathEqual [file normalize [file join dir.dir inside.file]]  [file normal
ize [file join dir.link inside.file]]

---- Result was:
not equal: C:/test/tcl8.5.13/win/dir.dir/inside.file C:/test/tcl8.5.13/win/dir.l
ink/inside.file
---- Result should have been (exact matching):
1
==== filesystem-1.4 FAILED

==== winFCmd-6.1 TclpRemoveDirectory: errno: EACCES FAILED
==== Contents of test case:

    cleanup
    file mkdir td1
    testchmod 000 td1
    catch {
        testfile rmdir td1
        file exists td1
    } r
    catch {
        testchmod 777 td1
        cleanup
    }
    set r

---- Result was:
td1 EACCES
---- Result should have been (exact matching):
0
==== winFCmd-6.1 FAILED



==== winFCmd-6.9 TclpRemoveDirectory: errno == EACCES FAILED
==== Contents of test case:

    cleanup
    file mkdir td1
    testchmod 000 td1
    catch {
        testfile rmdir td1
        file exists td1
    } r
    catch {
        testchmod 777 td1
        cleanup
    }
    set r

---- Result was:
td1 EACCES
---- Result should have been (exact matching):
0
==== winFCmd-6.9 FAILED

==== winFCmd-6.13 TclpRemoveDirectory: write-protected FAILED
==== Contents of test case:

    cleanup
    file mkdir td1
    testchmod 000 td1
    catch {
        testfile rmdir td1
        file exists td1
    } r
    catch {
        testchmod 777 td1
        cleanup
    }
    set r

---- Result was:
td1 EACCES
---- Result should have been (exact matching):
0
==== winFCmd-6.13 FAILED



==== winFCmd-9.3 TraversalDelete: DOTREE_PRED FAILED
==== Contents of test case:

    cleanup
    file mkdir td1/td2
    testchmod 000 td1
    catch {
        testfile rmdir -force td1
        file exists td1
    } r
    catch {
        testchmod 777 td1
        cleanup
    }
    set r

---- Result was:
td1 EACCES
---- Result should have been (exact matching):
0
==== winFCmd-9.3 FAILED

==== winFCmd-12.6 ConvertFileNameFormat: absolute path with drive FAILED
==== Contents of test case:

    catch {file delete -force -- c:/td1}
    close [open c:/td1 w]
    list [catch {string tolower [file attributes c:/td1 -longname]} msg] $msg [f
ile delete -force -- c:/td1]

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: couldn't open "c:/td1": invalid argument
    while executing
"open c:/td1 w"
    ("uplevel" body line 3)
    invoked from within
"uplevel 1 $script"
---- errorCode: POSIX EINVAL {invalid argument}
==== winFCmd-12.6 FAILED

The change in the "bug-3092089" branch fixes the reported
problem, but I cannot reproduce the test failures that Harald
mentions (using the originally provided patch). My patch
should fix that, but as I cannot reproduce it, I would like
Harald to confirm that the test failures are gone
in his environment too.

If everything OK, I would like to apply this patch to all
branches starting with Tcl 8.4.

Thanks for looking into this.  Passing to
someone who can at least test proposed fixes.

Harald, please have a look at the "bug-3092089" branch. Does that work for you?

Explanation: if GENERIC_READ doesn't work in some cases, and 0 doesn't
work in other cases, why don't we just try both and pick the one that works?
(I'm not even trying to understand what's going on here.......)

Regards,
          Jan Nijtmans

File Added - 457783: testlog_2012-12-13_hao.txt

I have tried the patch proposed at this comment:
Date: 2011-12-12 06:47:09 PST
Sender: dunkfan
with tcl8.5.13, msvc++ 6, Makefile.vc, win vista 32 bit.

Here is the applied patch:

--- C:/test/tcl8.5.13/win/tclWinFile_ori.cTue Nov 06 16:05:00 2012
+++ C:/test/tcl8.5.13/win/tclWinFile.cThu Dec 13 09:47:22 2012
@@ -704,7 +704,7 @@
     HANDLE hFile;
     DWORD returnedLength;
 
-    hFile = (*tclWinProcs->createFileProc)(linkDirPath, GENERIC_READ, 0,
+    hFile = (*tclWinProcs->createFileProc)(linkDirPath, 0, 0,
     NULL, OPEN_EXISTING,
     FILE_FLAG_OPEN_REPARSE_POINT|FILE_FLAG_BACKUP_SEMANTICS, NULL);
 

Results:

1) The issue is solved for me.

The demonstration in comment
Date: 2012-11-15 01:05:50 PST
Sender: oehhar
of tcl bug 3587096
now returns:
% file normalize c:/test2_junction/tcl860rc0
C:/test2/tcl860rc0
%  file normalize c:/test2_junction/tcl860rc0/bin
C:/test2/tcl860rc0/bin
%  file normalize c:/test2_junction/tcl860rc0/bin/junk

2) Bug 3587096 is also solved

3) Many many tests fail in the test suite with "permission denied on file access"
The test log is in the file section (200kB). Here the first failure as an example:

==== autoMkindex-4.1 platform indenpendant source commands FAILED
==== Contents of test case:

    file delete tclIndex
    auto_mkindex . pkg/samename.tcl
    set f [open tclIndex r]
    set dat [split [string trim [read $f]] "\n"]
    set len [llength $dat]
    set result [lsort [lrange $dat [expr {$len-2}] [expr {$len-1}]]]
    close $f
    set result

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: couldn't open "tclIndex": permission denied
    while executing
"open "tclIndex" w"
    (procedure "auto_mkindex" line 32)
    invoked from within
"auto_mkindex . pkg/samename.tcl"
    ("uplevel" body line 3)
    invoked from within
"uplevel 1 $script"
---- errorCode: POSIX EACCES {permission denied}
==== autoMkindex-4.1 FAILED


3) In a practical test, the modified version worked well for me.

Thank you,
Harald

Not new to 8.6; low user pressure. Not a release-blocker for 8.6 I guess => prio 8.

Sorry my patch fixes the junction problem C:/Documents and Settings, but not the latest C:/windows/system32/drivers/etc/hosts one. An explorer on my system does not show the C:/Documents and Settings folder, but it shows the C:/Windows/system32/drivers/etc folder. tk_chooseDirectory shows neither! Why the difference...?

File Added - 430721: normalize.patch

Found the culprit: tclWinFile.c and the NativeReadReparse call. 
When I replace GENERIC_READ with 0, in the CreateFileProc and remake then:

file normalize {C:\Documents and Settings\Guest} 

returns: 

C:\Users\Guest as expected, instead of the incorrect C:\Documents and Settings.
cd and pwd also work correctly after crossing the Junction. :-)

According to the MSDN using 0 implies theat the file is only queried and not accessed. That seems to cure it! Please patch!

Is there a patch available to make Tcl's
native filesystem support do the right thing?

This bug is in fact a big problem and required me to write a work around in C to access the Microsoft API directly.

I am using TCL/TK for an installer, and the typical target directories as seen in the Windows explorer contain at least one junction point, whose contents is not readable (this is the system default). E.g. C:\Programme\ on a German Windows is a junction point to C:\programs\. None of the proposals in the thread mentioned above actually helps.

File Added - 391350: otto371.c

No. I can reproduce this in 8.5.5 and 8.4.19. However, the problem is appearing more often as we are moving away from XP to Vista/Win7

Here is a snip from 8.4.19 running on Windows Vista Business 32 English

% cd {C:/Documents and Settings}
% cd Default
% file normalize .
C:/Documents and Settings
% info patchlevel
8.4.19
%

In a cmd I see the following entry for C:\Documents and Settings, this is causing the trouble

dir /a c:\

02/11/2006  15:02    <JUNCTION>     Documents and Settings [C:\Users]

One of the guys here had a go at solving the normalized problem with such paths having googled for an answer. I'm including a small prog that may be of use. Im not sure how it works or what it does but seeing the API calls used may be of help. A NULL arg to CreateFile helps get the job done.

Is this new (mis-)behavior in 8.5.9 ?  Or a problem
that has a longer history?