Tcl Source Code

This error occurs when running the macos build:

/Library/Developer/CommandLineTools/usr/bin/redo_prebinding: the __LINKEDIT segment does not cover the end of the file (can't be processed) in: .//Library/Frameworks/Tcl.framework/Versions/8.7/Tcl

References:
https://stackoverflow.com/questions/48324224/codesign-an-old-director-projector-on-os-x-10-13-perhaps-manipulate-linkedit

https://github.com/pyinstaller/pyinstaller/wiki/Recipe-OSX-Code-Signing (see pyinstaller fix implementation)

# some good information on how to create a new TEXT segment when linking.
https://github.com/HaxeFoundation/neko/issues/130

Basically, this ticket is now solved in two ways:

1) When using a MacOS framework, the zip-file is not attached any more

2) When building standalone executables, marc_culler's "macher" utility fixes the job: https://github.com/culler/macher. Just make sure "macher" is somewhere on your path, then the Tcl "configure" script will find it and use it.

So, this ticket can be closed.

I am not saying that the zipfs idea is totally useless on macOS, nor that
it is a non-starter.  I am saying that it should never be used in a framework
or in a standalone application.  But it does have a limited utility.

Where I do think the zipfs feature could be useful is for single file
executables which are meant to be run from the command line.

An example of such a thing is fossil.  I build my own fossil because Apple's
openssl is totally broken.  When I build fossil with Tk support (because I
find fossil diff -tk to be incredibly useful) I am not able to use it on a
machine which does not have Tcl installed in /Library/Frameworks. Specifically,
when I create a new macOS image in VMWare and I want to use fossil to clone
the Tcl and Tk repositories for the first time, I cannot use a fossil built
with Tk support.  So I build two versions of fossil, called fossil0 and fossil.
The first one does not have Tk support, and I can use it for my first clone.
Then I switch to using fossil.

But, once again, standalone macOS apps should be bundles and should never
use the zipfs feature.  Ditto for frameworks.

Just to expand on Marc's observations, the single-file-executable question has been addressed on the Mac for 20+ years with app bundles (app directories that appear to the user as single files), frameworks, and related technologies. The zipfs approach, as currently laid out, is completely incompatible with that and, in fact, seems to break Tcl/Tk on the Mac. I agree with Marc that the current approach is a non-starter on macOS.

Historically, tclkits have worked on the Mac by embedding everything, including the frameworks, within the virtual filesystem. Unpack a tclkit and Tcl.framework and Tk.framework are among the bits dumped out in to the vfs folder. While not very Mac-like, it does work, and can even be wrapped up in an app bundle as well so the user doesn't see a raw executable sitting on their desktop. Why are we not look at this approach for zipfs?

It does not make sense for a macOS framework build.

The macOS framework structure completely solves the problem of how a shared
library can find its initialization scripts.  And it solves that problem in
a way which is completely standardized throughout the operating system and
understood and expected by developers. It is also well thought out. I think
it is one of the better designed parts of macOS. It would be crazy to throw
that away in favor of some quirky one-off scheme that applies to Tcl and
nothing else and makes Tcl incompatible with everything else in the OS.

I have no issue with using the zip scheme for other OS's and I think it is
fine for single-file executables but please don't do that to the Tcl and Tk
frameworks.  That would be a big mistake.

Well, I think zips makes very much sense in a shared build! Just in a very different way than in static builds. In shared builds, the zip-file is attached to the shared library, not to the exe. So the shared library no longer has to search for its initialization scripts, it already has them in its own zip-file. Unfortunately, in Tk this implementation is only half-baked yet. My intention is to fix that first, and then we can see how to deal with that on MacOS. Thanks for all your info and the fix_macho script. That is very useful. Could we have such script in Tcl? :-; Thanks very much!

Just to expand on my comment about pointlessness ...

We have already disabled the zipfs option for framework builds on macOS but
we should also disable it for dynamic builds on any platform.  If a single
file executable requires a shared library then it is not a single file
executable.  This only make sense if tclsh is statically linked.

Appending a zip file to a mach-O binary corrupts the mach-O binary.
This is because a mach-O binary has structure.  It has a header followed
by a sequence of load commands.  Some load commands define segments,
such as __TEXT or __DATA. Some load commands define sections within
those segments.  The load command which defines a segment specifies
the offset and size of the segment within the file and within the
virtual image after loading. The last segment is the __LINKEDIT segment,
which is reserved for use by the dynamic linker. In the binary file this
segment is required to extend to the end of the file.  This is no longer
the case after you append data to the end of the binary.

It is possible to adjust the load commands in the binary so that the
extra data becomes part of the __LINKEDIT segment, and appears to the
loader as if it were part of the string table in the LC_SYMTAB section
of the __LINKEDIT segment.  I am attaching a python script which does
that, using the pypi package macholib. It is named fix_macho.

If you build the hello world example as described in TIP #430 and run
$ ./fix_macho hello
you will get a single file executable which runs and which also can be
processed by install_name_tool with no errors -- the @bll test passes.

NOTE: Building a "single file executable" for macOS using the recipe in
the TIP is a **truly pointless exercise**.  The executable that you get
needs to load the shared library /Library/Frameworks/Tcl.framework/Tcl.
If that shared library exists, then you could simply run the Tcl script
with tclsh.  If it does not exist then the executable is useless. It
will crash because it cannot load the shared library that it depends
upon.

Related tickets:

https://core.tcl-lang.org/tcl/tktview?name=16e9b297d

https://core.tcl-lang.org/tcl/info/92c2169782c2e006

Just an example that proves the library is corrupted.
I would have to re-run my build scripts against 8.7 to see if that error
pops up when install_name_tool is run on the executables.

Previously, I may have had some other package installed that required the
library load path be changed in a shared library.  I do not appear to 
have any such now.

The goal is simply to have the .zip file in a different location other
than appended to the shared library for Mac OS.  

In order to build with a different location for the .zip file, 
this ticket becomes relevant:
   https://core.tcl-lang.org/tcl/tktview?name=16e9b297d
As there needs to be a way to build Tcl such that the .zip file is located
elsewhere.

When Tcl is built as a framework, this location is easy, somewhere in the 
Resources/ directory, accessed via a relative path.
When built via ../unix/configure and the standard Makefile, the location may
need to be hard-coded.

Given that macOS has a concept of a "framework" which is designed for this purpose,
I agree that it would make more sense to use it.  Of course it is not one file,
it is a directory, but it is easily distributed as one file, either as a disk image
or a .pkg file. Using a zip filesystem would seem to be a Windows solution to this
problem which is not such a good match for macOS.

There is also a notion of a "subframework" in macOS.  An application bundle is
allowed to contain a Frameworks directory at the same level as the MacOS directory
that contains the main executable of the application.  When Tcl is built as a
framework, all of the needed files are already available to Tcl.  This requires
no change.  If you want to use the Tcl framework as a subframework placed in the
Frameworks directory of an application bundle then you can make the Tcl library
available to the application by setting the library load path of the main
executable of the application to include:
@executable_path/../Frameworks/Tcl.framework/Versions/Current/Tcl
or even
@executable_path/../Frameworks/Tcl.framework/Tcl
This is easily done with install_name_tool and works fine with code signing as
well.

So using a subframework would seem to be a more natural means of doing this
on macOS if I understand correctly what the goal is.

The appending of the zip file is done for TIP #430. It is - basically - all files normally stored on disk under $tcl_library, which are put into a zip-file and appended to the shared library. Tcl is able to access this zip-file at run-time, which means that we just have a single shared library containing everything in stead of relying on a bunch of files on disk. The shared library contains everything needed, instead of it needing to search where "init.tcl" is.

For MacOS, appending a zip-file to a shared library is not such a good idea. It would be better to store the zip-file in a resource, and adapt TIP #430 such that Tcl can access the resource instead of accessing the zip-file. That's what this ticket is about.

This is 'critical' IMHO for 8.7, but since 8.7 final is still far off it isn't something that needs to be solved now.

If you are only using install_name_tool to check whether a dynamic library is
corrupt then I think you would be better off using otool -l to see what the
actual structure of the mach-O library is.

But I can't contribute anything here until I find out what causes a zip file
to be appended to a dynamic library and why this is being done.

I really don't understand your invocation of install_name_tool.  You seem
to be trying to change the "identification name" of the .dylib file, not the
load path for any of the shared libraries that it depends upon. To do that
you are supposed to use the -id option.  I am able to change the id of the Tcl
library that I built using the standard make -C macosx command. The library
dependencies of that Tcl library are all system libraries, for which a static
global path should be fine.  They are:

/usr/lib/libz.1.dylib
/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation

I can't imagine why you would want to change any of those, which is all that
can be done with the -change option.

I also don't understand how when or why a zip file would be getting appended
to the Tcl dynamic library.  I can believe that there is a make option that
does that, but I don't know what it is. I can also believe that simply
appending a random file to the end of a mach-O dynamic library does not have
the effect of extending the TEXT segment.  I need a lot more explanation about
what the goal is here and also what the mechanism is for appending a zip file.

A dynamic library (often but not always named with the .dylib extension) can
be used for either linking or for loading dynamically.  It is not possible to
link against a "shared library" (usually .so or .bundle).  The Tcl library
created by the standard build is a dynamic library but it is just named Tcl.
My understanding is that the dynamic library's identification name is only
used when an executable or library is linked against it, and that it gets
copied into the list of library load paths for the executable, i.e. the paths
that get displayed by otool -L below the identification name.

bll-mac:/Volumes/Users/bll/t/localE/lib$ install_name_tool -change /Volumes/Users/bll/t/localE/lib/libtcl8.7.dylib @executable_path/../lib/libtcl8.7.dylib libtcl8.7.dylib 
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: fatal error: the __LINKEDIT segment does not cover the end of the file (can't be processed) in: libtcl8.7.dylib

I don't recall the conditions under which I built.
It's not so much a problem with what error may or may not print,
but rather the fact that appending a .zip file to the .dylib corrupts
the .dylib.

The test will be to run otool and install_name_tool on the newly
built libraries and make sure they are able to process the library.

I see that this ticket has been assigned to me, but I do not know what the
ticket is about.  I just build the tip of core-8-branch on Big Sur and there
were no errors, and no warnings.  I just used the usual command:

make -C macosx

Maybe this problem, whatever it was, got fixed?

Hmm, indeed... Or may be it is just some after effect, e. g. map/lib file should be regenerated after melting with zip-content, or there is a better zip-version for macos, that doesn't have this issue.

Anyway linking the zip as resource or object file would be definitely better strategy. A bit more complicated, because of mutual dependencies, but... it looks like here it could be the single proper way, possibly.

> As zip seems to do the correction of the entry offsets here, I don't understand
> why the offsets of __LINKEDIT segment should be still invalid. What do you see
> here (or do you see this or similar message)? 

I doubt very much that zip knows anything about Mac OS X dynamic library
structure.

XCode 8 is a bit old now...
I can also test with High Sierra/10.1 or Sierra/9.2.
MacPorts zip is the same version, perhaps different compilation options.

Mojave 10.14.1
XCode command line tools 10.1

bll-mac:/Users/bll/t/tcl-work/tcl$ fossil status
repository:   /Volumes/Users/bll/t/tcl-work/tcl/../../tcl.fossil
local-root:   /Volumes/Users/bll/t/tcl-work/tcl/
config-db:    /Users/bll/.fossil
checkout:     78fe053f4a93d61c5a6ef3901a346cf74b953eb0 2018-11-23 19:59:12 UTC
parent:       fc958af3807acc9eaadded3b0345be933f80071c 2018-11-23 11:59:53 UTC
merged-into:  56b5fa3d12208e41177d7257c335a1b986fc9fd9 2018-11-23 21:00:57 UTC
tags:         core-8-branch
comment:      make: better algo to wrap to absolute zip-path (for systems resp.
              toolchains without readlink/realpath), e. g. avoid xcode/macosx
              build error "readlink: illegal option -- m" (user: sebres)

# The zip that comes with Mojave...
zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.

make -C macosx PREFIX="" INSTALL_ROOT=$HOME/t/local-test  install

checking for zip... /usr/bin/zip 
Found INFO Zip in environment

ld: warning: option -prebind is obsolete and being ignored
Zip entry offsets appear off by 2827284 bytes - correcting...

/Library/Developer/CommandLineTools/usr/bin/redo_prebinding: the __LINKEDIT segment does not cover the end of the file (can't be processed) in: .//Library/Frameworks/Tcl.framework/Versions/8.7/Tcl

Do you have tried it with the latest fixes? Or could you try it with newest since [78fe053f4a], because it had previously some typo's and mistakes in unix/Makefile and in several configure scripts.

I'm not sure this is completely valid for macos, because is not tested there, but Xcode build (xcode8 C) seems to be OK (BUILD_DIR=unix as well as BUILD_DIR=macosx), at least on TravisCI.

As zip seems to do the correction of the entry offsets here, I don't understand why the offsets of __LINKEDIT segment should be still invalid. What do you see here (or do you see this or similar message)?

Which revision exactly produced this error?

Did you see more info better describing the error (failing group or line number where it is failed, some another errors/warnings)?

Which zip version exactly is used by build? e. g. is zip installed on yoour system or rather a minizip was used (built within tcl as zip-replacement fallback in build-directory if zip is missing)? Or what do you see by `zip --version`?

Because I'm not sure minizip can fix segments/sfx-info (especially by macosx), so could you install zip in order to test it works with?