Cygwin fork() rebase solution (was Re: dll_list::load_after_fork() blues ...)

Tue Dec 18 16:12:36 EST 2001

On Wed, Dec 12, 2001 at 08:57:44AM -0500, Jason Tishler wrote:
> On Mon, Dec 10, 2001 at 11:52:08AM -0500, Norman Vine wrote:
> > Michael Hudson writes:
> > >FWIW, and I don't know how much that is, all tests pass if I link _socket
> > >statically.  Oh, and this is building without threads, it seems.  I'll do
> > >a new build with threads and see if anything changes, but I doubt it.
> > 
> > GREAT IDEA !
> > 
> > I just rebuilt python 2.1.1 with threads and linking _socket statically
> > and all seems to work :-)
> > 
> > [snip]
> 
> I just tried the above and it works.  However, it only works if one has
> not fiddled around with rebasing their DLLs.
> 
> Although this is a good short-term workaround (and the one that I will
> probably use when I release Python 2.2), I think that we should focus
> our efforts on trying to solve this problem at its root cause -- Cygwin
> fork() and DLL base address conflicts.  Otherwise, when something else
> changes we will be back in the same situation.

I believe that I have found a rebase solution to the Cygwin fork()
problem that has been causing Cygwin Python some grief lately.  I added
an offset option to the attached rebase tool.  If I spread the DLLs out
by an extra 0x10000, then the fork() problem seems to be mitigate.

The following is the command line necessary to build rebase:

    $ g++ -O2 -o rebase rebase.cc -limagehlp

After rebasing the necessary Cygwin DLLs:

    $ cd /usr/bin
    $ rebase -d -b 0x68000000 -o 0x10000 cygXpm-X4.dll cygXpm-noX4.dll \
    cygbz21.0.dll cygcrypto.dll cygform5.dll cygform6.dll \
    cyggdbm.dll cyghistory4.dll cyghistory5.dll cygintl.dll \
    cygitcl30.dll cygitk30.dll cygjbig1.dll cygjpeg6b.dll cygmenu5.dll \
    cygmenu6.dll cygncurses++5.dll cygncurses++6.dll cygncurses5.dll \
    cygncurses6.dll cygpanel5.dll cygpanel6.dll cygpcre.dll \
    cygpcreposix.dll cygpng2.dll cygreadline4.dll cygreadline5.dll \
    cygregex.dll cygssl.dll cygtcl80.dll cygtclreg80.dll cygtiff3.dll \
    cygtk80.dll cygz.dll

I am able to build Python *without* the following patch:

    http://sourceforge.net/tracker/index.php?func=detail&aid=491107&group_id=5470&atid=305470

After rebasing the necessary Cygwin and Python DLLs:

    $ cd /usr/bin
    $ rebase -d -b 0x68000000 -o 0x10000 cygXpm-X4.dll cygXpm-noX4.dll \
    cygbz21.0.dll cygcrypto.dll cygform5.dll cygform6.dll \
    cyggdbm.dll cyghistory4.dll cyghistory5.dll cygintl.dll \
    cygitcl30.dll cygitk30.dll cygjbig1.dll cygjpeg6b.dll cygmenu5.dll \
    cygmenu6.dll cygncurses++5.dll cygncurses++6.dll cygncurses5.dll \
    cygncurses6.dll cygpanel5.dll cygpanel6.dll cygpcre.dll \
    cygpcreposix.dll cygpng2.dll cygreadline4.dll cygreadline5.dll \
    cygregex.dll cygssl.dll cygtcl80.dll cygtclreg80.dll cygtiff3.dll \
    cygtk80.dll cygz.dll ~/src/PythonCvs/nothreads/libpython2.2.dll \
    ~/src/PythonCvs/nothreads/build/lib.cygwin-1.3.6-i686-2.2/*.dll

I can run the full regression test without any failures *even though*
_socket is a shared module.

I encourage those interested in Cygwin Python to determine whether or not
the above rebase solution works for them too.  I'm particular interested
in 9x/Me reports since I do not have access to those platforms.  Please
post your results to the Cygwin and Python mailing lists.

If my findings are corroborated, then I will work with the Cygwin team
to integrate rebase into setup.exe so that rebasing automatically occurs
every time that setup.exe is run.

Thanks,
Jason
-------------- next part --------------
/*
 * Copyright (c) 2001 Jason Tishler
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * as published by the Free Software Foundation; either version 2
 * of the License, or (at your option) any later version.
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
 *
 * $Id: rebase.cc,v 1.5 2001/12/18 20:21:19 jtishler Exp $
 */

#include <iostream>
#include <sstream>
#include <string>
#include <time.h>
#include <stdlib.h>
#include <getopt.h>
#include <sys/cygwin.h>
#include <windows.h>
#include <imagehlp.h>

using namespace std;

string PosixToWin32(const string& aPosixPath);
void ParseArgs(int argc, char* argv[]);
unsigned long StringToUlong(const string& aString);
void Usage();

ULONG theImageBase = 0;
BOOL theDownFlag = FALSE;
ULONG theOffset = 0;
int theArgsIndex = 0;

void
main(int argc, char* argv[])
{
	ParseArgs(argc, argv);
	ULONG aNewImageBase = theImageBase;

	for (int i = theArgsIndex; i < argc; i++)
	{
		if (theDownFlag)
			aNewImageBase -= theOffset;

		string aFile = PosixToWin32(argv[i]);
		ULONG anOldImageSize, anOldImageBase, aNewImageSize;
		ULONG aPrevNewImageBase = aNewImageBase;
		BOOL aStatus = ReBaseImage(
			const_cast<char*>(aFile.c_str()), // CurrentImageName
			0, // SymbolPath
			TRUE, // fReBase
			FALSE, // fRebaseSysfileOk
			theDownFlag, // fGoingDown
			0, // CheckImageSize
			&anOldImageSize, // OldImageSize
			&anOldImageBase, // OldImageBase
			&aNewImageSize, // NewImageSize
			&aNewImageBase, // NewImageBase
			time(0)); // TimeStamp

		// ReBaseImage seems to never returns false!
		DWORD aStatus2 = GetLastError();
		if (aStatus2 != 0)
		{
			cerr << "ReBaseImage() failed with last error = " <<
				GetLastError() << endl;
			exit(2);
		}
		cout << aFile << hex << ": new base = " <<
			((theDownFlag) ? aNewImageBase : aPrevNewImageBase) <<
			", new size = " << aNewImageSize + theOffset << endl;

		if (!theDownFlag)
			aNewImageBase += theOffset;
	}

	exit(0);
}

string
PosixToWin32(const string& aPosixPath)
{
	char aWin32Path[MAX_PATH];
	cygwin_conv_to_win32_path(aPosixPath.c_str(), aWin32Path);
	return aWin32Path;
}

void
ParseArgs(int argc, char* argv[])
{
	const char* anOptions = "b:do:";
	for (int anOption; (anOption = getopt(argc, argv, anOptions)) != -1;)
	{
		switch (anOption)
		{
			case 'b':
				theImageBase = StringToUlong(optarg);
				break;
			case 'd':
				theDownFlag = TRUE;
				break;
			case 'o':
				theOffset = StringToUlong(optarg);
				break;
			default:
				Usage();
				exit(1);
				break;
		}
	}

	if (theImageBase == 0)
	{
		Usage();
		exit(1);
	}

	theArgsIndex = optind;
}

unsigned long
StringToUlong(const string& aString)
{
	stringstream ss;
	unsigned long aUlong;
	string::size_type anIndex = aString.find("0x");
	if (anIndex == 0)
		ss << hex << string(aString, 2, aString.size() - 2);
	else
		ss << aString;
	ss >> aUlong;
	return aUlong;
}

void
Usage()
{
	cerr << "usage: rebase -b BaseAddress [-d] -o Offset ImageFileName ..." << endl;
}