from __future__ import *

Kailash and Friends Kailash Kher Kaipa

Since the svnsync tool in Subversion 1.4 still lacks documentation, here's a quick example to get you started with it:

$ svnadmin create svn.mochikit.com
$ echo '#!/bin/sh' > svn.mochikit.com/hooks/pre-revprop-change
$ chmod +x svn.mochikit.com/hooks/pre-revprop-change
$ svnsync init file://$PWD/svn.mochikit.com http://svn.mochikit.com/
$ svnsync sync file://$PWD/svn.mochikit.com

The example performs the following tasks:

• Creates a new local repository named svn.mochikit.com
• Adds a trivial pre-revprop-change hook to the repository (required for svnsync)
• Initializes the svn.mochikit.com repository to be a mirror of http://svn.mochikit.com/
• Synchronizes the svn.mochikit.com repository with its initialized source

Note that this is point-in-time synchronization. To keep an up to date mirror you'll have to run svnsync sync on a regular basis (e.g. by cron job or post-commit hook).

If you're looking for an especially easy to install set of svn binaries for Mac OS X, I recommend Martin Ott's (of SubEthaEdit fame) build. I've been using it on all of my Macs for quite some time with no issues. Much less hassle than building it myself or using MacPorts (previously known as DarwinPorts).

After over three years with a 1ghz 15" titanium PowerBook I've finally upgraded to a 2ghz dual core 15" MacBook Pro! Compared to my old laptop, this machine is absolutely amazing. I've been waiting a LONG time for:

• Built-in Bluetooth, 802.11g, USB 2.0
• Higher resolution, contrast, brightness display
• Trackpad with the two-finger scrolling
• Much much improved video subsystem
• A machine fast enough to effectively use spotlight

Additionally, this new kit is going to save me a lot of VNC sessions since I can run other operating systems in virtualization via Parallels, and it will allow me to more easily build and properly test universal binaries. Parallels seems to work pretty well for a beta product, but I'm definitely looking forward to seeing some competition (especially open source competition).

Despite its awesomeness, there are still some issues with the MacBook Pro:

• The power supply whine is still there when the system isn't using a lot of juice, but apparently much less so than in earlier revisions.
• These machines run HOT. I probably wouldn't try and put it on my lap if I wasn't wearing jeans, and it does make your hands a little uncomfortably warm when using the built-in keyboard (though I'm sure I'll get used to that).
• The display doesn't tilt back quite as far as the old titanium powerbooks. Not a big deal, hopefully that just means they've solved the problem where the hinges would just snap after a year or two (and take out the backlight cable, argh).
• Intel code on Mac OS X supposedly uses more RAM than the PPC equivalent, which means you should buy more RAM than you are accustomed to from earlier machines.

The RAM usage increase is somewhat surprising, because i386 binaries are more compact (the i386 libSystem is about 16% smaller than PPC). I'm guessing the increase is due to the lack of usable registers on i386, which increases the amount of stack necessary per thread. I doubt it's an alignment issue, because PPC definitely cares more about alignment than i386 does. A quick sample shows that Terminal.app with one window does use about 5% more real memory than the PPC equivalent (and a LOT more virtual memory, but that might just be Quartz or something).

5% doesn't sound like much, but I've heard bigger numbers for other kinds of applications. I definitely plan to upgrade this machine from 1gb to 2gb, but the only times I've really felt RAM-hungry so far are when I'm using Parallels (which understandably needs a lot of real memory).

It's been a pretty long haul getting Python to do the right thing on Mac OS X with regard to Universal Binaries. The dust is settling, and it's about time to start publicizing the results.

In order to compile extensions with Universal MacPython, you absolutely must have the latest version of Xcode installed. When you install it, make absolutely sure that you Customize the installation and install the Mac OS X 10.4 Universal SDK. Universal MacPython requires this SDK to be installed, as it instructs the compiler and linker to use this specific SDK. There is a hook that will remove these special flags, but ONLY for legacy Mac OS X 10.3.9 users. Mac OS X 10.3.9 users should still be able to compile extensions, but the resultant binaries will be PPC only and shouldn't be distributed (as they will not work on Intel).

Besides the obvious endian fixes and distutils/configure hackery, Universal MacPython includes many usability enhancements over previous distributions of Python for Mac OS X:

• IDLE, the Python IDE, should now ship with Mac keybindings and menus by default
• Placing binaries in /usr/local/bin is now optional
• The bin directory from the framework is now (by default) automatically added first to your PATH environment variable. This means that when you open a new terminal session and type python, you will get Universal Python 2.4.3. Yes, it's ugly to do this, but it's a necessary evil. PATH related issues are by far the most frequent source of confusion on pythonmac-sig (between Python itself and distutils-installed scripts), and this is the solution. darwinports does something quite similar.
• pythonw is now an executable, instead of a shell script, so it may be used directly in a hash-bang line for a shell script (#!).
• Users no longer need to know or care about pythonw. python and pythonw are the same binary, which simply does an execve to a Python executable inside of a "fake" application bundle to placate the WindowServer. Yes, this introduces a tiny little bit of overhead when invoking python due to the extra syscall, but it is again a necessary evil to work around a design flaw in Mac OS X.
• Includes a working bsddb, readline, and curses.
• Known compatible with Mac OS X 10.3.9 and later, without any patching necessary on Mac OS X 10.4.

The latest Universal Python source code is currently a fork of Python 2.4.3, and currently resides outside of the official Python subversion repository. For those interested in the source, it is here:

• http://svn.pythonmac.org/python24/python24-fat/

The latest binary release of Universal Python is currently available in the Universal Python 2.4 section of pythonmac.org packages! There is/was an older Universal Python build available somewhere on python.org but it is NOT what you should be using (it's built without compiler optimization and is roughly 50% slower on either architecture!):

• http://pythonmac.org/packages/py24-fat

PyObjC and py2app users with Universal MacPython should install PyObjC from SVN trunk (which will net you a universal-ready py2app). There is currently no tagged release of either project that is suitable for this build of Python, but we hope to fix that soon:

• http://svn.red-bean.com/pyobjc/trunk/pyobjc/
• http://svn.red-bean.com/bob/py2app/branches/py2app-0.2-maint/

pythonmac.org packages has been split into two sections, one for universal packages, and one for legacy builds of Python. Currently there aren't many universal packages ready (largely because I don't have an Intel machine yet, so I can only test on PPC). The legacy packages for Python 2.4 are still compatible with Universal MacPython, however, they are NOT compatible with Intel.

Most packages should compile just fine, and setuptools/easy_install are already Universal MacPython aware. However, packages that depend on external libraries will not compile correctly of the dependent libraries aren't also Universal. This means that MySQL bindings, wxPython, and a few other popular packages are not yet readily available. This issue will resolve itself at some point due to the demand, but it's not resolved yet. There's a list of desired packages and their status at the UniversalPackages node on the pythonmac.org wiki.

Versions of Mac OS X prior to 10.4 did support universal binaries, but the shipping gcc did not, so it is possible to compile applications that will work on both 10.3 ppc (Panther) and 10.4 i386 (Tiger). To build Panther-compatible universal binaries, you'll need to use the following Custom Build Settings in Xcode 2.1 or later...

Per-architecture SDK Support for Universal Binaries in Xcode. This is according to documentation, but I couldn't get it to work:

SDKROOT_ppc = /Developer/SDKs/MacOSX10.3.9.sdk
SDKROOT_ppc64 = /Developer/SDKs/MacOSX10.4u.sdk
SDKROOT_i386 = /Developer/SDKs/MacOSX10.4u.sdk

I'm not sure how to translate per-architecture SDKs to gcc settings, but to build single-SDK Universal Binaries with GCC, there are some undocumented (not in the compiler man page, anyhow) flags that you can use. These flags are only available in GCC 4.0 and later, and probably only with Apple's toolchain.

Universal Binaries with GCC (in Makefile syntax):

SDK=/Developer/SDKs/MacOSX10.4u.sdk
CFLAGS= -isysroot ${SDK} -arch ppc -arch ppc64 -arch i386
LDFLAGS= -isysroot ${SDK} -syslibroot,${SDK}

Using lipo(1) to do per-architecture SDK builds with gcc should be possible, but painful. Hopefully it's not the only way! However, that's exactly what Xcode 2.1 does.

See Also:

• Universal Binary Programming Guidelines
• Xcode 2.1 Release Notes
• Cross-Development: Configuring Your Project for Cross Development
• lipo - create or operate on fat files

PyObjC

Ronald has customized the libffi export even further to work for Mach-O x86, mostly by removing the usage of autoconf. He has also managed to hack in some universal binaries support, but it really needs to move to distutils. All of the unit tests pass. This code should end up in ctypes too, probably.

The Xcode templates should work with Xcode 2.1 now, but they won't build universal binaries until py2app does.

objc.inject still doesn't work (since mach_inject isn't fixed yet).

Python

This is built as universal binaries by Apple. I haven't seen the patches yet, but whatever they're doing doesn't carry over to distutils, and it doesn't also include ppc64 support (though it certainly could, if Python didn't depend so much on autoconf crap). One way to fix this would be to create a custom pyconfig.h with reasonable compiler-determined values, and not using autoconf at all (an Xcode project might be a good idea). This has lots of problems of its own, because there are a lot of options that can be useful to configure, distutils reads in several autoconf-generated files, etc.

It's also going to take some time to figure out the gcc incantations to build against SDKs, as they're not documented. Especially if we're going to try and build against two different SDKs (i.e. 10.3 for PPC and 10.4.1 for x86).

If we do end up putting universal binaries support into distutils, it may require users to have the SDK installed in order to use distutils. Unfortunately this isn't the default when installing Xcode 2.1.

Four character codes, and probably other data structures, are still the wrong endian on x86.

The majority of bundlebuilder applications will not run on Intel machines without modification, even with Rosetta, because their executable is a script so Python is going to start up as an x86 binary. py2app-built applications don't have this problem.

It might make sense to add some intelligence to the extension importer to check for correct architecture before it tries to load the .so, or to have separate site-packages directories for each architecture, plus one for universal binaries. For example, it would make sense to have psyco available for x86, but maybe not anywhere else.

py2app

macholib needs to learn how to rewrite load commands of universal binaries.

If the py2app bootstraps are recompiled as universal binaries, then it should probably be able to detect whether the application needs to start up as ppc, because I don't want it to build incompatible apps like bundlebuilder does.

Python on Mac OS X for Intel is not going to be a seamless transition. Unlike Mathematica, there is going to be a lot more than 2 hours of effort involved. Why?

• Python uses autoconf. Autoconf is not Xcode. It does not have a checkbox for universal binary compatibility. Autoconf is a PITA.
• PyObjC depends on libffi. libffi doesn't know what the Mach-O calling convention for x86 is, so it doesn't work. libffi is very deep, magical, scary code. This has been mostly solved by Ronald over the past few days in the universal binaries lab.
• mach_inject (which PyObjC's objc.inject uses) depends on injecting code into other processes. On an OS that can be running code for two different architectures, how the heck does that work? How do you know which pid is running which architecture? Anyway, injecting from x86-x86 isn't going to work because mach_inject doesn't know x86 yet.
• py2app's macholib doesn't really support fat binaries yet. It understands them well enough to do the right thing with the PPC portion of the Mach-O header, but it ignores other architectures. It will need a semi-major refactoring in order to support this cleanly.
• Bgen stuff breaks. Specifically, all of your Carbon code that deals with four character codes is going to be broken due to endianness issues. Yet another reason to stay the heck away from this stuff.

So, thanks Apple, for giving us weeks worth of very hard and unintesting work to do. The least you could've done is put out a nice new laptop that I could buy to do this work on :)

I'm also not terribly interested in renting an Intel development machine from Apple so that I can do work that helps them more than me. If they sold it, or stated that we would get a grand worth of credit towards a real Intel machine when they're available, then I wouldn't complain. But no, we get to rent a machine from them, that we can't really talk about, publish benchmarks of, move to a location other than the shipped address (?!), etc. They also say that they're under no obligation to fix it if it breaks, and there are no refunds. Sweet deal!