from __future__ import *

It's been a pretty long haul getting Python to do the right thing on Mac OS X with regard to Universal Binaries. The dust is settling, and it's about time to start publicizing the results.

In order to compile extensions with Universal MacPython, you absolutely must have the latest version of Xcode installed. When you install it, make absolutely sure that you Customize the installation and install the Mac OS X 10.4 Universal SDK. Universal MacPython requires this SDK to be installed, as it instructs the compiler and linker to use this specific SDK. There is a hook that will remove these special flags, but ONLY for legacy Mac OS X 10.3.9 users. Mac OS X 10.3.9 users should still be able to compile extensions, but the resultant binaries will be PPC only and shouldn't be distributed (as they will not work on Intel).

Besides the obvious endian fixes and distutils/configure hackery, Universal MacPython includes many usability enhancements over previous distributions of Python for Mac OS X:

• IDLE, the Python IDE, should now ship with Mac keybindings and menus by default
• Placing binaries in /usr/local/bin is now optional
• The bin directory from the framework is now (by default) automatically added first to your PATH environment variable. This means that when you open a new terminal session and type python, you will get Universal Python 2.4.3. Yes, it's ugly to do this, but it's a necessary evil. PATH related issues are by far the most frequent source of confusion on pythonmac-sig (between Python itself and distutils-installed scripts), and this is the solution. darwinports does something quite similar.
• pythonw is now an executable, instead of a shell script, so it may be used directly in a hash-bang line for a shell script (#!).
• Users no longer need to know or care about pythonw. python and pythonw are the same binary, which simply does an execve to a Python executable inside of a "fake" application bundle to placate the WindowServer. Yes, this introduces a tiny little bit of overhead when invoking python due to the extra syscall, but it is again a necessary evil to work around a design flaw in Mac OS X.
• Includes a working bsddb, readline, and curses.
• Known compatible with Mac OS X 10.3.9 and later, without any patching necessary on Mac OS X 10.4.

The latest Universal Python source code is currently a fork of Python 2.4.3, and currently resides outside of the official Python subversion repository. For those interested in the source, it is here:

• http://svn.pythonmac.org/python24/python24-fat/

The latest binary release of Universal Python is currently available in the Universal Python 2.4 section of pythonmac.org packages! There is/was an older Universal Python build available somewhere on python.org but it is NOT what you should be using (it's built without compiler optimization and is roughly 50% slower on either architecture!):

• http://pythonmac.org/packages/py24-fat

PyObjC and py2app users with Universal MacPython should install PyObjC from SVN trunk (which will net you a universal-ready py2app). There is currently no tagged release of either project that is suitable for this build of Python, but we hope to fix that soon:

• http://svn.red-bean.com/pyobjc/trunk/pyobjc/
• http://svn.red-bean.com/bob/py2app/branches/py2app-0.2-maint/

pythonmac.org packages has been split into two sections, one for universal packages, and one for legacy builds of Python. Currently there aren't many universal packages ready (largely because I don't have an Intel machine yet, so I can only test on PPC). The legacy packages for Python 2.4 are still compatible with Universal MacPython, however, they are NOT compatible with Intel.

Most packages should compile just fine, and setuptools/easy_install are already Universal MacPython aware. However, packages that depend on external libraries will not compile correctly of the dependent libraries aren't also Universal. This means that MySQL bindings, wxPython, and a few other popular packages are not yet readily available. This issue will resolve itself at some point due to the demand, but it's not resolved yet. There's a list of desired packages and their status at the UniversalPackages node on the pythonmac.org wiki.

A few weeks ago I began my move from New York to San Francisco by... moving to Hawaii. For the summer, anyhow. In shipping, my trusty old Klipsch computer speakers seem to have eaten it. I tried hooking them up, and after some really painful shrieking, they worked for a few minutes.. until they caught fire. Really. My ex-favorite music spewing devices have found themselves a nice retirement home in a landfill somewhere on Maui.

I tried playing iPod DJ with the stereo here, but that doesn't really work very well for long stretches. It was time to break down and finally get an Airport Express. I looked around online a bit, and shipping was just ridiculous to Hawaii from most places (e.g. Smalldog wanted $95 S+H to ship a $120 refurb Airport Express to Kihei!). Fortunately, there is a Mac store in town, so I was able to pick one up at a reasonable price.

Immediately after bringing it home I plugged it in via ethernet, and tried to update the firmware. A few hundred times. It just wouldn't take. The Airport Admin utility would not do it. From either of my Macs. It'd try, and it would fail after a minute or two without even an error code. Eventually I found a link to download a standalone firmware updater application, which updated the firmware first try. Go figure. Certainly not the typical Apple experience.

Once it was updated and hooked up, it worked great. 90% of the time, anyway. iTunes will only play files that it natively understands through the Airport Express. Apparently, iTunes has decided that some of my MP3s were QuickTime files, and refused to play them through Airport Express, which means they were coming out of the built-in speakers. That didn't work out so well. I guess that sucks for people who care about OGG and other formats supported only via QuickTime.

It turns out that the reason these files were picked up as QuickTime is that iTunes has MPEG type code hate. MacAMP and SoundJam of yesteryear used MPEG, yet iTunes wants to see MPG3. It took me a lunch break to come up with a good way to track down all of these files without writing a lot of code. It was actually rather simple: the iTunes Music Library XML dump.

~/Music/iTunes/iTunes Music Library.xml is a brain-dump of iTunes that it creates every so often (on quit, I believe). It keeps track of almost everything that iTunes knows, all of your file's URLs, their type and creator codes, comments, metadata, you name it. It's also in plist format, which is extremely easy to work with from Python.

Assuming you have PyObjC installed, you can break into a Python interpreter and screw around with this rather easily:

% python
Python 2.4.1 (#2, Mar 31 2005, 00:05:10)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from Foundation import *
>>> import os
>>> dbFile = os.path.expanduser("~/Music/iTunes/iTunes Music Library.xml")
>>> db = NSDictionary.dictionaryWithContentsOfFile_(dbFile)
>>> type(db)
<objective-c class NSCFDictionary at 0x30a4b0>

With the database in hand, we can start poking around:

>>> track = db[u'Tracks'].itervalues().next()
>>> print track.description()
{
    Album = "Live Code";
    Artist = "Front 242";
    "Artwork Count" = 1;
    "Bit Rate" = 160;
    ...

In these track dictionaries, there are two keys relevant to this exercise: u'File Type', and u'Location'. File type is the integer representation of the type code of the track. Location is the URL of the track. With this, it's pretty easy to pick out the locations of the tracks that are set to the MPEG type code:

>>> import struct
>>> badPaths = [
...     NSURL.URLWithString_(track[u'Location']).path()
...     for track in db[u'Tracks'].itervalues()
...     if track.get(u'File Type') == struct.unpack('>I', 'MPEG')[0]
... ]
>>> badPaths[0]
/Volumes/...

With this, all we need to do is change the type and creator. Fortunately, that's easy. We'll use the iTunes creator code hook, and the expected MPG3 type code:

>>> import MacOS
>>> for path in badPaths:
...     MacOS.SetCreatorAndType(path, 'hook', 'MPG3')
...

And with that, my Airport Express does just about everything I'd have expected it to out of the box ;)

I've been neglecting Python a bit lately, but Ronald is going to cut a release of PyObjC 1.3.7 very soon, hopefully later this week. PyObjC 1.3.7 fixes a few bugs (particularly with Xcode 2.1 templates), has a more complete wrapping of Mac OS X 10.4 system frameworks (DiscRecording, SenTestingKit, SecurityFoundation), and adds a new TinyURLService example (which converts URLs on the pasteboard to tinyurl.com equivalents).

There's also a swath of new "First Steps" documentation in the PyObjC intro, which I wrote up earlier today:

[someObject doSomething:arg1 withSomethingElse:arg2];

doSomething:withSomethingElse:

doSomething_withSomethingElse_

someObject.doSomething_withSomethingElse_(arg1, arg2)

# note the trailing underscore
someObject.setValue_(aValue)

myObject = NSObject.alloc().init()

myData = NSData.alloc().initWithBytes_length_('the bytes', 9)

class MyClass(NSObject):

    def init(self):
        """
        Designated initializer for MyClass
        """
        # ALWAYS call the super's designated initializer.
        # Also, make sure to re-bind "self" just in case it
        # returns something else!
        self = super(MyClass, self).init()

        self.myVariable = 10

        # Unlike Python's __init__, initializers MUST return self,
        # because they are allowed to return any object!
        return self

class MyOtherClass(MyClass):

    def initWithOtherVariable_(self, otherVariable):
        """
        Designated initializer for MyOtherClass
        """
        self = super(MyOtherClass, self).init()
        self.otherVariable = otherVariable
        return self

myInstance = MyClass.alloc().init()
myOtherInstance = MyOtherClass.alloc().initWithOtherVariable_(20)

# This is equivalent to:
#
#   myObject = NSObject.alloc().init()
#
myObject = NSObject.new()

# This is equivalent to:
#
#   myDict = NSDictionary.alloc().init()
#
myDict = NSDictionary.dictionary()

# This is equivalent to:
#
#   myString = NSString.alloc().initWithString_(u'my string')
#
myString = NSString.stringWithString_(u'my string')

# Getting
#
# notice the method call
#
myValue = valueContainer.value()

# Setting
#
# notice the naming convention and trailing underscore
#
valueContainer.setValue_(myNewValue)

class MyValueHolder(NSObject):

    def initWithValue_(self, value):
        self = super(MyValueHolder, self).init()
        # It's recommended not to use typical Python convention here,
        # as instance variables prefixed with underscores are reserved
        # by the Objective-C runtime.  It still works if you use
        # underscores, however.
        self.ivar_value = value
        return self

    def value(self):
        return self.ivar_value

    def setValue_(self, value):
        self.ivar_value = value

PyObjC

Ronald has customized the libffi export even further to work for Mach-O x86, mostly by removing the usage of autoconf. He has also managed to hack in some universal binaries support, but it really needs to move to distutils. All of the unit tests pass. This code should end up in ctypes too, probably.

The Xcode templates should work with Xcode 2.1 now, but they won't build universal binaries until py2app does.

objc.inject still doesn't work (since mach_inject isn't fixed yet).

Python

This is built as universal binaries by Apple. I haven't seen the patches yet, but whatever they're doing doesn't carry over to distutils, and it doesn't also include ppc64 support (though it certainly could, if Python didn't depend so much on autoconf crap). One way to fix this would be to create a custom pyconfig.h with reasonable compiler-determined values, and not using autoconf at all (an Xcode project might be a good idea). This has lots of problems of its own, because there are a lot of options that can be useful to configure, distutils reads in several autoconf-generated files, etc.

It's also going to take some time to figure out the gcc incantations to build against SDKs, as they're not documented. Especially if we're going to try and build against two different SDKs (i.e. 10.3 for PPC and 10.4.1 for x86).

If we do end up putting universal binaries support into distutils, it may require users to have the SDK installed in order to use distutils. Unfortunately this isn't the default when installing Xcode 2.1.

Four character codes, and probably other data structures, are still the wrong endian on x86.

The majority of bundlebuilder applications will not run on Intel machines without modification, even with Rosetta, because their executable is a script so Python is going to start up as an x86 binary. py2app-built applications don't have this problem.

It might make sense to add some intelligence to the extension importer to check for correct architecture before it tries to load the .so, or to have separate site-packages directories for each architecture, plus one for universal binaries. For example, it would make sense to have psyco available for x86, but maybe not anywhere else.

py2app

macholib needs to learn how to rewrite load commands of universal binaries.

If the py2app bootstraps are recompiled as universal binaries, then it should probably be able to detect whether the application needs to start up as ppc, because I don't want it to build incompatible apps like bundlebuilder does.

The web presence for the Python on Mac OS X session at WWDC2005 is now up: http://pythonmac.org/wwdc2005/

The CodeImporter example and slides will be available as soon as I've made sure that it's OK to post them.

Python on Mac OS X for Intel is not going to be a seamless transition. Unlike Mathematica, there is going to be a lot more than 2 hours of effort involved. Why?

• Python uses autoconf. Autoconf is not Xcode. It does not have a checkbox for universal binary compatibility. Autoconf is a PITA.
• PyObjC depends on libffi. libffi doesn't know what the Mach-O calling convention for x86 is, so it doesn't work. libffi is very deep, magical, scary code. This has been mostly solved by Ronald over the past few days in the universal binaries lab.
• mach_inject (which PyObjC's objc.inject uses) depends on injecting code into other processes. On an OS that can be running code for two different architectures, how the heck does that work? How do you know which pid is running which architecture? Anyway, injecting from x86-x86 isn't going to work because mach_inject doesn't know x86 yet.
• py2app's macholib doesn't really support fat binaries yet. It understands them well enough to do the right thing with the PPC portion of the Mach-O header, but it ignores other architectures. It will need a semi-major refactoring in order to support this cleanly.
• Bgen stuff breaks. Specifically, all of your Carbon code that deals with four character codes is going to be broken due to endianness issues. Yet another reason to stay the heck away from this stuff.

So, thanks Apple, for giving us weeks worth of very hard and unintesting work to do. The least you could've done is put out a nice new laptop that I could buy to do this work on :)

I'm also not terribly interested in renting an Intel development machine from Apple so that I can do work that helps them more than me. If they sold it, or stated that we would get a grand worth of credit towards a real Intel machine when they're available, then I wouldn't complain. But no, we get to rent a machine from them, that we can't really talk about, publish benchmarks of, move to a location other than the shipped address (?!), etc. They also say that they're under no obligation to fix it if it breaks, and there are no refunds. Sweet deal!

For those of you attending WWDC 2005 next week, make sure to check out the Python related sessions:

602 A.M.P. on Mac OS X:

Tue, 9:00-10:30, North Beach

613 Python on Mac OS X:

Wed, 9:00-10:30, North Beach

Additionally, Guido is going to be speaking about Python at the Wednesday Brown Bag Lunches With O'Reilly (Wed, 12:45pm).

Oh, and if you're looking to complain about the lack of Python support (or public APIs so that third parties can do the hard work) in Xcode, here's where to express that opinion:

410 Developer Tools Feedback Forum:

Fri, 5:00-6:00, Russian Hill

There isn't a Python/PyObjC BOF arranged yet. If you're interested in making that happen, find me after session 613 on Wednesday morning. I don't really know the area, but bbum does (more than me, anyway), and he should be around at about the same time too.

We've (finally) updated talkingpanda.com today with a fresh new look and a new product: iBar.

iBar turns your iPod (any iPod with a screen and a dock connector) into an "ultimate bartending tool" with over a thousand drink recipes, mixing techniques, and even a couple history lessons. Like any good bar should be, it's stocked. Altogether, the Notes content adds up to over 3 megabytes (yes, that's text!) with more than 40 minutes of original, professionally recorded, audio.

Like its sibling iLingo, iBar ships with installers for Mac OS X and Windows XP/2000 (everywhere iTunes is supported) that make installation painless.

And for relevance, here's a little bit about how it's all put together:

• The Mac OS X installer was developed with Python 2.3, PyObjC 1.2, and bundled up with py2app.
• The Win32 installer was developed with Python 2.4, win32all, Tkinter, PIL and is bundled up with py2exe and then made self-contained with NSIS.
• The build scripts for the installer application actually use even more open source stuff. We use JExcelAPI to convert Excel spreadsheets to XML, and Win32::Exe to swap out the ICO resource out of win32 executables -- but no Java or Perl makes it into the redistributable :)

We're also blogging updates and podcasting some of our content, so point your NetNewsWire or Safari RSS over to Talking Panda News!

PyObjC 1.3.6 is now available for download at http://pyobjc.sourceforge.net/

This release fixes an issue with plugins.

py2app 0.2 is a minor bug fix release. You can download it from pythonmac.org packages. PyObjC 1.3.5 ships with py2app 0.2, so if you have that, you have already upgraded to py2app 0.2!

Functional changes:

• New datamodels option to support CoreData. Compiles .xcdatamodel files and places them in the Resources dir (as .mom).
• New use-pythonpath option. The py2app application bootstrap will no longer use entries from PYTHONPATH unless this option is used.
• py2app now persists information about the build environment (python version, executable, build style, etc.) in the Info.plist and will clean the executable before rebuilding if anything at all has changed.
• bdist_mpkg now builds packages with the full platform info, so that installing a package for one platform combination will not look like an upgrade to another platform combination.

Bug Fixes:

• Fixed a bug in standalone building, where a rebuild could cause an unlaunchable executable.
• Plugin bootstrap should compile/link correctly with gcc 4.
• Plugin bootstrap no longer sets PYTHONHOME and will restore PYTHONPATH after initialization.
• Plugin bootstrap swaps out thread state upon plug-in load if it is the first to initialize Python. This fixes threading issues.