from __future__ import *

For various reasons, I have a need to determine the version of an iPod. Previously, I only needed to know if an iPod was "firmware version 2" or later (aka "Dock-connector"), and there's a whole slew of obvious ways to determine that (presence of a Notes folder would be the most obvious). There are also ways to grab iPod information with SPI (via COM on win32 or the private iPod.framework on Mac OS X), but I like to avoid that whenever possible.

So far, the most reliable method I've found was with a little parsing of the SysInfo file. This file is located at "$(IPOD_VOLUME)/iPod_Control/Device/SysInfo". The iPod_Control folder is going to be hidden on either platform, so you may not have noticed it before. Many examples of SysInfo files can be found by googling for some of the keys, such as buildID, visibleBuildID, boardHwName, etc. The most complete resource I've found is the on the iPodLinux forums.

As-is, the SysInfo files don't make any of the useful information very obvious. However, with a little reverse-engineering of the iPod Updater for Mac OS X, it was relatively easy to figure out what it was using to determine the iPod model.

In the Resources folder of any iPod Updater, you'll find a lot of interesting things. The most interesting bits are UpdaterVersions.plist and the Updates folder.

UpdaterVersions.plist is a typical XML property list with a single root key: Versions. Under this key you'll find a dictionary that maps updaterFamily to a dictionary of information about the iPods in that updaterFamily, not unlike the information you will find in a SysInfo. The dicts with a displayInAbout key with a true value are the most interesting, all of the other entries are simply variations on the theme (i.e. the iPod U2 Special Edition or the various colors of iPod Mini). When using this filter, you should have exactly one entry per unique iPodFamily value.

The Updates folder is interesting because it contains icons for each iPod, as well as the firmware images. The icons are named either DeviceIcon-$(iPodFamily).icns or DeviceIcon-$(iPodFamily)-$(iPodColor).icns. I'm relatively certain that the color can be determined from the last three characters of the pszSerialNumber, but I don't have enough iPods around to test that theory.

With all of this information, it would almost be easy to determine the model of an iPod. However, it's not quite that simple. Not all SysInfo files contain an iPodFamily key, and some that do incorrectly report 0! The only reliable identifier for an iPod family is the following (given a dict from either the UpdaterVersions.plist or from a SysInfo file):

def buildPair(dct):
    buildID = dct.get('buildID', 0)
    visibleBuildID = dct.get('visibleBuildID') or dct.get('VisibleBuildID', 0)
    # grab these bits: 0xFF000000L
    return (buildID >> 24, visibleBuildID >> 24)

What I'm doing here is pulling the "major" version out of the version keys. It appears that the versioning convention is: 0x0ABC8000 where the firmware version is pretty-printed as "A.B.C", trimming "C" and possibly "B" if they are 0. This is similar to the hex-version-convention you see in gestaltSystemVersion ('sysv') and other places. It would be a little less ugly if the UpdaterVersions.plist used the same case as the SysInfo files for the visibileBuildID key! The reason that we need both the buildID and the visibleBuildID is that the iPod Mini and "3G iPod" both report a buildID major version of 2, however the iPod Mini has a visibleBuildID major version of 1.

Parsing the SysInfo file in a manner that will grab the information in the right way is pretty trivial, but perhaps not so obvious:

import os
def infoForMountedPod(path):
    path = os.path.join(path, u'iPod_Control', u'Device', u'SysInfo')
    lines = file(path).read().splitlines()
    d = {}
    for line in lines:
        try:
            key, value = line.split(': ', 1)
        except ValueError:
            continue
        try:
            value = long(value.split(None, 1)[0], 0)
        except (ValueError, IndexError):
            pass
        d[key] = value
    return d

infoForMountedPod simply looks for all lines that look like a key-value pair (containing a ": "). For each of these lines, it will set the key-value pair in the returned dictionary. For lines whose first "word" (split on whitespace) is parseable as an integer (typically as hex), it will be set as an integer. Otherwise, it is set to the string value of that line (i.e. pszSerialNumber).

Parsing UpdaterVersions.plist is pretty trivial too:

import plistlib
def parseUpdaterVersions(path):
    # this is Python 2.4 plistlib API
    # in 2.3 you can use plistlib.Plist.fromFile
    versions = plistlib.readPlist(path)
    buildPairs = {}
    families = {}
    for k,v in versions['Versions'].iteritems():
        # only pick out unique iPodFamily dicts
        if v['displayInAbout']:
            fam = v['iPodFamily']
            # skip pre-2.x iPods and iPod Shuffle.
            # This is specific to my use case, you may
            # not want this filter.
            if 1 < fam < 128:
                assert fam not in families
                families[fam] = v
                pair = buildPair(v)
                assert pair not in buildPairs
                buildPairs[pair] = v
    return buildPairs, families

Note that all of this code is cross-platform, but you'll need an UpdaterVersions.plist or equivalent from somewhere, and you'll need to pick up plistlib.py from Python's src/Lib/plat-mac dir if using it on some other platform. Alternatively, you could use ElementTree to parse the plist XML. Its iterparse documentation has an excellent example of how easily it can be used to parse these files.

A little known feature of PEP 234: Iterators (implemented in Python 2.1 and later) is the alternate form:

iter(callable, sentinel)

This form creates an iterator out of any callable, and will stop when an exception is raised or the sentinel value is returned by the callable. Additionally, raising StopIteration will stop the iteration as with a normal iterator. A sentinel value is a special value that indicates the end of a sequence. This can be any value, but the most commonly useful values are likely to be None, 0, or ''.

I haven't seen much code in the wild that takes advantage of this form of iter, but an excellent usage would be to replace this common idiom:

while True:
    data = fileobj.read(BLOCKSIZE)
    if not data:
        break
    # do something with data here

Using the sentinel form of iter, it can be rewritten as an iterator with more obvious control flow:

for data in iter(lambda: fileobj.read(BLOCKSIZE), ''):
    # do something with data here

Of course, nobody likes a lambda, but you could wrap this up as a little utility function, or you could wait for PEP 309: Partial Function Application (on the standards track, will be in Python 2.5). With partial it would look like this:

for data in iter(partial(fileobj.read, BLOCKSIZE), ''):
    # do something with data here

Versions of Mac OS X prior to 10.4 did support universal binaries, but the shipping gcc did not, so it is possible to compile applications that will work on both 10.3 ppc (Panther) and 10.4 i386 (Tiger). To build Panther-compatible universal binaries, you'll need to use the following Custom Build Settings in Xcode 2.1 or later...

Per-architecture SDK Support for Universal Binaries in Xcode. This is according to documentation, but I couldn't get it to work:

SDKROOT_ppc = /Developer/SDKs/MacOSX10.3.9.sdk
SDKROOT_ppc64 = /Developer/SDKs/MacOSX10.4u.sdk
SDKROOT_i386 = /Developer/SDKs/MacOSX10.4u.sdk

I'm not sure how to translate per-architecture SDKs to gcc settings, but to build single-SDK Universal Binaries with GCC, there are some undocumented (not in the compiler man page, anyhow) flags that you can use. These flags are only available in GCC 4.0 and later, and probably only with Apple's toolchain.

Universal Binaries with GCC (in Makefile syntax):

SDK=/Developer/SDKs/MacOSX10.4u.sdk
CFLAGS= -isysroot ${SDK} -arch ppc -arch ppc64 -arch i386
LDFLAGS= -isysroot ${SDK} -syslibroot,${SDK}

Using lipo(1) to do per-architecture SDK builds with gcc should be possible, but painful. Hopefully it's not the only way! However, that's exactly what Xcode 2.1 does.

See Also:

• Universal Binary Programming Guidelines
• Xcode 2.1 Release Notes
• Cross-Development: Configuring Your Project for Cross Development
• lipo - create or operate on fat files

PyObjC

Ronald has customized the libffi export even further to work for Mach-O x86, mostly by removing the usage of autoconf. He has also managed to hack in some universal binaries support, but it really needs to move to distutils. All of the unit tests pass. This code should end up in ctypes too, probably.

The Xcode templates should work with Xcode 2.1 now, but they won't build universal binaries until py2app does.

objc.inject still doesn't work (since mach_inject isn't fixed yet).

Python

This is built as universal binaries by Apple. I haven't seen the patches yet, but whatever they're doing doesn't carry over to distutils, and it doesn't also include ppc64 support (though it certainly could, if Python didn't depend so much on autoconf crap). One way to fix this would be to create a custom pyconfig.h with reasonable compiler-determined values, and not using autoconf at all (an Xcode project might be a good idea). This has lots of problems of its own, because there are a lot of options that can be useful to configure, distutils reads in several autoconf-generated files, etc.

It's also going to take some time to figure out the gcc incantations to build against SDKs, as they're not documented. Especially if we're going to try and build against two different SDKs (i.e. 10.3 for PPC and 10.4.1 for x86).

If we do end up putting universal binaries support into distutils, it may require users to have the SDK installed in order to use distutils. Unfortunately this isn't the default when installing Xcode 2.1.

Four character codes, and probably other data structures, are still the wrong endian on x86.

The majority of bundlebuilder applications will not run on Intel machines without modification, even with Rosetta, because their executable is a script so Python is going to start up as an x86 binary. py2app-built applications don't have this problem.

It might make sense to add some intelligence to the extension importer to check for correct architecture before it tries to load the .so, or to have separate site-packages directories for each architecture, plus one for universal binaries. For example, it would make sense to have psyco available for x86, but maybe not anywhere else.

py2app

macholib needs to learn how to rewrite load commands of universal binaries.

If the py2app bootstraps are recompiled as universal binaries, then it should probably be able to detect whether the application needs to start up as ppc, because I don't want it to build incompatible apps like bundlebuilder does.

The web presence for the Python on Mac OS X session at WWDC2005 is now up: http://pythonmac.org/wwdc2005/

The CodeImporter example and slides will be available as soon as I've made sure that it's OK to post them.

Python on Mac OS X for Intel is not going to be a seamless transition. Unlike Mathematica, there is going to be a lot more than 2 hours of effort involved. Why?

• Python uses autoconf. Autoconf is not Xcode. It does not have a checkbox for universal binary compatibility. Autoconf is a PITA.
• PyObjC depends on libffi. libffi doesn't know what the Mach-O calling convention for x86 is, so it doesn't work. libffi is very deep, magical, scary code. This has been mostly solved by Ronald over the past few days in the universal binaries lab.
• mach_inject (which PyObjC's objc.inject uses) depends on injecting code into other processes. On an OS that can be running code for two different architectures, how the heck does that work? How do you know which pid is running which architecture? Anyway, injecting from x86-x86 isn't going to work because mach_inject doesn't know x86 yet.
• py2app's macholib doesn't really support fat binaries yet. It understands them well enough to do the right thing with the PPC portion of the Mach-O header, but it ignores other architectures. It will need a semi-major refactoring in order to support this cleanly.
• Bgen stuff breaks. Specifically, all of your Carbon code that deals with four character codes is going to be broken due to endianness issues. Yet another reason to stay the heck away from this stuff.

So, thanks Apple, for giving us weeks worth of very hard and unintesting work to do. The least you could've done is put out a nice new laptop that I could buy to do this work on :)

I'm also not terribly interested in renting an Intel development machine from Apple so that I can do work that helps them more than me. If they sold it, or stated that we would get a grand worth of credit towards a real Intel machine when they're available, then I wouldn't complain. But no, we get to rent a machine from them, that we can't really talk about, publish benchmarks of, move to a location other than the shipped address (?!), etc. They also say that they're under no obligation to fix it if it breaks, and there are no refunds. Sweet deal!

JavaScript Sucks:

1. You can extend types by assigning stuff to their prototype, but that can easily break code since these new things can show up when iterating over the properties of an instance. There is no way to hide these new properties. Only magical built-in prototypes can define hidden properties.
2. Arrays are braindead because you can't trust the built-in ways to compare them. Strangely enough it seems that == is broken in most implementations, but the other comparison operators seem to do the right thing. However, that is observed behavior, and I don't trust it to do the same thing everywhere.
3. The only built-in mapping is the object, which means all keys must be strings.
4. Everything is an object, except for the three or forty things that aren't. Ugh.
5. Referencing names of things that don't exist is not an error, but returns undefined. Referencing a property of undefined is an error. It's a great way to get an exception very, very, far away from the code that was broken.
6. Some implementations don't like extraneous commas, so you can't sanely write multi-line literal objects or arrays. When you have them, you'll get strange errors and it will take you a long time to find and fix them.
7. There's no way to introspect the stack beyond the function that's currently running, and the function that called it. Your test failed, somewhere. Debugging JavaScript sucks.
8. Exceptions raised by "C code" often carry no information about the last JavaScript line that was running (if you were especially fortunate, it would simply crash the browser, in IE's case anyway). Finding where an error happened basically requires a binary search of the new code that you introduced with debugging statements until you find it.
9. Unit tests are good, the JavaScript unit test frameworks available aren't. The fact that exceptions are so hard to track down makes unit tests basically necessary in order to remain sane, why the hell isn't there something as good as py.test? There's a port of Perl's TestSimple, which is usable, but there's plenty of things not to like about it right now (which is to be expected, at 0.03).
10. There's no operator overloading.
11. There's no sprintf (to format numbers, mostly).
12. Generally, there are a lot of objects that behave like arrays, but aren't, so you either have to convert them to arrays or not expect to be able to use array methods everywhere. Fortunately, there isn't a quick and easy way to convert an array-like object to a real array. Awesome.
13. No varargs syntax, but you can call functions with variable arguments and painfully extract additional arguments from the arguments local variable. So, it's kinda like Perl subs, except since arguments isn't a real array, you can't even shift off the arguments or otherwise use it as a regular array. Ugh.
14. foo.bar() and var bar = foo.bar; bar(); do entirely different things. You can write a function that does return a "bound function", but then you have this: var bar = bind(foo.bar, foo);. Dumb!

continued...

For those of you attending WWDC 2005 next week, make sure to check out the Python related sessions:

602 A.M.P. on Mac OS X:

Tue, 9:00-10:30, North Beach

613 Python on Mac OS X:

Wed, 9:00-10:30, North Beach

Additionally, Guido is going to be speaking about Python at the Wednesday Brown Bag Lunches With O'Reilly (Wed, 12:45pm).

Oh, and if you're looking to complain about the lack of Python support (or public APIs so that third parties can do the hard work) in Xcode, here's where to express that opinion:

410 Developer Tools Feedback Forum:

Fri, 5:00-6:00, Russian Hill

There isn't a Python/PyObjC BOF arranged yet. If you're interested in making that happen, find me after session 613 on Wednesday morning. I don't really know the area, but bbum does (more than me, anyway), and he should be around at about the same time too.