from __future__ import *

Python 2.6 was released yesterday, which has tons of cool new features including a new json library and a new multiprocessing library. The json library is basically simplejson (from a few months ago) minus the Python 2.4/2.5 support and refactored to take advantage of the latest future-compatible features (such as the new str.format method instead of using % format interpolation).

The only downside is that I wasn't able to get the latest simplejson 2.0.1 performance enhancements into the first release of Python 2.6, but expect them for Python 2.6.1! Don't worry though, simplejson should install just fine with Python 2.6 if you need the speed. If not, you don't have any dependencies for JSON anymore, just change your imports:

# Use simplejson or Python 2.6 json, prefer simplejson.
try:
    import simplejson as json
except ImportError:
    import json

print repr(json.dumps({'key': 'serialize this!'}))
print repr(json.loads('{"key": "deserialize this!"}'))

One of my favorite features in json/simplejson is the shell command that will validate/pretty print JSON. Great for debugging, and it will be sweet to have it available on every box with Python 2.6+!

A really cool tip if you're using Mac OS X is that you can access the string version of the pasteboard from the shell with pbpaste and pbcopy. This will take the JSON from your pasteboard and replace it with a pretty-printed version! If you're using simplejson just change json.tool to simplejson.tool. I'm sure this would be super handy in a text editor macro too:

$ pbpaste | python -mjson.tool | pbcopy

Slides from my Exploring Erlang talk today at C4[1] are available here: http://undefined.org/c4-1/

Yes, it is definitely caturday.

For those of you interested in learning more about Erlang, I highly recommend buying Joe Armstrong's Programming Erlang and/or read his 2003 ph.d thesis, which is kinda similar in content to his book but is more technical and is freely available: Making Reliable Distributed Systems in the Presence of Software Errors. The thesis is a bit more dense, but it was what really made Erlang "click" for me.

If you want to play with the examples, go to CEAN and download "Erlang/OTP Full (Standard)" at the bottom for you version of Mac OS X and follow the "How do I install CEAN to /usr/local/bin" FAQ entry. Alternatively, if you want to wait longer, you could use MacPorts to download and compile it. Once you've got that set up just run erl from the examples directory in the slides download and it should all work.

Since the svnsync tool in Subversion 1.4 still lacks documentation, here's a quick example to get you started with it:

$ svnadmin create svn.mochikit.com
$ echo '#!/bin/sh' > svn.mochikit.com/hooks/pre-revprop-change
$ chmod +x svn.mochikit.com/hooks/pre-revprop-change
$ svnsync init file://$PWD/svn.mochikit.com http://svn.mochikit.com/
$ svnsync sync file://$PWD/svn.mochikit.com

The example performs the following tasks:

• Creates a new local repository named svn.mochikit.com
• Adds a trivial pre-revprop-change hook to the repository (required for svnsync)
• Initializes the svn.mochikit.com repository to be a mirror of http://svn.mochikit.com/
• Synchronizes the svn.mochikit.com repository with its initialized source

Note that this is point-in-time synchronization. To keep an up to date mirror you'll have to run svnsync sync on a regular basis (e.g. by cron job or post-commit hook).

If you're looking for an especially easy to install set of svn binaries for Mac OS X, I recommend Martin Ott's (of SubEthaEdit fame) build. I've been using it on all of my Macs for quite some time with no issues. Much less hassle than building it myself or using MacPorts (previously known as DarwinPorts).

After over three years with a 1ghz 15" titanium PowerBook I've finally upgraded to a 2ghz dual core 15" MacBook Pro! Compared to my old laptop, this machine is absolutely amazing. I've been waiting a LONG time for:

• Built-in Bluetooth, 802.11g, USB 2.0
• Higher resolution, contrast, brightness display
• Trackpad with the two-finger scrolling
• Much much improved video subsystem
• A machine fast enough to effectively use spotlight

Additionally, this new kit is going to save me a lot of VNC sessions since I can run other operating systems in virtualization via Parallels, and it will allow me to more easily build and properly test universal binaries. Parallels seems to work pretty well for a beta product, but I'm definitely looking forward to seeing some competition (especially open source competition).

Despite its awesomeness, there are still some issues with the MacBook Pro:

• The power supply whine is still there when the system isn't using a lot of juice, but apparently much less so than in earlier revisions.
• These machines run HOT. I probably wouldn't try and put it on my lap if I wasn't wearing jeans, and it does make your hands a little uncomfortably warm when using the built-in keyboard (though I'm sure I'll get used to that).
• The display doesn't tilt back quite as far as the old titanium powerbooks. Not a big deal, hopefully that just means they've solved the problem where the hinges would just snap after a year or two (and take out the backlight cable, argh).
• Intel code on Mac OS X supposedly uses more RAM than the PPC equivalent, which means you should buy more RAM than you are accustomed to from earlier machines.

The RAM usage increase is somewhat surprising, because i386 binaries are more compact (the i386 libSystem is about 16% smaller than PPC). I'm guessing the increase is due to the lack of usable registers on i386, which increases the amount of stack necessary per thread. I doubt it's an alignment issue, because PPC definitely cares more about alignment than i386 does. A quick sample shows that Terminal.app with one window does use about 5% more real memory than the PPC equivalent (and a LOT more virtual memory, but that might just be Quartz or something).

5% doesn't sound like much, but I've heard bigger numbers for other kinds of applications. I definitely plan to upgrade this machine from 1gb to 2gb, but the only times I've really felt RAM-hungry so far are when I'm using Parallels (which understandably needs a lot of real memory).

It's been a pretty long haul getting Python to do the right thing on Mac OS X with regard to Universal Binaries. The dust is settling, and it's about time to start publicizing the results.

In order to compile extensions with Universal MacPython, you absolutely must have the latest version of Xcode installed. When you install it, make absolutely sure that you Customize the installation and install the Mac OS X 10.4 Universal SDK. Universal MacPython requires this SDK to be installed, as it instructs the compiler and linker to use this specific SDK. There is a hook that will remove these special flags, but ONLY for legacy Mac OS X 10.3.9 users. Mac OS X 10.3.9 users should still be able to compile extensions, but the resultant binaries will be PPC only and shouldn't be distributed (as they will not work on Intel).

Besides the obvious endian fixes and distutils/configure hackery, Universal MacPython includes many usability enhancements over previous distributions of Python for Mac OS X:

• IDLE, the Python IDE, should now ship with Mac keybindings and menus by default
• Placing binaries in /usr/local/bin is now optional
• The bin directory from the framework is now (by default) automatically added first to your PATH environment variable. This means that when you open a new terminal session and type python, you will get Universal Python 2.4.3. Yes, it's ugly to do this, but it's a necessary evil. PATH related issues are by far the most frequent source of confusion on pythonmac-sig (between Python itself and distutils-installed scripts), and this is the solution. darwinports does something quite similar.
• pythonw is now an executable, instead of a shell script, so it may be used directly in a hash-bang line for a shell script (#!).
• Users no longer need to know or care about pythonw. python and pythonw are the same binary, which simply does an execve to a Python executable inside of a "fake" application bundle to placate the WindowServer. Yes, this introduces a tiny little bit of overhead when invoking python due to the extra syscall, but it is again a necessary evil to work around a design flaw in Mac OS X.
• Includes a working bsddb, readline, and curses.
• Known compatible with Mac OS X 10.3.9 and later, without any patching necessary on Mac OS X 10.4.

The latest Universal Python source code is currently a fork of Python 2.4.3, and currently resides outside of the official Python subversion repository. For those interested in the source, it is here:

• http://svn.pythonmac.org/python24/python24-fat/

The latest binary release of Universal Python is currently available in the Universal Python 2.4 section of pythonmac.org packages! There is/was an older Universal Python build available somewhere on python.org but it is NOT what you should be using (it's built without compiler optimization and is roughly 50% slower on either architecture!):

• http://pythonmac.org/packages/py24-fat

PyObjC and py2app users with Universal MacPython should install PyObjC from SVN trunk (which will net you a universal-ready py2app). There is currently no tagged release of either project that is suitable for this build of Python, but we hope to fix that soon:

• http://svn.red-bean.com/pyobjc/trunk/pyobjc/
• http://svn.red-bean.com/bob/py2app/branches/py2app-0.2-maint/

pythonmac.org packages has been split into two sections, one for universal packages, and one for legacy builds of Python. Currently there aren't many universal packages ready (largely because I don't have an Intel machine yet, so I can only test on PPC). The legacy packages for Python 2.4 are still compatible with Universal MacPython, however, they are NOT compatible with Intel.

Most packages should compile just fine, and setuptools/easy_install are already Universal MacPython aware. However, packages that depend on external libraries will not compile correctly of the dependent libraries aren't also Universal. This means that MySQL bindings, wxPython, and a few other popular packages are not yet readily available. This issue will resolve itself at some point due to the demand, but it's not resolved yet. There's a list of desired packages and their status at the UniversalPackages node on the pythonmac.org wiki.

xattr is a Python wrapper for the extended attributes APIs available in Darwin 8.0+ (Mac OS X 10.4) and Linux 2.6+. 0.2 added Linux compatibility, and API compatibility with the GPL pyxattr module. Unlike pyxattr, xattr supports both Darwin and Linux, has a dict-like API, and it is MIT licensed.

Usage looks like this:

>>> import xattr
>>> x = xattr.xattr('tiger_8a428_userdvd.dmg')
>>> x
<xattr file='tiger_8a428_userdvd.dmg'>
>>> x.keys()
[u'com.apple.diskimages.recentcksum']
>>> dict(x)
{u'com.apple.diskimages.recentcksum': 'i:969254 ...more stuff'}
>>> x['com.apple.diskimages.recentcksum']
'i:969254 ...more stuff'
>>> x['com.apple.adc'] = 'select'
>>> dict(x)
{u'com.apple.adc': 'select',
 u'com.apple.diskimages.recentcksum': 'i:969254 ...more stuff'}
>>> del x['com.apple.adc']
>>> dict(x)
{u'com.apple.diskimages.recentcksum': 'i:969254 ...more stuff'}

You can download xattr 0.2 from http://pythonmac.org/packages/ for Tiger's Python 2.3.5 or Python 2.4.1 (installed on Tiger).

svn trunk:

http://svn.red-bean.com/bob/xattr/trunk/

0.2 source:

http://svn.red-bean.com/bob/xattr/releases/xattr-0.2/

A few weeks ago I began my move from New York to San Francisco by... moving to Hawaii. For the summer, anyhow. In shipping, my trusty old Klipsch computer speakers seem to have eaten it. I tried hooking them up, and after some really painful shrieking, they worked for a few minutes.. until they caught fire. Really. My ex-favorite music spewing devices have found themselves a nice retirement home in a landfill somewhere on Maui.

I tried playing iPod DJ with the stereo here, but that doesn't really work very well for long stretches. It was time to break down and finally get an Airport Express. I looked around online a bit, and shipping was just ridiculous to Hawaii from most places (e.g. Smalldog wanted $95 S+H to ship a $120 refurb Airport Express to Kihei!). Fortunately, there is a Mac store in town, so I was able to pick one up at a reasonable price.

Immediately after bringing it home I plugged it in via ethernet, and tried to update the firmware. A few hundred times. It just wouldn't take. The Airport Admin utility would not do it. From either of my Macs. It'd try, and it would fail after a minute or two without even an error code. Eventually I found a link to download a standalone firmware updater application, which updated the firmware first try. Go figure. Certainly not the typical Apple experience.

Once it was updated and hooked up, it worked great. 90% of the time, anyway. iTunes will only play files that it natively understands through the Airport Express. Apparently, iTunes has decided that some of my MP3s were QuickTime files, and refused to play them through Airport Express, which means they were coming out of the built-in speakers. That didn't work out so well. I guess that sucks for people who care about OGG and other formats supported only via QuickTime.

It turns out that the reason these files were picked up as QuickTime is that iTunes has MPEG type code hate. MacAMP and SoundJam of yesteryear used MPEG, yet iTunes wants to see MPG3. It took me a lunch break to come up with a good way to track down all of these files without writing a lot of code. It was actually rather simple: the iTunes Music Library XML dump.

~/Music/iTunes/iTunes Music Library.xml is a brain-dump of iTunes that it creates every so often (on quit, I believe). It keeps track of almost everything that iTunes knows, all of your file's URLs, their type and creator codes, comments, metadata, you name it. It's also in plist format, which is extremely easy to work with from Python.

Assuming you have PyObjC installed, you can break into a Python interpreter and screw around with this rather easily:

% python
Python 2.4.1 (#2, Mar 31 2005, 00:05:10)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from Foundation import *
>>> import os
>>> dbFile = os.path.expanduser("~/Music/iTunes/iTunes Music Library.xml")
>>> db = NSDictionary.dictionaryWithContentsOfFile_(dbFile)
>>> type(db)
<objective-c class NSCFDictionary at 0x30a4b0>

With the database in hand, we can start poking around:

>>> track = db[u'Tracks'].itervalues().next()
>>> print track.description()
{
    Album = "Live Code";
    Artist = "Front 242";
    "Artwork Count" = 1;
    "Bit Rate" = 160;
    ...

In these track dictionaries, there are two keys relevant to this exercise: u'File Type', and u'Location'. File type is the integer representation of the type code of the track. Location is the URL of the track. With this, it's pretty easy to pick out the locations of the tracks that are set to the MPEG type code:

>>> import struct
>>> badPaths = [
...     NSURL.URLWithString_(track[u'Location']).path()
...     for track in db[u'Tracks'].itervalues()
...     if track.get(u'File Type') == struct.unpack('>I', 'MPEG')[0]
... ]
>>> badPaths[0]
/Volumes/...

With this, all we need to do is change the type and creator. Fortunately, that's easy. We'll use the iTunes creator code hook, and the expected MPG3 type code:

>>> import MacOS
>>> for path in badPaths:
...     MacOS.SetCreatorAndType(path, 'hook', 'MPG3')
...

And with that, my Airport Express does just about everything I'd have expected it to out of the box ;)

I've been neglecting Python a bit lately, but Ronald is going to cut a release of PyObjC 1.3.7 very soon, hopefully later this week. PyObjC 1.3.7 fixes a few bugs (particularly with Xcode 2.1 templates), has a more complete wrapping of Mac OS X 10.4 system frameworks (DiscRecording, SenTestingKit, SecurityFoundation), and adds a new TinyURLService example (which converts URLs on the pasteboard to tinyurl.com equivalents).

There's also a swath of new "First Steps" documentation in the PyObjC intro, which I wrote up earlier today:

[someObject doSomething:arg1 withSomethingElse:arg2];

doSomething:withSomethingElse:

doSomething_withSomethingElse_

someObject.doSomething_withSomethingElse_(arg1, arg2)

# note the trailing underscore
someObject.setValue_(aValue)

myObject = NSObject.alloc().init()

myData = NSData.alloc().initWithBytes_length_('the bytes', 9)

class MyClass(NSObject):

    def init(self):
        """
        Designated initializer for MyClass
        """
        # ALWAYS call the super's designated initializer.
        # Also, make sure to re-bind "self" just in case it
        # returns something else!
        self = super(MyClass, self).init()

        self.myVariable = 10

        # Unlike Python's __init__, initializers MUST return self,
        # because they are allowed to return any object!
        return self

class MyOtherClass(MyClass):

    def initWithOtherVariable_(self, otherVariable):
        """
        Designated initializer for MyOtherClass
        """
        self = super(MyOtherClass, self).init()
        self.otherVariable = otherVariable
        return self

myInstance = MyClass.alloc().init()
myOtherInstance = MyOtherClass.alloc().initWithOtherVariable_(20)

# This is equivalent to:
#
#   myObject = NSObject.alloc().init()
#
myObject = NSObject.new()

# This is equivalent to:
#
#   myDict = NSDictionary.alloc().init()
#
myDict = NSDictionary.dictionary()

# This is equivalent to:
#
#   myString = NSString.alloc().initWithString_(u'my string')
#
myString = NSString.stringWithString_(u'my string')

# Getting
#
# notice the method call
#
myValue = valueContainer.value()

# Setting
#
# notice the naming convention and trailing underscore
#
valueContainer.setValue_(myNewValue)

class MyValueHolder(NSObject):

    def initWithValue_(self, value):
        self = super(MyValueHolder, self).init()
        # It's recommended not to use typical Python convention here,
        # as instance variables prefixed with underscores are reserved
        # by the Objective-C runtime.  It still works if you use
        # underscores, however.
        self.ivar_value = value
        return self

    def value(self):
        return self.ivar_value

    def setValue_(self, value):
        self.ivar_value = value

For various reasons, I have a need to determine the version of an iPod. Previously, I only needed to know if an iPod was "firmware version 2" or later (aka "Dock-connector"), and there's a whole slew of obvious ways to determine that (presence of a Notes folder would be the most obvious). There are also ways to grab iPod information with SPI (via COM on win32 or the private iPod.framework on Mac OS X), but I like to avoid that whenever possible.

So far, the most reliable method I've found was with a little parsing of the SysInfo file. This file is located at "$(IPOD_VOLUME)/iPod_Control/Device/SysInfo". The iPod_Control folder is going to be hidden on either platform, so you may not have noticed it before. Many examples of SysInfo files can be found by googling for some of the keys, such as buildID, visibleBuildID, boardHwName, etc. The most complete resource I've found is the on the iPodLinux forums.

As-is, the SysInfo files don't make any of the useful information very obvious. However, with a little reverse-engineering of the iPod Updater for Mac OS X, it was relatively easy to figure out what it was using to determine the iPod model.

In the Resources folder of any iPod Updater, you'll find a lot of interesting things. The most interesting bits are UpdaterVersions.plist and the Updates folder.

UpdaterVersions.plist is a typical XML property list with a single root key: Versions. Under this key you'll find a dictionary that maps updaterFamily to a dictionary of information about the iPods in that updaterFamily, not unlike the information you will find in a SysInfo. The dicts with a displayInAbout key with a true value are the most interesting, all of the other entries are simply variations on the theme (i.e. the iPod U2 Special Edition or the various colors of iPod Mini). When using this filter, you should have exactly one entry per unique iPodFamily value.

The Updates folder is interesting because it contains icons for each iPod, as well as the firmware images. The icons are named either DeviceIcon-$(iPodFamily).icns or DeviceIcon-$(iPodFamily)-$(iPodColor).icns. I'm relatively certain that the color can be determined from the last three characters of the pszSerialNumber, but I don't have enough iPods around to test that theory.

With all of this information, it would almost be easy to determine the model of an iPod. However, it's not quite that simple. Not all SysInfo files contain an iPodFamily key, and some that do incorrectly report 0! The only reliable identifier for an iPod family is the following (given a dict from either the UpdaterVersions.plist or from a SysInfo file):

def buildPair(dct):
    buildID = dct.get('buildID', 0)
    visibleBuildID = dct.get('visibleBuildID') or dct.get('VisibleBuildID', 0)
    # grab these bits: 0xFF000000L
    return (buildID >> 24, visibleBuildID >> 24)

What I'm doing here is pulling the "major" version out of the version keys. It appears that the versioning convention is: 0x0ABC8000 where the firmware version is pretty-printed as "A.B.C", trimming "C" and possibly "B" if they are 0. This is similar to the hex-version-convention you see in gestaltSystemVersion ('sysv') and other places. It would be a little less ugly if the UpdaterVersions.plist used the same case as the SysInfo files for the visibileBuildID key! The reason that we need both the buildID and the visibleBuildID is that the iPod Mini and "3G iPod" both report a buildID major version of 2, however the iPod Mini has a visibleBuildID major version of 1.

Parsing the SysInfo file in a manner that will grab the information in the right way is pretty trivial, but perhaps not so obvious:

import os
def infoForMountedPod(path):
    path = os.path.join(path, u'iPod_Control', u'Device', u'SysInfo')
    lines = file(path).read().splitlines()
    d = {}
    for line in lines:
        try:
            key, value = line.split(': ', 1)
        except ValueError:
            continue
        try:
            value = long(value.split(None, 1)[0], 0)
        except (ValueError, IndexError):
            pass
        d[key] = value
    return d

infoForMountedPod simply looks for all lines that look like a key-value pair (containing a ": "). For each of these lines, it will set the key-value pair in the returned dictionary. For lines whose first "word" (split on whitespace) is parseable as an integer (typically as hex), it will be set as an integer. Otherwise, it is set to the string value of that line (i.e. pszSerialNumber).

Parsing UpdaterVersions.plist is pretty trivial too:

import plistlib
def parseUpdaterVersions(path):
    # this is Python 2.4 plistlib API
    # in 2.3 you can use plistlib.Plist.fromFile
    versions = plistlib.readPlist(path)
    buildPairs = {}
    families = {}
    for k,v in versions['Versions'].iteritems():
        # only pick out unique iPodFamily dicts
        if v['displayInAbout']:
            fam = v['iPodFamily']
            # skip pre-2.x iPods and iPod Shuffle.
            # This is specific to my use case, you may
            # not want this filter.
            if 1 < fam < 128:
                assert fam not in families
                families[fam] = v
                pair = buildPair(v)
                assert pair not in buildPairs
                buildPairs[pair] = v
    return buildPairs, families

Note that all of this code is cross-platform, but you'll need an UpdaterVersions.plist or equivalent from somewhere, and you'll need to pick up plistlib.py from Python's src/Lib/plat-mac dir if using it on some other platform. Alternatively, you could use ElementTree to parse the plist XML. Its iterparse documentation has an excellent example of how easily it can be used to parse these files.

Versions of Mac OS X prior to 10.4 did support universal binaries, but the shipping gcc did not, so it is possible to compile applications that will work on both 10.3 ppc (Panther) and 10.4 i386 (Tiger). To build Panther-compatible universal binaries, you'll need to use the following Custom Build Settings in Xcode 2.1 or later...

Per-architecture SDK Support for Universal Binaries in Xcode. This is according to documentation, but I couldn't get it to work:

SDKROOT_ppc = /Developer/SDKs/MacOSX10.3.9.sdk
SDKROOT_ppc64 = /Developer/SDKs/MacOSX10.4u.sdk
SDKROOT_i386 = /Developer/SDKs/MacOSX10.4u.sdk

I'm not sure how to translate per-architecture SDKs to gcc settings, but to build single-SDK Universal Binaries with GCC, there are some undocumented (not in the compiler man page, anyhow) flags that you can use. These flags are only available in GCC 4.0 and later, and probably only with Apple's toolchain.

Universal Binaries with GCC (in Makefile syntax):

SDK=/Developer/SDKs/MacOSX10.4u.sdk
CFLAGS= -isysroot ${SDK} -arch ppc -arch ppc64 -arch i386
LDFLAGS= -isysroot ${SDK} -syslibroot,${SDK}

Using lipo(1) to do per-architecture SDK builds with gcc should be possible, but painful. Hopefully it's not the only way! However, that's exactly what Xcode 2.1 does.

See Also:

• Universal Binary Programming Guidelines
• Xcode 2.1 Release Notes
• Cross-Development: Configuring Your Project for Cross Development
• lipo - create or operate on fat files