from __future__ import *

Kailash and Friends Kailash Kher Kaipa

It's been a pretty long haul getting Python to do the right thing on Mac OS X with regard to Universal Binaries. The dust is settling, and it's about time to start publicizing the results.

In order to compile extensions with Universal MacPython, you absolutely must have the latest version of Xcode installed. When you install it, make absolutely sure that you Customize the installation and install the Mac OS X 10.4 Universal SDK. Universal MacPython requires this SDK to be installed, as it instructs the compiler and linker to use this specific SDK. There is a hook that will remove these special flags, but ONLY for legacy Mac OS X 10.3.9 users. Mac OS X 10.3.9 users should still be able to compile extensions, but the resultant binaries will be PPC only and shouldn't be distributed (as they will not work on Intel).

Besides the obvious endian fixes and distutils/configure hackery, Universal MacPython includes many usability enhancements over previous distributions of Python for Mac OS X:

• IDLE, the Python IDE, should now ship with Mac keybindings and menus by default
• Placing binaries in /usr/local/bin is now optional
• The bin directory from the framework is now (by default) automatically added first to your PATH environment variable. This means that when you open a new terminal session and type python, you will get Universal Python 2.4.3. Yes, it's ugly to do this, but it's a necessary evil. PATH related issues are by far the most frequent source of confusion on pythonmac-sig (between Python itself and distutils-installed scripts), and this is the solution. darwinports does something quite similar.
• pythonw is now an executable, instead of a shell script, so it may be used directly in a hash-bang line for a shell script (#!).
• Users no longer need to know or care about pythonw. python and pythonw are the same binary, which simply does an execve to a Python executable inside of a "fake" application bundle to placate the WindowServer. Yes, this introduces a tiny little bit of overhead when invoking python due to the extra syscall, but it is again a necessary evil to work around a design flaw in Mac OS X.
• Includes a working bsddb, readline, and curses.
• Known compatible with Mac OS X 10.3.9 and later, without any patching necessary on Mac OS X 10.4.

The latest Universal Python source code is currently a fork of Python 2.4.3, and currently resides outside of the official Python subversion repository. For those interested in the source, it is here:

• http://svn.pythonmac.org/python24/python24-fat/

The latest binary release of Universal Python is currently available in the Universal Python 2.4 section of pythonmac.org packages! There is/was an older Universal Python build available somewhere on python.org but it is NOT what you should be using (it's built without compiler optimization and is roughly 50% slower on either architecture!):

• http://pythonmac.org/packages/py24-fat

PyObjC and py2app users with Universal MacPython should install PyObjC from SVN trunk (which will net you a universal-ready py2app). There is currently no tagged release of either project that is suitable for this build of Python, but we hope to fix that soon:

• http://svn.red-bean.com/pyobjc/trunk/pyobjc/
• http://svn.red-bean.com/bob/py2app/branches/py2app-0.2-maint/

pythonmac.org packages has been split into two sections, one for universal packages, and one for legacy builds of Python. Currently there aren't many universal packages ready (largely because I don't have an Intel machine yet, so I can only test on PPC). The legacy packages for Python 2.4 are still compatible with Universal MacPython, however, they are NOT compatible with Intel.

Most packages should compile just fine, and setuptools/easy_install are already Universal MacPython aware. However, packages that depend on external libraries will not compile correctly of the dependent libraries aren't also Universal. This means that MySQL bindings, wxPython, and a few other popular packages are not yet readily available. This issue will resolve itself at some point due to the demand, but it's not resolved yet. There's a list of desired packages and their status at the UniversalPackages node on the pythonmac.org wiki.

Python on Mac OS X for Intel is not going to be a seamless transition. Unlike Mathematica, there is going to be a lot more than 2 hours of effort involved. Why?

• Python uses autoconf. Autoconf is not Xcode. It does not have a checkbox for universal binary compatibility. Autoconf is a PITA.
• PyObjC depends on libffi. libffi doesn't know what the Mach-O calling convention for x86 is, so it doesn't work. libffi is very deep, magical, scary code. This has been mostly solved by Ronald over the past few days in the universal binaries lab.
• mach_inject (which PyObjC's objc.inject uses) depends on injecting code into other processes. On an OS that can be running code for two different architectures, how the heck does that work? How do you know which pid is running which architecture? Anyway, injecting from x86-x86 isn't going to work because mach_inject doesn't know x86 yet.
• py2app's macholib doesn't really support fat binaries yet. It understands them well enough to do the right thing with the PPC portion of the Mach-O header, but it ignores other architectures. It will need a semi-major refactoring in order to support this cleanly.
• Bgen stuff breaks. Specifically, all of your Carbon code that deals with four character codes is going to be broken due to endianness issues. Yet another reason to stay the heck away from this stuff.

So, thanks Apple, for giving us weeks worth of very hard and unintesting work to do. The least you could've done is put out a nice new laptop that I could buy to do this work on :)

I'm also not terribly interested in renting an Intel development machine from Apple so that I can do work that helps them more than me. If they sold it, or stated that we would get a grand worth of credit towards a real Intel machine when they're available, then I wouldn't complain. But no, we get to rent a machine from them, that we can't really talk about, publish benchmarks of, move to a location other than the shipped address (?!), etc. They also say that they're under no obligation to fix it if it breaks, and there are no refunds. Sweet deal!

We've (finally) updated talkingpanda.com today with a fresh new look and a new product: iBar.

iBar turns your iPod (any iPod with a screen and a dock connector) into an "ultimate bartending tool" with over a thousand drink recipes, mixing techniques, and even a couple history lessons. Like any good bar should be, it's stocked. Altogether, the Notes content adds up to over 3 megabytes (yes, that's text!) with more than 40 minutes of original, professionally recorded, audio.

Like its sibling iLingo, iBar ships with installers for Mac OS X and Windows XP/2000 (everywhere iTunes is supported) that make installation painless.

And for relevance, here's a little bit about how it's all put together:

• The Mac OS X installer was developed with Python 2.3, PyObjC 1.2, and bundled up with py2app.
• The Win32 installer was developed with Python 2.4, win32all, Tkinter, PIL and is bundled up with py2exe and then made self-contained with NSIS.
• The build scripts for the installer application actually use even more open source stuff. We use JExcelAPI to convert Excel spreadsheets to XML, and Win32::Exe to swap out the ICO resource out of win32 executables -- but no Java or Perl makes it into the redistributable :)

We're also blogging updates and podcasting some of our content, so point your NetNewsWire or Safari RSS over to Talking Panda News!

py2app 0.2 is a minor bug fix release. You can download it from pythonmac.org packages. PyObjC 1.3.5 ships with py2app 0.2, so if you have that, you have already upgraded to py2app 0.2!

Functional changes:

• New datamodels option to support CoreData. Compiles .xcdatamodel files and places them in the Resources dir (as .mom).
• New use-pythonpath option. The py2app application bootstrap will no longer use entries from PYTHONPATH unless this option is used.
• py2app now persists information about the build environment (python version, executable, build style, etc.) in the Info.plist and will clean the executable before rebuilding if anything at all has changed.
• bdist_mpkg now builds packages with the full platform info, so that installing a package for one platform combination will not look like an upgrade to another platform combination.

Bug Fixes:

• Fixed a bug in standalone building, where a rebuild could cause an unlaunchable executable.
• Plugin bootstrap should compile/link correctly with gcc 4.
• Plugin bootstrap no longer sets PYTHONHOME and will restore PYTHONPATH after initialization.
• Plugin bootstrap swaps out thread state upon plug-in load if it is the first to initialize Python. This fixes threading issues.

• Importing objc now ensures that Foundation is multi-threaded, previously it only ensured that Python was.

• New objc.RegisterCFSignature used to register CFTypeRef-like signatures with the runtime.

• PyObjCTools.Conversion functions now support all property list types with the following conversions:

  • NSData <-> buffer
  • NSDecimalNumber <-> decimal.Decimal (if present)
  • NSDate <-> datetime.datetime

  New toPythonDecimal, fromPythonDecimal functions which convert between NSDecimalNumber and decimal.Decimal using an intermediate string.

  New serializePropertyList and deserializePropertyList functions which serialize (Objective-C) property lists to and from NSData.

• OC_PythonObject, the proxy for Python objects that do not have an Objective-C superclass and are not otherwise special-cased, now act slightly more like typical Objective-C objects (supporting -isEqual:, -hash, and -compare:). This allows them to work with Key-Value Coding if they are contained by an Objective-C object, among other things.

• New objc.signature decorator that allows easier specification of objc.selector wrappers for functions when using Python 2.4:

  @objc.signature('i@:if')
  def methodWithX_andY_(self, x, y):
      return 0
  

• PyObjCTools.KeyValueCoding.getKeyPath now supports all of the Array Operators supported by Mac OS X 10.4.

• Key-Value Coding of Python objects (whether or not using an Objective-C base class) should act like Objective-C now. In previous versions there were inconsistencies with the use of capitalization, the underscore postfix in setters, and Key-Value Observing.

• The formal protocol list is now complete. A new internal function, objc.protocolsForProcess() enumerates over all mach headers and returns all of the protocols defined in the expected place. This fixes the scenario where an application uses a protocol but does not define any classes that conform to that protocol (i.e. to check plugin conformity). Previously it was not possible to reach these protocols simply by walking over all of the classes.

• A special value, objc.NULL, may now be passed in the place of 'in' and 'inout' arguments. This tells the bridge to pass a NULL pointer to the Objective-C method, instead of a pointer to the value. The return value will still be a tuple of the expected size.

• Some of the new Tiger frameworks now have wrappers:

  • AppleScriptKit
  • Automator
  • CoreData
  • DiscRecording
  • DiscRecordingUI
  • OSAKit
  • Quartz
  • QTKit
  • SyncServices
  • XgridFoundation

  Documentation and tests not yet written.

• New OutlineEditor example in Examples/CoreData, it is a Python version of the identically named Apple example.

• The last argument of selectors that end with ':error:' is now assumed to be 'out' if its type is an object pointer.

• More conveniences for list-like and dict-like objects: __reversed__, reverse, pop, remove, fromkeys.

• OC_PythonDictionary and OC_PythonArray now return NSNull to Objective-C callers as appropriate.

• New WebKitInterpreter example in Examples/Plugins. Uses the new WebKit Cocoa plugin API available in Safari 1.3 and later to embed a PyInterpreter in the browser.

• Fixed a CFBundleRef reference counting bug in Foundation._Foundation. The symptom of this is usually a crashing application after having loaded a PyObjC-based plugin into an otherwise Objective-C app.

• New PyObjCTools.AppHelper functions: callAfter and callLater, conveniences for calling Python functions on the main thread as soon as possible, or after a delay.

• Twisted examples changed to use threadedselectreactor instead of cfreactor. cfreactor is deprecated. Needs Twisted newer than 2.0 (svn r13575 or later).

• objc.inject now injects on main thread by default, and takes an optional third useMainThread argument to change this behavior. This is a complete rewrite which should be correct, stable, Tiger compatible, and synchronized with mach_* 1.1.

• Removed an NSAutoreleasePool category hack that has been deprecated for quite some time.

• New objc.removeAutoreleasePool function that will remove PyObjC's global NSAutoreleasePool, which may be useful for plugins.

• Fixed bug in the NSBundle hack that caused a NULL pointer dereference if looking up a non-existent class using NSBundle API.

• Added OC_PythonUnicode and OC_PythonString classes that preserve the identity of str and unicode objects across the bridge. The bridge for str now uses the default encoding of NSString, rather than sys.getdefaultencoding() from Python. For Mac OS X, this is typically MacRoman. The reason for this is that not all Python str instances could cross the bridge at all previously. objc.setStrBridgeEnabled(False) will still trigger warnings, if you are attempting to track down an encoding bug. However, the symptoms of the bug will be incorrectly encoded text, not an exception.

• New Xcode project template "PyObjC Mixed Application" that is a py2app based Python application that loads an Objective-C plug-in built as a separate target.

• New py2app based Xcode templates "PyObjC Application" and "PyObjC Document-based Application", these replace the older "Cocoa-Python Application" and "Cocoa-Python Document-based Application" respectively.

• New InjectBrowser example in Examples/Inject that demonstrates injection of the ClassBrowser example into another application using objc.inject.

• NSData and NSMutableData instances now support the Python buffer protocol.

• NSData instances now support a convenience API that allow them to act like a buffer instance for str() and slicing.

• Objects that support the Python buffer protocol, such as buffer and array.array (but not str or unicode) are now bridged as NSData subclasses.

py2app is the bundlebuilder replacement we've all been waiting for. It is implemented as a distutils command, similar to py2exe, that builds Mac OS X applications from Python scripts, extensions, and related data files. It tries very hard to include all dependencies it can find so that your application can be distributed standalone, as Mac OS X applications should be.

py2app 0.1.9 is included in the installer for PyObjC 1.3. If you have installed PyObjC 1.3, then you already have py2app 0.1.9 installed.

Download and related links are here: http://undefined.org/python/#py2app

py2app 0.1.9 is a minor bug fix release:

Bugs fixed:

• bdist_mpkg now builds zip files that are correctly unzipped by all known tools.
• The behavior of the bootstrap has changed slightly such that __file__ should now point to your main script, rather than the bootstrap. The main script has also moved to Resources, from Resources/Python, so that __file__ relative resource paths should still work.

I can finally take a breather now that my talks are over! For those of you that missed them, or want to take a closer look, here are the slides:

• Intro to PyObjC
• PyObjC Hacking

If you poke around in the svn repository that those link to, you can find the ReST source for the slides, the source imagery, the software that throws them together with docutils and Reportlab, etc.

NOTE:

This is an announcement for a BETA release of PyObjC. Though we know it to be quite stable, and have been using it on a daily basis for quite some time, use it at your own risk. 1.3.0 will be out in a matter of days, but it is essential that we get some eyes on this!

Installer package available from:

http://pythonmac.org/packages/ (note that the installer also contains py2app 0.1.8)

Source:

http://svn.red-bean.com/pyobjc/tags/pyobjc-1.3b1/

py2app is the bundlebuilder replacement we've all been waiting for. It is implemented as a distutils command, similar to py2exe, that builds Mac OS X applications from Python scripts, extensions, and related data files. It tries very hard to include all dependencies it can find so that your application can be distributed standalone, as Mac OS X applications should be.

py2app 0.1.8 will be included in the installer for PyObjC 1.3. If you have installed PyObjC 1.3, then you already have py2app 0.1.8 installed.

Download and related links are here: http://undefined.org/python/#py2app

py2app 0.1.8 is a major enhancements release:

Bugs fixed:

• Symlinks in included frameworks should be preserved correctly (fixes Tcl/Tk)
• Fixes some minor issues with alias bundles
• Removed implicit SpiderImagePlugin -> ImageTk reference in PIL recipe
• The --optimize option should work now
• weakref is now included by default
• anydbm's dynamic dependencies are now in the standard implies list
• Errors on app launch are brought to the front so the user does not miss them
• bdist_mpkg now compatible with pychecker (data_files had issues)

Options changed:

• deprecated --strip, it is now on by default
• new --no-strip option to turn off stripping of executables

New features:

• Looks for a hacked version of the PyOpenGL __init__.py so that it doesn't have to include the whole package in order to get at the stupid version file.
• New loader_files key that a recipe can return in order to ensure that non-code ends up in the .zip (the pygame recipe uses this)
• Now scans all files in the bundle and normalizes Mach-O load commands, not just extensions. This helps out when using the --package option, when including frameworks that have plugins, etc.
• An embedded Python interpreter is now included in the executable bundle (sys.executable points to it), this currently only works for framework builds of Python
• New macho_standalone tool
• New macho_find tool
• Major enhancements to the way plugins are built
• bdist_mpkg now has a --zipdist option to build zip files from the built package
• The bdist_mpkg "Installed to:" description is now based on the package install root, rather than the build root

The Python 2.3.0 distributed with Mac OS X 10.3 is missing three extensions typically found on other platforms:

_bsddb
_tkinter
readline

Since these are part of the standard library, it is somewhat annoying to build them after the fact. I have put together a little svn repository with the relevant setup scripts and sources:

http://svn.red-bean.com/bob/MacPythonPantherExtras/trunk/

Installer packages for these three extensions are available at:

http://pythonmac.org/packages/

Note that _bsddb and readline include their respective dependencies statically, where _tkinter requires TclTkAqua 8.4.

Many other packages are also available from this repository, including PantherPythonFix, wxPython, Numeric and numarray (with AltiVec support via Apple's Accelerate framework), PIL (with tiff, jpeg, and Tkinter support), and matplotlib (with Agg, wxPython, and Tkinter support contributed by Chris Barker).

The majority of these packages are only available for the Python 2.3.0 that ships with Mac OS X 10.3 (Panther), but a few are available for Mac OS X 10.2 (Jaguar). These Jaguar packages are compatible with the MacPython 2.3 distribution (even if used on Mac OS X 10.3 or later).

Most of the packages in this repository were built with the bdist_mpkg command of py2app. Building your own redistributable package installers can be as easy as:

bdist_mpkg setup.py --zipdist

The --zipdist option is currently only available in py2app svn trunk, but will be part of the 0.1.8 release later this month.