Python on Mac OS X (Intel)…
Python on Mac OS X for Intel is not going to be a seamless transition. Unlike Mathematica, there is going to be a lot more than 2 hours of effort involved. Why?
- Python uses autoconf. Autoconf is not Xcode. It does not have a checkbox for universal binary compatibility. Autoconf is a PITA.
- PyObjC depends on libffi. libffi doesn't know what the Mach-O calling convention for x86 is, so it doesn't work. libffi is very deep, magical, scary code. This has been mostly solved by Ronald over the past few days in the universal binaries lab.
- mach_inject (which PyObjC's objc.inject uses) depends on injecting code into other processes. On an OS that can be running code for two different architectures, how the heck does that work? How do you know which pid is running which architecture? Anyway, injecting from x86-x86 isn't going to work because mach_inject doesn't know x86 yet.
- py2app's macholib doesn't really support fat binaries yet. It understands them well enough to do the right thing with the PPC portion of the Mach-O header, but it ignores other architectures. It will need a semi-major refactoring in order to support this cleanly.
- Bgen stuff breaks. Specifically, all of your Carbon code that deals with four character codes is going to be broken due to endianness issues. Yet another reason to stay the heck away from this stuff.
So, thanks Apple, for giving us weeks worth of very hard and unintesting work to do. The least you could've done is put out a nice new laptop that I could buy to do this work on :)
I'm also not terribly interested in renting an Intel development machine from Apple so that I can do work that helps them more than me. If they sold it, or stated that we would get a grand worth of credit towards a real Intel machine when they're available, then I wouldn't complain. But no, we get to rent a machine from them, that we can't really talk about, publish benchmarks of, move to a location other than the shipped address (?!), etc. They also say that they're under no obligation to fix it if it breaks, and there are no refunds. Sweet deal!
jsolaithttp://jsolait.net/
seems to go some way toward rationalizing (and pythonizing) javascript
(But javascript drives me batty too. Imagine if you could program the browser with python…! )
Comment by Jon Schull — 2005-06-06 @ 11:14 pm
Since Apple uses and ships Python, wouldn’t they at least do some of the hard work?
Comment by Just van Rossum — 2005-06-07 @ 4:09 am
Well, solved once, solved forever, right? And I don’t think they’re going to make YET ANOTHER architectural transition after this one.
Comment by l0ne — 2005-06-07 @ 7:29 am
Seconding l0ne, I don’t think its unreasonable to have a major architecture change once a decade (the last one was 1996, this one will take place during 2006-2007). I doubt Apple finds this move to be ideal; rather, after getting burned with Motorola and then again with IBM, they had to retreat back to their if-not-for-this-the-company-will-die Plan D. Its pretty clear the way the Intel compatibility was talked about that it was viewed as Apple’s last chance, if all else went south.
Also, to correct a few misconceptions:
* the binary will have two complete sets of executable code. At runtime, the loader will just pick the appropriate one and launch it. There is one PID, the memory footprint is that of one architecture’s binary. The only downside is extra compliation time for the developer, and an increased HD footprint.
* on endianness; there is an API to insulate your program from byte-order issues. Use it.
* I’m sure more information on the binary format will come out; if the universal-binary format isn’t too closed, perhaps other Unices will adopt it. Solaris and Linus still prize multiplatform operation and might further wish to have binary compatibility with each other (and perhaps Darwin). Obviously this last would be challenging, with different kernel APIs, different versions of different libraries etc.
Comment by Rahul Sinha — 2005-06-07 @ 7:59 am
Solved once, solved forever? They were saying that twelve years ago. Of course there will be another architectural change, because the iX86 path is in its last mile.
Look out monoculture!
Comment by Joel Rees — 2005-06-07 @ 8:55 am
I would guess the Universal Binary “format” would be nothing more than the .app bundle that OSX has had since the start, would it not? Instead of there always being just the “Contents/MacOS” target, there’ll be a “Mactel”, “Inple”, “MacOSIntel”, “MacOSX86″, or something like that as a target. Just a guess…
Comment by Bob Maguire — 2005-06-07 @ 12:15 pm
Just FYI - they corrected the time required for porting Mathematica from 2 hours to 2 days.
Comment by Jos Yule — 2005-06-07 @ 12:58 pm
No — Universal Binary format is not implemented by bundles/packages, but by the mach-o format.
Multi-architectural support goes back to the NeXt/OpenStep days.
If you download xcode 2.1 update, you can even build a unix command line app as a fat binary –
packaged as a single mach-o file.
Comment by Steve Majewski — 2005-06-07 @ 1:10 pm
I doubt Apple is going to fix this stuff.
Yes, fixed once, fixed forever, but that doesn’t mean it’s any less of a pain in the ass for me.
If you didn’t notice, the last major architectural change was Mac OS X, which last I checked, didn’t happen in 1996 :)
The format used to facilitate this is the Mach-O file format, from NeXT, which is open. It’s NOT done at the .app bundle level. I doubt anyone else is going to use it; they like their ELF, and it’d be more likely for them to extend ELF than to replace it.
Comment by Bob Ippolito — 2005-06-07 @ 1:11 pm
See http://developer.apple.com/documentation/MacOSX/Conceptual/universal_binary/universal_binary.pdf — the ABI is in Appendix D.
For the flipping issues, there’s EndianU32_BtoN() etc. … or better, OSReadBigInt32() etc. … but yeah bgen would probably have to learn some new tricks.
Comment by Doug Wyatt — 2005-06-07 @ 1:30 pm
Oh, okay. Thanks for the correction on the format issue.
Comment by Bob Maguire — 2005-06-07 @ 1:42 pm
You don’t have to use Xcode. The Universal Binary guide says that you need to make sure to use the __BIG_ENDIAN__ macro instead of AC_C_BIGENDIAN with autoconf. Since they’re giving instructions for using autoconf, I’m pretty sure it should work with autoconf. Looks like it should be as simple as adding “-arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4.0.sdk -syslibroot,/Developer/SDKs/MacOSX10.4.0.sdk” to CFLAGS. I haven’t finished trying it yet though.
Does python not run on darwin/x86?
Comment by Rib Rdb — 2005-06-07 @ 9:53 pm
Python works fine on Darwin/x86, but the Mac OS X specific stuff does not (it compiles, but doesn’t work right with some data types).
You’d also have to mangle LDFLAGS, and these changes need to also work on older versions of the dev tools, etc.
It’s all going to happen, of course, but it’s going to be a pain in the ass for whoever ends up doing the work. That’s all.
Comment by Bob Ippolito — 2005-06-07 @ 10:03 pm
Here’s an open call to Apple to give you a machine and some real tech support. Not that it will do any good posted here, but Apple really should put some very modest resources into PyObjC and (though not mentioned above) wxPython. Anyone with sufficient connections at Apple should put a bug or two in their ear. The good news: there’s no particular rush….
Comment by Scott Lawton — 2005-06-07 @ 11:28 pm
Apperently the porting is already basically done, minus the mach_inject stuff, of course:
http://www.pycs.net/bbum/2005/6/8/
Comment by Evan Jones — 2005-06-08 @ 1:44 pm
It’s less done than it appears to be. The core PyObjC functionality is just a small piece of what’s necessary in order to have a nice environment.
Comment by Bob Ippolito — 2005-06-08 @ 1:46 pm
Right, that is a good point. Your point still stands: it will be a lot of work to be able to create “native” x86 Mac OS X Python applications. Still, it makes me happy that Python and my simple PyObjC scripts will likely work immediately.
Comment by Evan Jones — 2005-06-08 @ 5:07 pm
I just created a trivial one-liner patch to psyco to get it to compile on Darwin/x86. It’s fast!
Unfortunately as-shipped in 10.4.1, distutils doesn’t create universal binaries.. so THAT is going to be another “fun job”. Perhaps bdist_mpkg will learn how to do the distutils incantations twice and lipo everything together.
Comment by Bob Ippolito — 2005-06-08 @ 5:10 pm
I see from bbum’s post that the libffi issue has been resolved, so that’s good.
As for mach_inject, there is a way to determine if a given process is native or translated. I’m not sure if it’s in the documentation, however.
I don’t know what macholib looks like, but dealing with Mach-O headers, while slightly messy, really isn’t all that bad.
And as for four-char codes, while I don’t know what bgen is, they typically Just Work. The only exception I see on a regular basis is for OSType-to-string conversion, which is simple to fix.
Comment by Eric Albert — 2005-06-09 @ 4:02 am