from __future__ import *

Kailash and Friends Kailash Kher Kaipa

So it seems that I made the wrong decision for our uplink at Mochi Media. We have a gigabit line to Cogent at 365 Main. They've been mostly good so far (and cheap), but their recent spat with Telia has cut us off to parts of Europe (at least Sweden and Denmark). Not cool.

Anyone have any recommendations on other providers we should talk to? I don't want us to be dependent on Cogent for a minute longer than we have to be.

Jameson and I did an interview with Irina and Eddie from PodTech's LunchMeet yesterday at our office. Twenty minutes of video podcasty goodness right here: LunchMeet: Fueling Creativity with Mochi Media

Before anyone asks about the shirt, it's from the xkcd store (inspired by #54).

• Food is cheap in Taipei and Shanghai
• Alcohol isn't
• US internet connectivity in Taipei is kinda slow and high latency
• US internet connectivity in Shanghai is even worse; barely usable
• An earthquake in Taiwan can make Shanghai's US connectivity completely unusable for days. Going on 6 now; still slower than dial-up 10 years ago.
• The great firewall blocking sites like wikipedia, google's cache and wordpress can get annoying. It's also hard to tell if a site is just unreachable or is blocked by the great firewall.
• Google knows what the hell they're doing (and nobody else seems to). They're the only US company I've found that has maintained a usable infrastructure in Shanghai through this whole mess. Not Yahoo, not Microsoft, just Google. If Google had a proxy, I could've gotten by pretty nicely.
• Localized sites for US companies work fine because they're hosted domestically, but I can't read them (e.g. apple.com.cn).
• MacBook Pro power supplies really do melt, and they cost twice as much in Shanghai than retail from Apple in the US. No wonder people outside the US don't buy Apple!

We've been looking at solutions for hosting and serving large amounts of (small) static content to a global audience and are currently in the process of evaluating all of the CDNs out there.

One might think that Amazon S3 would be good for this purpose, but it's absolutely not. We tried it for a while, and it really only has two things going for it:

• It's very cheap
• Everything is automated and easy to set up

Aside from that, the performance is absolutely terrible outside of the US, which I really noticed since I've been in Taipei. In addition to that, it's simply not reliable. There are extended and unannounced periods of downtime, so you shouldn't be using it in production (except maybe for redundancy or backup purposes).

After we experienced some unacceptably long downtime I scoured the internet for a real CDN solution. There are a bunch out there, but only one of them is open enough to publish prices and allow you to sign up without doing a sales call: CacheFly. So far our experience with CacheFly has been pleasant. It's a little more expensive than Amazon S3, but you're paying for a far better service. Plus, there's a 30 day free trial. The trial appears to be "only available if you sign up before tomorrow", but it seems to be following the calendar.

There have been a few small snags with the CacheFly service so far:

• In order to use rsync/scp/sftp uploads you must file a support ticket; ftp is the only service enabled by default
• After that support ticket was resolved, scp was broken so I had to file another ticket (rsync and sftp worked)
• In order to use your own DNS you must file a support ticket, otherwise you're stuck with accountname.cachefly.net
• Their POP distribution technology seems suboptimal; they're serving the lion's share of our traffic from the US when most of it isn't coming from there (though still massively better than Amazon S3)

On the flip side, the experience has been good overall:

• Latency is MUCH better than Amazon S3 in our experience
• They give you pretty good reports about your traffic, without having to parse your own logs
• Responses to support tickets have been prompt
• It's the only real CDN with published prices that allows you to sign up without dealing with a sales department

All that said, CacheFly is the smallest fish in the CDN pond. It only has 5 POPs and we'll probably outgrow it within the next year and move over to one of the "Tier 1" providers such as Akamai or BitGravity . I still highly recommend CacheFly as a first step for anyone looking to improve their site's reliability and performance by using a CDN.

The first line of defense in scaling out a web solution is a load balancer, often implemented as a HTTP reverse proxy. In my particular situation, I wanted to be able to load balance based on HTTP headers (Host is a must, URL is a bonus) on a FreeBSD platform.

Apache 2.0 + mod_proxy

Pros:

We were already running Apache, so it was an obvious initial solution to the problem. Not too bad to configure, well documented, and pretty stable. It served us well for quite some time.

Cons:

The prefork MPM is definitely not ideal for this purpose and doesn't give us the scalability we want. Screwing around with other MPMs was not ideal because Apache is also doing other things that we didn't want to disturb. Maintaining a separately configured installation just for proxying isn't a good solution. The thread-using MPMs would only be a small win, anyway.

LightTPD

Pros:

Easy enough to configure, supports all of the features we need, pretty good performance and scalability. Single process event-driven model that can use kqueue.

Cons:

Spotty documentation. Totally bizarre configuration language. Not terribly mature compared to other solutions. Lots of bugs. The current version (1.4.11) leaks memory like a sieve (#758), so it's unfortunately not an option at this point.

Pound

Pros:

Solid documentation (but only in man page format). All of the features we need, not a whole lot of other junk. Good track record. This is what we're currently using for now, but it's no permanent solution. Better than Apache, and it doesn't seem to leak RAM.

Cons:

Uses LOTS of threads. That means lots of RAM, limited scalability and performance. Documentation is only available as a man page, had to install it to really evaluate it.

The other solutions I looked into, but didn't end up trying were

Squid:

Way too complicated. Too hard to figure out how to get it to do what we wanted to. Might be a good solution though, so I'll probably look into it again.

Perlbal:

No usable docs. FreeBSD port exists, but sucks (no rc file, no sample config). Couldn't figure out how to get it to reverse proxy with the features that I needed. Written in Perl, which I do grok, but don't like hacking on.

HAProxy:

Looks awesome, but doesn't have the features we need. Seems to only be able to do IP based load balancing, not by headers. More than just a TCP proxy, but not enough of a HTTP proxy.

Balance:

Just a TCP proxy.

Pen:

Just a TCP proxy.

We've (finally) updated talkingpanda.com today with a fresh new look and a new product: iBar.

iBar turns your iPod (any iPod with a screen and a dock connector) into an "ultimate bartending tool" with over a thousand drink recipes, mixing techniques, and even a couple history lessons. Like any good bar should be, it's stocked. Altogether, the Notes content adds up to over 3 megabytes (yes, that's text!) with more than 40 minutes of original, professionally recorded, audio.

Like its sibling iLingo, iBar ships with installers for Mac OS X and Windows XP/2000 (everywhere iTunes is supported) that make installation painless.

And for relevance, here's a little bit about how it's all put together:

• The Mac OS X installer was developed with Python 2.3, PyObjC 1.2, and bundled up with py2app.
• The Win32 installer was developed with Python 2.4, win32all, Tkinter, PIL and is bundled up with py2exe and then made self-contained with NSIS.
• The build scripts for the installer application actually use even more open source stuff. We use JExcelAPI to convert Excel spreadsheets to XML, and Win32::Exe to swap out the ICO resource out of win32 executables -- but no Java or Perl makes it into the redistributable :)

We're also blogging updates and podcasting some of our content, so point your NetNewsWire or Safari RSS over to Talking Panda News!

It has recently [1] come to my attention that using versioned frameworks without binary compatible APIs simply doesn't work all too well. Essentially, you may have an older version installed, but without swapping symlinks around, you you won't be able to use the older version for anything except runtime compatibility with software that is already linked.

This isn't a problem with the dyld runtime, simply a problem with gcc's compiler and linker (and supposedly there may be runtime bugs with CFBundle and/or NSBundle in this arena as well).

Frameworks

Mac OS X frameworks are structured in such a way that allow for versioned code, data, and headers. A typical multi-versioned framework looks something like this:

Foo.framework/
    Foo -> Versions/Current/Foo
    Headers -> Versions/Current/Headers
    Resources -> Versions/Current/Resources
    Versions/
        Current -> 2.0
        2.0/
            Foo
            Headers/
            Resources/
        1.0/
            Foo
            Headers/
            Resources/

Mach-O, MH_DYLIB, and the dyld runtime

The MH_DYLIB object file, the part that you link to, is "Foo". If this were not a framework, it would have the following layout, and it would be application-specific to put the headers and resources in the right place:

libFoo.dylib -> libFoo.2.0.dylib
libFoo.1.0.dylib
libFoo.2.0.dylib

Due to the way that MH_DYLIB object files work, each of these dylibs know their own (supposed) location on the filesystem. This is called the "install name" or the "id" of the dylib. Technically, this is a LC_ID_DYLIB load command in the Mach-O header. When you create an Mach-O object file (executable, dylib, bundle, etc.) that depends on another, it will generate a load command (LC_LOAD_DYLIB) referencing the other dylib. The data associated with the LC_LOAD_DYLIB is exactly the same data that was provided by the linked-to MH_DYLIB in its LC_ID_DYLIB load command, unless explicitly overrided with the -dylib_file option to ld(1) or rewritten post-link with a tool such as install_name_tool(1). The otool(1) tool can be used to view these load commands in a Mach-O file, among other things. It's very important to have these values set correctly because they are used by the dyld runtime to locate the intended MH_DYLIB.

gcc, ld, and frameworks

Apple's GCC includes many changes to support Objective C/C++ and framework based development. Unfortunately, none of them have any explicit support for versioned frameworks. For example, gcc(1) states that it uses the following algorithm for finding frameworks in headers:

-Fdir
    In Apple's version of GCC only, add the directory dir to the head
    of the list of directories to be searched for frameworks.

    The framework search algorithm is, for an inclusion of
    <Fmwk/Header.h>, to look for files named path/Fmwk.framework/Head-
    ers/Header.h or path/Fmwk.framework/PrivateHeaders/Header.h where
    path includes /System/Library/Frameworks/ /Library/Frameworks/, and
    /Local/Library/Frameworks/, plus any additional paths specified by
    -F.

    All the -F options are also passed to the linker.

ld(1) has an similar option, -framework, which has a similar algorithm:

-framework name[,suffix]
       Specifies a framework to link against.  Frameworks  are  dynamic
       shared  libraries,  but  they are stored in different locations,
       and therefore must be searched for differently. When this option
       is  specified,  ld  searches for framework `name.framework/name'
       first in any directories specified with the -F option,  then  in
       the  standard  framework  directories /Library/Frameworks, /Net-
       work/Library/Frameworks,  and  /System/Library/Frameworks.   The
       placement  of the -framework option is significant, as it deter-
       mines when and how the framework is searched.  If  the  optional
       suffix is specified the framework is first searched for the name
       with the suffix and then without.

Note that neither of these options allow for any consideration for the Versions directory in a framework, and therefore only link to whichever version was installed last, because the installation process for a framework will create the symlinks that point to locations inside the Versions directly.

Workaround

Unfortunately, since there is no support or hook that will allow proper usage of versioned frameworks, the workarounds are all ugly. I think the following workaround is the most appropriate: Just Don't Use GCC's Search Algorithms.

• Always use the compiler option -I/Path/To/Foo.framework/Versions/IntendedVersion/Headers
• use #include "Foo.h" instead of #include <Foo/Foo.h> in your sources
• Always use the direct path to the MH_DYLIB rather than any combination of -framework and -F.
• Or, if you're building extension bundles that will be used by an executable that already has the correct version of Foo linked in, use the -undefined dynamic_lookup linker option

A minimal compiler/link line for an executable would look like the following:

env MACOSX_DEPLOYMENT_TARGET=10.3 cc 
    -I/Path/To/Foo.framework/Versions/IntendedVersion/Headers 
    -o usesFoo 
    usesFoo.m /Path/To/Foo.framework/IntendedVersion/Foo

And a minimal compiler/link line for an extension bundle would look like:

env MACOSX_DEPLOYMENT_TARGET=10.3 cc 
    -I/Path/To/Foo.framework/Versions/IntendedVersion/Headers 
    -o fooUsingExtension.bundle 
    fooUsingExtension.m 
    -bundle -undefined dynamic_lookup

Quite ugly, eh? At least it works as intended. Unfortunately you'll need to emulate enough of GCC's search algorithms to find the framework, which is probably quite problematic from Xcode, but shouldn't be too hard from say, distutils, SCons, autoconf, etc. Note that you should also probably consider the NEXT_ROOT environment variable when building against an SDK.