CacheFly vs. Amazon S3
We've been looking at solutions for hosting and serving large amounts of (small) static content to a global audience and are currently in the process of evaluating all of the CDNs out there.
One might think that Amazon S3 would be good for this purpose, but it's absolutely not. We tried it for a while, and it really only has two things going for it:
- It's very cheap
- Everything is automated and easy to set up
Aside from that, the performance is absolutely terrible outside of the US, which I really noticed since I've been in Taipei. In addition to that, it's simply not reliable. There are extended and unannounced periods of downtime, so you shouldn't be using it in production (except maybe for redundancy or backup purposes).
After we experienced some unacceptably long downtime I scoured the internet for a real CDN solution. There are a bunch out there, but only one of them is open enough to publish prices and allow you to sign up without doing a sales call: CacheFly. So far our experience with CacheFly has been pleasant. It's a little more expensive than Amazon S3, but you're paying for a far better service. Plus, there's a 30 day free trial. The trial appears to be "only available if you sign up before tomorrow", but it seems to be following the calendar.
There have been a few small snags with the CacheFly service so far:
- In order to use rsync/scp/sftp uploads you must file a support ticket; ftp is the only service enabled by default
- After that support ticket was resolved, scp was broken so I had to file another ticket (rsync and sftp worked)
- In order to use your own DNS you must file a support ticket, otherwise you're stuck with accountname.cachefly.net
- Their POP distribution technology seems suboptimal; they're serving the lion's share of our traffic from the US when most of it isn't coming from there (though still massively better than Amazon S3)
On the flip side, the experience has been good overall:
- Latency is MUCH better than Amazon S3 in our experience
- They give you pretty good reports about your traffic, without having to parse your own logs
- Responses to support tickets have been prompt
- It's the only real CDN with published prices that allows you to sign up without dealing with a sales department
All that said, CacheFly is the smallest fish in the CDN pond. It only has 5 POPs and we'll probably outgrow it within the next year and move over to one of the "Tier 1" providers such as Akamai or BitGravity . I still highly recommend CacheFly as a first step for anyone looking to improve their site's reliability and performance by using a CDN.
Does MochiAds require so much traffic or this quest is for unrelated project?
Comment by Max Ischenko — 2006-12-06 @ 3:36 am
Yes, this is for MochiAds. It’s actually not a volume problem, but a latency problem. We can easily get tons of bandwidth, but all the bandwidth in the world isn’t going to do us any good if it takes *seconds* to load an ad. If you round trip more than once with 600ms latency you’re already doing pretty bad.
Here’s some echoping numbers for requesting a 20 KB file over HTTP from my hotel room in Taipei:
Our server (in San Francisco) - 0.803384 seconds
S3 (seems to be in Seattle) - 1.652920 seconds
CacheFly (maybe from the Tokyo POP) - 0.526801 seconds
Clearly S3’s performance is horrifying.
Comment by bob — 2006-12-07 @ 6:35 am
[OT]
bob,
svn.red-bean.com is not reachable. I’ve been trying to download the flashticle turobogears gateway code.
Please do the needful.
-pradeep
Comment by Pradeep — 2006-12-12 @ 6:38 am
It’s not my server. There’s nothing I can do for it. It should be back up soon though.
Comment by bob — 2006-12-12 @ 10:39 am
Was it one of the S3 servers that was down?
Comment by leftboom — 2006-12-13 @ 8:55 am
If you’re talking about the off-topic comment about a totally unrelated SVN server being down, no, that wasn’t S3. S3 doesn’t do subversion.
However, the whole post is about S3’s latency sucking hard and S3 having significant downtime. So yes, S3 does go down.
Comment by bob — 2006-12-13 @ 10:34 am
I notice you say that Bitgravity is a “teir 1″, obviously Akamai is top of the line, but isn’t Bitgravity brand new to the game or am I missing something?
Comment by leftboom — 2006-12-15 @ 8:26 am
Never heard of Bitgravity - what’s your experience with them to be able to rank them right up there with AKAM.
Comment by bawbzor — 2007-01-23 @ 6:25 pm
Hi Bob, I’d be interested in speaking with you more about this. Care to email me?
Thanks,
Beth
Comment by beth — 2007-03-05 @ 12:04 pm
Well even though this posting is a little dated I thought I would put my two cents worth in on it. I just started using S3 recently and I couldn’t be happier so far that is.
This may be because I have a rather small site and it does not see the amount of traffic that your site see’s. But I have really noticed a lot less latency since switching.
As far as Catchfly goes they look to be more expensive and the fact that if you don’t use little or no bandwidth you still pay the same amount seems to be a real seller for me and S3.
Comment by Redneck Techy — 2007-04-12 @ 3:24 pm
We are researching the actual throughput and response times from Amazon. Has anyone found anything of the sort?
Comment by nathan — 2007-05-31 @ 12:25 pm
We’re using both cachefly and S3 on projects and we’ve been happy with both. S3 has a simple web service interface where we can readily publish static content (that is added to extremely often) via the web application as opposed to cachefly which publishes using FTP interface and seems better set up to serve static content that doesn’t change that often (think large FLVs, CSS, JS, SWF, etal). The simple ability to CRUD objects via application on s3 is definitely a big advantage.
Anecdotally, we’re seeing latency on s3 of up to 500ms but mostly less than 100ms for 10kb objects. Way more than acceptable for our purposes so far, but certainly depends on your application.
Comment by senor — 2007-06-04 @ 6:37 am
CacheFly is up to 10 POPS, with an 11th to be added soon, none of which will result in a rate increase, for anyone concerned.
Comment by Mark — 2007-09-02 @ 5:02 pm
There are plenty of other smaller CDN’s comparing to Akamai and others with great rates I believe. Any experience with http://www.valuecdn.com ?
Thanks
Joik
Comment by Joik — 2007-11-05 @ 4:22 am
Hi,
Did you try Speedera. I have implemented it for a site for a big vendor. It is pretty good.
your static content will be served by the servers which are closest to the user all around the world.
For a user in China the static content will served by servers in china.
Cheers, Nag
Comment by Nag — 2007-12-20 @ 10:11 am