EC2 Ephemeral Disks vs EBS Volumes in RAID

January 02, 2010
tags: ec2 | aws

EC2 Disk Overview

Amazon’s EC2 service is really neat, but its disk subsystem has some peculiarities that are not initially obvious. Up until very recently, root directories (‘/’) at EC2 were limited to 10Gb, a limit defined by the maximum size of an Amazon Machine Image (AMI), essentially a template of an EC2 instance. In order to use more disk space, Amazon provides ephemeral disks that one can format and mount anywhere on the file system. However, in order to get persistent storage, one has to use network-attached EBS volumes, a sort of limitless in capacity but bound in I/O wonder of Amazon architecture. There are clear performance implications in choosing how to configure an EC2 instance’s disk subsystem, so I recently benchmarked some various ephemeral and EBS RAID configurations.

Ephemeral disks

Pros:

  • Free (included in cost of EC2 instance)
  • Stable, predictable performance on par with a standard physical hard disk
  • Abundant storage (up to 1.7TB on a c1.xlarge)

Cons:

  • Ephemeral - if the instance shuts down, all data is lost
  • Average random seek performance (6-7ms seek times per spindle)

EBS Volumes

Pros:

  • Highly available” - AWS claims to provide redundancy and a lower failure rate than physical disks
  • Portable - an EBS volume can be connected to any instance in a single availability zone
  • Backups - can easily create snapshots

Cons:

  • Extremely variable performance - seek times can range from .5ms to 10ms+
  • Maximum throughput of 1Gbit/s
  • Costs associated with storage and I/O

Testing

For this testing, c1.xlarge instances were used due to their high CPU performance, memory capacity, “I/O Performance: High” (according to Amazon), and 4 available 450GB ephemeral disks.

I created 5 c1.xlarge instances with 5 configurations: 4xEphemeral RAID0 local disk, single EBS, 2xEBS RAID0, 4xEBS RAID0, 8xEBS RAID0. All instances were created in the us-east-1b Availability Zone and all EBS volumes attached were newly created specifically for this test. Testing was done using bonnie++ on fast mode (-f flag, skips per-char tests).

mdraid was used to create RAID0 arrays with a chunk size of 256k, for example:

mdadm --create --verbose /dev/md0 ...

There are comments.

read more ...

From CET to CDT

View from Bismarckstrasse

Memory is really strange. On the one hand, I’m amazed at how fast the last three years of my life has gone by. I remember walking into my hotel room on the day I landed in Stuttgart, a full month before Rebecca would come, and putting down my bags and really wondering what I had gotten myself into. I had decided to take a job in a city I had visited only during my interviews, in a country I had only spent a few days in as a tourist, and here I am sitting down in my hotel room, in need of a shower, exhausted from jet-lag, only then grasping that I had committed myself and Rebecca to living in Stuttgart for at least three years. At that moment I could only hope that we had made the right choice to come. It turned out to be one of the best decisions of our lives.

Remembering specific events, though, and time doesn’t seem to move so quickly. I think about my first weekend in Germany, when a colleague invited me to an Onion Festival in the medieval town of Esslingen, and it seems appropriately placed about three years ago. Then I remember when our friend Laurel visited, our first visitor, I think, and how we went out to a besenwirtschaft (a uniquely south-west Germany gem, in which vineyard-owning families sell their own wine out of their living rooms) and got extremely intoxicated with a super friendly German couple. We ended up getting invited to their home for a few more bottles of wine, and Rebecca got sick in their bathroom just as our taxi pulled up. I remember all of the festivals - the Hamburg Fischmarkt, Karnival, the Weindorf, and of course the Bierfests (Germans love to find a reason, any reason, to have a festival). I remember all our visitors - our families and lots of friends from home - who took advantage of us living in Stuttgart and allowed us to share our newly found love of Germany with them. I think fondly of all the trips we took ...

There are comments.

read more ...

FOSDEM over; Crisis averted

lots of beer on tap

I went to Brussels last weekend for FOSDEM 2008, which was held at ULB Campus Solbosh. The free event was a good way to check in with the overall Open Source community and to see all of the interesting things people outside my normal circles are working on.

Friday Night Beer Event

Things got off to an memorable start on Friday night. I timed my arrival so that I could attend the Friday night “Pink Elephant” beer event held at the Delirium Cafe. I met up with a colleague, and we had a few good beers while chatting with other FOSDEM attendees. Lots of people had their gadgets out for others to play with. I got to play with a EeePC and a Nokia 810 while my iPhone was passed around. I even picked up the presence of a OLPC OX-1 over wifi, but was never actually able to find it.

After a few hours of drinking beer and talking about software, we met up with a few more friends to go to dinner at an underwhelming yet wistfully overpriced restaurant in the middle of the tourist trap. I had another beer or two over dinner, and so when we left the restaurant, I was a little toasted.

For some reason (playing with my phone?) I was straggling behind as we walked out when these two guys sidled up to me and started dancing, singing yelling, and doing some weird line dance kick between my legs. In my drunken state, I was a bit confused but thought they were just drunk too and danced along. After a few moments of this silliness, they walked off. I luckily had a moment of clarity and thought it best to check my pockets. Wait, my wallet is missing. Yup, it really is still missing. The two guys hadn’t taken more than 20 steps down the street, so I ran up to the nearest one, forcefully grabbed his shoulder, and demanded, “Give me back my wallet.” He looked a bit surprised and immediately pointed to his accomplice. I turned to him and ...

There are comments.

read more ...