2013 Year in Review

View from Brooklyn of the NYC New Year's Eve fireworks show
View from our living room of the NYC New Year’s Eve fireworks show

Last night Rebecca and I watched the NYC New Year’s Eve fireworks show from our Brooklyn apartment. We are fortunate enough to have a direct view of the Statue of Liberty from our living room, around which the fireworks were clustered. It was a nice, relaxing end to an eventful year.

We moved to New York City in August when an opportunity came up for me to work out of Bazaarvoice’s NYC office for a short period. I have always wanted to live in NYC, and as parents of a 1-something year old, this was perfect timing. Finn, our toddler, wasn’t old enough to have to worry about leaving any friends behind or having to deal with schools in NYC. It wasn’t difficult to tidy up the rest of our lives in Austin by renting out our house, loaning out our cars to friends, and putting everything we didn’t take with us to NYC in storage. I found a 1-month, furnished sublet on Craigslist for the month of August and we used the time to find our “permanent” apartment. Apartment hunting in NYC can be a bit of an ordeal, but in the end we found ourselves deciding between living in the Upper West Side or Park Slope, Brooklyn. Park Slope felt a little more family friendly with its nice parks, walkable streets, and abundance of restaurants and small shops. And so at the beginning of September, we found ourselves residents of a very hip part of Brooklyn.

2013 was our first full year as parents. Finn started off the year as a not-so-confident crawler and ended the year as a sprinter, with us having to chase him around Central Park to keep him from throwing himself off of boulders. He seems to have had a great time in NYC, with the city’s abundance of trucks, trains, and lots of other sights and sounds to keep his attention. It has been pretty amazing to watch him learn our neighborhood, which ...

There are comments.

read more ...



EC2 Ephemeral Disks vs EBS Volumes in RAID

January 02, 2010
tags: ec2 | aws

EC2 Disk Overview

Amazon’s EC2 service is really neat, but its disk subsystem has some peculiarities that are not initially obvious. Up until very recently, root directories (‘/’) at EC2 were limited to 10Gb, a limit defined by the maximum size of an Amazon Machine Image (AMI), essentially a template of an EC2 instance. In order to use more disk space, Amazon provides ephemeral disks that one can format and mount anywhere on the file system. However, in order to get persistent storage, one has to use network-attached EBS volumes, a sort of limitless in capacity but bound in I/O wonder of Amazon architecture. There are clear performance implications in choosing how to configure an EC2 instance’s disk subsystem, so I recently benchmarked some various ephemeral and EBS RAID configurations.

Ephemeral disks

Pros:

  • Free (included in cost of EC2 instance)
  • Stable, predictable performance on par with a standard physical hard disk
  • Abundant storage (up to 1.7TB on a c1.xlarge)

Cons:

  • Ephemeral - if the instance shuts down, all data is lost
  • Average random seek performance (6-7ms seek times per spindle)

EBS Volumes

Pros:

  • Highly available” - AWS claims to provide redundancy and a lower failure rate than physical disks
  • Portable - an EBS volume can be connected to any instance in a single availability zone
  • Backups - can easily create snapshots

Cons:

  • Extremely variable performance - seek times can range from .5ms to 10ms+
  • Maximum throughput of 1Gbit/s
  • Costs associated with storage and I/O

Testing

For this testing, c1.xlarge instances were used due to their high CPU performance, memory capacity, “I/O Performance: High” (according to Amazon), and 4 available 450GB ephemeral disks.

I created 5 c1.xlarge instances with 5 configurations: 4xEphemeral RAID0 local disk, single EBS, 2xEBS RAID0, 4xEBS RAID0, 8xEBS RAID0. All instances were created in the us-east-1b Availability Zone and all EBS volumes attached were newly created specifically for this test. Testing was done using bonnie++ on fast mode (-f flag, skips per-char tests).

mdraid was used to create RAID0 arrays with a chunk size of 256k, for example:

mdadm --create --verbose /dev/md0 ...

There are comments.

read more ...