Over the years, I’ve primarily used bash with occassional forays into zsh (and oh-my-zsh). zsh never seemed significantly better than bash, and it introduced enough friction in writing and running scripts that I’d inevitably switch back to trusty bash after a few months.

I’ve played around with fish in the past, but I never took the time to make an honest go at it. This time I’ve decided to spend some time to learn the shell, port my bash customizations, and see if fish will become my permanent choice.

First steps

Installing on archlinux is fairly straight forward, as the package is simply called fish. Then I ran chsh -s /usr/bin/fish to set fish as my default shell upon login.

Fish configuration location

Fish helpfully stores its primary configuration at $XDG_CONFIG_HOME/fish/config.fish, which is already a welcome change from putting random dot files in your home directory. Also, by default, fish looks for function files in $XDG_CONFIG_HOME/fish/functions/<function>.fish.

Setting vi mode

Setting vi mode in fish involvings invoking a built-in function.

# ~/.config/fish/config.fish

fish_vi_key_bindings

Auto-starting xorg

The next challenge is porting over my .bash_profile. Mine doesn’t do much except set an env var then start xorg, since I don’t use a display manager:

# ~/.bash_profile
[[ -f ~/.bashrc ]] && . ~/.bashrc

if [[ ! $DISPLAY && $XDG_VTNR -eq 1 ]]; then
  export QT_AUTO_SCREEN_SCALE_FACTOR=0
  exec startx
fi

Wanting to use functions, I did this:

# ~/.config/fish/config.fish
start_xserver

# ~/.config/fish/functions/start_xserver.fish
function start_xserver
    if test -z $DISPLAY && $XDG_VTNR -eq 1
        setenv QT_AUTO_SCREEN_SCALE_FACTOR 0
        exec startx
    end
end

fzf

fzf was a bit of a game changer for me after switching to fish. CTRL-R history search becomes fuzzy and so I can type the first half of a command followed by some other random bit of the command and have it selected properly.

# yay -S fzf
# curl -sL https://git.io/fisher | source && fisher install jorgebucaran/fisher
# fisher install jethrokuan/fzf 

I used https://github.com/jethrokuan/fzf instead of the more popular https://github.com/PatrickF1/fzf.fish since the former didn’t have any additional dependencies on fd and bat.

Intellij / Goland

I had to set the Settings -> Terminal -> Application Settings -> Shell Path:

/usr/bin/env XDG_CONFIG_HOME=/home/victor/.config /usr/bin/fish

This got fish running properly in the IntelliJ built-in terminal.

Update in March 2021

It’s been over 6 months since I’ve installed fish and I haven’t gone back to bash, so I suppose it’s been a successful switch.

I still sometimes switch back to bash to run some one-line loops because of muscle memory of bash syntax instead of learning the fish equivalent. I also find the fish vi mode somewhat inconsistent compared to bash’s vi emulation.

However, aside from that, I find fish to be very fast and configurable.

Back in early January my good friend Dave sent out an email to 20 or so people asking if anyone would be interested in doing the John Muir Trail with him. He mentioned something about going ultralight and doing it in 10 days, but I really didn’t think those were important details at the time. I had never gone backpacking for more than a long weekend, over a very short distance, but how hard could this be? I take my kids hiking in Austin all of the time. We go at least a mile or two. Of course I was in.

A couple of weeks later, I realized this was a pretty serious endeavor when Dave sent out a training regimen that had us getting up at 5am multiple times per week to run 5Ks and climb stairs up a 33-story building while wearing heavy backpacks. We “affectionately” dubbed the building Mount Windsor, after the name of the building itself.

So, for basically all of 2018, I’ve been dieting, reading r/ultralight, buying gear, or training in some way, including engaging the services of both a physical therapist to work on some knee pain and a trainer at Hyde Park Gym to work on my strength and conditioning. After consuming much of my thoughts, time, and too much money over the last 8 months, I have finally backpacked the John Muir Trail (not in its entirety; more on that later).

Logistics

In the end, there were 7 of us who agreed to go: 4 flying from Austin, 1 from Houston, 1 from Dallas, and 1 from Maine. Our ringleader, Dave, applied for the permit lottery in early February, and we ended up with North-bound (NOBO) permits leaving out of Cottonwood Lakes.

We planned to fly into Las Vegas, rent three cars, and drive to Lone Pine, California, crossing through Death Valley along the way. Joey, flying from Maine, had a conflict and could only join for the first 4 days of the trail.

Early setbacks

On the morning of Saturday, August 11th, we all arrived in Las Vegas at approximately the same time. Disaster struck almost immediately.

Eric, from Houston, did not see his bag come out at baggage claim. It was already around noon, and we needed to get to the ranger station in Lone Pine by 4:45pm in order to pick up permits. If we didn’t make the 4:45 closing time, we’d have to stick around in Lone Pine until the ranger station opened at 8am on Sunday, which meant that the soonest we could get on the trail at Cottonwood Lakes was after 9am, putting us seriously behind our aggressive schedule. Our plan had us getting in 8 miles on Saturday, not starting on Sunday after 9am. Getting an early start every morning was crucial in averaging 20+ miles per day and finishing the entire JMT in the 10 days that we intended.

After talking to Southwest, it was clear that Eric’s bag was not in Las Vegas. The earliest it could come was in a flight later in the afternoon, and even that was not assured. We decided to split up, leaving one car in LV to stick around until Eric’s bag showed up. The other two cars would try to pick up everyone’s permits before closing time. We also needed to drop off Joey’s rental car at the Taboose Pass trailhead, where he was planning on exiting the JMT.

Getting to Cottonwood Lakes

The drive through Death Valley was incredibly hot (registering 116F at one point), but the geology made for a really visually appealing drive. Even with a quick stop to experience the heat first-hand and to take some photos, we made it to the Lone Pine ranger station around 4pm - plenty of time to pick up our permits.

As we were getting permits, we got word from Las Vegas that Eric’s pack had arrived on a later flight, which was a huge relief. However, this put us sufficiently behind on our schedule that we had no choice but to spend our first night at the Cottonwood Lakes trailhead.

While we waited for Eric to arrive from Las Vegas, Joey and I drove to the Taboose Pass trailhead to do the car drop-off. The road from Highway 395 to the Taboose Pass trailhead was clearly intended for something higher off the ground than my Hyundai Elantra rental. The distance from the Highway 395 exit to the Taboose Pass trailhead is only 2-3 miles, but after carefully crawling along in the Elantra trying to avoid bottoming out on big rocks for over half an hour, we called it quits and found a spot to park Joey’s car along the dirt road, about a mile from the trailhead. The risk of getting a flat tire or getting the Elantra stuck on a rut or rock was too great. Well, I got stuck turning around. Fortunately, we were able to get it unstuck after cramming some rocks under the front wheels and rocking the ol’ Elantra back and forth.

Since we didn’t have time to hike at all that night, we took a “zero” day by simply having dinner in Lone Pine and didn’t get to the Cottonwood Lakes trailhead until after dark. We camped on ground sheets, cowboy style, to get an early start the next morning. As the camping gods would have it, it started drizzling at 3am, so we had a fun middle-of-the-night scramble setting up tents anyway.

Daily Journal

The following section is my day-by-day recollection of my JMT experience. If you’re considering doing your own JMT adventure, I hope you’ll find the recounting interesting and useful. At the end of each day’s section, I’ve included a link to my Strava segment with maps and elevations, as collected by my Garmin Fenix 5 watch.

Day 0 - Cottonwood Lakes to New Army Pass to Sky Blue Lake

We had a fairly late start the next morning. Since it was our first morning breaking down camp, everyone took a while to figure out their process - coffee, breakfast, order of operations packing up their gear, fitting the bear can into our pack, etc.

After at least an hour at camp, we started off with a fairly flat 2 mile warmup walk from the Cottonwood Lakes / Horseshoe Meadows campground. This turned into a gradual 2,000’ climb up New Army Pass, which we did in a little less than 3 hours.

From the top of New Army Pass, we hiked another 7 miles or so and decided to camp at Sky Blue Lake.

Since this was technically before Mount Whitney (the official start of the JMT), I decided to call this Day 0. It was, by far, our shortest day of hiking during the trip.

Day 1 - Crabtree Pass & Mount Whitney

We wanted to summit Mount Whitney the next day, but there was no easy way to get to Whitney from Sky Blue Lake. Going west towards Crabtree Meadows seemed like the best option, which meant scrambling through Class 2/Class 3 rocks over Crabtree Pass.

After we got up to the top of the 12,500’ pass, we realized how much of a challenge we had to get down. The north side of Crabtree Pass is no joke, and it took us about 2-3 hours of scrambling to get down to the lake (see the nearly vertical elevation graph on Strava). A misstep could have easily twisted an ankle or broken a leg. The rocks were loose, and steps would often sending you sliding down a couple of feet. We’re lucky that no one got hurt.

A few hours later, we made our way to Crabtree Meadows, where we decided to make a late afternoon Whitney attempt. By this time it was around 4:30pm, but we emptied our packs except for water, snacks, warm clothing, and headlamps, knowing full well that we wouldn’t be back until midnight at the earliest.

From Crabtree Meadows, we headed east towards Guitar Lake. By the time we arrived to the switchbacks up towards Whitney, it was already approaching 7-8pm, and the sun was starting to go down.

At this point, I wasn’t feeling particularly well, probably due to altitude, and the thought of hiking back to camp after midnight wasn’t super appealing. As the sun was disappearing over the ridge, I decided to throw in the towel on Whitney and return to camp. The group carried on. After 2-3 hours of headlamp hiking in nearly pitch darkness, I made it back to camp around 10pm. I ate dinner while watching shooting stars over the horizon, used the famous Crabtree Meadows camp toilet, and went to bed. The rest of the group staggered back well after I passed out, with a couple arriving around 1am and another two guys not getting back until 4am.

Day 2 - Forester Pass

Despite the extremely late night attempt at Whitney, we planned a long Day 2 over Forester Pass. Our alarms went off at 6am, meaning a couple of guys only had 2 hours of sleep. Fortunately, the hike From Crabtree Meadows to Forester Pass was a fairly easy climb, totalling 3,675’ over 14 miles and 6 hours.

The north side of Forester Pass has some spectacular views of the valley below. We took photos from the pass and started our descent as the sun was going down over the Sierras. By the time we got down into the valley it was completely dark, and we didn’t a great place to camp. We hiked another 5 miles using headlamps before finding a flat place to camp, somewhere along Bubbs Creek.

Day 3 - Glen Pass & Pinchot Pass

Day 3 was another tough day. Most people do 10-14 miles or a single pass in a day, but we continued on our tradition of setting the alarm for 5am and doing 20+ miles and 2 passes, which meant hiking well after the sun goes down. Day 3 included doing both Glen and Pinchot Pass. We walked more than 25 miles while climbing almost 6,500’ over 15 hours. I was beat. On the plus side, camping at Lake Marjorie was quite nice.

Day 4 - Mather Pass

This may have been my favorite day on the trail. After falling 30 minutes behind my group, I spent most of the day hiking solo. But the weather was perfect, so I didn’t bother trying to catch up. Instead, I took a nice comfortable pace that allowed me to really soak in the beautiful high Sierra scenery. I made a few videos to try to capture the feeling of hiking the JMT:

Having a nice break on the JMT

Going NOBO across Mather Pass was fairly straight forward - a climb of around 2,000’ over 6 miles to reach the top of the pass. Descending on the north side was quite a bit harder than the ascent on the south side (I imagine ascending the North side is just as hard for those going SOBO). Of particular difficulty is the Golden Staircase, part of the north side’s 4,000’ descent over 12 miles. The Golden Staircase is mostly big steps & boulders, which means your legs take an unrelenting pounding. Going down this section was where my knees really started bothering me, and my left foot started developing a blister, right in the middle of my forefoot. I sat down to tape my feet up, but cursed myself when I realized I had lent it out a few weeks back without re-supplying.

I found my group that night camped somewhere around Little Pete Meadow, at around mile 24 of the day.
My left knee was throbbing with every step, and I had a painful blister on the ball of my left foot. I stretched and rolled out my IT band (with a nalgene) as best as I could, but I knew the following day was going to be particularly hard.

Day 5 - Muir Pass

The alarm went off at 4am on Day 5. We were still 30 miles from the Muir Trail Ranch, where our first re-supply bucket was waiting. There was no chance we’d make it there by the 5pm closing time today, but the group wanted to get and camp somewhere close by so that we’d be able to pick up our re-supply when MTR opened at 8am the following morning (and enjoy “sleeping in”). Otherwise we’d have to get up early and hike the remaining miles to MTR to make it by 8am.

We quickly broke camp, and everyone except for Ryan was ready to go by 5am. Ryan had dropped his Leatherman CX somewhere at camp and was still looking for it when the rest of us started our hike for the day. When the sun came up, the five of us had already covered 4-5 miles.

My blister was still bothering me, but my knees felt slightly better. After a few miles, however, I hit a energy wall and couldn’t keep up with the group. I told them to go ahead and that I’d meet up later. This was fine with me, as I learned that I enjoyed hiking solo. As the morning wore on, though, my energy levels continued to drop while my knee pain increased. I was spent by the time I got to the base of Muir Pass. Each step up the pass was a challenge, and I needed to stop for a breather every few hundred feet. It was a baffling malaise that I couldn’t quite shake.

If I didn’t get it in gear, my chances of meeting up with the rest of my group were slim. No one was quite sure if we’d make it the entire 30 miles to MTR, so we hadn’t agreed upon a place to camp. Without any way of communicating with my group, I’d be forced to hike solo and hope that I’d find the group campsite later that night, somewhere along the trail, in the dark. The chances of that were slim. And if I didn’t catch up with them that night, I would have to get up early and hope to catch up with them at MTR. And if I didn’t catch up with them at MTR, I knew it would be even harder and harder to catch up with them at all. I had to catch up with my group in order to make the pre-arranged shuttle pickup at Yosemite Valley on Day 10. As this calculation swirled in my head, I was feeling worse as I forced my body up Muir Pass. Then I noticed that my pack’s hip-belt had started to rip and separate from the rest of my pack.

The combination of my hurting knees, blisters, fatigue, the logistics of hiking solo, and equipment failure - it seemed all too daunting at the time. I decided that I wanted to exit the JMT.

Ryan caught up with me on Muir Pass a little while later. He was a fast hiker and would easily catch up with the rest of the group. I told Ryan that I was bailing out and to inform the rest of the group, since I had no other way of sending a message to the rest of the group. We said our goodbyes, and Ryan walked on while I continued to trudge slowly up Muir Pass. There was no way for me to change my mind now.

The rest of the day was my hardest on the JMT. From the top of Muir Pass, I made it another 14 miles (about 7 miles shy of MTR), before making camp at Evolution Meadow around 8pm. I camped next to a wonderful couple from Seattle who gave me some fig newtons and delicious Trader Joe’s ginger cookies, which was out-of-this world good, since I had been eating keto for weeks.

Day 6 - Muir Trail Ranch & Piute Pass

The next morning, I wanted to get to MTR to talk to them about exit options. My hope was to go west via the Florence Lake ferry and somehow make it back to Lone Pine. I thought I only had about 7 miles to MTR, but I miscalculated and had to hike 10 miles to MTR.

It was 10-11am by the time I got to the MTR area. I had heard that the hot-spring at MTR were worth a visit, so I sought out the hot-spring before going down into the Ranch. That required walking another mile or so, fording a river, and then walking across a meadow. The hot-spring itself was basically a hot muddy pit and was pretty underwhelming. Having hobbled all of this extra way to get there, I went ahead and got into the warm, muddy water. I had enough of the experience after 5 minutes, but what made the stop worth it was fording the river on the way back. The fast-flowing river water was cold and crisp, especially after the hot spring, so I decided to just plop myself in the river and soak in the cold for a while. It felt fantastic.

The kind folks at MTR told me that I could exit the JMT west-bound via Florence Lake and hitch-hike towards Fresno, but then my public transportation options from Fresno back to Lone Pine would probably be a 2-day trip involving multiple Greyhounds and other transit systems. This seemed very unattractive.

The other option was to hike out east-bound via Piute Pass, requiring a 5 mile backtrack to where the JMT met up with the Piute Pass trail. From this junction, Piute Pass itself was a 3300’ climb over a distance of 20 miles to the Piute Pass Trailhead. From there, I’d still be 30 miles by road to the town of Bishop, and then I’d have to hitch-hike 60 miles to Lone Pine. And from Lone Pine, I still had to figure out how to get up to my rental car at Cottonwood Lakes. After walking for a week in one direction, I suppose you’re pretty far from where you started.

Both the west-bound and the east-bound options seemed bad, but after some hemming and hawing, I decided that a punishing hike to the East would ultimately be faster than going West to Fresno and navigating public transportation.

I backtracked to the JMT/Piute Pass intersection and hiked as much as I could of Piute Pass before setting up camp that evening. Fortunately, the weather was perfect, and it turned out to be a beautiful way to enjoy my last night on the trail.

Day 7 - Piute Pass to Trailhead

It was a bitter-sweet morning. I couldn’t help but feel a tinge of disappointment, knowing it was my last day on the trail. Months of training and thinking about the JMT were coming to an end. As I packed up my gear for the last time, I was overjoyed about the thought of seeing my family soon. However, I also wondered where my group was and how everyone was feeling, lamenting somewhat my decision to bail. I also thought about the logistics of hiking out and how I was going to get to my car, over 100 miles away.

After a slow breakfast, I headed towards civilization.

The Piute Pass trail was a very scenic route, but for me it was a tough hike out. My JMT maps didn’t include Piute Pass, so I wasn’t quite sure where I was or how much further I had to go. This probably contributed to how long it felt, but after what seemed like a very long day, I made it to the Piute Pass trail-head in the early afternoon.

There were lots of people at the Piute Pass trailhead. Within a few minutes of holding my thumb up, I was picked up by a friendly woman heading towards Bishop. She dropped me off right across from a McDonald’s, so I went in to have a quick wash up. I also couldn’t resist temptation and ate a Big Mac.

I did some research on my phone and learned that I would have to wait until the next morning for a bus to Lone Pine. I wanted to try to get to Vegas that night, so I decided to attempt hitchhiking. After 30-45 minutes of sticking my thumb out on Main street Bishop, a climber coming from Mammoth Mountain picked me up and dropped me off in Lone Pine.

In Lone Pine, I spent another hour or so trying to hitch-hike up to my car at Cottonwood Lakes, but only one car stopped, and they were on their way to Whitney Portal. It was getting late in the day, so I gave up on hitch-hiking up to Cottonwood Lakes and walked over to the Whitney Portal Hostel to ask if they had any suggestions. They gave me a phone number and told me to ask for Kurt. The famed Lone Pine Kurt (760-876-4811) came by 15 minutes later. Kurt was super friendly and we had a great 45 minute conversation as he drove me up to my car at Cottonwood Lakes, exactly where it was parked more than a week earlier. I packed up my things and drove back to the Whitney Portal Hostel to take a $7 shower before driving to Vegas.

I was on the road to Vegas by 9pm and on a flight back home the next afternoon.

Final Thoughts

While I’m a little disappointed that I didn’t finish the JMT, I’m super happy that I made an attempt and enjoyed the experience (thanks Dave!). Hiking 165 miles over 7 days, along some of the most beautiful scenery in the world, is something I’ll never forget.

The training alone was worth it - I was probably in the best shape of my life at the start of the trip. I’m significantly more aware of my body, having learned what tends to hurt, what causes the pain, and how to deal with it. While I don’t think I quite reached my physical and mental limits, I have a general idea of what they are.

I’m a much more competent and confident backpacker. I’m no longer intimated by the idea of going on long multi-day trips. I have a much better grasp of how to train, prepare, and pack for these types of trips (I’ve already started daydreaming about the Colorado Trail, the Tahoe Rim Trail, the Pacific Northwest Trail, and even Tour du Mont Blanc).

Our planned pace of 22-25 miles per day was simply too fast. At times, it felt like a heads-down, forced-march to make miles, which didn’t leave many opportunities to simply sunbath on a rock or catch some mountain trout by a serene high Sierras lake. In hindsight, a 15-18 miles per day pace would have been ideal, and I think most of my compatriots would agree. The reality, however, is that it’s simply hard to coordinate a group of guys to walk away, literally, from our familial and work lives for 3 weeks. We had all spent a tremendous amount of time throughout the year training, and I think it’s safe to say that we were all extremely appreciative of our significant others for giving us the time to dedicate to this trip.

Since getting back to every-day life, I’ve been surprised about how much I still think about the JMT. I’m still a member of a couple of the JMT Facebook groups and /r/JMT subreddit and enjoy seeing people’s pictures and posts about their own JMT adventures.

I sometimes think that maybe I should have pushed harder and longer through the pain to finish the JMT. I don’t know. But I do know I’ll be back on the JMT again.

More photos: https://photos.app.goo.gl/WbGcfb5Zi3bjfBB37
Lighterpack: https://lighterpack.com/r/8z8oy7

Leave a comment and let me know if you’d be interested in an equipment post that goes into detail about each piece of gear that I brought along.

A little late, but this is my 2014 Year in Review. It’s so hard to summarize an entire year into a few paragraphs and a handful of selected photos. I’ve spent hours over the last few weeks going through photos and videos that I took the year, some of which I’m looking through for the first time. I’m reminded of all the trips taken and places seen, some which seem so long ago but most just seem a few months ago, even if they were from the beginning of 2014. Time flies by as it always does, and the process of writing my Year in Review posts helps cement in my memory the things that I think ought to be remembered. So here goes…

Travel:

• Vietnam 1/19-1/25
• NYC-Austin 2/2
• AUS->CHS 5/24
• CHS->LAS 5/28
• AUS->PDX - 6/20
• AUS->JFK
• Big Bend 10/24
• AUS->CHS 12/25

Notable Restaurants:

• Eleven Madison Park - 1/7
• QUI 3/4
• Peninsula Grill 5/26
• Estiatorio Milos 5/30
• Starlight Theater

Random:

• Snowboarding at Mount Snow 1/16
• ROIKOI SXSW 3/8
• Last workday at BV - 6/17
• Last BV all-hands - 6/19
• Ecola State Park - 6/21
• Redwood National Park - 6/25
• Carter Lake National Park - 6/27
• J&J from Ireland on 7/3
• Last day at BV - 7/3
• ROIKOI - 7/7

2014 was a year of transition and change. In January, my stint working out of Bazaarvoice’s NYC office came to an end, as planned. Getting to spend 6 months in Brooklyn was such a fantastic experience. We got to really learn the city beyond the weekend here and there that we previously experienced as tourists. We spent most of January revisiting our favorite NYC places - restaurants, parks, museums, stores, etc, while casually planning our move back to Austin.

About a week before the movers were supposed to come, however, I received some saddening news - my grandmother in Vietnam passed away.

My grandmother was one of my last remaining strong connections to Vietnam. She was born in the rural Mekong Delta, in the deep south of Vietnam, and never ventured very far from her ancestral lands.
Late in life, however, she became somewhat of a jetsetter. She shuttled back and forth between Vietnam and the U.S. as she strove to spend time with her children and grandchildren, divided by a hemisphere. She never became comfortable with life in the U.S., where everyone worked constantly a nd going to buy vegetables required a vehicle. Conflicted with choosing one set of grandchildren over another, she eventually decided to return to Vietnam for good. There she could spend slow days in her hammock, enjoying the warm breeze rustling through the coconut and orange trees surrounding her house.

We where in the process of moving from NYC back to Austin when I learned of her passing, but I knew I had to go to Vietnam for her funeral.

I left NYC on the 19th of January and arrived on the 21st in Ho Chi Minh City, Vietnam. If one wants to experience awe at the rate at which an entire nation can be transformed, just go to Vietnam every couple of years. My first trip back to Vietnam as an adult in 1999 required driving on single-lane rural roads shared with pedestrians, bicyclists, and families of 5 on single mopeds, and two separate ferry trips. This time there was a brand new, 6-lane, elevated highway with modern bridges that cut through the rural countryside, the type of picturesque landscape of rice fields with farmers in straw cone hats and water buffalo that one sees on postcards of Vietnam. After three days of ceremony involving multiple feasts, chanting monks, and even fire-breathing, snake-charming performers (deserving of its own blog post), I flew back to NYC on the 25th in order to finish the move back to Austin. It was a whirlwind trip.

Back in NYC, we packed the last of our things and spent the final weekend in a hotel soaking in final bits of NYC as tourists, like finally doing the (old City Hall stop via the 6 train loop)[http://jalopnik.com/5684329/how-to-see-new-yorks-secret-city-hall-subway-stop].

We easily settled back into our old house and old life in Austin. It was really interesting to watch Finn walk into our house, to see how much of it he remembered. As with things intimately familiar, we found the grooves of our past Austin lives easily, in our favorite restaurants and in the company of our old Austin friends.

It would be a few months later when I found the siren of startup life call my name, and this time I could not resist any further. I left BV, the longest place I’ve ever worked, to work on roikoi.com, which at the time was just a questionable idea with a significant amount of funding. ROIKOI made an already eventful year even more exciting.

Last night Rebecca and I watched the NYC New Year’s Eve fireworks show from our Brooklyn apartment. We are fortunate enough to have a direct view of the Statue of Liberty from our living room, around which the fireworks were clustered. It was a nice, relaxing end to an eventful year.

We moved to New York City in August when an opportunity came up for me to work out of Bazaarvoice’s NYC office for a short period. I have always wanted to live in NYC, and as parents of a 1-something year old, this was perfect timing. Finn, our toddler, wasn’t old enough to have to worry about leaving any friends behind or having to deal with schools in NYC. It wasn’t difficult to tidy up the rest of our lives in Austin by renting out our house, loaning out our cars to friends, and putting everything we didn’t take with us to NYC in storage. I found a 1-month, furnished sublet on Craigslist for the month of August and we used the time to find our “permanent” apartment. Apartment hunting in NYC can be a bit of an ordeal, but in the end we found ourselves deciding between living in the Upper West Side or Park Slope, Brooklyn. Park Slope felt a little more family friendly with its nice parks, walkable streets, and abundance of restaurants and small shops. And so at the beginning of September, we found ourselves residents of a very hip part of Brooklyn.

2013 was our first full year as parents. Finn started off the year as a not-so-confident crawler and ended the year as a sprinter, with us having to chase him around Central Park to keep him from throwing himself off of boulders. He seems to have had a great time in NYC, with the city’s abundance of trucks, trains, and lots of other sights and sounds to keep his attention. It has been pretty amazing to watch him learn our neighborhood, which he often demonstrates by pointing out directions, from his stroller, to various bagel shops and the library that we frequent. We were also lucky enough to live within a couple blocks of a large neighborhood park, which, during the late summer, was perpetually filled with neighborhood kids alternating between running around the playground and cooling off under the water sprays.

New York fulfilled most of our expectations. It’s awe-inspiring to cross a Manhattan avenue and see two rows of perfectly aligned sky-scrapers, seemingly sloping down with the earth’s curvature until they blend into the city haze. It’s amazing to walk through Central Park and admire how such a fantastic retreat, with its miles of paths, lakes, and wonderful trees, can exist in the middle of such a loud, non-stop city. Entering Central Park always makes me appreciate the foresight of men like Frederick Olmsted. Of course, in New York we have some of the world’s best museums and restaurants, a mere subway ride away. With all of its amenities, you can’t help but feel that you are at the center of the universe, all of the time.

It took a few months, but I eventually felt a little at home in New York. The various subway lines of the MTA began to make sense to me, and sometimes I could even answer questions from tourists as if I knew exactly everything there is to know about NYC - which subway lines went where, which stop to get off and change at, and so forth. I even found myself sometimes getting annoyed at the slow tourists meandering in front of me, preventing me, an impatient New Yorker, from quickly going wherever I needed to go. This just means that when we leave, I’ll definitely miss the city. I think my favorite take-away will be the knowledge that when we come back to visit, it’ll be with a different perspective and appreciation for the city. I like to think I’ll feel comfortable, like a regular walking into to a private club, knowing exactly where everything is and how things work.

Travel

We started 2013 off with a return trip back to Austin, TX from Charleston, SC, where spent the 2012 Christmas holidays and New Years.

In May, I went to Denver, Colorado for Gluecon. Rebecca and Finn came up after the conference, and we spent a few extra days in Fort Collins and a night in the beautiful Colorado Rocky Mountains at a cabin owned by Rebecca’s relatives.

In June, I went out to Las Vegas to meet up with old friends for our annual Vegas weekend. This trip was timed so that we could play in a couple of the smaller World Series of Poker events. I ended up missing registration on my WSOP event but I did make the final tables on two out of two non-WSOP tournaments I played in, which paid for my trip and more.

Our big trip this year was to Scotland and Ireland, which we timed to slip between leaving Austin and moving to NYC. So in mid-July, we departed Austin for Scotland with a two night detour in NYC to drop off some stuff for our move and to give Finn a break from air-travel.

From NYC it was a pretty easy direct flight to Edinburg, Scotland.

We visited Rebecca’s sister in St. Andrews where we rented a nice apartment from HomeAway.

After a few days in St. Andrews, we drove up to Lochcarron, where we had a cottage for a 5 days or so.

From Lochcarron, we made some drives around the Scottish Highlands, including a very nice, beautiful day on Skye.

From Lochcarron, we drove down to Glasgow for a quick visit. No trip to Europe is complete without a stopover to Ireland to visit the relations, especially since this would be Finn’s first time meeting his Irish family. So we took a puddle jumper from Glasgow to Dublin, Ireland. After a nice visit with the family, we flew back to NYC, now as soon-to-be New Yorkers, from Dublin.

In August, I took a trip up to Manchester, NH to go to Dyn Inc’s Geek Summer Camp. Since it was only a single day event, Rebecca and Finn came along and we made a long weekend out of it, visiting Portland, Maine and a few other small towns along the way. In Kennebunkport, ME, we stopped by Nunan’s Lobster Hut, which is undoubtedly owned by people related to Rebecca.

In September, I went down to Atlanta, Georgia for a recruiting trip at Georgia Tech. Since I was “in the neighborhood”, I flew down to Charleston, SC where Rebecca met me with Finn. It’s always nice to visit Charleston, especially for Finn and his grandparents.

Rebecca had always wanted to see the autumn leaves changing in Vermont, so in October we took another long weekend up to Burlington and Woodstock, Vermont.

I attended AWS’s re:Invent event Las Vegas in November, making it two trips to Las Vegas for the year.

A week after I got back from Las Vegas, an ex-pat friend of ours living in China was attending a conference in Boston. So of course, we had to make a weekend trip out of it. It was our first time on Amtrak, which was surprisingly nice.

With so much travelling already in 2013, we stayed put for the holiday season. Both of our families visited New York in November, so we didn’t feel the need to go anywhere else and just enjoyed New York for the rest of the year.

Conferences and Events

I think this is a comprehensive list of tech conferences and events I attended this year.

• TEDxAustin
• GlueCon
• Texas Linux Fest
• Bazaarvoice I/O
• Dyn’s Geek Summer Camp
• World Maker Faire
• AWS re:Invent

Other Highlights

• Finn turned 1! And that means we concluded our first full year as parents and everyone still has 10 fingers and toes.

• After almost 5 years, we finally landscaped our Austin yard in March

• Rockies’ game at Coors Field
• New York Century Bike Ride. We rode from the top of Central Park down to Prospect Park in Brooklyn.
• Jazz Age Lawn Party on Governor’s Island
• US Open - went three days and saw both the Women’s Finals (Serena Williams def. Victoria Azarenka) and Men’s Finals (Rafael Nadal def. Novak Djokovic)
• NY Knick’s game at Madison Square Garden
• We went to the Daily Show!

• I played golf in Scotland.
• I bought a bike and while the weather was nice, I rode almost daily from Park Slope, Brooklyn to the Bazaarvoice NYC offices at Union Square, Manhattan.

Notable Meals

• Nunan’s Lobster Hut - Kennebunkport, ME. The folks that run this are undoubtedly related to Rebecca and her Irish family.
• Anstruther Fish Bar - Best fish & chips I’ve ever had, hands-down.
• The Varsity - Atlanta, GA - Notable because it was terrible.
• Jean Georges. Michelin 3-star.
• Gramercy Tavern - Michelin 1-star with a mellow vibe.
• Kabuto Edomae Sushi (2x) - Las Vegas, NV

2014

We started 2013 off in Austin, TX and ended the year in Brooklyn, New York with a bunch of great memories in between. It looks like we’ll be back in Austin soon, but who knows where the 2014 retrospective will be written from.

I’ve got big goals and aspirations for 2014. That’s what makes this all interesting and exciting.

EC2 Disk Overview

Amazon’s EC2 service is really neat, but its disk subsystem has some peculiarities that are not initially obvious. Up until very recently, root directories (’/’) at EC2 were limited to 10Gb, a limit defined by the maximum size of an Amazon Machine Image (AMI), essentially a template of an EC2 instance. In order to use more disk space, Amazon provides ephemeral disks that one can format and mount anywhere on the file system. However, in order to get persistent storage, one has to use network-attached EBS volumes, a sort of limitless in capacity but bound in I/O wonder of Amazon architecture. There are clear performance implications in choosing how to configure an EC2 instance’s disk subsystem, so I recently benchmarked some various ephemeral and EBS RAID configurations.

Ephemeral disks

Pros:

• Free (included in cost of EC2 instance)
• Stable, predictable performance on par with a standard physical hard disk
• Abundant storage (up to 1.7TB on a c1.xlarge)

Cons:

• Ephemeral - if the instance shuts down, all data is lost
• Average random seek performance (6-7ms seek times per spindle)

EBS Volumes

• "Highly available" - AWS claims to provide redundancy and a lower failure rate than physical disks
• Portable - an EBS volume can be connected to any instance in a single availability zone
• Backups - can easily create snapshots

Cons:

• Extremely variable performance - seek times can range from .5ms to 10ms+
• Maximum throughput of 1Gbit/s
• Costs associated with storage and I/O

Testing

I created 5 c1.xlarge instances with 5 configurations: 4xEphemeral RAID0 local disk, single EBS, 2xEBS RAID0, 4xEBS RAID0, 8xEBS RAID0. All instances were created in the us-east-1b Availability Zone and all EBS volumes attached were newly created specifically for this test. Testing was done using bonnie++ on fast mode (-f flag, skips per-char tests).

mdraid was used to create RAID0 arrays with a chunk size of 256k, for example:

mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=2 /dev/sdi1 /dev/sdi2

blockdev is used to set the read-ahead buffer to 64k:

blockdev --setra 65536 /dev/md0

XFS is used as the filesystem:

mkfs.xfs -f /dev/md0

Finally the RAID array is mounted with noatime at /mnt/md0:

mkdir -p /mnt/md0 && mount -o noatime /dev/md0 /mnt/md0

I logged the results of Sequential Writes, Sequential Reads, and Random Seeks. bonnie++ was run 6 times on each instance.

bonnie++ averages Instance Write Read 4xEphemeral RAID0 231680 176739 1xEBS 37827 48991 2xEBS RAID0 80578 75966 4xEBS RAID0 98836 89752 8xEBS RAID0 69921 93175

Sequential Throughput

The EBS results are a little less predictable. A single EBS does not have the throughput of a single ephemeral drive. The 2xEBS RAID0 shows almost twice the throughput of the single EBS volume, while the 4xEBS RAID0 and 8xEBS RAID0 instances do not scale much higher than the 2xEBS RAID0 instance for throughput. Since EBS volumes are access via network, this indicates that EBS volume throughput is limited by the gigabit interface.

Random Seek Times Instance Total Ran. Seek/s Per Volume 4xEphemeral RAID0 658 165 1xEBS 250 250 2xEBS RAID0 396 198 4xEBS RAID0 5138 1285 8xEBS RAID0 2554 319

Random Seeks

EBS random seek performance, however, is not easily predictable. The volumes that make up the 4xEBS RAID0 instance clearly are higher performing than those of the other instances. Is EBS performance more of a property of the EBS volumes or the instance?

Another interesting result I noticed (but didn’t include in these graphs) is the deviation of performance from one run to another. The standard deviation between the runs was much smaller for the ephemeral drives than for the EBS volumes.

Swapping EBS volumes to identify bottleneck

Results:

The I/O channel is more or less saturated, but we still see the same poor random seek performance that the 2xEBS RAID0 instance exhibited with these two same EBS volumes. This leads me to believe that the seek times are inherent to the individual EBS volumes themselves.

To confirm, I mounted the high performance volumes from the 4xEBS RAID0 instance and the poorly performing volumes from the 2xEBS RAID0 instance to the 8xEBS RAID0 instance. I wanted to test if we can “export” the high the performance from the 4xEBS RAID0 instance to the 8xEBS RAID0 instance. I then repeated the bonnie++ tests.

Results:

Once again, the 2xEBS volumes are still poorly performing and the previously fast 4xEBS volumes are still fast (they were even faster than before). At this point, the evidence is pretty clear that the performance of the EBS volumes are inherent to the volume itself, since they exhibit the same level of performance regardless of the EC2 instance that mounts them.

Instantaneous EBS performance over time

Results:

The same instance using the 4 same EBS volumes show a huge discrepancy in performance from week to week. During the first week, the 4 EBS volumes perform admirably. During the second week however, performance drops dramatically and the 4xEBS RAID0 volumes don’t even seem to saturate the gigabit channel. This doesn’t bode well for EBS performance predictability.

During the second week, I ran ‘iostat -x -m 240’ while I ran bonnie++ to see if I could identify the poor performance.

2xEBS RAID0:

           avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                     0.02    0.00  0.47    11.99    0.02     87.51
 
Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
md0               0.00     0.00  0.00 615.20     0.00    25.42    84.63     0.00    0.00   0.00   0.00
sdi1              0.00     0.06  0.00 307.41     0.00    12.71    84.67   148.55  483.20   3.25 100.02
sdi2              0.00     0.04  0.00 307.60     0.00    12.71    84.61    10.49   34.10   1.19  36.67
sdi3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdi4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Notice the particularly poor ‘await’ time of /dev/sdi1, one of the two members of the /dev/md0 RAID0 array. Because of how poorly /dev/sdi1 was performing, the entire /dev/md0 array exhibited poor performance. During a read request, the file file system sends a request to the /dev/md0 device, and mdadm determines that it needs to split the request into two separate requests, one that goes to /dev/sdi1 and /dev/sdi2. If one of the two members of the array is poorly performing, as in this situation, it becomes a bottleneck for the entire array. (As an aside, a few hours after seeing these poor numbers, I re-ran bonnie++ on these 4 EBS volumes, and they were once again fast.)

Conclusions

There are a few key takeways:

1. EBS volumes, even several in RAID0, have a throughput limit of 1 gigabit/second on an EC2 instance. Two EBS volumes in RAID0 have the ability to almost max out an instance's EBS I/O channel
2. EBS volumes can have random seek times ranging from .5x to 10x+ that of an ephemeral disk
3. Ephemeral disks in RAID0 are very fast
4. It is advisable to use both a mixture of ephemeral and EBS RAID arrays, depending on the situation
5. 4 or 8 EBS volumes in RAID0 seem to be enough to smooth out the peaks and valleys of individual EBS volumes' seek times
6. Increasing mdadm RAID0 chunk size to 256k and setting 'blockdev --setra 65536' seems to help sustain I/O throughput even when seek times are bad. Mounting EBS arrays with '-o noatime' should also theoretically help with poor seek performance.