bq.. Note: This is an old article from 2005 that I never got around to finishing. Considering it recounts my experiences working for the Democratic National Committee on Election Day 2004, I figured it was time to fill out what I remembered, edit it a bit and publish it.
Most of it was written in “Murky Coffee”:http://www.murkycoffee.com/, exactly one year after the ignominious end to three months of my life I’m not necessarily hankering to repeat.
Of course, this time I wasn’t up there for politics–my friend George’s son, George Jr., wanted to got to the Nine Inch Nail concert in DC. Since George was going to have to chaperone him, he invited me along to keep him amused.
“What the hell?”, I figured.
So I drove out that morning, got in around noon, filled my belly with rice and raw fish at “the best sushi joint I’ve ever been to”:http://sushitaro.com/, and then walked across the city to Murky.
p. Since I’m here, and since I never wrote about it before–first I was too exhausted, then I was too dispirited–here’s the geek’s-eye view of the boiler room at the Democratic National Committe’s Election Day HQ, 2004.
First, the night before ended pretty late, like 10pm, and I decided to work off some excess energy by walking back to the place I was staying–not from DNC HQ, though, but from the DNC boiler room, located in a building just off McPherson Square. This took an hour and a bit. So I got to sleep around midnight, with my phone set to go off at 3am, so I could be back at the DNC boiler room by 4:30am.
Of course, the Metro doesn’t run this early in the morning, and I had no car, so I walked back, although this time I brought a change of clothes and a pillow–very awkward, but very important, as I hoped to catch a nap at some point–on top of everything else.
(As a parenthetical note, I lost 10 pounds in DC from all the walking, gained them back by the new year, lost them again after my dental surgery in March, gained most of it back when we went on vacation in June, and then lost it again over the last four months. Hopefully for good).
I got to the HQ at 4:30am, met up with Alex and Bob and a couple of other people. We went to our assigned desks and started getting ready. We screwed up the network–it was set up for very high-security, with allowed MAC addresses assigned to individual ports, but we decided we wanted our seating set up differently–we wanted the dev team facing one another for easier communication–which broke stuff. We spent about the next half hour sorting all this out.
Almost immediately, the shit really hit the fan. Our primary database server–we had two for our two applications, both running PostgreSQL, with Slony-I providing replication–died. The SCSI controller started throwing errors and we lost the database.
We had only got replication set up two nights before. We had never actually tested fail-over.
At 5:30am, we found, to our great pleasure that failover worked.
The next few hours were consumed with rebuilding the server that had failed, and getting it back up as a hot standby for the (new) main DB server. Because I had gotten barely any sleep the night before, this took longer than it might have–I spent a lot of time wondering why things weren’t working, when all I needed to do was start the actual Slony replication process.
This PostgreSQL install was supporting two applications: my application was a web-based internal communication system for local coordinators to report polling place issues or irregularities. My friend Alex’s application was for tracking exit polling information to feed to the statistical model the DNC was using to predict the outcome.
By lunchtime things had stabilised, and we all had some lunch and grabbed some coffee.
At one point, Anne came by–she had driven into town in anticipation of having some good parties to go to that night and of packing me up and taking me home–and we talked a bit outside before I had to head in to get back to work.
More monitoring of the DB server and my application, as well as trying to help Alex with some performance problems with his application. Alex was doing a bunch of Ajax-y stuff with this app long before Ajax had been coined, but he had some database concurrency issues that were causing him problems–he had written his code to be too eager to lock rows, and was having conflicts.
We whiled away much of the afternoon, working on dealing with stuff, but pretty happy overall–I don’t know how many people remember, but Kerry was doing quite well for most of the day. We expected it was going to be a tight race, but definitely going to go for Kerry in the end.
By 8pm, I couldn’t stay awake any more. Kerry was ahead, everything looked good with the systems, so I took a nap–I dragged my pillow under a desk, stuck in my headphones and went comatose.
I crawled out, bleary-eyed, from underneath the desk at about 10:30pm. Everything had changed. Ohio, which had been predicted to break blue, was too close to call, and everything else in the electoral map which would give us enough votes was settled–and not in our favor.
We loitered around for a couple of hours, waiting for results from Ohio, and all we got was grim news–it was close, but as more and more votes were counted, it was looking like Bush was going to take the state.
Finally, around midnight, the decision was made that there was nothing else to do tonight, everyone should go get some sleep, and call in in the morning to find out what was going to happen next. I packed up all my stuff and headed out to the hotel where Anne was checked in. It was probably another hours’ walk.
Some time around 1:30 or 2am I got to sleep.
When I woke the next morning at 7am or so, things were still undecided. I called into the DNC, was told there was no hurry to get in, but there was going to be a conference call at 11am, and that we should be either at the DNC or at the Boiler Room.
Anne and I went in to the DNC after breakfast, so she could meet some people–especially Doug, my boss. Turns out they had a mutual friend who had come out of politics at UA to work for Zell Miller, back when Miller was really a Democrat.
While we were there, Mary Beth Cahill did the conference call. After thanking everyone for their hard work, she announced that Kerry was going to concede Ohio, burst into tears and basically hung up.
At that point, Anne and I decided to head up to McPherson Square to the Boiler Room to see if we could find the group of ex-Amazon volunteers I had fallen in with. While we were there, we all watched the concession.