Aug 20, 2010

When I first started here at SingleHop, over two years ago, one of my first tasks was to overhaul our aging DNS system. We had been creaking along using zone files, but their number was quickly growing out of control. Also, we rely heavily on automation here, and using scripts to automatically edit files can be quite error-prone.

So, back in April of 2008, we migrated to BIND-DLZ and moved all of our DNS records into MySQL databases. I released a DNS zone parsing script you can use to do the same thing in May.

This system has been incredibly stable. After a few initial kinks, we haven’t even had to think about our authoritative DNS servers at all. During our recent internal DB overhaul, however, we discovered that the version of MySQL hosting the DNS records was too old, and it couldn’t be updated because our DNS servers were still running CentOS 4.8. We were gonna have to migrate to an entirely new operating system, and to do that with the least downtime required setting up new servers.

Today, we finished setting up the new DNS servers and switching our infrastructure over to using them. When it came time to decommission our old nameservers, I decided to see just how long they had been running:

[root@ns1 ~]# uptime
16:55:01 up 593 days, 03:12, 2 users, load average: 0.11, 0.06, 0.04
---
[root@ns2 ~]# uptime
16:53:45 up 626 days, 10:40, 3 users, load average: 0.10, 0.08, 0.09

Like many unix nerds, I tend to get obsessed with uptime. For instance, my own personal server has been up for 99 days, and even my laptop hasn’t rebooted in over a month. So it was not without a heavy heart that I destroyed the epic 626-day uptime on ns2 when I shut it down today.

These long-running servers got me to thinking: how many of our other internal servers have been running for a really long time? It turns out, having servers that run for years isn’t unusual here at SingleHop. For instance, here is the uptime from our Quick Reaction monitoring server: 20:50:11 up 578 days, 1:30, 2 users, load average: 1.06, 1.25, 1.36; our customer database server: 20:33:42 up 647 days, 17:35, 1 user, load average: 0.29, 0.53, 0.63; the singlehop.com web server: 20:53:00 up 317 days, 12:40, 4 users, load average: 0.05, 0.07, 0.07.

However, the champion among all the servers I checked today is one of our oldest backup servers:

backup02:~# uptime
 20:44:48 up 899 days,  6:41,  1 user,  load average: 0.05, 0.02, 0.07

Considering that SingleHop is not even three years old, a two-and-a-half-year uptime is pretty impressive indeed.

All of this is possible thanks to the magic of Linux. Software updates happen automatically, and the computer does not need to be rebooted to take advantage of them — only the affected applications need be restarted. And, thanks to our KSplice service, we don’t even have to reboot our servers to do kernel updates! We can just keep our infrastructure humming without a care, for years at a time!

So, what’s YOUR longest-running server? Share your story on SingleHop’s community forum: http://community.singlehop.com/general-discussion-introduce-yourself/403-what-your-longest-running-server-we-just-reluctantly-rebooted-899-one.html#post2409

Comments

    One of our servers was running without rebooting for 554 days.

    Miguel

    Posted by Miguel on August 20, 2010 Reply

    interesting articles. I like this site. =)

    Posted by Servers on August 24, 2010 Reply

    We have got servers with more than 1200+ days uptime, which are mostly Solaris 9 servers but we also have Linux servers with quite an impressive uptime (800+ days).

    Posted by Web Hosting on August 21, 2011 Reply

    Thanks for a marvelous posting! I truly enjoyed reading it, you will be a great author.I will be sure to bookmark your blog and definitely will come back later on. I want to encourage you continue your great job, have a nice morning!

    Posted by weathervane cupola on September 19, 2011 Reply

    I’m really enjoying the theme/design of your blog. Do you ever run into any browser compatibility issues? A small number of my blog readers have complained about my site not operating correctly in Explorer but looks great in Opera. Do you have any solutions to help fix this problem?

    Posted by weathervanes on October 1, 2011 Reply

    I’m having some problems with Firefox, so I want to uninstall it and then reinstall it. But I want to save all my bookmarks, so can I save my bookmarks to my computer, and then import them back to Firefox once I reinstall it? Thanks for the help..

    Posted by Spencer Uhlir on November 26, 2011 Reply

    It’s arduous to seek out knowledgeable individuals on this topic, but you sound like you recognize what you’re speaking about! Thanks

    Posted by Estela Jody on July 21, 2012 Reply

    This is the correct Poindexter's Pond Blog | » Uptime Nostalgia journal for anyone who wants to attempt out out almost this subject. You observation so overmuch its almost exhausting to reason with you (not that I rattling would want…HaHa). You definitely put a new spin on a theme thats been handwritten almost for geezerhood. Prissy object, but uppercase!

    Posted by Android Apk Files on August 18, 2012 Reply

    I’m impressed, I have to say. Actually rarely do I encounter a weblog that’s each educative and entertaining, and let me tell you, you will have hit the nail on the head. Your idea is excellent; the issue is one thing that not sufficient people are talking intelligently about. I am very glad that I stumbled throughout this in my seek for one thing relating to this.

    Posted by Black Hat on June 6, 2013 Reply

Leave a Comment