Nov 14, 2011
Sam Bowling

Backups are important, even in the filesystem level!

I’ve been a Linux user for around 13 years now and am amazed with how progressive the overall experience has become. Thirteen years ago you were using either Slackware 3, Redhat 5.x or Mandrake usually. Being 14 I was one of the “newbies” stuck on Mandrake because my 56k modem was what is known as a softmodem – a modem that lacks quite a bit of hardware and relies on your computer’s resources to actually function. Back then to make these work in Linux was a complete nightmare and Mandrake was the only one that worked out of the box with softmodems.

Back in those days you didn’t have the package management tools you have today be it yum, aptitude, portage or any other various package management utilities. You had rpmfind.net to find your rpms while praying to god you found the right ones for your specific operating system as well as playing the dependency tracking game. Slackware was strictly source installs and the truly Linux proficient would pride themselves in how small of a Slackware install footprint they could get to have a running desktop.

Growing frustrated at not understanding the build process and being constantly referred to as a newbie who uses “N00bdrake” I forced myself into the depths of Linux and after a year or so had a working Slackware box with XFree86 running Enlightenment with sound and support for my modem. I learned an extensive amount about how Linux works, compiling your own kernel, searching mailing lists to find patches for bugs, applying patches to software and walking through your hardware to build proper .conf files so daemons would function specifically. It seems that this kind of knowledge is being lost with Linux users these days as they are not forced to drop down to the lowest level of Linux to make their systems function.

A good example of this is recently dealing with a hard drive with an ext3 filesystem that was showing no data on it. If you used the command df which shows partition disk usage the data was shown as taking up space, but you couldn’t see it. A lot of people conferred and figured that the data was completely lost for good while I sat there saying NOPE waiting for someone to give a correct answer. Unable to get one, I divulged that the reason this happened is because a special block what is known as a superblock had become corrupted and the journal on the filesystem lost all information. Issuing a fsck would not fix the issue and routinely would check out as “ok” as it is using the bad primary superblock. There are actually multiple superblocks on ext2, ext3 as well as ext4 partitions. These exist specifically for backup purposes should your main one become corrupt to correct such an issue. Having toasted Linux countless times playing around with things such as software raid and hard-locking due to a poorly configured kernel, I have probably spent more time than I should’ve in the past reading about how the ext filesystem works. You in the past might have overlooked this when creating a filesystem in Linux but you will see output similar to this when creating an ext filesystem:

Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

These are very, very important blocks and integral to maintaining data on your filesystem. Lost these numbers? You can still get them from a few methods as well:

1)  First you need to know what size of blocks you used on your filesystem. The default is 1k so unless you actually issued a specific command when using mkfs.ext3 then your blocksize is 1024.

2)  Now issue the command “mke2fs -n -b block-size /dev/sdc1”. This is assuming that sdc1 is the corrupt partition throwing no data. Since you’re issuing the -n flag this means the command will not actually make a new partition but will give you those precious backup superblocks.

3)  Now take any of those superblocks and make sure that your partition is unmounted. Issue “fsck -fy -c -b 163840 /dev/sdc1” to hopefully fix your partition. Once completed mount the drive and more-than-likely all your data will be in the folder lost+found. It might have lost the initial folder name but at least your data is there, and with a little bit of play you can figure out which folder is which.

Now take a breather, relax and be happy that your data is not completely gone. I suggest in the future pulling up a source-based distribution like Slackware and try setting up an entire system without using any package management. See how it goes, prepare to read a lot of documentation but in the end you will be thankful as you will learn more about Linux this way than any other method.

Leave a Comment