Thursday, March 15, 2007

Greater than 4GB files on an External USB Hard Drive

Several months ago, I purchased a Western Digital USB/Firewire external hard drive to backup my laptop's home directory. Considering that I was using it with Linux, specifically Fedora Core 6, I wasn't sure how things were going to work.

After plugging it in, and attaching it via the USB 2.0 cable, it mounted and was presented on the GNOME desktop, and I could browse the contents of the disk without issue. Trying to keep things simple, I merely used tar and created a gzipped tar of my home directory, making sure to preserve all the permissions of the files with the following command:

tar -czpf /tmp/[file name with date].tar.gz /home/[my home directory]

This works quite well, but it presented me with my very first issue. My home directory is quite large, and the very first tar file I created was larger than 4 GB, so I couldn't write it to the external drive. It couldn't be written for the simple reason that the drive was using the FAT file system, and it doesn't support file sizes larger than the 32-bit maximum of 4 GB.

So, I looked through my home directory, and I found some obvious culprits to my size problem, and deleted those files, because I no longer needed them. Mostly it was old ISO images, that I had burned to CD long ago, and didn't need anymore. Okay, problem solved right?

Well, not quite. This worked for several months, but I was still dangerously close to the 4 GB limit. Eventually I spilled over the limit, and really couldn't delete files to get back under it.

With this in mind, I decided to see if I could change the file system to one that supported files larger than 4 GB. Considering that I am only using this with Linux, cross platform compatibility was not an issue for me, so the obvious choice was to use the ext3 file system from Linux. This would give me the large file support I needed, and also be more reliable, as ext3 is more robust than FAT, and it supports journaling, so there is significantly less risk to losing data.

During my investigation of making this change, I found nothing but individuals having problems with trying to do this. Many individuals had even rendered their drives unusable. Considering this, I took a step back and wondered whether I should try this, or see if I could think of another resolution.

I really couldn't think of a better way to deal with this problem, and I wanted to keep things simple, so I went ahead and tried to make the file system change, and here is the procedure I used.

  • First, I moved all my backups of my home directory that were currently on the drive, and copied them to /tmp on my laptop.
  • Second, I fired up GParted, considering that it is a graphical partitioning tool, that also will format partitions. This proved to be an excellent choice, because it helped me to avoid one pitfall.
    • Considering that the drive was plugged into the USB port, and mounted under /media/My Book, GParted would not let me format the drive until I unmounted it.
    • I used GParted to unmount the drive, and then I selected from the menu "Format to->ext3".
    • I watched as it automatically changed the partition type to the correct one, and then formatted the partition with the ext3 file system.
    • It completed with no issues, but here is where one of the problems reared its ugly head.
      • After formatting, the drive would no longer auto mount, and show itself on the desktop. I could manually mount it with the mount command, and it was working. I even wrote some files to it just to make sure everything was fine, and it was.
      • The guys on the Fedora Core mailing list were most helpful with this problem.
      • As it turns out, I needed to label the new file system with the e2label utility, which I did with the following command:
        • e2label /dev/sdc1 "My Book"
  • Finally, I moved the backups I put in /tmp back to the drive with the new file system.
After, these simple steps, I had a newly formatted external USB hard drive that I could write files larger than 4 GB to, without issues. It would auto mount, just the way it did when it was a FAT file system, and I now have some very large backups on it, and didn't have to change my very simple backup procedure.

Tuesday, March 13, 2007

Glimmer of Hope for Desktop Linux?

In the last week or so, I have read three different articles that talked about different government agencies that are banning Microsoft's Vista operating system, along with other Microsoft products in some cases. The National Institute of Standards and Technology (NIST) is the latest, and this follows the US Department of Transportation (DOT) and the Federal Aviation Administration (FAA).

So, what makes this a glimmer of hope for desktop Linux. Well, at least in one of those cases, the FAA, is seriously looking at a combination of Linux desktops with Google's new enterprise applications as a replacement for Windows and Microsoft Office! When you combine this type of interest with other government initiatives to adopt open standard file formats, you can see a glimmer of hope that the Microsoft lock is being broken by some large government agencies.

You could say, so what! It's only some public sector organizations! What makes this a glimmer of hope, in my mind, is the carry over affect it could have on the private sector.

If enough government agencies start adopting open technologies like Linux and ODF, then the private sector companies that have to do business with them will have to adopt technologies that inter-operate. This in turn loosens the grip that Microsoft has on a larger portion of the market.

I sincerely hope that these government organizations aren't just bluffing to get concessions out of Microsoft. With large scale adoption of open technologies, such as Linux and ODF, we will all be better off. True competition on the market for desktop operating systems and applications could become a reality someday.