Greater than 4GB files on an External USB Hard Drive

Several months ago, I purchased a Western Digital USB/Firewire external hard drive to backup my laptop's home directory. Considering that I was using it with Linux, specifically Fedora Core 6, I wasn't sure how things were going to work.

After plugging it in, and attaching it via the USB 2.0 cable, it mounted and was presented on the GNOME desktop, and I could browse the contents of the disk without issue. Trying to keep things simple, I merely used tar and created a gzipped tar of my home directory, making sure to preserve all the permissions of the files with the following command:

tar -czpf /tmp/[file name with date].tar.gz /home/[my home directory]

This works quite well, but it presented me with my very first issue. My home directory is quite large, and the very first tar file I created was larger than 4 GB, so I couldn't write it to the external drive. It couldn't be written for the simple reason that the drive was using the FAT file system, and it doesn't support file sizes larger than the 32-bit maximum of 4 GB.

So, I looked through my home directory, and I found some obvious culprits to my size problem, and deleted those files, because I no longer needed them. Mostly it was old ISO images, that I had burned to CD long ago, and didn't need anymore. Okay, problem solved right?

Well, not quite. This worked for several months, but I was still dangerously close to the 4 GB limit. Eventually I spilled over the limit, and really couldn't delete files to get back under it.

With this in mind, I decided to see if I could change the file system to one that supported files larger than 4 GB. Considering that I am only using this with Linux, cross platform compatibility was not an issue for me, so the obvious choice was to use the ext3 file system from Linux. This would give me the large file support I needed, and also be more reliable, as ext3 is more robust than FAT, and it supports journaling, so there is significantly less risk to losing data.

During my investigation of making this change, I found nothing but individuals having problems with trying to do this. Many individuals had even rendered their drives unusable. Considering this, I took a step back and wondered whether I should try this, or see if I could think of another resolution.

I really couldn't think of a better way to deal with this problem, and I wanted to keep things simple, so I went ahead and tried to make the file system change, and here is the procedure I used.

• First, I moved all my backups of my home directory that were currently on the drive, and copied them to /tmp on my laptop.
• Second, I fired up GParted, considering that it is a graphical partitioning tool, that also will format partitions. This proved to be an excellent choice, because it helped me to avoid one pitfall.
• Considering that the drive was plugged into the USB port, and mounted under /media/My Book, GParted would not let me format the drive until I unmounted it.
• I used GParted to unmount the drive, and then I selected from the menu "Format to->ext3".
• I watched as it automatically changed the partition type to the correct one, and then formatted the partition with the ext3 file system.
• It completed with no issues, but here is where one of the problems reared its ugly head.
• After formatting, the drive would no longer auto mount, and show itself on the desktop. I could manually mount it with the mount command, and it was working. I even wrote some files to it just to make sure everything was fine, and it was.
• The guys on the Fedora Core mailing list were most helpful with this problem.
• As it turns out, I needed to label the new file system with the e2label utility, which I did with the following command:
• e2label /dev/sdc1 "My Book"
• Finally, I moved the backups I put in /tmp back to the drive with the new file system.
  

After, these simple steps, I had a newly formatted external USB hard drive that I could write files larger than 4 GB to, without issues. It would auto mount, just the way it did when it was a FAT file system, and I now have some very large backups on it, and didn't have to change my very simple backup procedure.