Forget the whole “Mac or PC” thing. Your household does both. Or maybe you’re nerdy (like me) and run Linux alongside the other two. You have a grand plan of using an external hard drive (or flash drive) to transfer/backup files between all of them. Is that even possible?
The answer is “yes”, and it doesn’t even depend on what hard/flash drive you have. Most drives come pre-formatted in a certain file system. Many folks don’t realize that *any* drive can be reformatted in *any* format you choose. Why not choose one that supports multiple OSes natively?
Survey of Native OS Support
A quick survey of Wikipedia’s Comparison of File Systems reveals that only a small number of file systems have native support for Windows, OS X, *and* Linux. Really, only two:
Let’s take a deeper look at these two.
FAT32 vs. UDF
The following table puts various differences between FAT32 and UDF side-by-side. Most of the data came from Wikipedia (as retrieved on 2015-Feb-20):
There are two others that are worth mentioning, along with why they didn’t quite make the cut:
- NTFS – In OS X, the kernel only has partial read-only support. Read/write support can be gained with third-party NTFS-3G.
- exFAT – In Linux, support requires third party driver.
UDF Details
In many regards, UDF seems to be superior to FAT. (This isn’t surprising, as UDF was designed to be a good open successor of FAT.)
Here is a more full listing of UDF’s features:
- UDF is an open standard.
- The design and evolution of UDF keeps compatibility in mind.
- UDF natively supports many modern file systems features:
- Large partition size (maximum 2 TiB with 512 B block size, or 16 TiB with 4 KiB block size)
- 64-bit file size
- Extended attributes (e.g., named streams, or forks) without size limitation
- Long file names (maximum 255 bytes, any character can appear in the name)
- Unicode encoding of file names
- Sparse file
- Hard links
- Symbolic links
- Metadata checksum
- UDF defines how different platforms interact with each other. For example, it defines how to store Mac Finder Info and Resource Fork, NTFS ACL, UNIX ACL, OS/2 EA, etc. It also requires platforms to preserve the information that they don’t understand.
- UDF is a truly universal file system. It can be used on all kinds of optical media, including read only (CD-ROM, DVD-ROM, BD-ROM (Blu-ray Disc Read-Only)), write once (CD-R, DVD-R, DVD+R, BD-R), rewritable (CD-RW, DVD-RW, DVD+RW, DVD-RAM, CD-MRW, DVD+MRW, BD-RE), and of course block device (hard drives). Even write-once media appears as a big overwritable floppy under UDF.
OS Support
UDF offers moderately wide support across many operating systems. See Wikipedia for current data, or this page for a good summary.
Formatting UDF
A few different (OS-specific) tools exist for formatting UDF:
However, each of these methods comes with deficiencies:
- Using Windows’ format method means that the resultant UDF drive cannot be recognized on OS X (at least as of OS X 10.10).
- Merely using newfs_udf on OS X means that the resultant UDF hard drive cannot be recognized on Windows (at least as of Windows 7).
- Merely using mkudffs on Linux means that the resultant UDF hard drive cannot be recognized on Windows (at least as of Windows 7).
Why can’t Windows recognize properly formatted UDF hard drives?
A Fake Partition Table to Fake Out Windows
As mentioned by Pieter here, Windows does not support hard disks without a partition table. This is strange because Windows does not apply the same limitation to flash drives.
To make matters worse, OS X only uses UDF disks that utilize the full disk (not just a partition).
The solution, as suggested by Pieter, is to place a fake partition table (via MBR) in the first block of the drive, which lists a single entire-disk partition. This works because UDF (perhaps intentionally) doesn’t utilize the first block. Unfortunately, until now, there has been no easy way to do this, while juggling all of the other variables (such as device type and device logical block size).
One-Stop Format Tool: format-udf
Great news! There is now an easy, reliable way to format your hard/flash drive in UDF. On OS X, Linux, or any environment with Bash, you can use the open source format-udf as a one-stop UDF formatting tool. Regardless of whether your block device is a hard drive or flash drive, and regardless of its logical block size, format-udf will produce a UDF drive that can be recognized automatically on Windows, OS X, and Linux.
Pieter posted a Perl script in 2010 that aimed to accomplish a similar purpose, but the script isn’t compatible with modern versions of Perl. Additionally, the Perl script makes a few incorrect assumptions (e.g. logical block size) and has a some mathematical/logical bugs. format-udf represents an improvement on Pieter’s Perl script, in the following ways:
- Auto-detects and uses the device’s logical block size (it’s not always 512 bytes anymore)
- Corrects some bugs in the math/logic for the MBR format (as well as partition table entries)
- Does not require (any version of) Perl
- Works in any Bash environment
- Has more robust error handling
- Presents a safety check for user verification
Since format-udf assumes a Bash environment, the tool does not run on Windows. Presumably, if you’re desiring to use a drive with multiple operating systems, this shouldn’t be a show-stopper, because you have multiple operating systems available. :)
Summary
Indeed, it is possible to have a portable disk which:
- Mounts automatically read-write in Windows, OS X, and Linux
- Supports large files, UNIX permissions, Unicode filenames, symlinks, hardlinks, et al.
- Is able to be formatted using a single tool, format-udf
Happy formatting!
Readme 