The Book

The Book — Extras

Software

Meta

About

elsewhere

A Debugger for HFS Plus Volumes

© Amit Singh. All Rights Reserved. Written in May 2004

hfsdebug

A debugger is a program that facilitates debugging, which may be casually defined as the process of finding and fixing "bugs" in the object of interest: usually a piece of software, firmware, or hardware (although debugging may apply to any domain).

hfsdebug is a filesystem debugger for Apple's HFS Plus (hereafter referred to as HFS+) volume format. The term "debugger" may be somewhat of a misnomer for hfsdebug, because a key feature (rather, a limitation) of it is that it operates on an HFS+ volume only in read-only mode. Therefore, it cannot currently be used by itself to repair any defects in a volume, or to even write to a volume in any manner whatsoever. Nevertheless, hfsdebug is meant to be a useful tool in exploring HFS+ internals, as it allows you to browse, inspect, and analyze various aspects of HFS+ volumes. It can also calculate and display certain filesystem statistics. My original intent in creating hfsdebug was to quantify filesystem fragmentation under HFS+.

Volume Formats Supported

hfsdebug supports only the HFS+ volume format. The older HFS format is not supported. It does, however, support the following HFS+ variants:

• HFS Plus
• Journaled HFS Plus
• HFS Plus (journaled or otherwise) embedded within an HFS wrapper
• HFSX

Note that hfsdebug should work with HFS+ volumes regardless of whether they reside on a real, physical disk (or disk slice), or on a "virtual" disk (such as a disk image).

Architecture

hfsdebug aims to provide accurate and complete details of various HFS+ internal structures, as well as filesystem statistics, most of which are not available through an API such as POSIX (or even the Carbon File Manager, which is far more flexible than POSIX when dealing with HFS+ volumes). hfsdebug works on an HFS+ volume by accessing its raw device directly. This means several things:

• You may need superuser access to use hfsdebug on certain volumes, in particular, the root volume on a Mac OS X system.
• You can use hfsdebug even on a mounted (live) HFS+ volume, in particular, the root volume on a running Mac OS X system.
• Since hfsdebug does not access the volume's data through a block device, or some higher level API, its operation does not trash the system's buffer cache (it wouldn't be a big deal in any case, because hfsdebug is read-only).
• Although hfsdebug is written on, and for Mac OS X, it does not use any Mac OS X specific features. It is a C program that uses the POSIX API, and should be readily portable to any Unix-like operating system.

Features

You can use hfsdebug to:

• View the Volume Header and the Master Directory Block (in case of "embedded" HFS+ volumes).
• View contents of the header nodes of HFS+ "special files". These files include the Catalog, Overflow Extents, and Hot Files B-Trees.
• List and view details of one or more types of records contained in the special file B-trees, such as:
  • File records, file thread records, folder records, and folder thread records (the Catalog B-Tree).
  • Extent records (the Extents Overflow B-Tree).
  • Hot file records and hot file thread records (the Hot File Clustering B-Tree).
• View detailed information about filesystem objects, such as files, folders, aliases, symbolic links, and hard links. An object may be looked up using its Catalog Node ID (typically, but not always, the same as the inode number reported by the POSIX API), a Carbon style specification (the object's "node name" and the Catalog Node ID of its parent), or using its POSIX path.
• View detailed information contained in a volume's journal, if any.
• Calculate and display volume statistics, such as:
  • Top N files ordered by size.
  • Top N files ordered by their degree of fragmentation.
  • Top N files ordered by how "hot" they are (in the context of Hot File Clustering).
  • A summary of the number of various kinds of filesystem objects present on a volume, their space usage, special cases such as invisible and empty files, and so on.
• Display details of all fragmented files on a volume.
• Display location and size of all free extents on a volume.

Usage

Specifying a volume

hfsdebug needs an HFS+ volume to work on. This volume may be specified in three ways:

• Implicitly: if no volume is explicitly specified on the command line, hfsdebug will attempt to determine the "appropriate" volume. If an option requires a filesystem object as an argument, then hfsdebug automatically uses the volume containing that object. If no volume is specified, and no option is present from which the volume can be deduced, hfsdebug uses the root volume. Thus, both the following commands will result in the root volume being used:

# hfsdebug /mach_kernel OPTIONS ... # hfsdebug OPTIONS ...

• Using the -d option which takes a string naming the volume's device. For active (mounted) volumes, this must be the raw device, but may be the block device for unmounted volumes.

# hfsdebug -d /dev/rdisk0s9 OPTIONS ...

• Using the -V option which takes a pathname to the volume's mount point, or any filesystem object within it.

# hfsdebug -V /Volumes/Music OPTIONS ... ... # hfsdebug -V /Volumes/Music/somefile.mp3 OPTIONS ...

Viewing the Volume Header

The following command displays the volume header of the specified device (in this example, the root device). If the volume has an HFS wrapper, the master directory block is also displayed. This option is similar to "hdiutil hfsanalyze", except that hfsdebug shows some additional details, while not showing the "alternate" structure(s).

# hfsdebug -v # HFS Plus Volume with HFS Wrapper Embedded offset = 11984 bytes Wrapper volume size = 78149032.00 KB/76317.41 MB/74.53 GB Embedded volume size = 78143052.00 KB/76311.57 MB/74.52 GB # HFS Wrapper Master Directory Block drSigWord = $4244 (BD) ... # HFS Plus Volume Header signature = 0x482b (H+) ... # Finder Info finderInfo[0] = 0x1077 (Macintosh HD:/System/Library/CoreServices) finderInfo[1] = 0 finderInfo[2] = 0 finderInfo[3] = 0x30894 (Macintosh HD:/System Folder) ... # Catalog File logicalSize = 251658240 bytes totalBlocks = 61440 fork temperature = /* Metadata Zone */ clumpSize = 8388608 bytes extents = startBlock blockCount % of file 0xc01 0x5800 36.67 % 0xed8c 0x800 3.33 % 0x1e70a 0x800 3.33 % 0x2add4 0x1000 6.67 % 0x4d4b4 0x800 3.33 % 0x55c39 0x800 3.33 % 0xdddb2 0x800 3.33 % 0x140b0e 0x800 3.33 % 38912 allocation blocks in 8 extents total. 4864.00 allocation blocks per extent on an average. ...

Viewing headers of the Volume B-Trees

HFS+ uses several B-Trees for storing metadata. hfsdebug allows you to view the contents of the header nodes of these data structures:

# hfsdebug -b catalog ... # hfsdebug -b extent ... # hfsdebug -b hotfile # HFS+ Hot File Clustering (HFC) B-Tree # B-Tree Node Descriptor fLink = 0 bLink = 0 kind = 1 (kBTHeaderNode) height = 0 numRecords = 3 reserved = 0 # B-Tree Header Record treeDepth = 2 rootNode = 3 leafRecords = 21244 firstLeafNode = 88 lastLeafNode = 1 nodeSize = 4096 bytes maxKeyLength = 10 bytes totalNodes = 144 freeNodes = 28 reserved1 = 0 clumpSize = 65536 (ignored) btreeType = 128 (kUserBTreeType) keyCompareType = 0 (unspecified/default) attributes = 00000000000000000000000000000010 . kBTBigKeys (keyLength is UInt16) # User Data Record magic = 0XFF28FF26 version = 1 duration = 216000 seconds timebase = Sun May 2 09:02:27 2004 timeleft = 79068 seconds threshold = 16 maxfileblks = 2560 blocks maxfilecnt = 1000 tag = CLUSTERED HOT FILES B-TREE

Viewing records contained in the Volume B-Trees

hfsdebug lets you list details of all leaf records residing in the Volume B-Trees. Since some trees have more than one type of leaf records, you may also specify the type(s) of records to list:

# hfsdebug -b catalog -l file ... # hfsdebug -b catalog -l file -l folder ... # hfsdebug -b catalog -l filethread ... # hfsdebug -b hotfile -l hfcfile ... # hfsdebug -b hotfile -l hfcthread ... # hfsdebug -b extent -l any ...

The "any" keyword causes all records to be listed for the specified B-Tree. Note that this may result in a very large amount of output. For example, running "hfsdebug -b catalog -l any" produces almost a Gigabyte of data for a volume with about 600,000 files.

Viewing details of filesystem objects

hfsdebug can be used to view details of files, folders, and other kinds of objects on HFS+ volumes, including those that are not exposed by the filesystem to the user (for example, indirect node files, hard link files, journal files, etc.) You may specify an object in three ways:

1. Its POSIX visible path (specified implicitly as the last argument, or using the -p option)

   # hfsdebug /usr/bin/gcc ... # hfsdebug -p /bin/ls ...

2. Its Catalog Node ID (specified using the -c option)

   # hfsdebug -c 2 ...

3. Its name and the Catalog Node ID of its parent (specified using the -F option)

   # hfsdebug -F 2:.journal ...

The third way is needed when an object's POSIX path is not visible at the user-level, and you don't know its Catalog Node ID: for example, the journal files.

# hfsdebug /System/Library/CoreServices/BootX path = Macintosh HD:/System/Library/CoreServices/BootX # Catalog File Record type = file file ID = 186444 flags = 0000000000000010 . File has a thread record in the catalog. reserved1 = 0 createDate = Mon Oct 6 19:28:33 2003 contentModDate = Fri Mar 26 04:23:24 2004 attributeModDate = Fri Mar 26 04:23:24 2004 accessDate = Mon Oct 6 20:08:00 2003 backupDate = Fri Jan 1 00:00:00 1904 # BSD Info ownerID = 0 (root) groupID = 0 (wheel) adminFlags = 00000000 ownerFlags = 00000000 fileMode = -rw-r--r-- linkCount = 0 textEncoding = 0 reserved2 = 0 # Finder Info fdType = 0x74627869 (tbxi) fdCreator = 0x63687270 (chrp) fdFlags = 0000000000000000 fdLocation = (v = 0, h = 0) opaque = 0 # Data Fork logicalSize = 157892 bytes totalBlocks = 39 fork temperature = not a hot file clumpSize = 0 bytes extents = startBlock blockCount % of file 0x4bfd6e 0x27 100.00 % 39 allocation blocks in 1 extents total. 39.00 allocation blocks per extent on an average. # Resource Fork logicalSize = 0 bytes

Note that as would be the correct behavior for a "debugger" of this kind, hfsdebug does not follow hard links and symbolic links automatically. It does, however, display the link target of a hard or symbolic link.

Viewing a journaled volume's journal

hfsdebug can be used to view information about, and dump the contents of, a volume's journal.

# hfsdebug -j # HFS+ Journal # Journal Info Block flags = 00000000000000000000000000000001 . Journal resides on local volume itself. device_signature = 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 offset = 104861696 bytes size = 8388608 bytes reserved = 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 # Journal Header magic = 0x4a4e4c78 endian = 0x12345678 start = 5090816 bytes end = 3207680 bytes size = 8388608 bytes blhdr_size = 12288 bytes checksum = 0x802a54c0 jhdr_size = 512 bytes # Block List Header max_blocks = 767 num_blocks = 2 bytes_used = 20480 checksum = 0 pad = 0 binfo[0].bp = 0 block_info[ 1] { bnum 0000000000a0b010 bsize 8192 bytes bp 0x17d5477c } ... output continues ...

Viewing the "hottest" files on a volume

We saw that you can list hot file records using "hfsdebug -b hotfile -l ...". You can also use hfsdebug to list the top N hottest files, as tracked and determined by Apple's Hot File Clustering scheme, that is available on Mac OS X 10.3.x boot volumes:

# hfsdebug -H -t 10 # Top 10 Hottest Files on the Volume rank temperature cnid path 1 469 9165 Macintosh HD:/private/var/db/netinfo/local.nidb/Store.128 2 372 237719 Macintosh HD:/usr/libexec/gcc/darwin/ppc/3.3/specs 3 359 1047016 Macintosh HD:/private/var/vm/app_profile/0_names 4 337 238241 Macintosh HD:/usr/libexec/gcc/darwin/ppc/3.3-fast/specs 5 305 9133 Macintosh HD:/usr/share/zoneinfo/US/Pacific 6 305 9166 Macintosh HD:/private/var/db/netinfo/local.nidb/Store.160 7 282 250281 Macintosh HD:/usr/include/ppc/types.h 8 282 250287 Macintosh HD:/usr/include/sys/appleapiopts.h 9 282 250295 Macintosh HD:/usr/include/sys/cdefs.h 10 282 250310 Macintosh HD:/usr/include/sys/fcntl.h 610 active Hot Files.

Viewing volume usage summary

hfsdebug can be used to calculate and display various usage statistics about a volume. It can optionally be instructed to list the top N largest sized files on the volume.

# Volume Summary Information files = 563142 folders = 89762 aliases = 4 hard links = 4394 symbolic links = 9856 invisible files = 602 empty files = 5269 # Data Forks non-zero data forks = 556267 fragmented data forks = 408 allocation blocks used = 11629019 allocated storage = 47632461824 bytes (46516076.00 KB/45425.86 MB/44.36 GB) actual usage = 46214263456 bytes (45131116.66 KB/44073.36 MB/43.04 GB) total extent records = 556319 total extent descriptors = 557328 overflow extent records = 52 overflow extent descriptors = 347 # Resource Forks non-zero resource forks = 7570 fragmented resource forks = 18 allocation blocks used = 93739 allocated storage = 383954944 bytes (374956.00 KB/366.17 MB/0.36 GB) actual usage = 363034266 bytes (354525.65 KB/346.22 MB/0.34 GB) total extent records = 7570 total extent descriptors = 7598 overflow extent records = 0 overflow extent descriptors = 0 # Largest Files on the Volume # Using 1 KB = 1024 bytes, 1 MB = 1024 KB, 1 GB = 1024 MB rank size cnid path 1 700.20 MB 988570 Macintosh HD:/work/Linux/KNOPPIX.toast ... output continues ...

Quantifying fragmentation on a volume

hfsdebug can be used to quantify how fragmented an HFS+ volume is, which was my original reason for creating it.

The -f option causes hfsdebug to list all files (forks, to be precise) that have non-zero fragmentation. Thus, any data or resource fork that occupies more than one extent is listed, along with the number and size of each of its fragments. Moreover, you can instruct hfsdebug to create a list of the top N files with the most extents.

Refer to Fragmentation in HFS+ Volumes for a more detailed discussion on this issue.

# hfsdebug -f -t 5 ... cnid=2364913 fork=data map=:1728:773: bytes=10241935 blocks=2501 extents=2 \ avg=1250.50 blks/ext path=Macintosh HD:/Applications/3rd Party/Microsoft \ Office 2004/Shared Applications/Proofing Tools/German Grammar Dictionary cnid=2364937 fork=data map=:192:168: bytes=1473424 blocks=360 extents=2 \ avg=180.00 blks/ext path=Macintosh HD:/Applications/3rd Party/Microsoft \ Office 2004/Shared Applications/Proofing Tools/Norwegian NYN Dictionary cnid=2364952 fork=data map=:240:333: bytes=2346969 blocks=573 extents=2 \ avg=286.50 blks/ext path=Macintosh HD:/Applications/3rd Party/Microsoft \ Office 2004/Shared Applications/Proofing Tools/Portuguese Thesaurus Dictionary cnid=2365831 fork=data map=:33:5:97:1:30:67:3:1:2:1:7:2:3:7:1:5:36: \ bytes=1229704 blocks=301 extents=17 avg=17.71 blks/ext path=Macintosh \ HD:/private/var/vm/app_profile/0_data ... # Top 10 Files with the Most Extents on the Volume rank extents blk/extents cnid path 1 694 28.43 1881709 Macintosh HD:/sw/var/mysql/lxr/useage.MYI 2 371 71.37 1881710 Macintosh HD:/sw/var/mysql/lxr/useage.MYD 3 281 104.16 1881707 Macintosh HD:/sw/var/mysql/lxr/symbols.MYD 4 267 61.36 2369116 Macintosh HD:/private/tmp/hfs+.dmg 5 104 78.33 45746 Macintosh HD:/System/Library/Caches/fontTablesAnnex Out of 527454 data forks total, 526428 (99.81 %) have no fragmentation. Out of 8119 resource forks total, 8081 (99.53 %) have no fragmentation.

Qualifying All Free Space

hfsdebug can go through the HFS+ Allocation File and enumerate all free extents: an indicator of "free space fragmentation", if you will.

The -0 option will print a list of triplets for the volume, with each triplet consisting of the number of free contiguous extents, the block on the volume at which the extent starts, and the amount of space it represents. For example:

# hfsdebug -0 # Free Contiguous Starting @ Space 128928 68520 503.62 MB 1 197456 4.00 KB 1 197462 4.00 KB 1 197920 4.00 KB 1 198191 4.00 KB ... 15412 12819025 60.20 MB 6701325 12834438 25.56 GB Allocation block size = 4096 bytes Allocation blocks total = 19535763 (0x12a1793) Allocation blocks free = 8163575 (0x7c90f7)

Note that the free extents list is printed in block order: it is not sorted by the amount of free space the extents represent. You can obtain a sorted listed by piping the output through sort:

# hfsdebug -0 | sort -n ... 31850 12611790 124.41 MB 45861 12689954 179.14 MB 81709 213206 319.18 MB 128928 68520 503.62 MB 6701325 12834438 25.56 GB

This way, you can list the free extents with the biggest ones last. In this example, we can see that the largest amount of contiguous free space on the volume is 25.56 GB.

Further Reading

I add new features to hfsdebug every once in a while. To know what more you can do with hfsdebug, take a look at the following links.

• Extending HFSDebug through Filters—You can extend hfsdebug by adding your own code to its processing pipeline.
• HFSDebug 4 and New HFS+ Features—hfsdebug 4 supports several new features including support for directory hard links, hard link chains, lookup-by-path on offline volumes, and many new built-in filters.
• A Note on Pathname Processing in HFSDebug

Summary of Options

hfsdebug: Command-line Options

Download

hfsdebug is obsolete and has been retired. Take a look at fileXray. You may also want to look at this comparison between fileXray and hfsdebug.

All contents of this site, unless otherwise noted, are ©1994-2014 Amit Singh. All Rights Reserved. Terms of Use