“K64″ is what Apple refers to as the 64-bit version of the kernel beginning with Snow Leopard. As an end user, you really should not worry about the bitness of the kernel. If your Apple computer is not booting into K64 by default, you don’t need it—unless, of course, you know that you need it. (Say, because you are a kernel developer or an otherwise system-level developer and want to test something against a 64-bit kernel.) In particular, the 32-bit kernel, which is the default on most existing x86-based Apple computers, runs 64-bit applications just fine. Therefore, as long as you have a 64-bit processor, your Snow Leopard installation is 64-bit from the typical end-user standpoint.

An easy way to tell if you are running a K64 kernel is to use the uname command-line program. The "x86_64" in the excerpt below means that we are running a 64-bit kernel. If the output showed "i386" instead, that would mean a 32-bit kernel.

$ uname -a
Darwin... root:xnu-1456.1.25~1/RELEASE_X86_64 x86_64

If you are averse to using command-line programs (Do you really care about a K64 kernel in that case?), you could instead launch the System Profiler application, either directly, or by clicking on "More Info…" in the "About This Mac" panel. In System Profiler, you can click on the "Software" section in the sidebar. There will be something about the presence or absence of 64-bit Kernel and Extensions. You could also launch the Activity Monitor application and look for kernel_task. The "Kind" column will say if the kernel task (and consequently the kernel) is 64-bit.

As alluded to earlier, a 64-bit processor is required to run a K64 kernel. To boot into K64, you could do one of several things:

• Press the 6 and the 4 keys simultaneously at power-on time. This indicates to the EFI boot loader (boot.efi) that you wish to boot a 64-bit kernel.
• Set the boot-args firmware variable, say, through the nvram command-line program. To boot K64, the specific command-line would be:
  

  $ sudo nvram boot-args="arch=x86_64"

• Edit /Library/Preferences/SystemConfiguration/com.apple.Boot.plist and add arch=x86_64 to the value of the Kernel Flags key. By default, this value is an empty string.
  

  $ cat /Library/Preferences/SystemConfiguration/com.apple.Boot.plist
  ...
  <dict>
  	<key>Kernel</key>
  	<string>mach_kernel</string>
  	<key>Kernel Flags</key>
  	<string>arch=x86_64</string>
  </dict>
  ...
  

Another way is to use the -setkernelbootarchitecture argument of the systemsetup(8) command-line program.

Additionally, you could tell the kernel to boot verbosely if you are interested in catching a 64-bit boot early on. Note that one of the early kernel messages is "64 bit mode enabled". This does not mean K64—it just means the kernel has identified the processor to be 64-bit and is going to use certain 64-bit features. In the case of a K64 boot, the message to look for is "Kernel is LP64".

Not so fast though.

Unfortunately, a 64-bit processor alone doesn’t suffice. Out of the box, boot.efi will not boot K64 even if you have a 64-bit processor and explicitly request K64 if at least one of the following is true.

1. The machine has 32-bit EFI.
2. The machine’s model is prohibited from booting K64 through a hardcoded list within the boot loader. (A cursory look suggests that the list excludes "non-Pro" machines.)

Both of these "limitations" are technically artificial, albeit to different degrees.

The first limitation actually does have merit and is arguably not all that artificial. Although a 32-bit EFI could launch a 64-bit kernel, the kernel, when running, would not be able to use firmware services. In particular, you wouldn’t have NVRAM. For kernel developers merely wanting to run a 64-bit kernel for testing and debugging, this may not be an issue, but it’s understandable why the limitation is in place.

The second limitation is annoying. As a developer, if you knowingly wish to boot into K64 to test something, you can’t on certain machines even though they are technically perfectly capable. I ran into this on a Unibody MacBook, which has 64-bit EFI but is not a "Pro" machine. Also, it’s ironic that you can, in fact, boot Snow Leopard into K64 on the very same computer when you run it as a guest operating system in a virtual machine.

If you really need to boot into K64 on such a machine with 64-bit EFI, you can—at your own peril—"fix" things within boot.efi by setting the appropriate bits in the hardcoded list of models. To ensure that we’re talking about the same multiarchitecture version of boot.efi, compare the SHA-1 checksum of that file.

$ shasum boot.efi
2fb9fc10e5b4bb06f62c38b01bd9836a433897f8    boot.efi

Then, change the 1 byte at the corresponding model-specific position in the following table to the corresponding new value. Rather than overwriting the original boot.efi, we will copy the original to a new file, say, boot-k64.efi, and edit the latter.

For the specific case of the MacBook, which is the only one I've actually tried, the before and after bytes will look like the following:

38 47 01 00  00 00 00 00  00 00 00 00  00 00 00 00

38 47 01 00  00 00 00 00  04 00 00 00  00 00 00 00

We'll place the boot-k64.efi file somewhere on the root volume—/System/Library/CoreServices/ is fine. Then, we need to reset volume bootability through the bless command-line program. Optionally, we can also set the ownership and user immutable flag on the file to "proper" values.

$ sudo cp boot-k64.efi /System/Library/CoreServices/
$ cd /System/Library/CoreServices/
$ sudo chown root:wheel boot-k64.efi
$ sudo chflags uchg boot-k64.efi
$ sudo bless --folder /System/Library/CoreServices \
  --file /System/Library/CoreServices/boot-k64.efi

Your mileage may vary depending on whether your installation has 64-bit versions of all necessary drivers for the model of your specific machine. Since I have not tried any other "excluded" machine besides a 64-bit MacBook, I don't know about other models. (Unavailability or instability of certain 64-bit drivers could be a plausible reason for these models to be excluded in the first place.)

If you do render your system unbootable, you can simply run bless again to restore volume bootability as it was before. That is, you can tell bless to use the original boot.efi. Of course, to do that, you'll need to either boot from a different volume (a system install disc would be fine), or be able to access and write to the unbootable volume from another computer.

$ sudo bless --folder /Volumes/BrokenMac/System/Library/CoreServices \
  --file /Volumes/BrokenMac/System/Library/CoreServices/boot.efi

The other day I came across this web page in which the author describes his experiment to create a tiny ELF executable that will run on Linux. The result: a 45-byte ELF executable that executes and returns a value. The executable is functionally equivalent to the one generated from compiling the following C program.

  /* tiny.c */
  int main(void) { return 42; }

Apparently recent Linux kernels do stricter checks on ELF executables, because of which the aforementioned 45-byte executable no longer works. A slightly larger, 64-byte version still works at the time of this writing.

Anyway, as far as tiny executables go, ELF on Linux is taken care of. It would be interesting to repeat a similar experiment for Mach-O executables on Mac OS X.

Let us first see how large the executable generated from the C program is on Mac OS X.

$ cat tiny.c
main() { return 42; }
$ sw_vers
...
ProductVersion: 10.5.6
...
$ gcc -Oz -o tiny tiny.c
$ strip tiny
$ ls -las tiny
32 -rwxr-xr-x  1 singh  wheel  12348 Mar 15 17:26 tiny

The following assembly language program can be compiled to generate a 165-byte Mach-O executable that runs on Mac OS X and returns the wisely chosen value 42.

; tiny.asm for Mac OS X (Mach-O Object File Format)
; nasm -f bin -o tiny tiny.asm

BITS 32
        org   0x1000

        db    0xce, 0xfa, 0xed, 0xfe       ; magic
        dd    7                            ; cputype (CPU_TYPE_X86)
        dd    3                            ; cpusubtype (CPU_SUBTYPE_I386_ALL)
        dd    2                            ; filetype (MH_EXECUTE)
        dd    2                            ; ncmds
        dd    _start - _cmds               ; cmdsize
        dd    0                            ; flags
_cmds:
        dd    1                            ; cmd (LC_SEGMENT)
        dd    44                           ; cmdsize
        db    "__TEXT"                     ; segname
        db    0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ; segname
        dd    0x1000                       ; vmaddr
        dd    0x1000                       ; vmsize
        dd    0                            ; fileoff
        dd    filesize                     ; filesize
        dd    7                            ; maxprot

        dd    5                            ; cmd (LC_UNIXTHREAD)
        dd    80                           ; cmdsize
        dd    1                            ; flvaor (i386_THREAD_STATE)
        dd    16                           ; count (i386_THREAD_STATE_COUNT)
        dd    0, 0, 0, 0, 0, 0, 0, 0       ; state
        dd    0, 0, _start, 0, 0, 0, 0, 0  ; state
_start:
        xor   eax,eax
        inc   eax
	push  byte 42
        sub   esp, 4
        int   0x80                         ; _exit(42)

filesize equ  $ - $$

We can compile and run this program as follows, verifying that its return value is indeed 42.

$ nasm -f bin -o tiny tiny.asm
$ ls -las tiny
8 -rw-r--r--  1 singh  admin  165 Mar 15 12:21 tiny
$ chmod 755 tiny
$ ./tiny
$ echo $?
42

Some points to note:

• The executable—Mach-O header, load commands, the text segment—is manually crafted in assembly using the nasm 80x86 assembler. The C compiler toolchain is not involved.
• The executable is unusual for Mac OS X in that the dynamc link editor (dyld) is not involved in running it. No dynamic libraries are involved either.
• The program makes a "direct" system call through the int 0x80 interface. This is a big no-no on Mac OS X—production code should not be bypassing the C library for making system calls, but then hopefully you won't be writing production code using such techniques. A specific caveat is that system call implementation may do things differently—in terms of arguments, return values, and such—from the user-callable interface. The implementation may also change across system revisions, so such code may break with a system update.
• The int 0x80 system call path is a legacy path on Mac OS X. It may be removed from Mac OS X some day, in which case the program would need to be modified to use newer alternatives such as sysenter.
• The executable is not a "correct" Mach-O file, even though the kernel can parse and run it in our case. The reason it's incorrect is because Mach-O load commands have been deliberately made to overlap each other to save some bytes. The otool object-file introspection command will be quite unhappy with this executable. In the following output, inconsistencies are shown in red.

$ otool -l tiny
tiny:
Load command 0
      cmd LC_SEGMENT
  cmdsize 44 Inconsistent size
  segname __TEXT
   vmaddr 0x00001000
   vmsize 0x00001000
  fileoff 0
 filesize 165
  maxprot 0x00000007
 initprot 0x00000005
   nsects 80
    flags 0x1
Section
  sectname
   segname  (does not match segment)
      addr 0x00000000
      size 0x00000000
    offset 0
     align 2^4248 (16777216)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
section structure command extends past end of load commands
Section
  sectname
   segname  (does not match segment)
      addr 0x00000000
      size 0x00000000
    offset 0
     align 2^0 (1)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0

At the cost of roughly 80 additional bytes, we can create a "more correct" Mach-O executable that otool will be happy with.

; nicertiny.asm for Mac OS X (Mach-O Object File Format)
; nasm -f bin -o nicertiny nicertiny.asm

BITS 32
        org   0x1000

        db    0xce, 0xfa, 0xed, 0xfe       ; magic
        dd    7                            ; cputype (CPU_TYPE_X86)
        dd    3                            ; cpusubtype (CPU_SUBTYPE_I386_ALL)
        dd    2                            ; filetype (MH_EXECUTE)
        dd    2                            ; ncmds
        dd    _start - _cmds               ; cmdsize
        dd    0                            ; flags
_cmds:
        dd    1                            ; cmd (LC_SEGMENT)
        dd    124                          ; cmdsize
        db    "__TEXT"                     ; segname
        db    0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ; segname
        dd    0x1000                       ; vmaddr
        dd    0x1000                       ; vmsize
        dd    0                            ; fileoff
        dd    filesize                     ; filesize
        dd    7                            ; maxprot
        dd    5                            ; initprot
        dd    1                            ; nsects
        dd    0                            ; flags
        db    "__text"                     ; sectname
        db    0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ; sectname
        db    "__TEXT"                     ; segname
        db    0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ; segname
        dd    _start                       ; addr
        dd    _end - _start                ; size;
        dd    _start - 0x1000              ; offset
        dd    2                            ; align
        dd    0                            ; reloff
        dd    0                            ; nreloc
        dd    0                            ; flags
        dd    0                            ; reserved1
        dd    0                            ; reserved2

        dd    5                            ; cmd (LC_UNIXTHREAD)
        dd    80                           ; cmdsize
        dd    1                            ; flavor (i386_THREAD_STATE)
        dd    16                           ; count (i386_THREAD_STATE_COUNT)
        dd    0, 0, 0, 0, 0, 0, 0, 0       ; state
        dd    0, 0, _start, 0, 0, 0, 0, 0  ; state
_start:
        xor   eax, eax
        inc   eax
        push  dword 42
        sub   esp, 4
        int   0x80                         ; _exit(42)
_end:
filesize equ  $ - $$

Let us compile and run this version, and see what otool has to say.

$ nasm -f bin -o nicertiny nicertiny.asm
$ ls -las nicertiny
8 -rw-r--r--  1 singh  admin  248 Mar 15 14:49 nicertiny
$ chmod 755 nicertiny
$ ./nicertiny
$ echo $?
42
$ otool -l nicertiny
nicertiny:
Load command 0
      cmd LC_SEGMENT
  cmdsize 124
  segname __TEXT
   vmaddr 0x00001000
   vmsize 0x00001000
  fileoff 0
 filesize 248
  maxprot 0x00000007
 initprot 0x00000005
   nsects 1
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x000010e8
      size 0x00000010
    offset 232
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 1
        cmd LC_UNIXTHREAD
    cmdsize 80
     flavor i386_THREAD_STATE
      count i386_THREAD_STATE_COUNT
	    eax 0x00000000 ebx    0x00000000 ecx 0x00000000 edx 0x00000000
	    edi 0x00000000 esi    0x00000000 ebp 0x00000000 esp 0x00000000
	    ss  0x00000000 eflags 0x00000000 eip 0x000010e8 cs  0x00000000
	    ds  0x00000000 es     0x00000000 fs  0x00000000 gs  0x00000000

It would be a nice exercise for the reader to try to shrink tiny.asm and nicertiny.asm even further, while retaining the high-level behavior of the corresponding executables. There are plenty of zeros lurking in there.

Suppose you have a Macintosh without a TPM. This, of course, is highly likely because only the first few x86-based Macintosh models had TPMs. Now suppose you really want to experiment with Trusted Computing or features of the TPM in general. Your needs could be development-related or they could be purely academic. Well, you can do the next best thing to having a real TPM…

I often hear users—and even developers, for that matter—grumbling about the fact that they are "required" to reboot their systems after installing or upgrading MacFUSE. I’ve even heard explanations that because MacFUSE "does something with the kernel," a reboot is necessary. Well, this whole rebooting-required thing is a myth. Lets clear up some misconceptions.

When you install the official MacFUSE package, you see the following rebooting-related notice.

To begin with, it is recommended (not required) that you restart your system. Unlike in the case of Mac OS X software installations that do necessitate a restart, you can simply close the Installer window after you are done installing MacFUSE.

Now, why is it even recommended that you restart? Let us get some context.

If you are installing MacFUSE for the first time (that is, you are not upgrading a MacFUSE installation), restarting is entirely unnecessary.

If you are upgrading a MacFUSE installation, restarting is a heavy-handed way for users to avoid the confusing situation described below—it is still not required to restart if you know how to avoid the confusing situation and are willing to deal with the extra steps needed.

MacFUSE consists of both kernel-resident and user-space-resident software components. The MacFUSE kernel extension is both dynamically loadable and dynamically unloadable. That is, you can simply unload it if it is not in use and reload a new version.

There is one important thing to know: the kernel-space and user-space components of MacFUSE are in lockstep. You cannot use an older user-space MacFUSE component with a newer MacFUSE kernel extension and vice versa. The aforementioned confusing situation arises when you upgrade MacFUSE while you have a MacFUSE-based file system mounted. The busy kernel extension cannot be unloaded, and the upgraded user-space components will refuse to work with the old kernel extension.

The user-space MacFUSE components actually ensure that the loaded kernel extension is the matching one. In fact, if you upgrade an installation while the MacFUSE kernel extension is loaded, the user-space components will try to unload the old extension and then try to load the new extension. This works, except in the case when the already loaded kernel extension is "busy," which would be the case if you do have a mounted MacFUSE-based file system.

If you first ensure that no MacFUSE-based file systems are mounted, you can upgrade MacFUSE, not restart, and once upgraded, start using the new MacFUSE. The lsvfs command-line tool can be used to determine if a MacFUSE-based file system is mounted.

$ lsvfs
Filesystem                        Refs Flags
-------------------------------- ----- ---------------
ufs                                  0 local
nfs                                  1
fdesc                                1
cd9660                               0 local
unionfs                              0
hfs                                  2 local, dovolfs
devfs                                1
autofs                               2
msdos                                0 local
fusefs                               0

If there is no entry for fusefs or if the corresponding value in the Refs column is 0, you shouldn’t have to do anything. If the value is non-zero, you need to unmount that many MacFUSE-based volumes.

To sum up, a restart is recommended so that end users don’t have to understand these details. Even then, the only time a restart will actually matter is the case when you upgrade MacFUSE while a MacFUSE-based file system is mounted and then you attempt to mount yet another MacFUSE-based file system using the upgraded version of MacFUSE.

The other day I needed to know within one of my experimental programs if the host x86 processor supports certain features. In many cases, the operating system provides interfaces that can answer such questions. Sometimes, the interfaces may not have the answer, or you may wish to avoid them for other reasons. (Say, you don’t wish to depend on anything operating-system-specific.) Of course, one can turn to the x86 processor’s venerable CPUID instruction. I took some code from the xnu kernel and made it into a user-space program that displays processor information. For academic needs, it is always good to have a program at hand that does things from first principles.

Here is the source for cpuinfo_x86.

The following is an example of the program in action. You must compile it as a 32-bit program. Besides Mac OS X, the program should also work on Linux, FreeBSD, and perhaps some other operating systems.

$ gcc -march=i386 -m32 -o cpuinfo_x86 cpuinfo_x86.c
$ ./cpuinfo_x86
# Identification
Vendor                : GenuineIntel
Brand String          : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
Model Number          : 26 (Nehalem)
Family Code           : 6
Extended Model        : 1
Extended Family       : 0
Stepping ID           : 5
Signature             : 67237

# Address Bits
Physical Addressing   : 40
Virtual Addressing    : 48

# Multi-Core Information
Logical Processors (Threads) per Physical Processor : 16
Cores per Physical Package                          : 8

# Caches
## L1 Instruction Cache
Size                  : 32K
Line Size             : 64B
Sharing               : shared between 2 processor threads
Sets                  : 128
Partitions            : 1
Associativity         : 4

## L1 Data Cache
Size                  : 32K
Line Size             : 64B
Sharing               : shared between 2 processor threads
Sets                  : 64
Partitions            : 1
Associativity         : 8

## L2 Unified Cache
Size                  : 256K
Line Size             : 64B
Sharing               : shared between 2 processor threads
Sets                  : 512
Partitions            : 1
Associativity         : 8

## L3 Unified Cache
Size                  : 8M
Line Size             : 64B
Sharing               : shared between 16 processor threads
Sets                  : 8192
Partitions            : 1
Associativity         : 16

# Translation Lookaside Buffers
Instruction TLBs      : 7 large, 0 small
Data TLBs             : 32 large, 64 small

# Features
ACPI                  : Thermal Monitor and Software Controlled Clock
APIC                  : On-Chip APIC Hardware
CLFSH                 : CLFLUSH Instruction
CMOV                  : Conditional Move Instruction
CX16                  : CMPXCHG16B Instruction
CX8                   : CMPXCHG8 Instruction
DCA                   : Direct Cache Access
DE                    : Debugging Extension
DS                    : Debug Store
DS-CPL                : CPL Qualified Debug Store
DTES64                : 64-Bit Debug Store
EST                   : Enhanced Intel SpeedStep Technology
FPU                   : Floating-Point Unit On-Chip
FXSR                  : FXSAVE and FXSTOR Instructions
HTT                   : HyperThreading
MCA                   : Machine-Check Architecture
MCE                   : Machine-Check Exception
MMX                   : MMX Technology
MONITOR               : MONITOR/MWAIT Instructions
MSR                   : Model Specific Registers
MTRR                  : Memory Type Range Registers
PAE                   : Physical Address Extension
PAT                   : Page Attribute Table
PBE                   : Pending Break Enable
PDCM                  : Perfmon and Debug Capability
PGE                   : Page Global Enable
POPCNT                : POPCNT Instruction
PSE                   : Page Size Extension
PSE-36                : 36-Bit Page Size Extension
SEP                   : Fast System Call
SS                    : Self-Snoop
SSE                   : Streaming SIMD Extensions
SSE2                  : Streaming SIMD Extensions 2
SSE3                  : Streaming SIMD Extensions 3
SSSE3                 : Supplemental Streaming SIMD Extensions 3
SSE4.1                : Streaming SIMD Extensions 4.1
SSE4.2                : Streaming SIMD Extensions 4.2
TM                    : Thermal Monitor
TM2                   : Thermal Monitor 2
TSC                   : Time Stamp Counter
VME                   : Virtual Mode Extension
VMX                   : Virtual Machine Extensions
xTPR                  : xTPR Update Control

# Extended Features
EM64T                 : Intel Extended Memory 64 Technology
XD                    : Execution Disable

Despite the detailed information cpuinfo_x86 shows, it is still does not retrieve all possible information you can get through CPUID. Moreover, because I wrote it for the x86 version of Mac OS X, the program assumes that you have a relatively recent x86 processor and will likely not behave too well on old processors. Writing an exhaustive and legacy-aware CPUID program is a more rigorous programming exercise.

The Apple Kernel Debug Kit comes with a kernel gdb macros file (kgmacros) that contains numerous macros useful during low-level development and analysis. One of the macros is showbootermemorymap, which dumps the physical memory map from EFI. The information in this map is very useful for certain types of development.

Since I am often mobile—without ready access to a two-machine debugging set up—I can’t use real kernel debugging much of the time. For introspection-only kernel "debugging", one can use the darwin-kernel gdb target on Mac OS X. It does require that you have a /dev/kmem device available. /dev/kmem is not enabled by default on Mac OS X, but you can enable it at boot time by including kmem=1 in the kernel’s boot arguments.

$ sudo nvram boot-args="-v kmem=1 ..."
$ sudo reboot
/* system reboots */
$ cd /path/to/kernel/debug/kit/for/current/system/
$ sudo gdb ./mach_kernel
...
(gdb) source kgmacros
Loading Kernel GDB Macros package.  Type "help kgm" for more info.
(gdb) target darwin-kernel
(gdb) attach
Connected.
(gdb) showbootermemorymap
Type       Physical Start   Number of Pages
available  0000000000000000 000000000000008f
ACPI_NVS   000000000008f000 0000000000000001
available  0000000000090000 0000000000000010
...
RT_data    0000000000a8c000 000000000000002b
RT_data    0000000000ab7000 0000000000000001
(gdb)

A more convenient alternative is to simply use DTrace to print this information. The following is a DTrace script that mimics the behavior of the showbootermemorymap kernel debug macro.

#! /usr/sbin/dtrace -s

/*
 * showbootermemorymap - DTrace script that prints out the physical memory
 *                       map from EFI. Mimics the output format of the
 *                       kernel debugging macro of the same name.
 *
 * Amit Singh
 * http://osxbook.com
 */

#pragma D option quiet

BEGIN
{
    self->inited = 1;

    self->kgm_boot_args = ((struct boot_args*)(`PE_state).bootArgs);
    self->kgm_msize = self->kgm_boot_args->MemoryMapDescriptorSize;
    self->kgm_mcount = self->kgm_boot_args->MemoryMapSize / self->kgm_msize;

    printf("Type       Physical Start   Number of Pages\n");

    self->kgm_i = 0;
}

fbt:::entry
/self->inited && self->kgm_i < self->kgm_mcount/
{
    this->kgm_mptr = (struct EfiMemoryRange*)
    ((unsigned long)self->kgm_boot_args->MemoryMap +
                    self->kgm_i * self->kgm_msize);

    self->kgm_i++;

    printf("%s", (this->kgm_mptr->Type == 0)  ? "reserved  " :
                 (this->kgm_mptr->Type == 1)  ? "LoaderCode" :
                 (this->kgm_mptr->Type == 2)  ? "LoaderData" :
                 (this->kgm_mptr->Type == 3)  ? "BS_code   " :
                 (this->kgm_mptr->Type == 4)  ? "BS_data   " :
                 (this->kgm_mptr->Type == 5)  ? "RT_code   " :
                 (this->kgm_mptr->Type == 6)  ? "RT_data   " :
                 (this->kgm_mptr->Type == 7)  ? "available " :
                 (this->kgm_mptr->Type == 8)  ? "Unusable  " :
                 (this->kgm_mptr->Type == 9)  ? "ACPI_recl " :
                 (this->kgm_mptr->Type == 10) ? "ACPI_NVS  " :
                 (this->kgm_mptr->Type == 11) ? "MemMapIO  " :
                 (this->kgm_mptr->Type == 12) ? "MemPortIO " :
                 (this->kgm_mptr->Type == 13) ? "PAL_code  " :
                                                "UNKNOWN   ");
    printf(" %016llx %016llx\n",
           this->kgm_mptr->PhysicalStart, this->kgm_mptr->NumberOfPages);
}

fbt:::return
/self->inited && self->kgm_i >= self->kgm_mcount/
{
    exit(0);
}

XBinary is software that lets you add kernel-level support for executing arbitrary binary formats on Mac OS X. To read more about it and to download it, visit the XBinary page.

By popular demand, I’ve "ported" AncientFS to Linux and FreeBSD. It was reasonably straightforward. Largely on purpose, AncientFS depends on the cross-platform interfaces of MacFUSE for the most part.

You should now be able to build the AncientFS source tree on Mac OS X, Linux, and FreeBSD. You must have the FUSE implementation for your platform installed to build and use AncientFS. Then, it should take a single make command to build.

$ svn co http://macfuse.googlecode.com/svn/trunk/filesystems/unixfs
$ cd unixfs/ancientfs
$ make # GNU make required. Use gmake on FreeBSD.
...

If it doesn’t build out-of-the-box on your system, you might want to tweak platform-specific settings in unixfs/ancientfs/Makefile first. The default settings assume that fuse is installed under /usr on Linux and under /usr/local on FreeBSD.

Because libfuse wants to use sem_init(), you will need to ensure on FreeBSD that the POSIX semaphore implementation is either statically compiled into the kernel (not the default on recent FreeBSD systems) or the sem kernel module is loaded. See sem(4) for details.

I’m traveling and don’t have good/much access to Linux or FreeBSD machines. Therefore, I haven’t been able to test this much. I do know that it builds and mounts some things on at least Linux 2.6.18 (Ubuntu) and FreeBSD 7.1-BETA2.

Here are some extended notes on understanding, compiling, and using AncientFS, the file system I talked about in the previous blog post.

There has been much buzz about new features and functionality in MacFUSE 2.0. Besides the MacFUSE 2.0 release, there are still more new and interesting things to be discussed in the realm of user-space file systems. As I described and briefly demonstrated during my recent talk at the Googleplex, I wrote several new user-space file systems to "celebrate" two years of MacFUSE.

Now that the talk’s video is available. I’ve written some notes on the new file systems themselves. There is a lot of new information and new code for those interested in file systems, Mac OS X, and operating systems in general. Much of it is academically and practically useful, and some could have significant potential down the road. You could also think of it as a new year gift to the Mac OS X and open-source communities.

This is a meta note on how you can go about discovering and understanding the what, how, and why of everything I’m talking about.

1. MacFUSE State of the Union Talk (2008) Video

If you were not at the talk, watching the video would provide useful context and should help greatly in understanding what follows.

2. AncientFS

Next, you could look at AncientFS, which I introduced, among other things, during the talk. AncientFS lets you mount ancient (and in some cases, current-day) "data containers" as regular volumes on Mac OS X. It supports the following formats.

• v1tap—DECtape tap tape archive; UNIX V1
• v2tap—DECtape tap tape archive; UNIX V2
• v3tap—DECtape tap tape archive; UNIX V3
• ntap—DECtape/magtape tap tape archive; 1970 epoch
• tp—DECtape/magtape tp tape archive
• itp—UNIX itp tape archive
• dtp—UNIX dtp tape archive
• dump—Incremental file system dump (512-byte blocks, V7/bsd)
• dump1kI—Incremental file system dump (1024-byte blocks, V7/bsd)
• dump-vn—Incremental file system dump (512-byte blocks, bsd-vn)
• dump1k-vn—Incremental file system dump (1024-byte blocks, bsd-vn)
• v1ar—Very old (0177555) archive (.a) from First Edition UNIX
• v2ar—Very old (0177555) archive (.a) from Second Edition UNIX
• v3ar—Very old (0177555) archive (.a) from Third Edition UNIX
• ar—Current (!<arch>\n), old (0177545), or very old (0177555) archive (.a); use (v1|v2|v3)ar for UNIX V1/V2/V3 archives
• bcpio—Binary cpio archive (old); may be byte-swapped
• cpio_odc—ASCII (odc) cpio archive
• cpio_newc—New ASCII (newc) cpio archive
• cpio_newcrc—New ASCII (newc) cpio archive with checksum
• tar—ustar, pre-POSIX ustar, or V7 tar archive
• v1—First Edition UNIX file system
• v2—Second Edition UNIX file system
• v3—Third Edition UNIX file system
• v4—Fourth Edition UNIX file system
• v5—Fifth Edition UNIX file system
• v6—Sixth Edition UNIX file system
• v7—Seventh Edition UNIX file system
• v10—Tenth Edition UNIX file system
• 32v—UNIX/32V file system
• bsd—BSD file system (V7-style with fixed-length file names; e.g. 2.9BSD or 4.0BSD)
• bsd-vn—BSD file system (pre fast-file-system “UFS” with variable-length file names; e.g. 2.11BSD for PDP-11)

To learn more about AncientFS and how to use it, please read the AncientFS article. Enjoy seeing ancient data seamlessly in the modern namespaces of Mac OS X!

3. The UnixFS Layer

AncientFS also led to UnixFS, a general-purpose abstraction layer that proved useful in getting several other "alien" file systems up and running on Mac OS X. It is particularly useful for "Unix-style" file systems, where you need concepts such as on-disk and in-memory superblocks and inodes. As I mentioned during the talk, I used UnixFS in conjunction with ad-hoc "Linux emulation" to bring support for the UFS, System V, and Minix file system families on Mac OS X. That amounts to a large number of new file systems, but it was easier than it sounds because the idea was to take existing Linux kernel implementations of these file system families and make them work in user space on Mac OS X!

UnixFS is currently rather "beta" and not a formal API by any means. Since it is a programming interface and doesn’t do anything by itself, it will be of interest only to developers at this point. In future, it may also evolve into a "LinuxFS" layer that could make it even easier and faster to systematically make Linux kernel-based file systems work on other platforms with very few code changes. For now, you can browse some bits of code.

You can, however, use the aforementioned UnixFS-based file system families: UFS, System V, and Minix. To do so, check out the relevant part of the MacFUSE source tree and compile one or more (or all) of the UnixFS-based file systems—it is quite straightforward: a single make in the filesystems/unixfs/ subdirectory in the MacFUSE source tree should build all of them. (You must have MacFUSE installed, of course.)

$ svn co http://macfuse.googlecode.com/svn/trunk/filesystems/unixfs
$ cd unixfs
$ ls -F
Makefile	common/		sysvfs/
ancientfs/	minixfs/	ufs/
$ make
...
$

4. The UFS Family

This is a user-space implementation (read-only) of the UFS file system family. Most of the UFS-specific code comes from the Linux kernel and is largely unchanged. Specific UFS flavors supported are as follows.

• old—the oldest UFS format
• sun—used in SunOS/Solaris
• sunx86—used in the x86 versions of SunOS/Solaris
• hp—used in HP-UX
• nextstep—used in NEXTSTEP
• nextstep-cd—used in NEXTSTEP CDROMs
• openstep—used in OPENSTEP
• 44bsd—used in FreeBSD, NetBSD, OpenBSD, and Mac OS X
• ufs2—used in FreeBSD 5.x

5. The System V Family

This is a user-space implementation (read-only) of the System V file system family. Most of the sysvfs-specific code comes from the Linux kernel and is largely unchanged. Specific sysvfs flavors supported are as follows.

• svr2—used in SVR2
• svr4—used in SVR4
• xenix—used in Xenix
• coherent—used in Coherent Unix

6. The Minix Family

This is a user-space implementation (read-only) of the Minix file system family. Most of the minixfs-specific code comes from the Linux kernel and is largely unchanged.

The following image summarizes the new file system capabilities Mac OS X gets as a result of these exercises.

7. The "One More Thing" Thing

Although it isn’t directly file system related, the last thing I demonstrated during the talk was the ability to run ancient Unix (PDP-11) binaries seamlessly on Mac OS X. PDP-11 aside, the techniques used to do so are generally useful for research and experimentation because unlike Linux, Mac OS X does not allow developers to extend the set of binary formats that can be "natively" executed by the kernel. The demo shows the Fifth Edition Unix kernel being compiled on Mac OS X using the original C compiler toolchain from a Fifth Edition disk image mounted using AncientFS. Additionally, there’s an "authentic" reproduction of the following ominous error message by running the original mv executable from Sixth Edition Unix.

	values of β will give rise to dom!

For more details, please watch the last section of the talk video. This is very preliminary work for which no source code or binaries are available yet.

Mac OS X, like many other Unix-like operating systems, includes the “autofs” file system layer that make automatic on-demand mounting of remote resources possible. See the man page for automount(8) for more details.

Such automatic mounting is orthogonal to and possible with MacFUSE. (NB: You will need MacFUSE 2.0 or above for this to work properly since older versions of MacFUSE filter out the “automounted” mount-time argument.) Consider sshfs, a user-space SFTP file system implementation that works with MacFUSE. The following is a quick-and-dirty example of how you could set up an autofs mount for sshfs. (There are other ways to set up autofs mounts.)

Create an /etc/fstab file (or add to an existing one) with the following entry. We will create what’s called a “static map” in autofs parlance.

$ cat /etc/fstab
dummy:user@host:/remotedir /Network/name sshfs volname=volname,allow_other 0 0

You will have to customize the green highlighted components. user, host, and remotedir are the SFTP user name, the SFTP server host name, and the remote directory on the SFTP server, respectively. You can choose some reasonable value for name and volname. The local mount point will be /Network/name.

Next, to keep things simple, configure key-based authentication to the SFTP server so you can log in without having to type your password.

The keyword “sshfs” in the /etc/fstab entry is the type of the file system. Given a file system type foo, the automounter will expect a mount_foo file-system-specific mounting program to exist. In our case, we don’t have a separate mounting program for sshfs. However, because of the format of the entry and how the automounter passes arguments to the mounting program, it will work if you simply copy the command-line sshfs program to /sbin/mount_sshfs. Alternatively, you can create a symbolic link as follows.

$ which sshfs
$ /usr/local/bin/sshfs
$ sudo ln -s /usr/local/bin/sshfs /sbin/mount_sshfs

That should be it. Run the automount program to update the state of things. The -c argument tells the automount daemon to flush any cached information.

$ sudo automount -c

If everything went well, the new mount should appear in the output of the mount command. In the following example, we used SSH as the name component of the mount point.

$ mount
...
map -static on /Network/SSH (autofs, automounted)

Now, if you simply access /Network/SSH, the SFTP file system should be automatically mounted.

$ ls /Network/SSH
Applications		Volumes			work
Desktop DB		bin			private
...

If there is an error in mounting (say, the remote host is not reachable), you will not be permitted to access the /Network/SSH directory.

$ ls /Network/SSH
ls: SSH: Operation not permitted

You can specify a timeout period after which an automounted file system will be unmounted if it has not been accessed within that period. Either use the -t argument of automount or see the /etc/autofs.conf file.

VeryBigFS is a trivial MacFUSE file system—about 60 lines of C code—that creates a huge volume with a huge file in it. “Huge” would be 512TB in this case.

This is useful if you want to see how a program will deal with unusually large files. Since it is extremely unlikely for the majority of us to be able to actually create files that are 512TB in size any time in the near future, faking is the way to go. Assuming you have MacFUSE installed, here is how you can try this file system.

$ ls
Makefile	verybigfs.c
$ make
...
$ mkdir "/Volumes/Very Big HD"
$ ./verybigfs "/Volumes/Very Big HD"
$ ls -lh
...
-r--r--r--  1 singh  wheel   512T Nov 28 05:38 copyme.txt

As an aside, note that HFS+ does not support sparse files. In fact, if you create a large scratch file on HFS+, it will be zero filled “soon”. This can take a long time depending upon the file size, the hardware, and the resources available. Mac OS X provides a way for a privileged process to set a file’s size without zero filling: it’s the F_SETSIZE command of the fcntl() system call.

Here is what the Finder tells you about the file copyme.txt inside the volume.

Note that although the tantalizingly named file is fictitious in that it doesn’t occupy any real disk space, you will read 512TB of zero-filled data if you choose to do so.

$ od -Xv copyme.txt
...
1357500          00000000        00000000        00000000        00000000
1357520          00000000        00000000        00000000        00000000
1357540          00000000        00000000        00000000        00000000
1357560          00000000        00000000        00000000        00000000
1357600          00000000        00000000        00000000        00000000
...
# A very, very, very long undetermined length of time...

It was a little over two years ago that I gave serious thought to making user-space file systems a reality on Mac OS X. The result of that work, MacFUSE, was introduced at the Macworld conference in January 2007. Since then, MacFUSE has come a long way. It’s been used in projects big and small and has made numerous existing (on other platforms) and new file systems possible on Mac OS X.

MacFUSE 2.0 is a major update to MacFUSE. We’ll be discussing what’s new in MacFUSE 2.0 in a talk at Google’s Mountain View headquarters today. For those who can’t be at the talk, there are two versions of the “what’s new” description.

The Apple-style version is easy to state: “improvements and bug-fixes.”

Alternatively, you can read the CHANGELOG in its entirety at the project’s web site.

Next Monday (December 8, 2008), there will be an open-to-all talk on MacFUSE at Google’s Mountain View headquarters. Here is a more detailed announcement.

A couple of weeks ago, I released HFSDebug 4. I’ve updated it to make HFSDebug’s pathname processing a little more sophisticated. Depending on how (and how seriously) you use HFSDebug, knowing the details could be useful.

When you specify a file system object to HFSDebug using a pathname, how HFSDebug will treat the pathname usually depends on other arguments, or the lack of other arguments.

Something to Read and Forget: The “Legacy” Mode

A typical invocation is quite simple: you simply give HFSDebug a path. If the path exists on an HFS+ volume, HFSDebug will use the underlying volume as the one to operate upon.

$ sudo hfsdebug /mach_kernel
...

In this case, HFSDebug will begin by doing a stat(2) call on the path. (As you will see shortly, this and the entire “legacy mode” is optional—you can make HFSDebug do even this part “from scratch” and not use stat(2) or other file system calls on the volume.) Since the goal is to make it possible to examine file system’s internal structure as opposed to what the user “sees” through layers of interfaces, it matters whether the object in question is a file hard link, directory hard link, symbolic link, etc. Specifically, we need to get at the link reference that the pathname represents—not the target it resolves to. For symbolic links, we can use lstat(2) to give us the node ID of the reference. For file and directory hard links, we have to use something else.

How do we know if something is a hard link? HFSDebug examines the st_nlink (link count) and st_mode fields of the resultant stat structure.

In the case of regular files, a link count of 2 or higher means the object is a “known” hard link.

In the case of directories, what “link count” means is quite context-sensitive on HFS+. The stat structure’s st_nlink for a directory normally represents the directory’s item count. The directory itself (the "." entry) and its parent (the ".." entry) together add 2 to the count. Thus, if you have a directory with, say, 4 files and 4 subdirectories in it, a stat(2) call would report an st_nlink value of 10. However, if the folder count bit is enabled on the mount, the meaning of st_nlink changes: it then represents a count of only the subdirectories. In the aforementioned example, the link count would be 6 instead of 10. The folder count bit is currently only enabled for case-sensitive (HFSX) volumes. Besides the stat structure’s st_nlink field, HFS+ can separately provide the children count and the real hard link count for directories. The getattrlist(2) call can be used to retrieve these: they are the ATTR_DIR_ENTRYCOUNT and ATTR_DIR_LINKCOUNT directory attributes, respectively. Once we do know a directory’s “real” hard link count, again, a value of 2 or higher means it is a “known” hard link.

By a “known” file or directory hard link, I mean that we know that it currently is a hard link. That means the object we are looking at is a link reference and the “visible” node ID isn’t that of the reference but is that of its target. (See my previous post for more information.) In this case, HFSDebug will retrieve the object’s parent folder’s node ID and use it in conjunction with the object’s name to do a “from scratch” lookup of the object’s node ID. (This is a traditional { parent_nodeid, name } ==> nodeid lookup.)

However, things are not simpler if the link count of the object is 1. That’s because it could have had a higher link count in the past and the other links were deleted. Again, as I described earlier, the on-disk object continues to be a reference to the real content that lives in a special hidden HFS+ folder. If that’s the case though, the hidden folder will have a file or a folder whose name is formed from the object’s visible node ID: iNode%d or dir_%d for files and folders, respectively. If no such file or folder exists, as HFSDebug can look up from scratch, the object is not a current or past hard link.

If this sounds unnecessarily complex, well, some of it is. Until version 4, HFSDebug did not have the ability to process complete pathnames from scratch. With version 4, you can do things like the following on both mounted and unmounted volumes alike.

$ sudo hfsdebug -d /dev/disk0s2 /mach_kernel
...

Simpler, Better

In the new mode, HFSDebug will no longer do a stat(2)/lstat(2) or involve the file system otherwise. Obviously, if it has to support unmounted volumes, that’s how it has to be. It will take the pathname and process it component-by-component, which is easier to conceptualize than the “legacy” mode I described above. (Well, to be fair to the legacy mode, it became uglier with the advent of directory hard links.)

The following are examples of how HFSDebug will handle things based on some of its arguments.

$ sudo hfsdebug /foo/bar/baz
# legacy mode; will use stat(2)/lstat(2) to kick things off
...
$ sudo hfsdebug -d /dev/diskN /foo/bar/baz
# new mode; volume can be mounted or unmounted
...
$ sudo hfsdebug -P /foo/bar/baz
# new mode; uses root volume, which is obviously mounted
...
$ sudo hfsdebug -d /dev/diskN -P /foo/bar/baz
# new mode; volume can be mounted or unmounted

Just remember that you can use the -P argument (note that it’s the capital P) to specify a path and HFSDebug will use the new mode on both mounted and unmounted volumes. The path must be absolute.

Some Rules

Now, if HFSDebug is not involving the file system at all, it better be able to handle arbitrarily convoluted pathnames. Using things like realpath(3) is not an option since realpath(3) would want to call stat(2)/lstat(2). Say, we could have a path like the following.

/foo////././bar///../baz/../blah/..//////.././dir/are/you/../crazy/

Besides, there could be components in the path that are symbolic or hard links. Symbolic links could point to targets that have equally crazy pathnames. It’s not just a matter of canonicalizing the dots and the slashes. We have several requirements as illustrated by the following examples. (Some of these are simply HFSDebug conventions.)

• We must ensure that all intermediate components resolve to directories. They can be actual directories, valid symbolic links to directories, or directory hard links. Remember that in the case of symbolic or hard links, the on-disk object will be a “file”—HFSDebug will need to resolve them from scratch too.
• If there is a “..”, we must not blindly go “up” one level: we must ensure that what we are going back from is a directory. realpath(3) actually doesn’t care about this: it will canonicalize /path/to/file.txt/../file.txt to /path/to/file.txt.
• Although we do want to resolve intermediate components that are links, we must not resolve the terminal component if it happens to be a hard link or a symbolic link. That’s because our goal is to look at what’s on disk for the given path. Besides, in the case of a link, the details shown by HFSDebug will include the full pathname to the link’s target. If we wanted further details on the target, we could run HFSDebug on it.
• If the path has a terminal slash, HFSDebug will ensure that the component is a directory. If it happens to be a directory hard link or a symbolic link, HFSDebug will resolve it in this case. Consider an example: suppose there is a symbolic link /tmp/somesymlink. The following is what HFSDebug will do depending on the arguments and what the link points to.

# somesymlink points to a file
#
$ sudo hfsdebug -P /tmp/somesymlink
... # will show details of the link itself
$ sudo hfsdebug -P /tmp/somesymlink/
... # will complain that the link target is not a directory

# somesymlink points to a directory
#
$ sudo hfsdebug -P /tmp/somesymlink
... # will show details of the link itself
$ sudo hfsdebug -P /tmp/somesymlink/
... # will show details of the directory somesymlink points to

# somesymlink points to a non-existent target
#
$ sudo hfsdebug -P /tmp/somesymlink
... # will show details of the link itself
$ sudo hfsdebug -P /tmp/somesymlink/
... # will complain that path /tmp/somesymlink/ was not found on the volume

I wrote HFSDebug in early 2004. I initially made it available as a software tool to help understand fragmentation in HFS+ volumes, although it could also be used to analyze several implementation details of HFS+. Eventually, I extended HFSDebug to be able to analyze all on-disk aspects of HFS+, along with the ability to compute more types of volume statistics and to even retrieve some in-memory details of mounted HFS+ volumes.

HFSDebug has been an extremely useful tool for me. I’ve used it to help explain the workings of HFS+ in the Mac OS X Internals book, to understand occasional mysterious behavior in HFS+ volumes, to search for file system objects, to generate interesting file system statistics (top N largest files, top N fragmented files, resource forks vs data forks, contiguous free space, and so on), and to create interesting demos. (For example, to show HFS+ mechanisms such as Hot File Clustering and On-the-Fly Defragmentation at work.)

HFS+ is the preferred and default volume format on Mac OS X. Even with exciting new developments such as ZFS support in Mac OS X, I don’t expect HFS+ to become obsolete any time soon. Today’s Macintosh computers, iPods, iPhones, and AppleTV’s all use HFS+.

With most major releases of Mac OS X, HFS+ has gained new capabilities. Features such as metadata journaling, on-the-fly file defragmentation, hot file clustering, extended attributes, access control lists, hard link chains (tracking hard links), and directory hard links have come to HFS+ in recent years. There have been “news reports” of compression being an upcoming feature in HFS+.

The most interesting technical things about HFS+ are not its features, but how several of the newer features are implemented. With the goal of retaining backward compatibility, new features have often been retrofitted, or shoehorned, if you will, into HFS+. Knowing such implementation details evokes different reactions in different people, ranging from “That’s a nifty way to implement this!” to “Gross!” This is something you can decide for yourself with the help of HFSDebug, which can show you exactly how the file system works.

New Features in HFSDebug

Now, every time a new feature is added to HFS+, HFSDebug likely (but not always) needs to be updated, say, to recognize and parse a new type of on-disk object such as a directory hard link. I’m releasing a new version of HFSDebug that has the following improvements.

• Ability to show details of directory hard links.
• Ability to show details of hard link chains.
• New built-in filters: atime (find files by access time), dirhardlink (list directory hard links), hardlink (list file hard links), and sxid (list setuid/setgid files).
• Ability to do component-wise path lookup from scratch, allowing you to analyze individual file system objects by path even on unmounted HFS+ volume.
• Support for Snow Leopard.
• Numerous subtle improvements and some bugfixes.

Let us take the new HFSDebug features for a spin.

Hard Link Chains

Although support for file hard links has been there in HFS+ before Leopard, the new chaining feature in Leopard can keep track of hard link chains, which are doubly linked list of file IDs connecting hard links together. Hard links to a file on HFS+ are conceptually similar to those on Unix systems: They represent multiple directory entries referring to common file content. Implementation-wise, HFS+ hard links use a special hard-link file for each directory entry. The common file content is stored in a special file: the indirect-node file. All indirect-node files are stored in the private metadata folder, a special directory (/\xE2\x90\x80\xE2\x90\x80\xE2\x90\x80\xE2\x90\x80HFS+ Private Data) that’s both normally invisible to the user and has a name that’s “hard” to type. It’s much easier to understand this through HFSDebug.

We begin by creating a file called file1. Before we create a hard link to this file, we examine its details using HFSDebug. That way, we can tell if anything about the file changes after link creation.

$ mkdir /tmp/test/
$ cd /tmp/test
$ echo "This is file1" > file1
$ sudo hfsdebug file1
...
  path                 = Leopard HD:/private/tmp/test/file1
# Catalog File Record
  type                 = file
  file ID              = 1927091
  flags                = 0000000000000010
...
  # BSD Info
  ownerID              = 501 (singh)
  groupID              = 0 (wheel)
  adminFlags           = 00000000
  ownerFlags           = 00000000
  fileMode             = -rw-r--r--
  linkCount            = 1
  textEncoding         = 0
  attrBlocks           = 0
  # Finder Info
  fdType               = 0
  fdCreator            = 0
  fdFlags              = 0000000000000000
...

Let us now make a hard link to this file.

$ ln file1 file2

Let us see if anything has changed about file1 now that we made a hard link to it.

$ sudo hfsdebug file1
...
  path                 = Leopard HD:/private/tmp/test/file1
# Catalog File Record
  type                 = file (hard link)
  indirect node file   = Leopard HD:/%0000%0000%0000%0000HFS+ Private Data/iNode1927091
  file ID              = 1927094
  flags                = 0000000000100010
                       . File has a thread record in the catalog.
                       . File has hardlink chain.
...
  # BSD Info
  ownerID              = 1927095 (previous link ID)
  groupID              = 0 (next link ID)
  adminFlags           = 00000000
  ownerFlags           = 00000010
                       . UF_IMMUTABLE (file may not be changed)
  fileMode             = -r--r--r--
  iNodeNum             = 1927091 (link reference number)
  textEncoding         = 0
  attrBlocks           = 0
  # Finder Info
  fdType               = 0x686c6e6b (hlnk)
  fdCreator            = 0x6866732b (hfs+)
  fdFlags              = 0000000100000000
                       . kHasBeenInited
...
  # Data Fork
  logicalSize          = 0 bytes
  # Resource Fork
  logicalSize          = 0 bytes

We see that a lot has changed! The on-disk nature of file1 has completely transformed. The original content has actually “moved” to an indirect-node file. What was file1 before has been replaced with a new directory entry altogether: one that has a new file ID within the file system. The new directory entry is also a file, but with several special properties. Its “type” and “creator” (as stored in the Finder Info) are hlnk and hfs+, respectively. It has been marked immutable. It has no content in either its data fork or its resource fork. Moreover, the owner and group ID on-disk fields have been repurposed to act as the previous and next links, respectively, in the hard link chain. We see that the previous link ID is 1927095. Let us use HFSDebug to show us information for that ID.

$ sudo hfsdebug -c 1927095
...
  path                 = Leopard HD:/private/tmp/test/file2
# Catalog File Record
  type                 = file (hard link)
  indirect node file   = Leopard HD:/%0000%0000%0000%0000HFS+ Private Data/iNode1927091
  file ID              = 1927095
  flags                = 0000000000100010
                       . File has a thread record in the catalog.
                       . File has hardlink chain.
...
  # BSD Info
  ownerID              = 0 (previous link ID)
  groupID              = 1927094 (next link ID)
  adminFlags           = 00000000
  ownerFlags           = 00000010
                       . UF_IMMUTABLE (file may not be changed)
  fileMode             = -r--r--r--
  iNodeNum             = 1927091 (link reference number)
  textEncoding         = 0
  attrBlocks           = 0
  # Finder Info
  fdType               = 0x686c6e6b (hlnk)
  fdCreator            = 0x6866732b (hfs+)
  fdFlags              = 0000000100000000
                       . kHasBeenInited
...
  # Data Fork
  logicalSize          = 0 bytes
  # Resource Fork
  logicalSize          = 0 bytes

We see that ID 1927095 corresponds to the other reference we just created: file2. The properties of this reference are similar to those of the other reference file1. They do differ in their file IDs. (They are indeed two separate on-disk file system objects.) They also differ in their previous and next links in the hard link chain. (We can confirm that file1 and file2 are connected together.)

The file’s content is in the indirect node file, which also now is the on-disk object with the original file ID (1927091). Let us use HFSDebug to look at that file.

$ sudo hfsdebug -c 1927091
...
  path                 = Leopard HD:/%0000%0000%0000%0000HFS+ Private Data/iNode1927091
# Catalog File Record
  type                 = file
  file ID              = 1927095
  flags                = 0000000000100010
                       . File has a thread record in the catalog.
                       . File has hardlink chain.
  reserved1            = 1927095 (first link ID)
...
  # BSD Info
  ownerID              = 501 (singh)
  groupID              = 0 (wheel)
  adminFlags           = 00000000
  ownerFlags           = 00000000
...
  # Finder Info
  fdType               = 0x686c6e6b (hlnk)
  fdCreator            = 0x6866732b (hfs+)
  fdFlags              = 0000000100000000
                       . kHasBeenInited
...
  # Data Fork
  logicalSize          = 14 bytes
  totalBlocks          = 1
  fork temperature     = no HFC record in B-Tree
  clumpSize            = 0
  extents              =   startBlock   blockCount      % of file
                             0xbb04b7          0x1       100.00 %
                         1 allocation blocks in 1 extents total.
                         1.00 allocation blocks per extent on an average.
  # Resource Fork
  logicalSize          = 0 bytes

As we see, the indirect-node file acts as the container for several of the original file’s properties. In particular, it has the original file’s content, the owner ID, and the group ID. A reserved field (reserved1) even contains the ID of the head of the hard link chain.

Of course, these are implementation details. HFS+ will show you the expected hard link semantics when you look at these files through the usual file system interfaces.

$ ls -las file1 file2
1927091 8 -rw-r--r--  2 singh  wheel  14 Nov  3 21:55 file1
1927091 8 -rw-r--r--  2 singh  wheel  14 Nov  3 21:55 file2
$ cat file1 file2
This is file1
This is file1

We see that both file1 and file2 show up with identical metadata, including the same “inode” number as you would expect. They also “have” the same content.

Note that if you now delete one of the hard links, say, file2, things will not revert back to how they were to begin with. You will have file1 as the only hard-link file along with the indirect-node file.

Let us look at directory hard links next.

Directory Hard Links

It’s not straightforward to create a directory hard link on Mac OS X. Well, that shouldn’t be surprising: directory hard links aren’t meant for third party developers, let alone users. They are essentially an implementation detail needed to make the Time Machine feature of Leopard work. Since we are talking about implementation details here, we will have to create a directory hard link or two—for experimentation, of course.

Leopard at the time of this writing requires the following conditions to be met for a directory hard link’s creation to be allowed. In the following list, “source” refers to the existing directory that will be pointed at by the new directory hard link “destination” that’s being created.

• The file system must be journaled HFS+.
• The parent directories of the source and destination must be different.
• The source’s parent must not be the root directory.
• The destination must not be in the root directory.
• The destination must not be a descendent of the source.
• The destination must not have any ancestor that’s a directory hard link.

If you meet all these conditions, you could create a directory hard link on an HFS+ volume under Mac OS X 10.5 and above. It’s then a matter of writing a program that uses the link() system call.

/* dirlink.c */

#include <stdio.h>
#include <unistd.h>

int
main(int argc, char** argv)
{
    int ret = -1;
    if (argc == 3) {
        ret = link(argv[1], argv[2]);
        if (ret) {
            perror("link");
        }
    }
    return ret;
}

In our /tmp/test/ testing directory, we’ll create a directory dir1 and a subdirectory subdir. It’s in subdir that we’ll create a hard link dir2 to dir1. This is because dir1 and dir2 can’t have the same parent.

$ gcc -Wall -o dirlink dirlink.c
$ mkdir dir1
$ mkdir subdir

Before we create the directory hard link, let us use HFSDebug to peek at the current on-disk details of dir1.

$ sudo hfsdebug dir1
...
  path                 = Leopard HD:/private/tmp/test/dir1
# Catalog Folder Record
  type                 = folder
  folder ID            = 1927398
  flags                = 0000000000000000
  valence              = 0
...
  # BSD Info
  ownerID              = 501 (singh)
  groupID              = 0 (wheel)
  adminFlags           = 00000000
  ownerFlags           = 00000000
  fileMode             = drwxr-xr-x
  linkCount            = 1
  textEncoding         = 0
  attrBlocks           = 0
  # Finder Info
  frRect               = (top = 0, left = 0), (bottom = 0, right = 0)
  frFlags              = 0000000000000000
  frLocation           = (v = 0, h = 0)
  opaque               = 0
  # Opaque Finder Info
  scrollPosition       = (v = 0, h = 0)
  reserved1            = 0
  Opaque Finder Flags  = 0000000000000000
  reserved2            = 0
  putAwayFolderID      = 0

Let us create the link and confirm that our expectations of directory hard link semantics are met.

$ ./dirlink dir1 subdir/dir2
$ ls -lasdi dir1 subdir/dir2
1927398 0 drwxr-xr-x  2 singh  wheel  68 Nov  3 22:59 dir1
1927398 0 drwxr-xr-x  2 singh  wheel  68 Nov  3 22:59 subdir/dir2
$ echo Hello > dir1/file
$ cat subdir/dir2/file
Hello

Everything looks in order. Let us now use HFSDebug to see what actually happened inside the file system. We looked at dir1‘s on-disk details earlier. We can now see what changed after we created a directory hard link to dir1.

$ sudo hfsdebug dir1
...
  path                 = Leopard HD:/private/tmp/test/dir1
# Catalog File Record
  type                 = file (alias, directory hard link)
  indirect folder      = Leopard HD:/.HFS+ Private Directory Data%000d/dir_1927398
  file ID              = 1927407
  flags                = 0000000000100010
                       . File has a thread record in the catalog.
                       . File has hardlink chain.
...
  # BSD Info
  ownerID              = 1927408 (previous link ID)
  groupID              = 0 (next link ID)
  adminFlags           = 00000000
  ownerFlags           = 00000010
                       . UF_IMMUTABLE (file may not be changed)
  fileMode             = -r--r--r--
  iNodeNum             = 1927398 (link reference number)
  textEncoding         = 0
  attrBlocks           = 0
  # Finder Info
  fdType               = 0x66647270 (fdrp)
  fdCreator            = 0x4d414353 (MACS)
  fdFlags              = 1000000000000000
                       . kIsAlias
  fdLocation           = (v = 0, h = 0)
  opaque               = 0
  # Data Fork
  logicalSize          = 0 bytes
  # Resource Fork
  logicalSize          = 464 bytes
  totalBlocks          = 1
  fork temperature     = no HFC record in B-Tree
  clumpSize            = 0
  extents              =   startBlock   blockCount      % of file
                             0xbae746          0x1       100.00 %
                         1 allocation blocks in 1 extents total.
                         1.00 allocation blocks per extent on an average.

  rsrc contents        = (up to 464 bytes)
       00 00 01 00 00 00 01 9e 00 00 00 9e 00 00 00 32 00 00 00 00 00 00 00 00
                                                     2
...
       00 00 00 00 00 00 00 1c 00 32 00 00 61 6c 69 73 00 00 00 0a 00 00 ff ff
                                   2        a  l  i  s
       00 00 00 00 00 00 00 00

We see that dir1‘s transformation is more drastic than what we had observed in the case of file hard links. After we created a directory hard link to dir1, it’s no longer a directory inside the file system. In fact, the “real” directory (that is, the link target) has moved to a special folder (/.HFS+ Private Directory Data\xd), just as the link target of a file hard link had moved to a (different) special folder. Its name within the special folder is dir_1927398, where the number represents the original “inode” number of dir1. However, dir1 hasn’t been replaced by another directory that points to the link target—it has been replaced by a file, or specifically, an alias. (Backward compatibility!) The immutable alias file has fdrp and MACS as its type and creator codes, respectively. It also as a resource fork. Moreover, we see that as in the case of file hard links, there exists a hard link chain.

Let us also examine the link target using HFSDebug. The path would be “hard” to type because of the characters in the special folder’s name. We can use the folder ID instead, which would be the original ID of dir1.

$ sudo hfsdebug -c 1927398
...
  path                 = Leopard HD:/.HFS+ Private Directory Data%000d/dir_1928557
# Catalog Folder Record
  type                 = folder
  folder ID            = 1927398
  flags                = 0000000000100100
                       . Folder has extended attributes.
                       . Folder has hardlink chain.
  valence              = 0
...
  # BSD Info
  ownerID              = 501 (singh)
  groupID              = 0 (wheel)
  adminFlags           = 00000000
  ownerFlags           = 00000000
  fileMode             = drwxr-xr-x
  linkCount            = 2
  textEncoding         = 0
  attrBlocks           = 0
  # Finder Info
  frRect               = (top = 0, left = 0), (bottom = 0, right = 0)
  frFlags              = 0000000000000000
  frLocation           = (v = 0, h = 0)
  opaque               = 0
...
# Attributes
...
  # Attribute Key
  keyLength            = 72
  pad                  = 0
  fileID               = 1927398
  startBlock           = 0
  attrNameLen          = 30
  attrName             = com.apple.system.hfs.firstlink
  # Inline Data
  recordType           = 0x10
  reserved[0]          = 0
  reserved[1]          = 0
  attrSize             = 8 bytes
  attrData             = 31 39 32 37 34 30 38 00
                          1  9  2  7  4  0  8     

We see mostly what we would expect given our previous observation of the implementation details of file hard links. There is one more thing in this case though: the folder has an extended attribute whose name is com.apple.system.hfs.firstlink and whose value is an encoding of the “inode” number of the head of the directory hard link chain.

HFSDebug Filters

At this point, you could use the built-in dirhardlink filter in HFSDebug to enumerate all directory hard links on the volume.

$ sudo hfsdebug --filter=builtin:dirhardlink
2 links -> dir_1927398
Leopard HD:/private/tmp/test/dir1 -> dir_1927398
Leopard HD:/private/tmp/test/subdir/dir2 -> dir_1927398

The filter prints both link targets and link references. For a link target, the number of references to it is printed before it. For a link reference, the target that it points to is printed after it.

By the way, filters are a very useful recent addition to HFSDebug. A fundamental capability of HFSDebug is to go over all the entries in the HFS+ catalog file. It uses this capability to generate many types of statistics. The recently added filter support makes it possible for you to write a program that plugs into HFSDebug and receives a callback for each catalog file entry. That way, you can examine each entry, apply arbitrary criteria, and show (or not show) details about that entry. Say, you wish to list all setuid/setgid files on an HFS+ volume. Sure, you could run a find command to do that. On one of my HFS+ volumes with about a million files and 200K folders, find takes a while to do this.

$ time sudo find / -xdev -type f \( -perm -4000 -o -perm -2000 \)
...
6.41s user 94.12s system 34% cpu 4:53.35 total
$

You could do this much faster with the sxid built-in HFSDebug filter, whose implementation is a mere ten lines of C code. (Of course, the absolute time taken will also depend on the underlying hardware, but we are only interested in the relative time difference.)

$ time sudo hfsdebug --filter=builtin:sxid
...
2.86s user 9.33s system 17% cpu 1:08.04 total
$

Note that many types of searches on HFS+ can also be done through the searchfs() system call, although it can be quite cumbersome to use. Of course, searchfs() cannot be done on an unmounted volume.

Specifying File System Objects by Path on Unmounted Volumes

As we have seen, a common use for HFSDebug is to have it display implementation details of individual file system objects. You could specify the object of interest in several ways: by providing its catalog node ID (CNID), by providing an “fsspec” style pair consisting of the parent folder’s CNID and the object’s name, or by providing a POSIX-style path to the object. The latter is often the easiest and most convenient to specify. However, until now, HFSDebug did not do component-wise path lookups itself—it used the operating system to convert the path to an inode number. This results in a few caveats. To begin with, it’s against the HFSDebug philosophy of not relying on the operating system for any HFS+-related operations. It also means that if the volume in question is not mounted (say, it’s corrupt and can’t be mounted or you are investigating something and don’t want to mount it), you can’t use paths to look at individual objects. You will have to dump all objects on the file system and then find the node ID of the object of interest. Moreover, even on a mounted volume, the operating system disallows path-based access to several files. (See Chapter 12 of Mac OS X Internals.) In such cases, again you will need to know the node ID of the object of interest, even on a mounted volume.

I’ve “fixed this issue” (or “added the feature”, depending on how you look at it) in the new version of HFSDebug. Say, if you have an unmounted volume on /dev/disk5s1 and you want to examine /tmp/foo/bar on it. Now you can simply do:

$ sudo hfsdebug -d /dev/disk5s1 /tmp/foo/bar
...

The semantics of symbolic link resolution are as follows. If the object (bar in this example) is a symbolic link itself, then HFSDebug will show you properties of bar and not what it points to. This is in line with HFSDebug philosophy and also how things work today on mounted volumes. If, however, a nonterminal component of the path is a symbolic link, HFSDebug will resolve it. Again, this is desirable.

That’s about it.

One More Thing

I can’t talk about HFSDebug’s Snow Leopard-specific features since the latter is under NDA. If you do have access to the latest Snow Leopard seed, try HFSDebug on it. For example, examine some standard Mac OS X files using HFSDebug.

$ sudo hfsdebug /bin/ls
...
$ sudo hfsdebug /etc/asl.conf
...
$ sudo hfsdebug /Applications/Mail.app/Contents/PkgInfo
...