External Drive Migration with rsync

#rsync #macos #external-drive #migration #backup # Overview Procedure for migrating the `dataold` external drive to the new `data` drive on macOS. Three passes: dry run, real copy, verify. Nothing gets erased until the verify pass comes back clean. Related: [[rsync cheat sheat]] for general flag reference. # Configuration ## Use Homebrew rsync, not Apple's Apple ships rsync 2.6.9 (or openrsync on newer macOS) — both handle extended attributes and ACLs poorly. ```bash brew install rsync rsync --version # want 3.4.x, "protocol version 32" ``` If `which -a rsync` shows `/usr/bin/rsync` first, Homebrew's bin isn't ahead of the system paths. `path_helper` puts `/usr/bin` first, so prepend via `~/.zprofile`: ```bash echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile exec zsh -l ``` Use `/usr/local/bin/brew` on Intel. If the old binary still resolves in the current shell, run `hash -r` to clear zsh's command cache. ## Flags | Flag | Meaning | |---|---| | `-a` | Archive — recursive, preserves perms, times, symlinks | | `-H` | Preserve hard links | | `-A` | Preserve ACLs | | `-X` | Preserve extended attributes / resource forks | | `-n` | Dry run | | `-c` | Compare by checksum, not size+mtime | | `--info=progress2` | Whole-transfer progress (rsync 3.x only) | | `--partial` | Keep partial files so an interrupted run can resume | **Trailing slash is critical.** `dataold/` copies the *contents* into `data/`. Without it you get `/Volumes/data/dataold/`. # Procedure ## 1. Dry run ```bash rsync -aHAXn --stats \ --exclude='.DocumentRevisions-V100' \ --exclude='.Spotlight-V100' \ --exclude='.fseventsd' \ --exclude='.Trashes' \ --exclude='.TemporaryItems' \ --exclude='.DS_Store' \ /Volumes/dataold/ /Volumes/data/ ``` ## 2. The copy ```bash caffeinate -i rsync -aHAX --info=progress2 --partial \ --log-file="$HOME/rsync-migrate.log" \ --exclude='.DocumentRevisions-V100' \ --exclude='.Spotlight-V100' \ --exclude='.fseventsd' \ --exclude='.Trashes' \ --exclude='.TemporaryItems' \ --exclude='.DS_Store' \ /Volumes/dataold/ /Volumes/data/ ``` `caffeinate -i` stops the Mac sleeping mid-transfer. If interrupted, rerun the identical command — it resumes. ## 3. Verify **Drop `-AX` for the verify pass.** Keeping `-X` here produces a false `x` on every single file — see Troubleshooting below. Keep the same excludes, or the verify reports differences for folders that were skipped on purpose. ```bash rsync -aHn --itemize-changes \ --exclude='.DocumentRevisions-V100' \ --exclude='.Spotlight-V100' \ --exclude='.fseventsd' \ --exclude='.Trashes' \ --exclude='.TemporaryItems' \ --exclude='.DS_Store' \ /Volumes/dataold/ /Volumes/data/ ``` Any output = a file that differs. Clean run prints only the summary, confirming every file matches on content, size, mtime, permissions, and ownership. `-X` still belongs on the **copy** pass — it moves the real metadata (Finder tags, resource forks, `kMDItemWhereFroms`). It's only verification that needs it dropped to be readable. Paranoid version — reads every byte on both drives, slow but worth it before erasing anything: ```bash rsync -aHAXnc --itemize-changes /Volumes/dataold/ /Volumes/data/ ``` Sanity counts: ```bash echo $? # 0 = clean exit du -sh /Volumes/dataold /Volumes/data find /Volumes/dataold -type f | wc -l find /Volumes/data -type f | wc -l grep -Ei 'error|failed|denied' "$HOME/rsync-migrate.log" ``` ## 4. Photos library — special case A live Photos library can't be copied cleanly. Its SQLite write-ahead logs (`Photos.sqlite-wal`, `psi.sqlite-wal`, `store.cloudphotodb-wal`) are mid-write, and iCloud keeps pulling new assets in behind the copy. Quit the app and wait for the daemons to idle out: ```bash osascript -e 'quit app "Photos"' sleep 15 pgrep -l 'Photos|photolibraryd|photoanalysisd|cloudphotod' ``` `photolibraryd` is the one that matters — it holds the SQLite handles and exits shortly after the app. Don't kill it while Photos is open. Then re-sync just the bundle: ```bash caffeinate -i rsync -aHAX --delete --info=progress2 --partial \ --log-file="$HOME/rsync-photos.log" \ "/Volumes/dataold/Photos Library.photoslibrary/" \ "/Volumes/data/Photos Library.photoslibrary/" ``` Trailing slash on **both** sides — with `--delete` scoped to a single bundle, that's the safe form. ### Only some of the bundle actually matters Photos regenerates most of it on first open, so don't chase byte-perfect parity: | Path | Matters? | |---|---| | `originals/` | **Yes — the irreplaceable data** | | `database/Photos.sqlite` | **Yes — needs a consistent snapshot** | | `resources/derivatives/`, `resources/caches/` | No — regenerated | | `database/search/Spotlight/` | No — reindexed | | `resources/cpl/cloudsync.noindex/` | No — CloudKit sync state, rebuilt | The verify that answers the real question: ```bash rsync -aHn --itemize-changes \ "/Volumes/dataold/Photos Library.photoslibrary/originals/" \ "/Volumes/data/Photos Library.photoslibrary/originals/" ``` **Before trusting any of it:** if the library uses iCloud Photos with *Optimize Mac Storage*, the bundle holds thumbnails, not full-resolution originals. Set Photos → Settings → iCloud to **Download Originals to this Mac** and let it finish first. Don't open the copied library on `data` until `dataold` is retired — two libraries with the same UUID both syncing to iCloud gets messy. # Decisions - **No `--delete`.** Not needed when migrating to an empty drive, and it's the one flag that can destroy data if a path is fumbled. Only add it if `data` has stale content from a previous partial run *and* the paths have been verified. - **Excludes over sudo.** The macOS system metadata folders are SIP-protected; running the whole migration as root to reach them brings more risk than the metadata is worth. All of it is volume-specific and regenerates on the new drive. # Retiring the Old Drive Gate: don't wipe until the `originals/` verify is silent **and** the new library on `data` has been opened once with the expected photo count. A sold drive is a bad place to discover a problem. ## Identify the device, not the volume ```bash diskutil list external physical ``` Target the whole device (`/dev/disk6`), not a volume slice (`/dev/disk6s2`) — erasing the volume leaves the partition map and hidden containers intact. Triple-check the identifier; there's no undo. ```bash diskutil info /dev/diskN | grep -i 'solid state' ``` ## SSD — crypto-erase `diskutil secureErase` refuses to run on SSDs — wear-leveling means the controller may never touch the cells holding old data. Two valid approaches, and **the order matters enormously**. ### Check first — it may already be handled ```bash diskutil info /dev/diskN | grep -iE 'solid state|trim|encrypt' ``` - Already FileVault/APFS encrypted → a single plain erase *is* a complete crypto-erase. Done. - SSD reporting TRIM support → the erase already unmapped every block and the controller returns zeros. Most USB bridges don't pass TRIM, so verify rather than assume. ### Crypto-erase — only works BEFORE erasing ==Encrypt the volume while the data is still on it.== Encrypting an *empty* volume after an erase protects nothing — the old blocks were never touched by the new key. Learned this the hard way on 2026-09-02. ```bash # dataold still mounted and full, AFTER migration is verified diskutil apfs encryptVolume /Volumes/dataold -user disk diskutil apfs list # poll until conversion reads 100% diskutil eraseDisk exFAT WIPED GPT /dev/diskN ``` Conversion is proportional to data size — hours for a large drive. A partial conversion leaves the remainder in plaintext, so let it finish. APFS volumes only; HFS+ needs deprecated CoreStorage conversion and exFAT can't do it at all. **Gotcha:** `"APFS (Encrypted)"` is the Disk Utility *GUI* label, not a `diskutil` personality — `eraseDisk` rejects it. Only what `diskutil listFilesystems` prints is valid, and no encrypted variants appear there. ### Already erased? Overwrite the raw device The remedy once the crypto-erase window has passed: ```bash diskutil list external physical # confirm identifier, no undo diskutil unmountDisk /dev/diskN # unmount, not eject sudo caffeinate -i dd if=/dev/zero of=/dev/rdiskN bs=1m diskutil eraseDisk exFAT WIPED GPT /dev/diskN ``` `/dev/rdiskN` (raw) is several times faster than `/dev/diskN` and bypasses the filesystem, so no sparse-file or compression games. `sudo` before `caffeinate` so the password prompt comes up front and the sleep assertion covers the whole run. **macOS `dd` is BSD, not GNU — `status=progress` fails with `dd: unknown operand status`.** Press **Ctrl+T** during the run for a progress line (BSD `dd` answers SIGINFO). For a live counter instead: ```bash brew install coreutils sudo caffeinate -i gdd if=/dev/zero of=/dev/rdiskN bs=1M status=progress ``` `bs=1M` capitalized for GNU `gdd`, lowercase `bs=1m` for BSD `dd`. Ending with "No space left on device" is success. Roughly 1.5 hours per TB over USB 3. One pass is enough. Reaches every exposed LBA but not the ~7–10% overprovisioned pool; extracting that needs chip-off forensics. Fine for a consumer resale. ### Don't bother encrypting between the zero pass and repartition Same trap as above: the volume is empty at that point, so the key only encrypts empty space. Purely ceremonial. If more thoroughness is genuinely wanted, the thing with real (if marginal) value is a **second full-device pass with random data** — a second write cycle forces another round of cell allocation, pushing more overprovisioned cells through garbage collection. `/dev/urandom` is ~50–100 MB/s on macOS and would take days. Use a cipher as a keystream to run at drive speed: ```bash openssl enc -aes-256-ctr -nosalt \ -pass pass:"$(head -c 32 /dev/urandom | base64)" </dev/zero 2>/dev/null \ | sudo caffeinate -i dd of=/dev/rdiskN bs=1m ``` ## Spinning HDD — single-pass zeros ```bash diskutil secureErase 0 /dev/diskN diskutil eraseDisk exFAT WIPED GPT /dev/diskN ``` Level `0` is sufficient at modern platter densities. Levels 2 and 3 (7-pass DoD, 35-pass Gutmann) take days on a large drive and buy nothing. ## Leaving it ready for the buyer - Format **exFAT + GPT** — mounts on Mac and Windows with no effort from them. - If the enclosure has *hardware* encryption (some Samsung T-series, WD My Passport), also reset drive security via the vendor utility so the buyer isn't stuck at a password prompt. - Using Disk Utility instead? Enable **View → Show All Devices** first, or you'll only see volumes and won't fully clear the disk. # Troubleshooting ## `llistxattr(".DocumentRevisions-V100") failed: Permission denied (13)` Harmless. That's macOS's Versions database — SIP-protected, volume-specific, contains no user data, and the new drive builds its own. Worth excluding anyway: the errors make rsync exit **23** ("partial transfer due to error"), which would otherwise mask a real failure later. With the excludes in place, `echo $?` is trustworthy. ## Verify pass reports `.f........x` on nearly every file Decode the itemize columns before panicking: ``` .f........x ││└──────┴─ attribute columns: c s t p o g u a x │└───────── f = regular file └────────── . = no data transfer needed ``` Only the trailing `x` is set — **extended attributes differ, nothing else**. Content, size, mtime, permissions, owner, and group all match. The file data is fine. ### Diagnosis chain (2026-09-02) Two dead ends worth recording so they aren't re-walked: 1. `diff <(xattr -l src) <(xattr -l dst)` came back **empty** — misleading. `xattr -l` renders binary values as unprintable/blank, so differing attributes compare as identical. Always use `-lx` for hex. 2. `com.apple.macl` was absent from both files — not the cause here, though it is a common one (per-volume sandbox record, unwritable by userspace). The real culprit, found with: ```bash F='MessagesDataBackup/2024/03/19/IMG_3751.png' diff <(xattr -lx "/Volumes/dataold/$F") <(xattr -lx "/Volumes/data/$F") ``` ``` 0a1,3 > com.apple.provenance: > 00000000 01 02 00 F6 D6 7F 41 93 91 67 A5 |......A..g.| ``` `com.apple.provenance` exists **only on the destination**. It's a Gatekeeper/TCC record macOS stamps per-volume on arrival; rsync can neither replicate it from a source that lacks it nor delete it. So `-X` reports a mismatch on every file, permanently — re-running never converges. **Resolution:** cosmetic, ignore it. Verify with `-aHn` (no `-AX`) as in step 3. ## `sudo rsync` runs the wrong binary `sudo` ignores the user PATH and resolves `/usr/bin/rsync`. Use the absolute path (`sudo /opt/homebrew/bin/rsync …`) if sudo is ever needed. ## Photos daemons respawn after `killall` `photolibraryd`, `photoanalysisd`, and `cloudphotod` are on-demand launchd jobs — anything touching the library re-bootstraps them over XPC, so killing them is whack-a-mole. The usual hidden trigger is **`PhotosReliveWid`**, the Photos Memories widget. It polls the library on a timer, which wakes `photolibraryd`, which wakes `photoanalysisd`. Remove the Photos/Memories widget from the desktop and Notification Center and the loop stops. If they still come back, boot them out of the user domain (reversible, and everything returns at next login): ```bash launchctl list | grep -i photo # get the real labels first launchctl bootout gui/$(id -u)/com.apple.photoanalysisd launchctl bootout gui/$(id -u)/com.apple.cloudphotod ``` `launchctl kickstart -k gui/$(id -u)/<label>` restarts one immediately. Don't use `launchctl disable` — it persists and is easy to forget about. ## `.bzvol` shows as changed every day Backblaze's volume marker; it writes a fresh `ping_YYYYMMDD_*.log` daily. Add `--exclude='.bzvol'` to routine runs. ==Separate decision: `.bzvol` carries the volume GUID Backblaze uses to identify the drive. Copying it preserves backup continuity, but Backblaze gets confused badly if both drives are ever mounted at once. Don't leave it duplicated long-term.== ## Target drive is exFAT or FAT Drop `-AX` and use `-rltDv` instead. Hard links won't survive. APFS/HFS+ → keep the full `-aHAX`. # Build Log ## 2026-09-02 - Installed rsync 3.x via Homebrew; sorted PATH ordering so `which rsync` resolves to `/opt/homebrew/bin/rsync` - First dry run threw `llistxattr` permission errors on `.DocumentRevisions-V100` — added to the exclude list along with the other macOS volume metadata folders - Ran the copy pass with `-aHAX` - Verify pass with `-aHAXn` flagged `.f........x` on nearly every file. Traced it to `com.apple.provenance` being stamped on the destination volume by macOS — cosmetic, unfixable, and not a data problem. Corrected the documented verify command to drop `-AX`. Full chain in Troubleshooting. - Re-ran verify with `-aHn`: **everything on the drive verified clean except `.bzvol` and the Photos library**. Migration of general data confirmed good. - Photos library still churning — new `originals/` asset pulled by iCloud after the copy, plus live SQLite `-wal` differences. Photos.app was closed but `photolibraryd`/`cloudphotod`/`photoanalysisd` kept respawning; traced the trigger to the `PhotosReliveWid` Memories widget. - Plan set for retiring the drive. Two corrections along the way: `"APFS (Encrypted)"` isn't a valid `diskutil eraseDisk` personality, and — more importantly — crypto-erase has to happen *before* the first erase. Erased `dataold` first, so falling back to a raw-device zero pass instead. # Lessons - **Verify before erasing, section by section.** The general data verified clean, but the Photos library never got its quiet-moment re-copy — and the source was erased anyway. The gate at the top of Retiring the Old Drive exists for a reason. - **Crypto-erase is order-dependent.** Encrypt while the data is present. After an erase, the only remedy left is a full raw-device overwrite. # References - [[rsync cheat sheat]]