FreeBSD ZFS: Performance degration, disk replace

Christian Kruse,

My home NAS is based on a FreeBSD server with root ZFS on 4 physical disks and two disks redundancy (raidz-2). I'm pretty satisfied with this solution until a few weeks ago where the performance of the raid degraded. It was really unbearable. After googling a bit I found several tuning guides, but none of them helped. But one of them suggested to check the drive health, and smartctl -a showed that one of my disks had read errors. To test this I took the disk out of the array:

$ zpool offline zroot ada2p3
    NAME                     STATE     READ WRITE CKSUM
    zroot                    DEGRADED     0     0     0
      raidz2-0               DEGRADED     0     0     0
        ada0p3               ONLINE       0     0     0
        2021608558689218476  OFFLINE      0     0     0  was /dev/ada2p3
        ada3p3               ONLINE       0     0     0

Immediately the read and write performance increased. So I bought a new disk, copied the partition table via

$ gpart backup ada0 | gpart restore -F ada2
$ gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2

And re-attached it with the following command:

$ zpool replace zroot /dev/ada2p3

Now my raid is rebuilding:

  pool: zroot
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Wed Dec  5 10:25:12 2012
    35.3G scanned out of 2.00T at 67.4M/s, 8h28m to go
    8.79G resilvered, 1.73% done
config:

    NAME                       STATE     READ WRITE CKSUM
    zroot                      DEGRADED     0     0     0
      raidz2-0                 DEGRADED     0     0     0
        ada0p3                 ONLINE       0     0     0
        ada1p3                 ONLINE       0     0     0
        replacing-2            OFFLINE      0     0     0
          2021608558689218476  OFFLINE      0     0     0  was /dev/ada2p3/old
          ada2p3               ONLINE       0     0     0  (resilvering)
        ada3p3                 ONLINE       0     0     0

Yay, that was easy! I really like ZFS.