I recently tried to mirror my hard drives in a new machine. The Handbook instructions, and those in my own Absolute FreeBSD, didn’t work well. (The Handbook now warns about this in a big, friendly, hard-to-miss red box.) So how can I mirror my disk? By using per-partition mirroring rather than full-disk mirroring.
I should note up front that this article is the result of my researches and testing. I am not a filesystem developer. I’m not even a FreeBSD committer any more. You should check the FreeBSD Handbook for updated documentation before trying this approach.
First, I need to partition my disks identically. My system has two disks, da0 and da1. da0 has an installed system, da1 is blank. Use gpart(8) to copy the GPT.
# gpart backup da0 > da0.gpt
The file should look something like this:
# cat da0.gpt
GPT 128
1 freebsd-boot 34 128
2 freebsd-ufs 162 201326464
3 freebsd-swap 201326626 8388540
Now copy this to the second disk.
# gpart restore -F /dev/da1 < da0.gpt
The -F flag tells gpart to destroy any existing GPT on the target disk. Now verify the GPT on both disks.
# gpart show
=> 34 209715133 da0 GPT (100G)
34 128 1 freebsd-boot (64k)
162 201326464 2 freebsd-ufs (96G)
201326626 8388540 3 freebsd-swap (4G)
209715166 1 - free - (512B)
=> 34 209715133 da1 GPT (100G)
34 128 1 freebsd-boot (64k)
162 201326464 2 freebsd-ufs (96G)
201326626 8388540 3 freebsd-swap (4G)
209715166 1 - free - (512B)
These look pretty identical to me. I now have a separate device node for each partition on each disk. Mirroring these works much like mirroring an entire disk.
Unlike mirroring MBR disks, to mirror GPT partitions you must be in single-user mode. Now that GPT is the default partitioning scheme I’m sure someone will figure out a clever way around this, but for now reboot into single-user mode.
# gmirror label -vb round-robin p1 /dev/da0p1
# gmirror label -vb round-robin p2 /dev/da0p2
# gmirror label -vb round-robin p3 /dev/da0p3
/dev/da0p3 is the default swap partition. You must decide if you want to mirror your swap partitions as well. I’m choosing to do so. If one of my disks fails, the system has a fighting chance to continue running. Having two independent swap areas, one on each disk, means that a disk failure will yank a swap space out from under the otherwise-working system.
Now add the second disk’s partitions to your mirror devices.
# gmirror insert p1 /dev/da1p1
# gmirror insert p2 /dev/da1p2
# gmirror insert p3 /dev/da1p3
This will mirror your boot blocks, your root disk, and your swap space.
Now you must update /etc/fstab to boot from your mirror. The tricky bit here is that the system now has /dev/da0p2 mounted as root, read-only. You don’t want to write to /dev/da0p2 again; all writes should go to the mirror. Instead, mount /dev/mirror/p2 to a temporary location.
# mount /dev/mirror/p2 /mnt
# cd /mnt/etc
# cp fstab fstab-old
# ee fstab
vi requires a read-write /var/tmp, but ee works just fine.
Let the system run until the disks are synchronized. Check your disk status with “gmirror status.”
# gmirror status
Name Status Components
mirror/p1 COMPLETE da0p1 (ACTIVE)
da1p1 (ACTIVE)
mirror/p2 DEGRADED da0p2 (ACTIVE)
da1p2 (SYNCHRONIZING, 17%)
mirror/p3 DEGRADED da0p3 (ACTIVE)
da1p3 (SYNCHRONIZING, 7%)
After your disks synchronize, reboot. Don’t just exit single-user mode, as you have the wrong root partition mounted. (I found that rebooting before mirror synchronization meant the system came up with a read-only /.)
Your drives are now mirrored. Don’t forget to add mirror checks to your daily status mails. Add the following to /etc/periodic.conf:
daily_status_gmirror_enable="YES"
Hopefully, we’ll have a faster way to do this soon.
UPDATE 7/12/2011: You must have geom_mirror_load=YES in /boot/loader.conf to run gmirror commands.
What? You’re no longer a committer? 🙁
There are advantages and disadvantages to both situations.
My time is better spent writing *BSD books than running “cvs commit.” I’m still on developers@, and they tolerate me hanging around.
Tried this with my working system and it works great, thanks for pointing out this information 🙂
Mr Lucas: Good choice. I’m on dev@ too… I have no idea why they thought *that* would be a good idea. 😉
A question: Why it always has 4GB swap space? My computer has 16GB memory, is 4GB too less to dump all of the memory? I cannot understand.
# gmirror insert p1 /dev/da1p1
# gmirror insert p2 /dev/da1p2
# gmirror insert p3 /dev/da1p3
I am using FreeBSD 9.0RC2. It reports gmirror class not found error.
Frank,
You probably don’t have the geom_gmirror.ko kernel module loaded. You’ll need that in /boot/loader.conf .
I’d much rather gmirror up the whole device – since I’m practically never going to want to replace JUST a partition if a disk fails. It also has the benefit of making for one resync rather than 3 parallel, disk-thrashing resyncs.
Frank – that’s because Michael missed mentioning “gmirror load” after the labelling.
Resyncing now, will see how this goes… I think I’d still rather do whole-disk, but I’ll try this for now.
Myke,
I’d rather mirror the whole disk too. Not an option, yet. I’m sure it will be soon.
Why not use an MBR partition scheme instead ?
George,
The installer uses a GPT partition. I could do a manual install, but I’m not sure the new installer will even let you choose a MBR partition any more.
Hi Michael, last time I did an installation using the new bsdinstaller was back in April. I remember that there was an option to create MBR in manual partition tool. I am not sure if it works though.
Tested on 2 identical disks and can boot with either one on a simulating one disk failure. To do so-
Before Single USER reboot
==========================
Copy the MBR(512 bytes) also from da0 to da1 to make da1 bootable!
dd if=/dev/da0 of=/dev/da1 bs=512 count=1
Add these to /etc/sysctl.conf
kern.geom.debugflags=17
kern.geom.part.check_integrity=0
and as later updated by Mike
echo ‘geom_mirror_load=”YES”‘ >> /boot/defaults/loader.conf
>Now add the second disk’s partitions to your mirror devices.
#gmirror load ===========================This is needed. Otherwise it won’t work.
># gmirror insert p1 /dev/da1p1
># gmirror insert p2 /dev/da1p2
># gmirror insert p3 /dev/da1p3
You may need also:
># gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/da1
iplus, that line gives “Operation not permitted”. Hopefully whatever that line does is copied when that partition is mirrored. I will test to make sure.
Good short guide but very helpfull. Info that we can mirror partition – opposite to whole disks should be put in freeeBSD Handbook below warning about incompatibility MBR and GPT disks mirroring. I read about 10 instructions about mirror disks under GPT – yours is the straightforward the best.
Hello! The fstab is not clear to me. What lines i should put inside the new fstab?
Thanks for the tutorial!
I did all that was mentioned here and went through the comments. I ended up with a fail on reboot.
I will just type what I believe to be the relevant parts.
mounting from ufs;/dev/mirror/ada0p2 failed with error 19
You can see a picture I took here:
http://i.imgur.com/Oxzp7.jpg
About fstab issues here.
Current example creates mirror devices with full path /dev/mirror/p1, /dev/mirror/p2 and /dev/mirror/p3.
And they are coming from “gmirror label -vb round-robin p1 /dev/da0p1” command. See “man gmirror” for more information (gmirror label [-Fhnv] [-b balance] [-s slice] name prov …)
@ML – This guide make things very simple and worked. (I did it in VMware Fusion.)
However, when I removed the 1st drive from the VM the second drive did not boot… until I ran iplus’ recommendation:
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/da0Notice that, whereas I had removed the 1st disk, the second disk now shows up as
da0.I was able to successfully run the ‘
gpart bootcode‘ command from the Live CD and it corrected the problem.Now to trick VMware Fusion into properly ordering the drive I add to the VM.
This is /etc/defaults/periodic.conf not /etc/periodic.conf
Thanks it’s very clear