Read only root partition with aufs PDF Print E-mail
Friday, 09 April 2010 15:38

One of my machines runs linux from an SDCard but I want to be able to pull the power without shutting down gracefully.  It struck me that the best way to protect the SDCard was to mount the root filesystem read-only and use unionfs or aufs to merge it with a RAMDisk.  This allows the system to write to /etc/resolv.conf, /etc/mtab, /etc/udev and anything else it needs to alter without bothering the SDCard.

I decided to use aufs for this for one very simple reason:  I found someone else on the internet who had already done it.  I needed to make some changes from his method so I'm reproducing it here, but all credit goes to the author of that post.  The main change I made from his version is that I removed the modprobe as (a) the -Q flag was causing problems and (b) I'm running Ubuntu 9.10 which uses kernel 2.6.31-14-generic, so aufs is already in the kernel.


  1. Save this script as /etc/initramfs-tools/scripts/init-bottom/mount_root_aufs and make it executable
    • root@feegle:~# chmod 755 /etc/initramfs-tools/scripts/init-bottom/mount_root_aufs
  2. Run update-initramfs to recreate the initrd image for booting
    • root@feegle:~# update-initramfs -k all -u
      update-initramfs: Generating /boot/initrd.img-2.6.31-14-generic
      root@feegle:~#
  3. Lastly update /boot/grub/grub.cfg to add aufs=tmpfs to the end of the linux kernel parameters.  Salient part of my grub.cfg below:
    1. ### BEGIN /etc/grub.d/10_linux ###
      menuentry "Ubuntu, Linux 2.6.31-14-generic" {
              recordfail=1
              if [ -n ${have_grubenv} ]; then save_env recordfail; fi
              set quiet=1
              insmod ext2
              set root=(hd0,1)
              search --no-floppy --fs-uuid --set 5a39304a-db00-47ee-a305-7124b96a834d
              linux   /boot/vmlinuz-2.6.31-14-generic root=UUID=5a39304a-db00-47ee-a305-7124b96a834d ro   console=tty0 console=ttyS0,115200n8 aufs=tmpfs
              initrd  /boot/initrd.img-2.6.31-14-generic
      }
      ### END /etc/grub.d/10_linux ###
  4. Reboot and you have a root partition that you can write to without writing to the SDCard, which should make it much more resilient to sudden power failure.  Output of mount is shown below: 
    • root@feegle:~# mount
      /aufs on / type aufs (rw)
      proc on /proc type proc (rw)
      none on /sys type sysfs (rw,noexec,nosuid,nodev)
      none on /sys/kernel/security type securityfs (rw)
      udev on /dev type tmpfs (rw,mode=0755)
      /dev/sda1 on /ro type ext3 (rw)
      /rw on /rw type tmpfs (rw)
      none on /sys/fs/fuse/connections type fusectl (rw)
      none on /sys/kernel/debug type debugfs (rw)
      none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
      none on /dev/shm type tmpfs (rw,nosuid,nodev)
      tmpfs on /tmp type tmpfs (rw,noatime,mode=1777)
      none on /var/run type tmpfs (rw,nosuid,mode=0755)
      none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
      none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
      /dev/sda3 on /writeable_etc type ext3 (rw,noatime,errors=remount-ro,commit=120)
      none on /var/log type tmpfs (rw,noatime,mode=0755)
      binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)
      root@feegle:~#

    /dev/sda1 is the original root filesystem and /dev/sda3 is a separate writeable parition where I store files from / that I want to permanently change (such as /etc/shadow).  I then copy these over at boot using another script which will make it onto this page eventually.


Gotchas to watch out for:

  • I initially called the script mount_root_aufs.sh but for some reason update-initramfs didn't pick it up - I neither know nor care why - so make sure its called mount_root_aufs.
  • If you are likely to update your system using Ubuntu's built-in mechanisms do not edit grub.cfg by hand as I did above.  Edit the files in /etc/grub.d and/or /etc/default/grub and run update-grub like you are supposed to. 


UPDATE:  I discovered that the dhcp server would no longer run after I did this, as it apparently couldn't find the libraries it needed such as libc (which seemed unlikely to be missing):

    root@feegle:~# /etc/init.d/dhcp3-server start
    dhcpd self-test failed. Please fix the config file.
    The error was:
    /usr/sbin/dhcpd3: error while loading shared libraries: libcap.so.2: cannot open shared object file: No such file or directory
    root@feegle:~#
    root@feegle:~# ldd /usr/sbin/dhcpd3
            linux-gate.so.1 =>  (0x00b22000)
            libcap.so.2 => not found
            libc.so.6 => not found
    root@feegle:~#

Strangely enough when I printed the ldconfig cache the missing libraries where there.

    root@WANdroid:~# ldconfig -p | grep libc.so.6
            libc.so.6 (libc6, hwcap: 0x8008000000008000, OS ABI: Linux 2.6.15) => /lib/tls/i686/cmov/libc.so.6
            libc.so.6 (libc6, OS ABI: Linux 2.6.15) => /lib/libc.so.6
    root@WANdroid:~#

There were some very odd messages in the output of dmesg too:

    [  110.389084] type=1503 audit(1271327240.104:28): operation="open" pid=1708 parent=1707 profile="/usr/sbin/dhcpd3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/ro/etc/ld.so.cache"
    [  110.389797] type=1503 audit(1271327240.104:29): operation="open" pid=1708 parent=1707 profile="/usr/sbin/dhcpd3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/ro/lib/libcap.so.2.16"
    [  110.391714] type=1503 audit(1271327240.104:30): operation="open" pid=1708 parent=1707 profile="/usr/sbin/dhcpd3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/ro/lib/tls/i686/cmov/libc-2.10.1.so"
    [  110.392355] type=1503 audit(1271327240.108:31): operation="open" pid=1708 parent=1707 profile="/usr/sbin/dhcpd3" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/ro/lib/libc-2.10.1.so"


The root cause turned out to be AppArmor, which was objecting to the altered paths which now contain /ro and /rw.  There are three solutions to this:

  1. Read up on AppArmor and create a proper profile for dhcp3-server.  Hahahahahahaha..........yeah, right.
  2. Disable AppArmor for dhcp3 by calling 
    1. sudo ln -s /etc/apparmor.d/usr.sbin.dhcp3 /etc/apparmor.d/disable/
  3. Use the rather more permanent solution of 
    1. sudo apt-get purge apparmor

#1 was never going to happen and I was hacked off with AppArmor at this point so I went with #3.  Ubuntu documenation advises against it so "on your head be it" yada yada yada etc. etc.

 
Joomla Templates by Joomlashack