Remote Cloud Server Encryption

Imagine using a hosting provider that suddenly goes out of business. It happens, unfortunately. What an inconvenience!

Now imagine that the server you were renting space on is being sold to the secondary market and somebody is going to buy it with your data on it, free for them to peruse or sell to your competitors. Ought the now-defunct hosting company wipe all the data? Sure. Should every good recycler DBAN every disk before listing it on eBay? Absolutely. Do they? Ha—try buying used storage on eBay! Now imagine that none of the above applies and that the servers are just stolen by thugs.

So, what to do about this? Encrypt all your disks of course! With all modern processors supporting crypto in hardware, the performance penalty is negligible for most workloads and linux is plenty capable.

So, what’s the challenge? A large number of hosting providers pre-install OS images (which you should verify) and do not give you access to alternate boot media. If you do have access to switch boot order and boot an ISO for ‘recovery’—great—you should probably do that instead.

Let’s talk strategy.

Most modern Linux distros boot from a disk image, the initramfs, on a /boot partition. This minimal system image is there to set up everything needed to get the main system image, the / partition, running, including setting up crypto or repairing problems. We’ll be looking at a Debian system here—the process will be similar for other distros.

Our first problem: many of the tools you’ll need to effectively do disk management are not included by default on Debian in the initramfs. This used to make more sense when disk space and memory were scarce. It still makes sense for tiny or embedded systems, but every year it seems like a worse choice for servers. Anyway, very little is set in stone so we can fix it.

First, let’s install some necessary packages. The examples will be in Puppet language, for devops people to copy and paste, and they should be entirely self-explanatory for other sysadmins to install with apt, cp, cat, etc. Sorry, the world isn’t ready for n00bs to deploy this—though it absolutely ought to be standard. Keep practicing and follow none of these examples verbatim without thinking first! Sharps parts inside.

Status message: this was written after encrypting yet another server but hasn’t been followed by the author on the next one yet, so double-check the work. This message will disappear after being verified.

  package { 'dropbear-initramfs' :
    ensure => installed;
  }

  package { 'cryptsetup-initramfs' :
    ensure => installed;
  }

The cryptsetup package contains the tools needed for encryption. Note that installing cryptsetup-initramfs won’t get you cryptsetup on your initramfs unless you actually have encryption already set up (gah!). To fix this, set CRYPTSETUP=y in /etc/cryptsetup-initramfs/conf-hook. Supposedly this will be the default behavior in the future (it can’t happen soon enough).

Dropbear is an SSH server that you will use to remotely unlock your instance. Dropbear should generate host keys on installation (otherwise see the man page). Note that all the configuration needs to go in /etc/dropbear-initramfs . This has changed in recent years so many extant tutorials are wrong now. Here’s a puppet stanza for devops:

 file { '/etc/dropbear-initramfs' :
   source => "puppet:///modules/cryptroot/etc_dropbear-initramfs/${hostname}",
   recurse => true,
   purge   => false,
   mode    => 'u=+rwX,g=-rwx,o=-rwx',
   owner   => 'root',
   group   => 'root',
   notify  => Exec['update_initramfs'],
 }

where update_initramfs is defined as update-initramfs -u.

There are two files that need editing in that directory. One is authorized_keys – same as for openssh. Put your management key in there. The other is config, and we’ll be editing two lines:

DROPBEAR_OPTIONS="-p 12345 -s -j -k"
IFDOWN=*

Prosaically, this is the port to listen to, don’t allow passwords or port forwarding, and take down the interface after unlocking—so systemd doesn’t get confused during normal startup. You should use a different port than normal, so you can use the nagios negate plugin to monitor it and throw an alert when it appears. That means your hosting provider has rebooted a server you didn’t know about! To be safe, check your provider’s status page to make sure there’s an actual outage. The biggest risk here is that somebody steals your dropbear keys and sets up their server as a MitM. If your provider doesn’t know about an outage, check for shenanigans.

If you really want to be super-duper secure, randomize the port, per-server, and keep a configuration file with the appropriate keys and ports for each vm. That way if an attacker gets one of your servers, he doesn’t know that piece of information about another. In most cases, the dropbear exposure will be minimal and you should calculate your odds for where to put your time.

Next, we have to configure the kernel to have an IP address on startup. The initramfs config file suggests that you could put the configuration in there. I really wish that worked. For now, edit your /etc/default/grub file to contain an ip= stanza and set if.netnames to 0. The latter will revert your device names to eth0, eth1, etc. . make sure your /etc/network/interfaces matches the eth* standard.

The thought with dynamic network interface names is that it makes hardware interfaces easier to manage because PCI bus-enumeration order doesn’t matter. Now that there are more cloud vm’s than physical machines, the ancient default makes more sense than the ‘modern’ one. I’ve observed hosted servers have different device names on reboot (thanks, weirdo hypervisor) and then different names again on the next boot! eth0 is all most vm’s will have in the common case. Here’s a sample kernel parameter. Devops folk should templatize this (e.g. using Puppet facts):

net.ifnames=0 ip=203.0.113.42::203.0.113.1:255.255.255.0:secure.bfccomputing.com:eth0

Where the format is: server_ip::gateway:netmask:hostname:interface . Note the double-colon, which could have been an nfs server to get a root fs from, but it’s omitted for this use. Remember to run update-grub if your devops doesn’t do it automatically.

Note that you will need to have the appropriate network drivers in your initramfs. If you’re using a pure hardware-emulation hypervisor you will probably be OK, since Debian is heavily weighted towards hardware installs, but if your hosting provider uses paravirtualization (more efficient) then the default set of Debian initramfs modules won’t cut it. For example, for Xen you can add the xen_netfront driver. Use dmesg and lsmod to figure it out what to put in your module file. e.g.:

 file { '/etc/initramfs-tools/modules' :
   ensure  => file,
   content => 'xen_netfront
',
   mode    => '0644',
   owner   => 'root',
   group   => 'root',
   notify  => Exec['update_initramfs'],
 }

OK, lastly we need to add the tools we need to the initramfs image. To do this, create an executable file called something like /etc/initramfs-tools/hooks/management which will have contents like this (adapt to your preferences):

#!/bin/sh

set -e

PREREQS=""

prereqs() { echo "$PREREQS"; }

case "$1" in
    prereqs)
        prereqs
        exit 0
    ;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /sbin/e2fsck
copy_exec /sbin/mkfs.ext4
copy_exec /sbin/tune2fs
copy_exec /sbin/resize2fs
copy_exec /usr/bin/rsync

exit 0

When you run update-initramfs -u, it will copy those management tools and any libraries they need to the initramfs image. Add -v if you like to watch. Only you know your preferred data and layout, so copy what you need, being conscientious of minimizing total size. e.g. you may prefer dump to rsync. Do as much management as you can from your live system first.

OK, so hopefully your hosting provider gives you remote console through VNC. If so, you will need to reboot into the grub bootloader, edit the linux line on your default entry and add the parameter break to it. Then hit F10 to continue booting. If you can’t get any kind of console from your hosting provider, then you’ll need to add it to your grub line and take it out later. That’s a fingers-crossed kinda situation but you can always reimage if you have to!

break is a shortcut for break=premount which means to drop to a shell before the disk mounting steps happen in initramfs’s init script (which you can read for more details). You’ll get a prompt that says (initramfs) where you can type disk management commands. You’re doing this either at the console or via ssh to your dropbear instance.

Here’s the basic strategy:

back up your root partition. This can be a ramdisk or another partition on your hosted storage. If you have insufficient RAM and no additional partition space or room to resize your root, see if your hosting provider will let you rent a disk volume by the hour to get some extra space. If you use rsync, mount both filesystems and run it like, e.g.:

rsync -axlSWH /root/mnt/real/ /root/mnt/temp/

so that you get one filesystem with symlinks, hardlinks, and sparse files correctly. dump should handle this for you.
Unmount your old root if you need to and encrypt your disk. e.g.:

cryptsetup luksFormat /dev/xvda3
It will ask you for confirmation and the passphrase you want to use. If you are typing at a console, you can use a temporary one here and change it later. This passphrase encrypts the disk key, not the disk, and LUKS can have several ‘slots’ for key-encrypting-keys.
Open the LUKS volume. Following the Debian installer convention:

cryptsetup luksOpen /dev/xvda3 xvda3_crypt

enter your passphrase.
Format the LUKS volume. Don’t forget to match the UUID. e.g.:

/sbin/mkfs.ext4 -L secure.root -U `cut -f 2 -d ' ' < /proc/cmdline | cut -f 3 -d =` /dev/mapper/xvda3_crypt

You could also restore a dump .
Mount the LUKS volume. e.g.:

mount /dev/mapper/xvda3_crypt /root/mnt/old
dump or rsync the files back. e.g.:

rsync -axlSWH /root/mnt/temp/ /root/mnt/real/
umount everything you mounted and type exit to continue with a normal boot.
edit your /etc/crypttab file to reflect the encrypted root. e.g.:

xvda3_crypt UUID=76b93b85-89c2-a43d-8853-755f3ae5b1bb none luks,discard
This UUID is the uuid of the LUKS volume, not the ext4 it’s holding. Find it like:

# blkid|grep xvda3 /dev/xvda3: UUID="76b93b85-89c2-a43d-8853-755f3ae5b1bb" TYPE="crypto_LUKS" PARTUUID="d6a70d47-03"
again, update-initramfs -u . This will make sure your crypttab is now inside your initramfs.
Test time! Reboot, connect with ssh e.g.:

ssh -i my.id_rsa -p 12345 root@secure.bfccomputing.com
and run:

cryptroot-unlock
It should find your root volume, ask for your password, then mount it, down the if, and relinquish control. You will see your ssh session disconnected – that is normal. ssh back in to confirm.

Once that has all succeeded, don’t forget to add a decent passphrase for your volume. Look into the luksAddKey and luksRemoveKey commands in cryptsetup. Since most web VNC clients do not support pasting data it is really hard to get a great key going from a console, but over ssh you don’t have that limitation.

Good luck and leave comments. Also, go forth and make this the new normal!