Instance store HVM AMIs for Amazon EC2

At SmugMug, we have a rule for EC2 instances: do not use EBS for production systems unless instance store is not an option for the instance type. EBS is networked storage, which has many useful capabilities, but in our production environment we have automated builds for our servers and have no need for the added complexity of networked storage.

We’ve been successful with this goal for the most part, but the one area that we couldn’t use instance store is instances that required HVM virtualization, such as cc2.8xlarge and i2.* instances. EC2 offers two ways to boot instances, off of an instance store (like local disk) or via EBS, and two types of virtualization, Hardware Virtual Machine (HVM) and Paravirtual (PV). Recently AWS added instance store volumes to these HVM-only instances, so we wanted to take advantage of that. To create instance-store HVM AMIs, AWS guided us to boot an HVM AMI and convert it to instance store.

Before beginning, make sure you have both access keys and X.509 certificates for use. You must have both types of credentials for these steps to work. See bundle AMI prerequisites if you need guidance.

Here are the steps for creating instance-store HVM AMIs:

  1. Boot an EBS HVM instance. I used ami-dfa98cb6 (Ubuntu 12.04.3) on a c3.large instance (do not use 20131205 AMIs as they don’t boot on i2.* due to a bug)
  2. Set up the instance with your environment; we use masterless Puppet to configure the host with our base environment
  3. Copy the below make-hvm-s3.sh script to the instance
  4. Copy your cert.pem and key.pem to /tmp
  5. export AWS_SECRET_KEY="foo"
  6. Run the make-hvm-s3.sh script. The script creates the machine image, uploads it to s3, and registers it with EC2 so it is available for use
  7. Document the AMI created somewhere

make-hvm-s3.sh script:

#!/bin/bash
set -e

USER='root'
if [ `whoami` != $USER ]; then
    sudo -u $USER -H AWS_SECRET_KEY=$AWS_SECRET_KEY $0 "$@"
    exit $?
fi

if [ -z "$AWS_SECRET_KEY" ] ; then
    echo "ERROR: \$AWS_SECRET_KEY not set"
    echo "    export AWS_SECRET_KEY=foo"
    exit 2
fi

AWS_ACCESS_KEY="foo"
AWS_ACCOUNT="123412341234"
REGION="us-east-1"
BUCKET="s3-bucket"

STAMP=`/bin/date +%s`
PREFIX="hvm-s3"
export EC2_HOME=/opt/ec2-ami-tools-1.4.0.10/
export EC2_AMITOOL_HOME=/opt/ec2-ami-tools-1.4.0.10/

apt-get -y install ruby1.8 gdisk kpartx grub unzip python-pip
apt-get -y autoremove

if ! [ -d /opt/ec2-ami-tools-1.4.0.10 ] ; then
    curl -o/tmp/ec2-ami-tools-1.4.0.10.zip http://s3.amazonaws.com/aws-dev-support/beta/ec2-ami-tools-1.4.0.10.zip
    unzip /tmp/ec2-ami-tools-1.4.0.10.zip -d /opt
fi

/bin/rm -f /var/cache/apt/archives/*deb

sed -i 's;ro console=hvc0;ro console=ttyS0 xen_emul_unplug=unnecessary;' /boot/grub/menu.lst

$EC2_HOME/bin/ec2-bundle-vol \
    --privatekey /tmp/key.pem \
    --user $AWS_ACCOUNT \
    --cert /tmp/cert.pem \
    --arch x86_64 \
    --partition mbr \
    --prefix $PREFIX-$STAMP \
    --block-device-mapping ami=sda,root=/dev/sda1,ephemeral0=sdb,ephemeral1=sdc,ephemeral2=sdd,ephemeral3=sde \
    --exclude `find /tmp | tail -n+2 | tr '\n' ','` \
    --include `find / -name '*.gpg' -o -name '*.pem' -o -name 'authorized_keys' | grep -v '^/mnt\|^/tmp' | tr '\n' ','`

$EC2_HOME/bin/ec2-upload-bundle \
    --bucket $BUCKET \
    --manifest /tmp/$PREFIX-$STAMP.manifest.xml \
    --access-key $AWS_ACCESS_KEY \
    --secret-key $AWS_SECRET_KEY \
    --batch \
    --location US \
    --retry

echo -e "[default]\nregion = $REGION\naws_access_key_id = $AWS_ACCESS_KEY\naws_secret_access_key = $AWS_SECRET_KEY" > /tmp/aws
export AWS_CONFIG_FILE="/tmp/aws"
pip install awscli
aws ec2 register-image \
    --image-location $BUCKET/$PREFIX-$STAMP.manifest.xml \
    --name $PREFIX-$STAMP \
    --virtualization-type hvm

Notes for the script:

  • Set up the variables on lines 16-19 to match your environment
  • We’re using a beta version of ec2-ami-tools that has support for HVM on instance store, previous versions of these tools will not work: EC2 AMI tools v1.4.0.10 (beta)
  • AWS_SECRET_KEY is passed through your shell so it doesn’t accidentally get baked into the AMI
  • You may need to adjust the include/exclude lines for ec2-bundle-vol as appropriate for your setup
  • With Ubuntu/Debian, you need to include *.gpg, *.pem, and authorized_keys files otherwise you’ll have problems connecting and performing apt-get operations
  • Adjusting menu.lst ended up being critical for getting this working, as well as using an older grub (0.97). Without these two changes, our AMIs would not boot
  • AWS list of AMI types to instance types

Thanks to Joshua F. from AWS support for help getting this going.

— Shane Meyers, SmugMug Operations