Home > DevOps > Instance store HVM AMIs for Amazon EC2

Instance store HVM AMIs for Amazon EC2

January 29, 2014

At SmugMug, we have a rule for EC2 instances: do not use EBS for production systems unless instance store is not an option for the instance type. EBS is networked storage, which has many useful capabilities, but in our production environment we have automated builds for our servers and have no need for the added complexity of networked storage.

We’ve been successful with this goal for the most part, but the one area that we couldn’t use instance store is instances that required HVM virtualization, such as cc2.8xlarge and i2.* instances. EC2 offers two ways to boot instances, off of an instance store (like local disk) or via EBS, and two types of virtualization, Hardware Virtual Machine (HVM) and Paravirtual (PV). Recently AWS added instance store volumes to these HVM-only instances, so we wanted to take advantage of that. To create instance-store HVM AMIs, AWS guided us to boot an HVM AMI and convert it to instance store.

Before beginning, make sure you have both access keys and X.509 certificates for use. You must have both types of credentials for these steps to work. See bundle AMI prerequisites if you need guidance.

Here are the steps for creating instance-store HVM AMIs:

  1. Boot an EBS HVM instance. I used ami-dfa98cb6 (Ubuntu 12.04.3) on a c3.large instance (do not use 20131205 AMIs as they don’t boot on i2.* due to a bug)
  2. Set up the instance with your environment; we use masterless Puppet to configure the host with our base environment
  3. Copy the below make-hvm-s3.sh script to the instance
  4. Copy your cert.pem and key.pem to /tmp
  5. export AWS_SECRET_KEY="foo"
  6. Run the make-hvm-s3.sh script. The script creates the machine image, uploads it to s3, and registers it with EC2 so it is available for use
  7. Document the AMI created somewhere

make-hvm-s3.sh script:

#!/bin/bash
set -e

USER='root'
if [ `whoami` != $USER ]; then
    sudo -u $USER -H AWS_SECRET_KEY=$AWS_SECRET_KEY $0 "$@"
    exit $?
fi

if [ -z "$AWS_SECRET_KEY" ] ; then
    echo "ERROR: \$AWS_SECRET_KEY not set"
    echo "    export AWS_SECRET_KEY=foo"
    exit 2
fi

AWS_ACCESS_KEY="foo"
AWS_ACCOUNT="123412341234"
REGION="us-east-1"
BUCKET="s3-bucket"

STAMP=`/bin/date +%s`
PREFIX="hvm-s3"
export EC2_HOME=/opt/ec2-ami-tools-1.4.0.10/
export EC2_AMITOOL_HOME=/opt/ec2-ami-tools-1.4.0.10/

apt-get -y install ruby1.8 gdisk kpartx grub unzip python-pip
apt-get -y autoremove

if ! [ -d /opt/ec2-ami-tools-1.4.0.10 ] ; then
    curl -o/tmp/ec2-ami-tools-1.4.0.10.zip http://s3.amazonaws.com/aws-dev-support/beta/ec2-ami-tools-1.4.0.10.zip
    unzip /tmp/ec2-ami-tools-1.4.0.10.zip -d /opt
fi

/bin/rm -f /var/cache/apt/archives/*deb

sed -i 's;ro console=hvc0;ro console=ttyS0 xen_emul_unplug=unnecessary;' /boot/grub/menu.lst

$EC2_HOME/bin/ec2-bundle-vol \
    --privatekey /tmp/key.pem \
    --user $AWS_ACCOUNT \
    --cert /tmp/cert.pem \
    --arch x86_64 \
    --partition mbr \
    --prefix $PREFIX-$STAMP \
    --block-device-mapping ami=sda,root=/dev/sda1,ephemeral0=sdb,ephemeral1=sdc,ephemeral2=sdd,ephemeral3=sde \
    --exclude `find /tmp | tail -n+2 | tr '\n' ','` \
    --include `find / -name '*.gpg' -o -name '*.pem' -o -name 'authorized_keys' | grep -v '^/mnt\|^/tmp' | tr '\n' ','`

$EC2_HOME/bin/ec2-upload-bundle \
    --bucket $BUCKET \
    --manifest /tmp/$PREFIX-$STAMP.manifest.xml \
    --access-key $AWS_ACCESS_KEY \
    --secret-key $AWS_SECRET_KEY \
    --batch \
    --location US \
    --retry

echo -e "[default]\nregion = $REGION\naws_access_key_id = $AWS_ACCESS_KEY\naws_secret_access_key = $AWS_SECRET_KEY" > /tmp/aws
export AWS_CONFIG_FILE="/tmp/aws"
pip install awscli
aws ec2 register-image \
    --image-location $BUCKET/$PREFIX-$STAMP.manifest.xml \
    --name $PREFIX-$STAMP \
    --virtualization-type hvm

Notes for the script:

  • Set up the variables on lines 16-19 to match your environment
  • We’re using a beta version of ec2-ami-tools that has support for HVM on instance store, previous versions of these tools will not work: EC2 AMI tools v1.4.0.10 (beta)
  • AWS_SECRET_KEY is passed through your shell so it doesn’t accidentally get baked into the AMI
  • You may need to adjust the include/exclude lines for ec2-bundle-vol as appropriate for your setup
  • With Ubuntu/Debian, you need to include *.gpg, *.pem, and authorized_keys files otherwise you’ll have problems connecting and performing apt-get operations
  • Adjusting menu.lst ended up being critical for getting this working, as well as using an older grub (0.97). Without these two changes, our AMIs would not boot
  • AWS list of AMI types to instance types

Thanks to Joshua F. from AWS support for help getting this going.

— Shane Meyers, SmugMug Operations

Categories: DevOps
  1. ToKy
    February 21, 2014 at 9:07 pm

    Why would you put your AMIs, in S3? Doesn’t that make the process of spinning up a new instance even longer? We do have to share the AMIs with all our VPCs but that is cheaper than using S3 and quicker.
    We do use the instance storage on both cc2.8xls and hs1.8xls. We build our AMIs monthly (which I hate). We did run into problems with disk resizing in HVM instances, lost a good day on figuring that one out.

    • February 23, 2014 at 9:46 am

      In our experience, spinning up an instance store is still relatively quick. Instances launch within one minute, which is fast enough for us. Keep in mind, EBS snapshots are loaded from S3 as well, so either way S3 is being used.

      We’ve been using EC2 for a long time (before EBS was even an option), so we’re used to the advantages and disadvantages of s3-backed instances. Almost all of our instances are disposable, so the long term persistence that EBS offers is unnecessary for our needs.

  2. dseth
    March 13, 2014 at 5:05 pm

    Works like a charm. I was struggling to boot the instance (HVM with instance store) until I found this blog. Great work!

  3. Pradeep
    October 28, 2014 at 5:35 am

    Hi,

    I am trying your blog to convert EBS-backed HVM instance to instance-store-backed. I am working with centos 6.5

    I am getting the following error:

    /opt/ec2-ami-tools-1.4.0.10/lib/ec2/platform/linux/image.rb:653:in `execute’: Failed to execute: ‘modprobe loop’ (FatalError)
    from /opt/ec2-ami-tools-1.4.0.10/lib/ec2/platform/linux/image.rb:293:in `format_image’
    from /opt/ec2-ami-tools-1.4.0.10/lib/ec2/platform/linux/image.rb:97:in `make’
    from /opt/ec2-ami-tools-1.4.0.10//lib/ec2/amitools/bundlevol.rb:183:in `bundle_vol’
    from /opt/ec2-ami-tools-1.4.0.10//lib/ec2/amitools/bundlevol.rb:230:in `main’
    from /opt/ec2-ami-tools-1.4.0.10/lib/ec2/amitools/tool_base.rb:201:in `run’
    from /opt/ec2-ami-tools-1.4.0.10//lib/ec2/amitools/bundlevol.rb:238:in `’

    Can you please help me.

    Regards,
    Pradeep

  1. No trackbacks yet.
Comments are closed.
%d bloggers like this: