Scaling Puppet in EC2

At SmugMug, we’re constantly scaling up and down the number of machines running in ec2. We’re big fans of puppet, using it to configure all of our machines from scratch. We use generic Ubuntu AMIs provided by Canonical as our base, saving ourselves the trouble of building custom AMIs every time there is a new operating system release.

To help us scale automatically without intervention (we use AutoScaling), we run puppet in a nodeless configuration. This means we do not run a puppet master on any machines in our infrastructure. All machines run puppet independent of any other, removing dependencies and improving reliability.

I will first explain nodeless puppet, then I will dive into how we use it.

Understanding Nodeless Puppet

Most instructions for setting up puppet tell you to create a puppet master instance that all puppet agents talk to. When an agent needs to apply a configuration, the master compiles a config and hands it back to the agent. With nodeless, the puppet agent compiles its own configuration and applies it to the host.

We start with a simple puppet.conf file that is pretty generic:


Then we create a top-level manifest called mainrun.pp. In our setup this manifest lives in a directory called /manifests. An example mainrun.pp:

include ntp
include puppet
include ssh
include sudo

if $hostname == "foo" {
    include apache2

There is also a /modules directory that contains puppet modules. Each include statement in the mainrun.pp manifest exists as a module.

Once we have all of our modules created and listed appropriately in mainrun.pp, we run puppet with the apply command: sudo puppet apply /etc/puppet/manifests/mainrun.pp. Puppet will then run and do all that our manifests tell it to.

Scaling Puppet

Upon booting, all machines download their entire puppet manifest code tree from an Amazon S3 bucket. Then puppet is run and the machine is configured. By using S3, we’re leveraging Amazon’s ability to provide highly available access to files despite server or data center outages.

To help keep our changes to puppet sane, we use a git repository. When anyone does a push to the central repository server, it copies our files to our Amazon S3 bucket. The S3 bucket has custom IAM access rules applied so puppet can only see its bucket and no other.

When we launch a new instance in ec2, we use the --user-data-file option in ec2-run-instances to run a first-boot script that sets us up with puppet.

A simple first-boot script:


apt-get update
apt-get --yes --force-yes install puppet s3cmd
wget --output-document=$S3CMDCFG$BUCKET_PUPPET/s3cmd.cfg
sed -i -e "s#__AWS_ACCESS_KEY__#$AWS_ACCESS_KEY#" \
chmod 400 $S3CMDCFG

until \
    s3cmd -c $S3CMDCFG sync --no-progress --delete-removed \
    s3://$BUCKET_PUPPET/ /etc/puppet/ && \
    /usr/bin/puppet apply /etc/puppet/manifests/mainrun.pp ; \
do sleep 5 ; done

s3cmd.cfg in s3 is a publicly accessible template file, containing placeholders for the AccessKey and SecretKey that is supplied by the first-boot script. As s3cmd.cfg is a publicly accessible file, do not place any real credential data in it.

Puppet will install additional tools for keeping puppet running on all machines.

Keeping Puppet Running

As puppet is not running in agent mode, it does not wake up from a sleeping state to apply manifests that have changed since booting. We use cron to run puppet every 30 minutes. Our cron entry:

*/30 * * * * sleep `perl -e 'print int(rand(300));'` && /usr/local/sbin/ > /dev/null

We have the 5 minute sleep in the cron to ensure that machines run puppet at a staggered interval. This is to prevent all machines from restarting a service at the same moment (Apache for example), causing an interruption for our customers.

We have three simple scripts that are also installed by puppet for puppet:



/usr/bin/s3cmd -c /etc/s3cmd.cfg sync --no-progress \
    --delete-removed s3://$BUCKET_PUPPET/ /etc/puppet/


/usr/bin/puppet apply /etc/puppet/manifests/mainrun.pp

We split puppet runs into three scripts to help with manual maintenance of a box. We run sudo if we simply want puppet to run immediately. sudo is handy when making manual changes to the puppet manifests and modules for testing purposes; once testing is complete we copy our changes back into the git repository. sudo is infrequently run manually, mostly for resetting the puppet config when making manual testing changes.

Final Thoughts

As you can imagine there is a lot more involved in our manifests. There are a large number of conditional operators that enable and disable different parts of the manifests depending on what role an instance has.

EC2 tags have proven to be invaluable for us; each machine is assigned two tags that exactly describe any role. A script for reading the ec2 tags at boot combined with a custom fact is used to expose the ec2 tags to puppet.

Future posts about puppet may include:

  • How we use EC2 tags for determining instance roles
  • Speeding up initial booting of instances
  • Using custom facts to enable one-off configurations for testing or debugging

— Shane Meyers, SmugMug Operations

11 thoughts on “Scaling Puppet in EC2

  1. blalor January 14, 2013 / 3:43 pm

    Thanks for posting this, Shane. I’m new to Puppet and I really like the nodeless concept, but this is the first write-up I’ve seen of how to bootstrap the process using something like –user-data-file. Would you mind sharing the source for the custom fact for reading ec2 tags?

    • shane January 14, 2013 / 5:39 pm

      The custom fact for reading ec2 tags will be shared in a future blog post. I need to clean up the code so it is suitable for open sourcing, but once that’s done you’ll read about it here. Hopefully I’ll have that done within a few weeks (but no promises).

  2. Joe Block January 15, 2013 / 11:00 am

    Shane, this is a good intro to running masterless. I published very similar code to github at a previous job – I wrote that to allow me to manage our Macintosh fleet without them having to be on the corp network, and for some of our linux servers, but switched jobs and never got around to writing a post about it.

    I packed the manifests into a deb using fpm for linux, and into a pkg (wrapped in a dmg) on the Macs, then stuff them into s3. This makes it easier to have multiple environments, since the different machines can load the appropriate deb/pkg, but since I wasn’t using a git hook it involved manually running the rake tasks to generate the packages.

  3. JMS April 26, 2013 / 2:19 pm

    any chance on getting a sample of code for handling custom facts in a masterless configuration. That is the one big blocker to rolling out masterless puppet in an enterprise environment…all the other shortcomings are either irrelevant or easily overcome, but puppet 3.x does not appear to handle custom facts without syncing to a server.

    • shane April 26, 2013 / 2:50 pm

      I’ll see what I can do. I’m have a lot going on right now, but once I have some time, I’ll write up a post explaining how to handle custom facts. *Maybe* within a week or two.

    • blalor April 26, 2013 / 3:05 pm

      You can set facts via a json, yaml or text file in /etc/facter/facts.d if you’re using the puppet stdlib module. It’s also really easy to shell out to a command from within a custom facter plugin. The plugin would live in a module on the puppet modulepath. If you don’t need the dynamic aspect of a fact being determined at runtime, facts.d is an awesome solution I’m using with great success right now.

  4. Madhurranjan Mohaan January 27, 2014 / 4:58 am

    This is quite cool. How are you managing passwords ? We have passwords in our hiera . Would like to know your approach. Thanks.

    • shane January 29, 2014 / 1:06 pm

      We use IAM roles for permissions with AWS resources. As for things like user logins, we only store the public key in puppet.

  5. blainehilton (@blainehilton) May 28, 2014 / 11:50 am

    This is a great article. I’m wondering if you are going to be providing more details soon regarding the connection between AWS EC2 tags and Puppet. I am able to create puppet facts from EC2 tags properly, however I’m not sure how to pull those into the puppet world. I’m trying the has_role() function from for instance and am having some problems with it.

  6. Dennis gearon August 9, 2014 / 8:52 pm

    Ever get around to writing up followups to this?

Comments are closed.