We have started to use Netflix’s *amazing* Asgard to manage our deployments of our service clusters. If you haven’t tried it yet, I strongly recommend it. It is designed to allow you to build and deploy autoscaled clusters of machines with tremendous ease. Having said that, any solution we developed for maintaining configuration management with chef needed to fit into a paradigm that would work well with Asgard. The Cloud Formation documentation which is supplied by Opscode gave us a good pattern to use for our configuration:

1. Store validation and secret keys in S3
2. Create a user data script which can Chef bootstrap a server
3. Launch servers in Autoscaling groups for great good

As a general model, this worked perfectly, however we needed to add some of our own tweaks to completely realize this solution.

We created a bucket in S3 which holds our validation.pem and encrypted_data_bag_secret keys. This needs to be accessible to our server. We also store our first_run.json files there, but more about that later.

Step 2 actually requires work . In the end, we developed 2 scripts. One is the script which is stored in user data. This is a simple bash script whose primary purpose is to download the *actual* bootstrap script. Here is an example of what that bash script looks like:

This is a sample of what you would put into your user-data:

#/bin/bash

export AWS_ACCESS_KEY_ID=********************
export AWS_SECRET_ACCESS_KEY_ID=***************************
export LOGSTASH_HOST=*****************
export LOGSTASH_PORT=*****************
export GITHUB_TOKEN=***********************

curl -o /tmp/virtualenv.py https://raw.github.com/pypa/virtualenv/master/virtualenv.py
python /tmp/virtualenv.py /tmp/install
/tmp/install/bin/pip install boto
/tmp/install/bin/pip install sh
/tmp/install/bin/pip install requests

curl -o /tmp/bootstrap.tar.gz https://gist.github.com/gists/4218533/download
mkdir /tmp/bootstrap
cd /bootstrap
tar -xzvf bootstrap.tar.gz
mv */bootstrap.py /tmp

/tmp/install/bin/python /tmp/bootstrap.py -b your-bucket -t instance-type -e stg -c http://your.chef.url -k $AWS_ACCESS_KEY_ID -s $AWS_SECRET_ACCESS_KEY_ID -l $LOGSTASH_HOST -L $LOGSTASH_PORT -g $GITHUB_TOKEN > /tmp/bootstrap.log 2> /tmp/bootstrap.error

rm -rf /tmp/install /tmp/bootstrap.tar.gz /tmp/bootstrap.py

Most of this should look fairly straight forward, although a couple things are somewhat unique to our setup. At the beginning we define some environment that are needed to bootstrap the machine. The first two are AWS credentials which will be used to grab the validation.pem and encrypted_data_bag_secret file from S3. A note about these credentials: because the user data script is available to all users on the instance (via the EC2 metadata url), these credentials should have very limited permissions. Basically, they need to have permission to read items from the S3 bucket containing the chef secrets and permission to edit tags on EC2 instances.

We use Logstash to process all of our logs so, we supply the host and port to our Logstash server to collect certain data. Finally, we are posting the bootstrap reports to private gist files, so we supply a github OAuth token with “gist” permission.

The script next creates a virtualenv which will be used to run the actual bootstrap script. It also installs a couple of dependencies which we use to make the bootstrap script simpler. Next, we download the gist which contains the bootstrap script and store it in /tmp.

Then, we actually invoke the bootstrap script. This does all the magic. Lastly, we delete all of the temporary files we have created.

This is a pretty general purpose user-data script. The bootstrap script could pretty much do anything, but in our case we are bootstrapping chef.

Instead of pasting the entire bootstrap script here, I’ll just point at the gist and then discuss a few specific parts. The entire script can be found here.

The first thing the script does is add a generate the details about the newly created server. This includes the name, run_list, environment and generic server type.

response = urllib2.urlopen(INSTANCE_ID_URL)
instance_id = response.read()
server_name = "i_{0}_{1}_{2}".format(environment, server_class, instance_id)
key = Key(bucket)
key.key = 'first-boot-scripts/{0}/{1}.json'.format(environment, server_class)
key.get_contents_to_filename('/etc/chef/first-boot.json')

with open('/etc/chef/first-boot.json', 'r') as handle:
run_list = json.loads(handle.read())['run_list']

return {
'name': server_name,
'instance_id': instance_id,
'environment': environment,
'class': server_class,
'run_list': run_list
}

This grabs the first-boot.json file from S3. This contains the run_list for the server plus any attributes which need to be set at the node level. It also generates a name for the server.

Next we add a Name tag the instance to the instance. This means that you see a human friendly name in the AWS console.

After we have generated and stored all of the metadata about the new instance, we configure and run Chef for the first time.

response = urllib2.urlopen('http://www.opscode.com/chef/install.sh')
with open('/tmp/chef-install.sh', 'w') as handle:
handle.write(response.read())
chef_install_log = sh.bash('/tmp/chef-install.sh')
configure_chef(server_details, chef_server, bucket)
chef_client = sh.Command('/opt/chef/bin/chef-client')
chef_run_log = chef_client(config='/etc/chef/client.rb', environment=server_details['environment'], j='/etc/chef/first-boot.json')

This installs Chef using the default installer, then runs chef-client using the configuration stored earlier.

Finally, we store the output from the chef-client run in a private gist and send the url in a message to Logstash. That is all straight-forward http and socket work, but it gets us all of the information we need about the status of the bootstrap.

At this point our new server has been automatically bootstrapped with chef and should be working.

In a future post, I’ll discuss how we integrate this with our Asgard deployment.

It’s actually pretty straightforward.

Starting from a data perspective, you can think of insight as a product of noticing relationships between facts that are not entirely clear by looking at each fact by itself. Insight is going beyond the data and even beyond drawing the facts from it, to assessing how those facts come together to form an overall picture of what has, and is, happening.

For example, the fact that a company has declining revenues is interesting but not insightful. Likewise, the fact that its costs are going up alone doesn’t paint a picture. Nor does that fact that labor costs are on the rise because of a scarcity of expertise. But together, we can read the story of a company who is on the ropes because it is competing for a resource that every other company in its industry is hunting as well. And, true insight is to understand that this situation often leads to consolidation, which is likely on the horizon.

Of course, since we can describe this situation means that we can encode it along side similar patterns of fact and data relations that give rise to insight. Once encoded and turned into data analytics, Quill can use those analytics to move from data, to fact, to insight. That is to say, it can express human insight at machine scale.

Now some may say that insight is a human trait that goes beyond the numbers. I would agree with them. Insight goes beyond the numbers to the facts and the relationships between them. Numbers are the “grist for the mill” of insight. They are part of the human experience of assessing the world and determining what has happened and will happen next. That Quill can take this human skill and execute it at tremendous speed, scale and quality is less a testament to the machine and more to our ability to understand ourselves and how we think.