My rabbit hole(s) of enforcing multi-factor authentication in AWS

I’ve been using AWS since EC2 was in limited beta.  At 42Lines we’ve built a lot of infrastructure on AWS that pre-dates most current AWS features like IAM.  While we’ve been using IAM for all new projects, we recently decided to enforce multi-factor authentication (MFA) for all infrastructure and cease all use of account (root) keys.  This led to the following adventure; the theme you’ll notice is adding support for security tokens.  Security token support is required as using keys issued with MFA have three parts (access_key, secret_access_key, security_token) rather than the two (access_key, secret_access_key) most AWS users would find familiar.

We needed a way to easily authenticate IAM users with MFA while in their shell environments and manage their (many) related shell environment variables.  This led to the creation of some code that will be open-sourced shortly.

We needed a way for our existing Perl-based orchestration/launching system to assume IAM roles.  Net::Amazon::EC2 doesn’t support the IAM API, so I decided to add a Python script to make the AssumeRole call and emit the temporary credentials as JSON.  I really dislike having multiple languages for a tool, but our long term plan is to move this tool to Python anyway, so this step aligns with our long term plans.  The Python script for making the AssumeRole call will also be a part of the aforementioned open-sourcing of code.

However, boto‘s STS does not support passing a security token.  So, I proposed a fix for that.

Our Perl tool would be run (in most cases) by MFA IAM users, so the Perl tool needed to pass a security token for the calls it makes.  I previously added support for IAM temporary security credentials from the on-instance metadata service and support for launching with IAM roles to Net::Amazon::EC2.  I now needed to add passing the security token when creating the connection object; hence this proposed change.

Further, as our Perl AWS tool was from nearly the beginning of AWS time it used SimpleDB to keep track of various instance information.  This is because it was written before tags existed in AWS.  I didn’t bother to see if the SimpleDB Perl library had been updated to support STS issued temporary credentials; I took a side trip to move all information formerly stored in SimpleDB to EC2 instance tags and remove all use of SimpleDB.

Finally there were some boto-using command line utilities (including the new MFA-related ones I wrote) that now needed to pass a security token to boto.  I’d much rather handle this via environment variables (as opposed to adding an option to every script that is passed to the boto connection object), so tonight I finished this adventure by writing and submitting another proposed patch to boto.

The end result of this overly long adventure is all staff using MFA tokens for all AWS access.  Hopefully all of my patches are adopted upstream and I can remove the custom-built packages from our repos.

If the holidays allow I’ll publish a Github repo of my new MFA helper scripts in the next few days.

Layer2 for Route53 in Boto

Finally pushed my take on the Slick53 integration proposal for Boto; where does the time go?



ssh-agent-proxy is my new SSH love

Normally Shane Meyers clues me into nifty tricks about SSH, but this one I found on my own.  Fortunately I searched for an existing solution before coding it myself.  The new love of my SSH life is ssh-agent-proxy.  It solves the problem of ensuring the SSH_AUTH_SOCK environment variable in your remote screen sessions is connected to the agent on your connecting machine (i.e. laptop.)  I’ve seen many hacks for making this connection, usually fragments spit out at login that have to be executed in each child shell.  ssh-agent-proxy solves this problem seamlessly.

When run ssh-agent-proxy daemonizes and creates a listening socket at a well known location (default: /tmp/sshUID/agent.sock).  When it receives a connection on that socket it connects it to the upstream SSH agent.  It does this by checking all files in /tmp for the correct, live socket for a currently connected SSH connection and proxies all communication.  It also reaps dead socket files for the user in /tmp.

This means I can re-connect to a host, run screen -r and all SSH agent requests from all of my existing shells are connected to my current SSH session.  I downloaded the ssh-agent-proxy tarball, reviewed the code, and executed it two months ago; I’ve nearly forgotten it is running and am enjoying my pain-free ssh-agent-proxy life.


Boto bug with AWS ELBs

The past two weeks I’ve been spending more time tending to our cloud infrastructure.  It seems that we use areas of AWS that most boto users don’t seem to as I usually find a bug in boto each time I circle back to our cloud assets.  Last time it was Route53 WRRs.  This time I found that elbadmin was emitting strange output for some of our ELBs.  It turns out that boto does not handle ELBs with policies correctly.  I submitted a bug report and patch.  I love working with Boto and Mitch does a great job with it; I’m always happy to be able to contribute code for a missing feature or bug fix.


New and updated Ubuntu packages for AWS

I attended my first Ubuntu related conference earlier this month, the Ubuntu Developer Summit for the Quantal release.  We use Ubuntu heavily at 42lines, so it is a bit embarrassing that I didn’t even know about UDS until I saw it mentioned on Eric Hammond’s blog.  I was planning on being in SF a few days after UDS was being held, so I adjusted my travel plans and dropped by Oakland.  The main attraction for me was the session on packaging more AWS tools for Ubuntu.  I had worked with Scott Moser of Canonical earlier in the year on having the ELB tools packaged for Ubuntu.  At this session I signed up for updating the RDS and IAM tools to the latest upstream version and packaging the ELB and Import/Export tools.  So far the first three items are done.  The results can be seen in the awstools-dev Launchpad repos and soon should be seen in the awstools-dev PPAs and Quantal itself.


Listing Amazon EC2 events

I have a post over on our work blog about listing pending events in Amazon’s EC2 using python and boto.  Maybe I’ll post to the work blog more regularly than I post here.


Atlassian Crowd, Apache, and cookie-based authentication

We and some of our customers use Atlassian’s Crowd as a SSO solution.  I was surprised to find that Atlassian does not provide code to enable Apache to use cookie-based authentication.  Their only Apache solution is based around HTTP Basic Authentication.  I found an open JIRA issue about this and from there found some code examples laying around.  None of them were an all-in-one solution, so I combined the various bits into a single solution.  Hopefully Atlassian will provide an official solution in the near future, or at least patch their official Perl libraries to include the calls utilized by the code that does implement a solution.

Munin, mysql, and mk-heartbeat

I’ve been playing around a bit with Munin and the great plug-in mysql_ I wrote a patch that adds support for mk-heartbeat from the Maatkit toolkit.

Matching up private and public SSH keys

I’m assuming we’re talking about OpenSSH.  There are other things out there like Putty and (incredibly) commercial versions of SSH, but thankfully I don’t have to touch those.

The easiest mechanism for matching up the keys are the fingerprints.  The fingerprint is always of the public key.  There is no fingerprint for the private key.  There doesn’t need to be, as the private key contains enough information to generate the public key as well (for example, ssh-keygen -y -f private.key).  So, when you ssh-add a private key and then run ssh-add -l,  you are seeing the fingerprint of the public key.  You can then use ssh-keygen -l -f publickeyfile on the remote host to generate a fingerprint to compare against the fingerprint returned by ssh-add -l.

Annoyingly the -f option to ssh-keygen does not support STDIN, so you have to write the test key to a file before generating a fingerprint.  This complicates, say, generating fingerprints for all of the public keys in an authorized_keys file on a host or fingerprinting a private key with  ssh-keygen -y private.key | ssh-keygen -l -f –

How Amazon calculates private key fingerprints

Short answer: I still don’t know as I haven’t looked at the code (or if it is even available.)

Even shorter answer: ec2-fingerprint-key

Why such an obvious post?  Well, I spent far too much time trying to figure out how to match up the information returned by ec2-describe-keypairs with the files I had on disk.  This would have been a two minute operation had Amazon saw fit to mention ec2-fingerprint-key in the “related operations” section of the documentation for ec2-describe-keypairs (or ec2-add-keypair, or ec2-delete-keypair.)  I’ve submitted feedback to Amazon’s documentation people about this omission.  I imagine most people, like myself, do not read the Amazon documentation cover to cover.

Perhaps this was me being too smart for my own good.  Instead of browsing the API calls, I set about trying to figure out how private key fingerprints were calculated based on my knowledge of how they are calculated for public keys.  I’m quite familiar with public key fingerprints from the use of SSH, primarily when using ssh-agent.  I assumed that private key fingerprinting, while perhaps a bit more obscure, was commonplace.  It turns out only Amazon does this and probably made up their own method.  I did find one reference to how (sha1 of the DER of the private key), but my quick attempt at

openssl rsa -in test.key -outform DER -pubout | openssl sha1 -c

didn’t match.  Since I have the whole public-key fingerprinting stuff swapped into my head, I think I’ll write a quick post about it.