Mesos & Django (and Aurora and CircleCI too)

I gave a talk at Python Ireland about using Mesos, Django, Aurora and CircleCI together for deployment bliss. These are some of my notes.

Setting up

First follow the Aurora Vagrant instructions to set up Aurora. This gets you a working Zookeeper, Mesos master + slave and Aurora itself. I tried using Ansible to jinny up my own little cluster but that was a little bit too much to get going while trying to understand the Aurora build process at the same time :)

So the first step was a vagrant up.

Next I installed a few useful packages:

apt-get install python-dev ntp htop unzip

Then some useful python stuff:

curl https://bootstrap.pypa.io/get-pip.py | sudo python
sudo /usr/local/bin/pip2.7 install -U setuptools pip wheel virtualenv awscli

These bits were needed for running my Django app in virtualenvs (one per Mesos container). Normally I'd advocate using nothing from the host machine (so put everything you need in the slug or use a container), but this is handier for a demo :)

The Django Code

You can find a trivial Django app I called Acaversity here: https://github.com/micktwomey/acaversity. It's a really small Django app, the most interesting bit is the circle.yml.

This is used to build the app in CircleCI, you can see the latest builds here.

The general build process is:

  1. Pip install and test as normal
  2. Make a slug directory
  3. Put all the wheel dependencies in there
  4. Tar it up
  5. Upload the slug to S3

The key missing bit here is some kind of hook which pokes the server controlling the aurora deployment config (more on that later). A really simple approach is to write out a config and upload that, keeping it together with the code (and thus allowing for branching). Then you fetch and run that config.

The reason I use slugs is the Aurora deployment doesn't have to know what you are doing, all it needs to know is to untar, run bin/setup.sh and then bin/start.sh. This makes it possible to completely switch around what you are doing to start a process.

Aurora

Apart from setting up Aurora above the other bit you need is the acaversity.aurora (and S3 keys from me ;)). I have the code below, and here as well.

To do the deploy you ssh onto the vagrant box with aurora and run something like this to deploy.

aurora update start devcluster/www-data/devel/acaversity acaversity.aurora

That's pretty much it, this is most of what I demoed (I plan to keep working on this stack to make a more fully fledged drop in stack people can just use).

acaversity.aurora