Configuring a Static Asset Pipeline with Django and Heroku

Note: this is a repost of something I originally wrote while at Refer.ly. You can find the original post here.

Recently, I was tasked with overhauling the asset pipeline for a Django application being hosted on Heroku. The idea being that we want static files to deploy seamlessly when we push new code and yet maintain a different set of static files while working in development, staging, and on production. Additionally, user-uploaded media should work as expected: on production it should be uploaded to S3 and in development it should be stored locally.

This was my first major task in this code base, and needless to say, there was a lot to learn. While I have experience running Django on a standard stack, this would be my first time working with Heroku, and I was surprised to find that this wasn’t nearly as straightforward as I expected. Hopefully my experiences here can help others get this set up correctly from the start.

Some Background: Django & Static Files

When you’re working in the development environment (

1	DEBUG = True

and using

1	manage.py runserver

), Django serves all static content for you from the

1	STATIC_URL

, allowing you to make changes to the static files and view them with a simple refresh.

Now, when you deploy to production and set

1	DEBUG = False

, Django no longer serves static assets from

1	STATIC_URL

– instead, it expects you to serve these asset files using your webserver. This is good, because using Django to serve static files is a huge waste of resources. However, it means that every time you deploy new static assets, you have to manually copy them to the static root your website has configured– something you’ll probably want to automate in your deploy process.

But wait: someone already figured out the code for deployments, and you don’t have to! As long as you’re using the default

1	django.contrib.staticfiles

app, all you have to do is run

1	python manage.py collectstatic

, which collects all of your static assets and places them in the

1	STATIC_ROOT

you defined, which can then be served by your faithful webserver of choice. But what if you don’t control the webserver?

Enter Heroku

Now, Heroku is smart enough to automatically run collectstatic for you when you deploy code, and even includes some troubleshooting advice, but unless you define a Django view to serve files from the

1	STATIC_ROOT

, you won’t actually be serving these files. And if you do use Django to serve these files, you are wasting valuable dyno hours on static files.

Unfortunately, Heroku’s guide on using Django and static files has no actual details on how you should configure Django to serve static files. Even worse, a quick google search for “heroku static files django” returns a number of solutions that directly contradict the advice given in the Django docs, which is deserves to be quoted verbatim here(emphasis theirs):

This view[static file serve] will only work if DEBUG is True.

That’s because this view is grossly inefficient and probably insecure. This is only intended for local development, and should never be used in production.

Why Django even includes instructions for using this code in production is beyond me. Thankfully, a google search for “proper way to handle static files for django on heroku” turns up a stack overflow answer from intenex with the correct solution: use

1	django-storages

. By implementing a different storage backend, such as S3,

1	collectstatic

will automatically collect your static files to S3 and serve them faithfully.

So here’s how to set it up:

1. Add

   1	django-storages

   and

   1

   boto

   to your

   1	requirements.txt

   and run

   1	pip install -r requirements.txt

   to install them.

2. Add

   1

   storages

   to your

   1	INSTALLED_APPS

   in

   1	settings.py

   .

3. Add AWS keys to your Heroku config if you haven’t already:

   1
   2
   heroku config:add AWS_ACCESS_KEY_ID=YOURACCESSKEY
   heroku config:add AWS_SECRET_ACCESS_KEY=YOURSECRETACCESSKEY
   

4. Create a file called

   1	s3storages.py

   in the project root dir and add:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   from storages.backends.s3boto import S3BotoStorage
       
   # Define bucket and folder for static files.
   StaticStorage = lambda: S3BotoStorage(
       bucket='yourbucket', 
       location='assets')
       
   # Define bucket and folder for media files.
   MediaStorage  = lambda: S3BotoStorage(
       bucket='yourbucket',
       location='media')
   

5. update

   1	settings.py

   with the following:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   if DEBUG:
       # Development storage using local files.
       STATIC_URL = '/static/'
       MEDIA_URL = '/media/'
       MEDIA_ROOT = '/path/to/development/media'
       
   if not DEBUG:
       # Production storage using s3.
       DEFAULT_FILE_STORAGE = 's3storages.MediaStorage'
       STATICFILES_STORAGE = 's3storages.StaticStorage'
       STATIC_URL = 'https://s3.amazonaws.com/yourbucket/assets'
       MEDIA_URL = 'https://s3.amazonaws.com/yourbucket/static/'
   

6. Inside of templates, use

   1

   and

   as your roots and they will work seamlessly on production and development.

7. You’ll need to tell heroku to allow deployment scripts to access the AWS credentials in the environment by running

   1	heroku labs:enable user-env-compile -a myapp

Some Final Notes

What about user uploaded media? If you’re using the Django forms module, and using FileField on your models, django-storages will automatically store your media in the correct place: S3 for production, and the local filesystem for development.

Use absolute S3 paths in js/css: Since you can’t use template variables in css/js, you’ll have to upload images to S3 and use the full path to the image, i.e.

1	background-image: url(http://s3.amazonaws.com/yourbucket/assets/image.png);

. Using a preprocessor may allow you to get around this, but I’m going to save that for a follow-up post.

Careful: if you’re not using

1

render()

(which automatically adds the default

1	RequestContext

) or specifying a

1	context_instance

in your

1	render_to_response()

calls, you need to pass

1

MEDIA_URL

and

1	STATIC_URL

values to the template. If you’re using templates to generate emails, double check them!

Email to Picasa

This weekend I got back from a family vacation and found that when I got back to my computer, my inbox was filled with pictures of the vacation– despite facebook, flickr, instagram, and all the other social media tools around, my family finds it easiest to share photos via email. And who could blame them? It’s simple, and they’ve been attaching things to email for years. We already use a mailing list for emails to the family, and that’s easier to maintain than your circles/friends-list/whatever.

But down at the bottom of an email, one cousin chimed in:

Why can’t we set up a flickr account to upload pictures to so we can see them all in one place?

That got me thinking. What if I set up something to capture all these photos sent via email, and put them into a single photo album? I have a lot of Picasa storage available, so that’s what I’d prefer to use. A quick google found that Picasa supports email upload but it doesn’t allow you to default them to a single album, and I didn’t want family pictures showing up in the wrong place.

So, after a solid day of tinkering, I created email-to-picasa. It’s a webapp that allows you to upload photos via email and have them be stored in a specific album. Add that email to your distribution list, and bam: all photos send to the group are automatically uploaded!

Check it out.

Welcome to Jekyll

I put together this blog on Jekyll– it’s what github pages supports and it’s easy to use. I never thought the file system would be such a capable replacement for a simple CMS.

Also: static html is very simple to host.