A Rails Development Environment with Docker and Vagrant
Onboarding new developers to a Rails projects is still a far harder task than it should be. A big part of this is that setting up a development environment for an app or suite of apps, getting the correct ruby versions, database versions etc running locally, can in many cases take upwards of a day. A combination of Vagrant and Docker can make this a thing of the past.
Vagrant alone has already gone a long way to alleviating this but with Docker we can go one step further. Not only can we have a fully functional development environment (for both new and existing devs) up in a matter of a minutes, we can use almost the same containers we create in development to deploy to production. This goes even further to avoiding the classic "it worked in dev" problem.
In this tutorial, I'll show you how to use a combination of Vagrant and Docker to setup a fully functional Rails + Postgres + Redis development environment. In a follow up tutorial I'll demonstrate how to deploy the containers we create here to production.
Vagrant Docker Provider
Vagrant 1.6 added native support for Docker as a provider. If you're developing on a Linux machine, it will run Docker natively, otherwise it will transparently spin up a virtual machine to use as the Docker host.
I won't be using the providers DSL in this tutorial. This is for two reasons:
- Later, when designing a production Docker configuration, it's really important to understand the Docker options and command line switches. The Vagrant provider abstracts this into a DSL which, while Ruby friendly, is no simpler than Docker's own command line switches
- By simply using Vagrant to setup an Ubuntu VM which matches our eventual production VM, we can be sure that the development configuration is as close to identical to the production configuration as possible
Here, therefore, Vagrant is used to setup a standard Ubuntu VM and install Docker, while everything else is done using standard Docker shell commands.
The End Result
The final system will require a simple
to setup a complete Docker-based development environment. This development environment will consist of:
- A VirtualMachine running Ubuntu with Docker installed
- Separate Docker containers for the Rails application, PostgreSQL and Redis
- A shared folder linked to the Docker container so you can carry on editing Rails code on your development machine as you do now, and see those changes instantly reflected on
- A simple interface for running all the normal Rails commands (
rake db: migrate,
rails cetc) in the Docker environment
- Vagrant 1.6+ https://www.vagrantup.com/downloads.html
- Virtualbox 4.3.10+ https://www.virtualbox.org/wiki/Downloads
I'm assuming a basic understanding of what Docker is. If this is the first you've heard of Docker, they have a great interactive tutorial of the basics on their website https://www.docker.io/gettingstarted/.
I've tested this tutorial on OSX and Ubuntu 12.04. It should work on other nix flavours, but it may require more tweaking for use on Windows.
Dockerising the App
I started with a standard Rails 4.1.0 application generated with
rails new then added a single model + scaffolding and converted it to use PostgreSQL. The final source for this is available at https://github.com/TalkingQuickly/docker_rails_dev_env_example
The Rails app stores all secret values (API keys, anything in
secrets.yml etc) in environment variables and uses the dotenv gem for loading these in development. Note that in the example application, the
.env file is included in version control, for any real application, particularly one in a public repository, this should be added to
The PostgreSQL access details will be inferred directly from the database container. For more on this, see the section "Environment Variables in Linked Containers" later in this tutorial. In the example application,
database.yml looks like this:
default: &default adapter: postgresql pool: 5 timeout: 5000 development: <<: *default encoding: unicode database: dpa_development pool: 5 username: <%= ENV['DB_ENV_POSTGRESQL_USER'] %> password: <%= ENV['DB_ENV_POSTGRESQL_PASS'] %> host: <%= ENV['DB_PORT_5432_TCP_ADDR'] %> test: <<: *default encoding: unicode database: dpa_test pool: 5 username: <%= ENV['DB_ENV_POSTGRESQL_USER'] %> password: <%= ENV['DB_ENV_POSTGRESQL_PASS'] %> host: <%= ENV['DB_PORT_5432_TCP_ADDR'] %>
Dockerfiles and Scripts
The example configuration requires three Dockerfiles: one for Rails, one for Redis and one for PostgreSQL. The Rails Dockerfile is stored in the root of the Rails project, the others are stored in sub-folders of the
Templates for all of these files are available here: https://github.com/TalkingQuickly/docker_rails_dev_env
To begin with, copy all of the files and folders from the above repository into the root of your Rails project. If you're adding this to an existing project, the docker specific files and folders you'll be creating are:
├── Dockerfile ├── Vagrantfile ├── d └── docker/ ├── postgres/ ├── rails/ ├── redis/ └── scripts/
Setting Up Vagrant
If you're completely new to Vagrant it's worth briefly going through their getting started tutorial http://docs.vagrantup.com/v2/getting-started/ at least through to the end of the provisioning section.
The Vagrantfile should be stored in the root of your Rails project and look like this:
# Commands required to setup working docker environment, link # containers etc. $setup = <<SCRIPT # Stop and remove any existing containers docker stop $(docker ps -a -q) docker rm $(docker ps -a -q) # Build containers from Dockerfiles docker build -t postgres /app/docker/postgres docker build -t rails /app docker build -t redis /app/docker/redis/ # Run and link the containers docker run -d --name postgres -e POSTGRESQL_USER=docker -e POSTGRESQL_PASS=docker postgres:latest docker run -d --name redis redis:latest docker run -d -p 3000:3000 -v /app:/app --link redis:redis --link postgres:db --name rails rails:latest SCRIPT # Commands required to ensure correct docker containers # are started when the vm is rebooted. $start = <<SCRIPT docker start postgres docker start redis docker start rails SCRIPT VAGRANTFILE_API_VERSION = "2" Vagrant.configure("2") do |config| # Setup resource requirements config.vm.provider "virtualbox" do |v| v.memory = 2048 v.cpus = 2 end # need a private network for NFS shares to work config.vm.network "private_network", ip: "192.168.50.4" # Rails Server Port Forwarding config.vm.network "forwarded_port", guest: 3000, host: 3000 # Ubuntu config.vm.box = "precise64" # Install latest docker config.vm.provision "docker" # Must use NFS for this otherwise rails # performance will be awful config.vm.synced_folder ".", "/app", type: "nfs" # Setup the containers when the VM is first # created config.vm.provision "shell", inline: $setup # Make sure the correct containers are running # every time we start the VM. config.vm.provision "shell", run: "always", inline: $start end
The Vagrant Shared Folder
We want our Docker container to use the Rails app directly from our local filesystem so we can make changes on our development machine as we normally would and have these changes instantly reflected on our development server.
Since we're using Vagrant, first we have to share this folder from the local filesystem to the Vagrant virtual machine, which in turn shares this to the Docker container. If this is done using the default Virtualbox shared folders, then Disk IO and, consequently, Rails performance will be terrible. In my tests, it took something like 20 - 30 seconds to render a simple view.
This can be resolved by using NFS shares, which are much faster but require some additional setup and entering the sudo password when starting the virtual machine. The following entry in your
Vagrantfile ensures NFS is used:
config.vm.synced_folder ".", "/app", type: "nfs"
For more on NFS shares and what's required to set them up, see https://docs.vagrantup.com/v2/synced-folders/nfs.html. On OSX it should work out of the box, on Linux you may need to install
vagrant up in the root of your project to start the new development environment.
You'll be asked for your sudo password, this is required for NFS shared folders.
Due to a bug in Virtualbox 4.3.10, you may run into the below error on mounting shared folders the first time you run
Failed to mount folders in Linux guest. This is usually because the "vboxsf" file system is not available. Please verify that the guest additions are properly installed in the guest and can work properly. The command attempted was: mount -t vboxsf -o uid=`id -u vagrant`,gid=`getent group vagrant | cut -d: -f3` vagrant /vagrant mount -t vboxsf -o uid=`id -u vagrant`,gid=`id -g vagrant` vagrant /vagrant
You can resolve this by running:
vagrant ssh -c 'sudo ln -s /opt/VBoxGuestAdditions-4.3.10/lib/VBoxGuestAdditions /usr/lib/VBoxGuestAdditions' vagrant reload
UPDATE (11/6/2014): This is resolved in Vagrant 1.6.3 so it's definitely worth upgrading.
This will create an Ubuntu Virtual Machine, install Docker on it and proceed to running the script defined in your
$setup variable in the Vagrantfile. In this example, for completeness, we build all of the containers from scratch rather than pulling them from an Index so the first time you run this, it will take a while.
The Setup Script
The first time we start the VM, this line in the Vagrantfile:
config.vm.provision "shell", inline: $setup
Causes the shell script defined in the
$setup variable at the top of the file to be executed.
This starts by stopping and removing any running Docker containers, just in case we're rebuilding an existing system:
docker stop $(docker ps -a -q) docker rm $(docker ps -a -q)
It then proceeds to build our Docker images from the Dockerfiles and tag them with user friendly names (
docker build -t postgres /app/docker/postgres docker build -t rails /app docker build -t redis /app/docker/redis/
/app on the Ubuntu virtual machine is shared back to the root of your Rails directory, so this is using the Dockerfiles that can be viewed and edited as we normally would any file in a Rails project.
This process can take a long time. Something I've encountered on quite a few occasions with the OSX + Vagrant + Docker combination is that any sort of interruption in network connection can cause the build process to hang indefinitely.
If this happens, there's no harm in killing the Vagrant provisioner (
c twice), logging in with
vagrant ssh and then running the commands manually.
Once the three images are built, the script starts containers from those images. The first two of these are quite simple:
docker run -d --name postgres -e POSTGRESQL_USER=docker -e POSTGRESQL_PASS=docker postgres:latest docker run -d --name redis redis:latest
Breaking these down:
-d means run in the background
--name xyz gives the container the friendly name
xyz which we can use to refer to it later when we want to stop it or link it to another container
-e allows us to set environment variables in the container we're creating. In this case, we're setting the PostgreSQL username and password. See the section "Environment Variables in Linked Containers" for how we later access these credentials from our Rails app without hard-coding them.
xyz:latest means start the container from the latest image tagged with
I find it useful to think of a Docker image like a class definition. We use a Dockerfile (basically a list of shell commands) and the
docker build command to create an image.
We then use the
docker run command to create a container from that image. The container is like an instance of a class. We can create multiple containers (a new one every time we use
docker run) from a single image. Each container (instance) is completely isolated from every other container, even if they are created from the same image.
That said, we can also do things like create images based on the state of a container, so the analogy shouldn't be extended much further(!)
docker run is a little more complicated:
docker run -d -p 3000:3000 -v /app:/app --link redis:redis --link postgres:db --name rails rails:latest
In addition to the operations already discussed for the Postgres and Redis containers:
-p 3000:3000 makes port 3000 from the container available as port 3000 on the host (the Virtualbox VM). Since we have Vagrant configured to forward port 3000 of the VM to your local machine 3000, you can access this container on port 3000 on your development machine as you would the normal Rails dev server (e.g.
--link postgres:db establishes a link between the container you're starting (your Rails app) and the Postgres container you started previously. This is in the format
name:alias and will make ports exposed by the Postgres container available to the Rails container.
Environment Variables in Linked Containers
Linking will also make the environment variables from the Postgres container available to the Rails container with the prefix
ALIAS. When you expose a port in a container, a corresponding environment variable is created within that container.
The Postgres container exposes port 5432 which leads to a corresponding environment variable
PORT_5432_TCP_ADDR which will contain the IP address of the Postgres container. We use this in our
database.yml to automatically connect to the Postgres container database, irrespective of whether its IP has changed.
Since we used
db as our alias for this container, in our Rails container, we will therefore have an environment variable
DB_PORT_5432_TCP_ADDR available which contains the IP of this container.
Therefore, we use
ENV['DB_PORT_5432_TCP_ADDR'] to access this value in
The command to build the Postgres image includes:
-e POSTGRESQL_USER=docker -e POSTGRESQL_PASS=docker
which sets environment variables in the docker container with the database access credentials. These will be available in your Rails container as
DB_ENV_POSTGRESQL_PASS respectively (as seen in
database.yml). Notice the format
It's worth reading http://docs.docker.io/reference/run/#env-environment-variables for more on the environment variables available. It's also interesting to inspect the contents of
ENV from a Rails console once the full environment is up and running.
Why make this a shell script?
The few lines in our
$setup script are everything we need to build and run our application on any machine with Docker installed. If you wanted to run this application in development on a Linode, you could just create a new node, install Docker, upload your code and run this same script, and you'd have a working development version of your application on this server.
Later in this series of tutorials I'll demonstrate how the commands in this script can be adapted to form the basis of a production deployment with Docker. Getting familiar with the commands as part of the day-to-day development workflow means that working with the production stack is much less of a learning curve for any developer on the team.
Interacting with the Rails Application
Once the above process is complete, the Rails application should be available in a browser on your local development machine at
http://localhost:3000. However, the first thing you're likely to see is an error that the database does not exist.
Normally at this stage we could simply use
bundle exec rake db:create db:migrate to create the database and apply any migrations. Now that we're running our application in a Docker container, the process is slightly different.
In this configuration, each Docker container runs a single process. In the
$setup script, the container for the Rails server is started with:
docker run -d -p 3000:3000 -v /app:/app --link redis:redis --link postgres:db --name rails rails:latest
The Docker daemon expects the second, non-parametrised argument to the
run command to be the command to be executed within the container. Since we don't specify a command to be run within the container, the default command from the
Dockerfile is run. This is specified in
Which makes the default action of the container to run the script in
#!/bin/bash cd /app bundle install bundle exec unicorn -p 3000
This is equivalent to starting the container with:
docker run -d -p 3000:3000 -v /app:/app --link redis:redis --link postgres:db --name rails rails:latest bash -c "cd /app && bundle install && bundle exec unicorn -p 3000"
To run other commands in the container based on your Rails image, we can construct equivalent
docker run commands and run them from within the Vagrant virtual machine (
vagrant ssh). So to run
bundle exec rake db:create db:migrate we could ssh into the Vagrant host and then use:
docker run -i -t -v /app:/app --link redis:redis --link postgres:db --rm rails:latest bash -c "cd /app && bundle exec rake db:create db:migrate"
This starts a new container based on the Rails image and runs
db:migrate with it. Notice the additional command line flags:
-i -t attaches the console to Standard In, Out and Error, then assigns a TTY so that we can interact with it. This is required when running interactive commands such as
bundle exec rails console
--rm means that the container will be removed once execution completes.
Doing this every time is cumbersome, so the example configuration includes some simple shell scripts to automate this. The first part of this is the file
d in the root of the Rails project. This automates sshing into the vagrant host and executing a single command, for example:
vagrant ssh -c "sh /app/docker/scripts/rc.sh"
Which is the same as sshing into the Vagrant host and executing
/app/docker/scripts/rc.sh contains a
docker run command:
docker run -i -t -v /app:/app --link redis:redis --link postgres:db --rm rails:latest bash -c "cd /app && bundle exec rails c"
Which has the effect of starting a Rails console in a new container.
You can run
./d without any arguments to see which functions it provides shortcuts for. It also provides a generic
./d cmd "bundle exec any_command"`
which allows arbitrary commands to be executed within the
/app directory of a new Rails container. So we could, for example, execute:
./d cmd "bundle exec rake db:create db:migrate"
to create and migrate the database.
These don't have to be Rails commands, we could equally execute:
./d cmd "ls"
to get the directory listing of
When interacting with the Rails application like this, it's important to remember that every command is executed in a new container and hence is completely isolated from any other command. Therefore any local resources created in one container are not normally available to any other container and will be lost when the command terminates and the container is discarded.
The exception in the case of this development configuration is the
/app/ directory, which is a shared folder and, therefore, any files created here will persist and be available to all containers. It should be kept in mind that, in a production configuration, there should be no such shared local storage and so this should not be relied upon for any sort of shared state. It's worth reading the Twelve Factor App guide for more on how sharing state should be approached http://12factor.net/.
Bootstrapping the Database
Generally, when setting up a development environment, I want the development database populated with a recent dump of data from production. Sometimes this is a direct dump and sometimes it's a segment of the production data, or modified version of production with sensitive information removed. Either way, this normally takes the form of a .sql file (assuming MySQL or Postgres). It's useful to be able to quickly restore future dumps to the development database.
./d convenience script provides a simple interface to this:
- Drop, create and migrate the development database
- Look for a zip file called
db/current.sql.zipin the Rails directory structure
- Unzip it within a new Docker container
- Import the file
current.sqlfrom the unzipped files to the development database using the
Note the naming, it expects a zip archive at
db/current.sql.zip which contains a single file called
I'll be releasing further tutorials over the next few months on the topics below, you can follow me on twitter @talkingquickly for updates. If you run into any issues with this tutorial, feel free to tweet or email me, [email protected]
- Setting up a private index so you can build your Docker images centrally and then just pull them when you start a new development environment, rather than rebuilding each time
- Building production images
- Automating building production images and pushing them to a private index
- Deploying production images with Capistrano + a Private Index