Python on Docker

In this article you will learn about setting up a Python-based project on Codeship Pro. We will use nosetest and py.test for setting up the build, but the same works for any other test framework as well.

Services and Steps

Before reading through the documentation please take a look at the Services and Steps documentation page so you have a good understanding how services and steps on Codeship work.

Dockerfile

We will start by creating Dockerfiles that lets you run your Python based test suite in Codeship.

Please take a look at our Dockerfile Caching best practices article first to make sure you build your Dockerfile in a way that only invalidates the Docker cache when necessary.

# Starting from Python 2.7 base image
FROM python:2.7

# Set the WORKDIR to /app so all following commands run in /app
WORKDIR /app

# Adding requirements files before installing requirements
COPY requirements.txt dev-requirements.txt ./

# Install requirements and dev requirements through pip. Those should include
# nostest, pytest or any other test framework you use
RUN pip install -r requirements.txt -r dev-requirements.txt

# Adding the whole repository to the container
COPY . ./

This Dockerfile will give you a good starting point to install any further packages or tools you might need. Take a look at our browser testing documentation to find and install any further tools you might need for your build.

codeship-services.yml

The following example will use the Dockerfile we created to set up a container we call project_name (please change to your specific project name) that will run your build. We’re adding a PostgreSQL container and Redis container so the tests have access to those two services.

When accessing other containers please be aware that those services do not run on localhost, but on a different hostname, e.g. “postgres” or “mysql”. If you reference localhost in any of your configuration files you have to change that to point to the hostname of the service you want to access. Setting them through environment variables and using those inside of your configuration files is the cleanest approach to setting up your build environment.

project_name:
  build:
    image: organisation_name/project_name
    dockerfile_path: Dockerfile
  # Linking Redis and Postgres into the container
  links:
    - redis
    - postgres
  # Set environment variables to connect to the service you need for your build.
  # Those environment variables can overwrite settings from your configuration
  # files if configured. Make sure that your environment variables and
  # configuration files work work together as expected.
  environment:
    - DATABASE_URL=postgres://postgres@postgres/YOUR_DATABASE_NAME
    - REDIS_URL=redis://redis
# Service definition that specify a version of the service through container tags
redis:
  image: redis:2.8
postgres:
  image: postgres:9.4

For more information about other services you can use with Codeship check out our services and databases documentation.

codeship-steps.yml

Now we’re going to set up our codeship-steps.yml file. Every step in a build gets its own clean container and linked services. Any setup commands that are necessary to setup a linked container (e.g. database migrations) need to be run for every step. While this duplicates some of the work, it also makes sure your parallelized test commands run completely independently.

- name: ci
  type: parallel
  steps:
  - service: project_name
    command: nosetests tests/unit
  - service: project_name
    command: nosetests tests/acceptance

The same setup can be used with py.test.

- name: ci
  type: parallel
  steps:
  - service: project_name
    command: py.test tests/unit
  - service: project_name
    command: py.test tests/acceptance

Parallelising your build

As a step can only have a single command the best way to set up the parallelised build is through scripts in your repository. You can then call those scripts in your step file.

The following step file calls a scripts/ci file that we will create in two parallelised steps.

- name: ci
  type: parallel
  steps:
  - service: project_name
    command: bash -c "CODESHIP_NODE_INDEX=0 ./scripts/ci"
  - service: project_name
    command: bash -c "CODESHIP_NODE_INDEX=1 ./scripts/ci"

Make sure to set the CODESHIP_NODE_TOTAL environment variable in the codeship-services.yml file.

environment:
  - DATABASE_URL=postgres://postgres@postgres/YOUR_DATABASE_NAME
  - REDIS_URL=redis://redis
  - CODESHIP_NODE_TOTAL=2

And here is the corresponding script file you can put into script/ci. It finds all files named test*.py and then splits them up between the parallel containers. Make sure to change the command at the end of the following script that starts either nosetest or py.test.

#!/bin/bash

set -e

NODE_TOTAL=${CODESHIP_NODE_TOTAL:-1}
NODE_INDEX=${CODESHIP_NODE_INDEX:-0}

# Find files named test*.py in the repository and sort them. We're setting a requirements.txt as the randomisation source so the randomisation is repeatable. You can add any file instead of requirements.txt.
files=$(find ./ -name "test*.py" | sort -R --random-source=requirements.txt)

i=0

tests=()
for file in $files
do
  if [ $(($i % ${NODE_TOTAL})) -eq ${NODE_INDEX} ]
  then
    tests+="${file} "
  fi
  ((i+=1))
done

# Remove # from your test framework line
# nosetests $tests
# py.test $tests