Deploy Existing Django app to Ubuntu

For many newbies to django, the deployment process to production can seem to be a daunting one. Fueled with questions of how to ensure the app is always running to monitoring that app is always running even if something goes wrong. Fear not, as I will try my best to explain in great detail what each step consists of, why it is needed and the indepth background of each tool used.

The basic steps I will be following are as follows:

1. Setting up OS + User
2. Setup Database + Users
3. Setup bower/npm/yarn
4. Setup virtualenv + environment
5. Clone git project
6. Manage project packages
7. Setup gunicorn
8. Setup supervisor
9. Setup nginx
10. Manage static + media folders
11. Project cleanup + testing

Here we go…

1. Setting up OS + User

I normally use Digital Ocean to run most of my apps, but there are other various service providers (ramnode)you can choose from. But more on that here on my blog where I covered hosting that particular blog, and various other vps hosting sites.

The first thing I always do when working on a fresh server is to run update, it has been a habit which I believe is good so lets do that…

apt-get update
apt-get upgrade

We need to use a user with sudo rights. To do that we add a new user and give them sudo privileges.

adduser NEWUSER
usermod -aG sudo NEWUSER

Now that we have sorted all of that out let us switch to that user

su NEWUSER

The project I am working on deploy for this example is written in python3 and django 1.11.1. You might be using different versions of python and django so just keep that in mind for the rest of these install instructions.

I will be pulling my app from a private bitbucket repo, with postgres as the main database store. Usually when one starts a django project it is wired to sqlite, however in a production environment one might want to use something like postgres or mysql for example.

So lets install git and some of the other required packages for this project.

sudo apt-get install git
sudo apt-get install python3-pip python3-dev libpq-dev postgresql postgresql-contrib nginx

2. Setup Database + Users

Now that our basic setup of what we will need we can proceed to setting up the database and database users.

We need to login as postgres user as that is the only user with the necessary privileges to create db users and assigning roles.

sudo su - postgres
createuser --interactive -P
Enter name of role to add: prod_user
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n

Now that all the db user stuff is sorted we can create a database and assign the created user above as the owner.

createdb --owner prod_user prod_db
logout

If you were using mysql you would probably have to do almost the same. Before I used to install a web manager like webmin and use the web interface to do tasks like this, but thats another added couple of mins.

3. Setup bower/npm/yarn

For this project I am using as an example I used bower for front-end dependency management. However you could have used something different, so you can follow steps to install that here. The debate on the best dependency management is not one for me to try police or offer my 2 cents worth. But here goes, if you used npm this will also help. Just do not install bower on the last step.

sudo apt-get install python-software-properties
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install nodejs
sudo npm install -g bower

Now that we have our dependency management sorted we can begin to get the project onto the server.

4. Setup virtualenv + environment

Virtualenv is a tool that is used to isolate python environments. It makes it easy and safe to have multiple projects running from one server running different packages with different versions just fine.

sudo apt-get install python-virtualenv

Now that we have installed virtualenv we can create our environment.

virtualenv -p python3 prod_env

Now that we have created our environment we can activate it and proceed to get the project in there and install all our project dependencies.

source prod_env/bin/activate

5. Clone git project

My sample project is sitting on a private bitbucket repo, however you could be using github. The process is exactly the same, we clone the project into a folder.

sudo git clone https://USERNAME@bitbucket.org/ninjak/prod_project.git prod_project

And we are done with getting our project.

If you…

ls

You should see two folders, prod_env and prod_project.

This is my practice to have the two folders in this structure, I have seen some people have the env folder within the project however this seems to work best for ME. Plus I follow this format of instructions all the time so it just works out perfectly for me.

6. Manage project packages

Hopefully you ran pip freeze during dev phase and you got a requirements.txt file. What I like to do is "mock"a prod requirements file as opposed to having the full list as I did during dev. Now I must mention that you do not need to run the command as pip3 when in virtualenv. We installed the environment as python3 already.

cd prod_project
pip install -r requirements-prod.txt

Here is an example of what my prod requirements file looks like. Some of these are very important such as gunicorn. You will see why in the next step.

appdirs==1.4.3
boto3==1.4.4
botocore==1.5.80
cachetools==2.0.0
certifi==2017.4.17
chardet==3.0.4
Django==1.11.1
django-grappelli==2.10.1
django-watson==1.3.1
djangorestframework==3.6.3
docutils==0.13.1
gunicorn==19.7.1
httplib2==0.10.3
idna==2.5
jmespath==0.9.3
packaging==16.8
protobuf==3.3.0
psycopg2==2.7.1
pyasn1==0.2.3
pyasn1-modules==0.0.9
pyparsing==2.2.0
#python-apt==0.7.8
python-dateutil==2.6.0
pytz==2017.2
requests==2.18.1
rsa==3.4.2
s3transfer==0.1.10
six==1.10.0
urllib3==1.21.1

Now that you have successfully gotten all the required dependencies of the project we can proceed to running migrations, we will revisit collecting the static files a bit later and you will see why when setting up nginx.

python3 manage.py migrate

We can now do our bower installation then collect all of those into our static folder. The next two steps are a bit tricky with nginx and the static folder. So we need to define a static path in nginx and also in the settings file for our django app. This has to be the same folder.

bower install

Now we collect static.

python3 manage.py collectstatic

Lastly we can now create our super user, you will need to enter a username, password, email etc.

python3 manage.py createsuperuser

7. Setup gunicorn

This is where it starts getting a bit hairy but if you copy this script there should not be much that you need to change nor fear.

cd into your project root directory. We will create a gunicorn bash file here. This is what will start django and put a socket file where nginx will communicate with django via gunicorn.

Lets create the file now.

sudo nano prod_gunicorn.bash

Now past this into that file changing the bolded areas then save (ctrl + x)

#!/bin/bash
NAME="prod_project"
DJANGODIR=/home/prod_user/prod_project
SOCKFILE=/home/prod_user/prod_env/run/gunicorn.sock
USER=prod_user
GROUP=sudo
NUM_WORKERS=3
DJANGO_SETTINGS_MODULE=DJANGOAPP.settings
DJANGO_WSGI_MODULE=DJANGOAPP.wsgi


cd $DJANGODIR
source /home/prod_user/prod_env/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH

RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

exec gunicorn ${DJANGO_WSGI_MODULE}:application \
  --name $NAME \
  --workers $NUM_WORKERS \
  --user=$USER --group=$GROUP \
  --bind=unix:$SOCKFILE \
  --log-level=debug \
  --log-file=-

Once we have created the file we must now make it executable by the OS.

sudo chmod u+x prod_gunicorn.bash

Now we can test and see if it works.

sudo ./prod_gunicorn.bash

Hopefully you get something like this…

[1253] [INFO] Starting gunicorn 19.7.1
[DEBUG] Arbiter booted
[INFO] Listening at: unix:/home/prod_user/prod_env/run/gunicorn.sock (1253)
[INFO] Using worker: sync
[1259] [INFO] Booting worker with pid: 1259
[1260] [INFO] Booting worker with pid: 1260
[1262] [INFO] Booting worker with pid: 1262
[1253] [DEBUG] 3 workers

Click ctrl + c to get out of that run.

8. Setup supervisor

I normally like to keep my logs inside the project folder in a folder called logs, that way if I have multiple projects I do not get too confused. So lets create the folder and file.

Assuming you are still in the project directory.

sudo mkdir logs
sudo touch logs/prod_gunicorn.log

Now that thats out the way we can continue to install supervisor and wire out log file and folder in our project config.

sudo apt-get install supervisor

Now we can setup our project inside supervisor

sudo nano /etc/supervisor/conf.d/prod_project.conf

If you have followed each of my steps exactly as they are then there should be no need to change the contents of this file. However if you named your project/user etc differently then you can change accordingly.

[program:prod_project]
command=/home/prod_user/prod_project/prod_gunicorn.bash
user=root
stdout_logfile=/home/prod_user/prod_project/logs/prod_gunicorn.log
redirect_stderr=true
autostart=true
autorestart=true
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8

Now we can run our supervisor process and see what happens.

sudo systemctl restart supervisor
sudo systemctl enable supervisor

Now lets check the status of our supervisor process

sudo supervisorctl status prod_project

Hopefully you get a response like

prod_project                    RUNNING   pid 1922, uptime 0:00:26

9. Setup nginx

Now that we have setup every we need to serve the app to the outside world. So for this we will use nginx.

Lets start by installing it.

sudo apt-get install nginx

Now we can define the config file for nginx

sudo nano /etc/nginx/sites-available/prod_project.conf

Now we can paste these in the conf file and save it (ctrl + x).

Remember I said we might need to revisit the static folder? Well if you look at the script below you will see the location of the static folder and how it will be routed to the outside world. That then points to the inside directory, that should point to where you pasted your static files to and bower install. This will also need to be exactly the same as your django settings file.

server {
    listen 80;
    server_name prodproject.net;

    location = /favicon.ico {
            access_log off;
            log_not_found off;
        }
    location ^~  /static {
            root /home/prod_user/prod_ject;
    }

    location / {
        include proxy_params;
        proxy_pass http://unix:/home/prod_user/prod_env/run/gunicorn.sock;
    }
}

Lastly we create a symbolic link

sudo ln -s /etc/nginx/sites-available/prod_project.conf /etc/nginx/sites-enabled/prod_project.conf

Now we can start nginx and browse to our website.

sudo service nginx restart

If all went well you should be able to run http://yourdjangoprodenv.com and get your website live in your shiny new prod deployment.

This process took me a while to get right but I now have scripts I just mod and try keep up to date as possible so it is now super easy for me with loads of practice.

Good luck to you!