Sitesetup documentation

sitesetup: script for checking a site’s setup

Usage: add it to your buildout’s [console_scripts] part and run bin/sitesetup afterwards.

sitesetup looks at your site’s setup and checks for things like:

  • Is buildout.cfg symlinked? Is there a development.cfg and production.cfg?
  • Is gunicorn available?
  • Is there a fabfile? An nginx configuration template?

It helps a lot in converting an old deploy.cfg-style site! See the online documentation for the full explanation of all the checks.

Fabric scripts

By adding ‘fabric’ to the console scripts section of buildout.cfg, you gain a ‘bin/fab’ command that runs tasks from a fabfile.py in your main project directory. Fabric is a tool to run commands on remote servers.

The sitesetup.fab module contains tasks to deploy sites to different server environments, making lots of assumptions based on the Nelen & Schuurmans setup to make our life easier. The goal is to initially deploy a site with one command, update it from then on with one command, and have some useful tools to do with server configuration.

There can be several server environments: “production” is the production site, “staging” is used for final testing before a production release, “integration” has the current development situation of the whole team, and “development” is a single developer’s machine. The defaults assume that you have an individual buildout file and Django settings file for each of these situations, but everything can be configured. It’s also not necessary to use all of these environments, it’s fine to only have settings for production. It is also very easy to add new environments if necessary.

There is one line that every fabfile.py needs:

from sitesetup.fab.tasks import *

This makes the sitesetup tasks available to your bin/fab command. You also need some minimum amount of configuration. There are two ways to set options, in Python code or in a .cfg file. The .cfg file looks neater but some options (ones that require Python data structures like lists, or that need computation of some sort) can only be set the Python way. Both ways can be combined.

The Python way looks like this:

fabfile.py:

from sitesetup.fab.config import init_config

init_config(production={
        'web_host': 'p-web-ws-00-d4.external-nens.local',
        },
            staging={
        'web_host': 's-web-ws-00-d3.external-nens.local',
        })

The same thing using a config file looks like this:

fabfile.py:

from sitesetup.fab.config import init_file

init_file('fabfile.cfg')

fabfile.cfg:

[production]
web_host=p-web-ws-00-d4.external-nens.local

[staging]
web_host=s-web-ws-00-d3.external-nens.local

The ‘web_host’ setting defines the server that the site will be deployed to. It is the only setting without a default and therefore this is the minimum configuration necessary for a setup with a production and a staging environment.

Tasks

Fabric (and bin/fab) takes a number of tasks as arguments. They are run in the order they are given, and there is no mechanism to give command line options to the tasks. Therefore, in our script the first task that runs is always one that defines which environment we are going to run in; this task can setup the configuration for the next tasks.

So, commands are always of the form:

bin/fab <environment> <task>

Where <environment> is ‘production’, ‘staging’ etc. There are three main commands. One of them is ‘help’, which shows some help text and is the default task in case bin/fab is called without arguments. The others are ‘init’, which is used the first time a site is deployed, the other is ‘update’, which is called each time after.

Each of those tasks is made up of a number of detail tasks. If an error occurs during one of the tasks, bin/fab stops, and the best way to continue is often to run the final detail tasks by hand. The detail tasks have names like ‘detail.sync_and_migrate’ and can be run from the command line like any task.

As of this writing, the following tasks are run by the main tasks:

init:
    detail.server_install_ubuntu_packages
    detail.server_install_backup_and_logrotate_scripts
    detail.initial_create_srv_dir
    detail.initial_create_databases
    detail.switch_and_buildout
    detail.initial_symlink_django_logrotate
    detail.initial_nginx_symlinks
    detail.copy_var_data_from_production
    detail.copy_databases
    detail.sync_and_migrate
    detail.start_or_restart

update:
    detail.switch_and_buildout
    detail.sync_and_migrate
    detail.start_or_restart

Configuration options

The scripts have many configuration options, many of which are not meant to be changed but only used internally. Here we mention the ones that may prove useful.

web_host: Already mentioned, the host to which the site will be deployed. Mandatory, no default.

user: the user as which the fabric script should connect to the server. Default is the current user.

checkout: either ‘trunk’, which uses the latest version of the main branch, or ‘tag’ which finds the latest tag and uses that.

tag: if a tag option is given, use that instead of the latest tag.

buildout-file: which file shall be symlinked to buildout.cfg on the server. Default is to use <environment>.cfg.

Many other settings (the site name, the project name, the Django settings file, database login information for copying databases) are found using the buildout file’s values.

Changelog of sitesetup

0.10 (unreleased)

  • Nothing changed yet.

0.9 (2012-03-07)

  • Update does not offer to copy var/ files or the production database anymore.

0.8 (2012-03-05)

  • Implemented new functionality to run a sequence of Django commands (nens/vss/#18).

0.7.2 (2012-01-11)

  • Fabfile deployment type ‘staging’ does not automatically inherit from production anymore, and other types not from staging anymore. Inheritance led to bugs.

0.7.1 (2012-01-11)

  • Fix typo.

0.7 (2012-01-11)

  • Changed the design of config so that it can support different deployment environments in a more natural way.
  • It is now possible to do a trunk checkout instead of a tag, by setting the ‘checkout’ option to ‘trunk’. It is also possible to checkout a tag other than the latest one by setting the ‘tag’ option.
  • It is now possible to set config options by means of an “ini” config file.

0.6 (2012-01-09)

  • Added rudimentary support for an ‘integration’ environment, that can be configured separately and takes defaults from the staging environment. Its default buildout file is ‘integration.cfg’, otherwise there is no functionality implemented for it.

0.5 (2012-01-05)

  • Option to add a ‘user’ config option for the SSH connection.

0.4 (2011-11-25)

  • Fabfiles:
    • using production config during staging deployment (e.g. to figure out where to get var files and database from)
    • automated copying of var files
    • database copying functions
    • Correctly creates staging site for demo.lizard.net and correctly updates it.
    • Start of using commands in development mode (instead of staging or production), not ready for common use yet

0.3 (2011-11-22)

  • Added more functionality to the fabfiles:
    • Staging and production deployments, called like bin/fab staging <command> and bin/fab production <command>.
    • Fine-grained configuration is possible, but the defaults should usually be good enough
    • Added high-level tasks like ‘init’ that call detail tasks
  • Started a TODO list

0.2 (2011-11-10)

  • Fabfile: showing the tag before switching to it.
  • Doing a “git fetch” instead of a “git pull” on the server to prevent merge problems with the “detached head” state we’re in on the server. Git pull also tries to do a merge, which is what we don’t need.
  • Renamed the tasks to group them in ‘server’, ‘initial’ and ‘regular’. The non-tasks are hidden now.

0.1 (2011-11-09)

  • Added server.cfg, logrotate/nginx configs for copying into your site.
  • Checking buildout configs, fabric/gunicorn setup and so on.
  • Initial library skeleton created by nensskel.

Credits