An Excellent Development Server for a Team of Developers

Flattr the An Excellent Development Server for a Team of Developers post

Introduction

When working in a team it is very useful to have a central web server with multiple environments and a configuration as close to the live server as possible. This can be a bit of a nightmare though if you need to setup a new VirtualHost container in Apache every time a new project is brought on or when a developer wants to work on a version of the site in their own environment.

The good news is that this can all be handled automatically and new sites can be setup by simply adding a new directory to the file system. There are at least two ways of getting this going; the first of which is the mod_vhost_alias module for Apache and the second is enabled via mod_rewrite. I prefer to use the second method as it is more flexible and it allows you tap into the ability of mod_rewrite to introduce environment variables and redirect requests (this is particularly useful for robots.txt - you’ll see).

The Apache2 Manual does have a very good page dedicated to overcoming this problem, but I will be sharing with you all the settings I am using which you will need to stop Google et. al. from crawling your sites served from the staging environment for example.

Intended Audience

This article is targeted at people who have a fairly good working knowledge of Apache and VirtualHost container configuration. I have previously written a more basic and step by step development server configuration article, but it is focused at single developers as it uses a virtual machine.

It is still a good introduction though and takes you through all the installation steps on an Ubuntu machine. If you are not confident installing and configuring a LAMP server then head over and give my A Good Windows Development Environment and Ubuntu Virtualbox article a read first. Skip to step eight if you are not going to be using a virtual machine.

My Setup

In my environment I have three subdomains that I use to host various aspects of a project:

  • proof.example.org
  • *.dev.example.org
  • staging.dev.example.org

They are all essentially handled the same by Apache. When a request comes into the server from one of the above domains mod_rewrite will direct the request to the correct file location. I use the following (respectively):

  • /var/vhosts/proof
  • /home/*/www/
  • /var/vhosts/staging

The *.dev subdomain is used for each users individual development area. So simon.dev.example.org would map to /home/simon/www/ in the file system. Staging is for demonstrating sites to clients and proof is a location used to show designs & wireframes for sign off.

I will now run through how you would create a staging site using this system. Simply:

  1. Create a new directory in /var/vhosts/staging using the name of your site (eg. mysitename).
  2. Now create a directory called pub inside your newly created directory (/var/vhosts/staging/mysitename).
  3. Copy or create your public files in pub (/var/vhosts/staging/mysitename/pub). You may need to include a .htaccess file here with a ``RewriteBase /`` line in it.
  4. Navigate to mysitename.staging.example.org to see the served page.

Having the pub directory allows you have the bulk of your application outside of public HTTP folder.

So now you can see how easy, quick and simple it is to use lets delve into how it works.

Development Server Configuration

Initial Include

The first file that is necessary is a simple link to our main development server configuration file. This keeps the main Apache configuration file simple and clean, which will make upgrades further down the line less painful.

# This file is /etc/apache2/httpd.conf
# This file is automatically included by Ubuntu in /etc/apache2/apache2.conf

Include /etc/apache2/dev-server.conf

The above file is at /etc/apache2/httpd.conf by default in Ubuntu, but your distribution may differ in which case it might be /etc/httpd/httpd.conf. Essentially you need to add the Include line to your default Apache configuration.

VirtualHost Containers

This leads us to the VirtualHost configuration file. This includes two VirtualHost containers, one for port 80 traffic and one for SSL port 443 traffic.

Both containers are essentially the same except for the actual SSL options so I have included the main or common elements in a separate file (/etc/apache2/dev-server-vhost.conf) so they can easily be reused in both. This saves a lot of duplication and allows you to easily update them both in once place. This will make more sense when I describe the dev-server-vhost.conf file further on.

# This file is /etc/apache2/dev-server.conf
# This file is included by /etc/apache2/httpd.conf
# Hide server information and setup VirtualHost container skeletons for HTTP and HTTPS

# Hide server vitals from responses
ServerSignature Off
ServerTokens Min

# Mass Virtual Hosting
<VirtualHost *:80>
    <IfModule mod_ssl.c>
        SSLEngine off
    </IfModule>
    Include /etc/apache2/dev-server-vhost.conf
</VirtualHost>


# SSL Mass Virtual Hosting
<VirtualHost *:443>
    <IfModule mod_ssl.c>
        SSLEngine on
        SSLOptions +StrictRequire
        SSLCertificateFile /etc/ssl/certs/server.crt
        SSLCertificateKeyFile /etc/ssl/private/server.key
    </IfModule>
    Include /etc/apache2/dev-server-vhost.conf
</VirtualHost>

In this file I firstly configure the server to broadcast as little as possible about itself as a simple way of slowing down possible attackers and hopefull deterring script kiddies. Next the first VirtualHost container is specified for standard port eighty traffic.

In the VirtualHost container I have ensured that SSL is definitely deactivated for this port and then included the common elements from the dev-server-vhost.conf file mentioned earlier.

The second VirtualHost container is basically the same except for the mod_ssl options. Firstly it enables SSL and forces all traffic on this port through SSL encryption. Also specified here are the SSL certificate and the encryption key that the server should use for communication with the client. If you do not know how to create an SSL certificate then the Apache2 manual has you covered. Finally the dev-server-vhost.conf file is included again.

VirtualHost Container Common Configuration Include

The third file is the dev-server-vhost.conf file which is included in both of our VirtualHost containers as mentioned above. This is quite a large set of configuration options, but nothing overly complex.

Basic Configuration

The first few options should be familiar to you; simply setting the DocumentRoot and ServerName etc. After this there is a simple line to block any known robots just incase they do not intend to respect the robots.txt. Followed by Directory directives that enforce the bad_bot blocking. They also stop the contents of directories being displayed or indexed.

# This file is /etc/apache2/dev-server-vhost.conf
# This file is included from the virtual hosts in /etc/apache2/conf.d/dev-server.conf
# The guts of the VirtualHost container with robot blocking and mass virtual hosting via mod_rewrite

ServerName dev.example.org
ServerAdmin example@example.org
ServerAlias *.dev *.dev.example.org staging.example.org *.staging *.staging.example.org proof.example.org *.proof *.proof.example.org
DocumentRoot /var/www
LimitInternalRecursion 15


# Detect bots by user agent
SetEnvIfNoCase User-Agent ".*(Googlebot|msnbot|Yahoo! Slurp|YahooSeeker|Yahoo-Blogs|bot|robot|spider|Ask Jeeves|ArchitextSpider|Scooter|AltaVista|Slurp|Crawler|WebCrawler|Lycos).*" bad_bot

# Detect incorrect website access by referer. This is to stop anybody
# clicking on a link from the following search engine's listings.
SetEnvIfNoCase Referer ".*(Google\.|Yahoo\.|Bing\.|Ask\.|Excite\.|Lycos\.|AltaVista\.|WebCrawler\.).*" bad_bot
<Directory /home/>
    Options Indexes FollowSymLinks Multiviews
    AllowOverride All
    Order allow,deny
    Allow from all
    Deny from env=bad_bot
</Directory>
<Directory /var/vhosts/virtual>
   Options Indexes FollowSymLinks Multiviews
    AllowOverride All
    Order allow,deny
    Allow from all
    Deny from env=bad_bot
</Directory>
<Directory /var/vhosts/proof>
    Options Indexes FollowSymLinks Multiviews
    AllowOverride All
    Order allow,deny
    Allow from all
    Deny from env=bad_bot
</Directory>

RewriteEngine on

# Rewrite log settings
RewriteLog /var/log/apache2/dev-rewrite.log
#0-9: 0 = none 9 = verbose
RewriteLogLevel 0

# Block robots with a central robots.txt for all sites
RewriteRule ^/robots.txt$ /var/vhosts/robots.txt [L]

# Filter to parse URLs to lowercase
RewriteMap lowercase int:tolower

# Dev sites
# With the format "user    /home/dir/"
RewriteMap vhost txt:/etc/apache2/dev-server-rewrite.map
RewriteCond ${lowercase:%{SERVER_NAME}} ([^.]+)\.([^.]+)\.dev\.

# Windows:
# RewriteCond %1.${vhost:%2}£%2 ^([a-z]+)\.([d].*)£([a-z]+)$

# Linux:
RewriteCond %1.${vhost:%2}£%2 ^([a-z]+)\.(/.*)£([a-z]+)$

# The £%2 is only there so that the developer
# username can be passed between RewriteConds.
# The £ symbol is just a delimiter.
RewriteRule ^/(.*)$ %2/%1/pub/$1 [L,E=VIRTUAL_DOCUMENT_ROOT:%2/%1/pub,E=WE_ARE_ON_STAGING:TRUE,E=DEVELOPER_USERNAME:%3]

# Staging sites
RewriteCond ${lowercase:%{SERVER_NAME}} ([^.]+)\.staging\.
RewriteRule ^/(.*)$ /var/vhosts/staging/%1/pub/$1 [L,E=VIRTUAL_DOCUMENT_ROOT:/var/vhosts/staging/%1/pub,E=WE_ARE_ON_STAGING:TRUE]

# Proof sites
RewriteCond ${lowercase:%{SERVER_NAME}} ([^.]+)\.proof\.
RewriteRule ^/(.*)$ /var/vhosts/proof/%1/pub/$1 [L,E=VIRTUAL_DOCUMENT_ROOT:/var/vhosts/proof/%1/pub,E=WE_ARE_ON_STAGING:TRUE]

# Logging
LogLevel debug
LogFormat "%{Host}i %h %l %u %t \"%r\" %s %b" vcommon
CustomLog /var/log/apache2/dev-access.log vcommon
ErrorLog /var/log/apache2/dev-error.log

mod_rewrite Setup and Logging

Next mod_rewrite is employed and a log file is specified for debugging rewrite requests. I have set the log level to zero by default to disable logging but bump it up to 9 and it will give you full request logging.

Central Robots.txt

All requests to robots.txt on a particular site are rewritten to a central file that simply denies all robot traffic to all URLs. If you are not sure how to specify this then check out the documentation on Robotstxt.org.

# This file is /var/vhosts/robots.txt
# The central robots.txt file.

User-agent: *
Disallow: /

Development Environments for Individuals

Further on we have the first rewrite rule for *.dev.example.org sites. These are for each developers individual development environment. It uses a RewriteMap file to match developers name in the URL to a location in the file system. This looks like the following and it is self explanatory:

# This file is /etc/apache2/dev-server-rewrite.map
# This file is included from the virtual hosts in /etc/apache2/dev-server-vhost.conf
# Maps a username to a location in the file system uses the following format:
# name<tab>/dir/location

simon       /home/simon/www
joe     /home/joe/www
jane        /home/jane/www

As you can see from the directory layout above you will need to setup a new user for each developer and add their path and username to the map file.

Environment Variables for Bootstrapping

When the developers environment is rewritten three environment variables are set for use in your scripts (in PHP this is available in $_SERVER). The variables would look like the following if you were to access the site mysitename.simon.dev.example.org:

  1. VIRTUAL_DOCUMENT_ROOT = /home/simon/www/mysitename/pub
  2. WE_ARE_ON_STAGING = TRUE
  3. DEVELOPER_USERNAME = simon

VIRTUAL_DOCUMENT_ROOT

The VIRTUAL_DOCUMENT_ROOT is the location of the files that are being served and can be used in place of DOCUMENT_ROOT in your scripts.

This can get tedious however so I have setup an auto_prepend file in my PHP configuration that converts VIRTUAL_DOCUMENT_ROOT to DOCUMENT_ROOT automatically.

<?php
// This file is /etc/php/auto_prepend.php
// It is actioned by setting auto_prepend_file="/etc/php/auto_prepend.php" in your PHP INI/Config
// Converts the virtual document root to be the standard document root

if(isset($_SERVER['VIRTUAL_DOCUMENT_ROOT']) and
   !empty($_SERVER['VIRTUAL_DOCUMENT_ROOT'])) {
    $_SERVER['DOCUMENT_ROOT'] = $_SERVER['VIRTUAL_DOCUMENT_ROOT'];
}

Then set the PHP INI configuration variable auto_prepend_file to the path of your file. In my case above this would look like: auto_prepend_file="/etc/php/auto_prepend.php"

This is the only way I have found to override the value in the DOCUMENT_ROOT environment variable. Attempting to do so in the RewriteRule will not work.

WE_ARE_ON_STAGING and DEVELOPER_USERNAME

The next two environment variables WE_ARE_ON_STAGING and DEVELOPER_USERNAME are used to automatically setup the correct configuration environment in development, staging and production.

So I currently use an XML configuration file that is interpreted by the ZendFramework Zend_Config class. Zend_Config has the concept of multiple environments that can extend each other. This is described in more detail in the ZendFramework manual, but you could just as easily write your own or tie it into any frameworks bootstrap.

Essentially these variables allow me to work out which set of configuration data I should use when bootstrapping the application. Each developer maybe working from their own database for example so if DEVELOPER_USERNAME is set we can automatically use their configuration.

If the WE_ARE_ON_STAGING variable is not set then we are on the production server so we can use the live configuration options.

This works for me but you may want to tweak it to suit your work flow.

Apache Logging

I have this set to the maximum logging level and I have redirected all logging for sites going through this VirtualHost container to a central error and access log. This is all explained in more depth in the Apache2 manual section on Log Files.

Other Niceties

There are a few the other helpful things I have running on the server as well to aid development. I would go into more detail about them but they are fairly easy to install and configure plus they are not required for the main functioning of the server.

  1. Install and setup Samba shares for the dev, staging and proof directories so Windows users can easily access the files and create new sites.
  2. Setup a mail server so that your applications can send out test emails. I use Postfix for this and there is a nice help tutorial for Ubuntu to help you along.
  3. If you are using PHP then seriously consider installing XDebug and looking into is profiling capabilities.
  4. Also on PHP install APC to speed up PHP.
  5. Consider setting up mod_cache to test your long life caches.
  6. I have added memcached and Redis to the mix as well.

I have recently written about installing memcached and APC on my blog for both RedHat and Ubuntu.

Conclusion

I have found this to be a very effective development environment when working with groups of people. You can develop in your own environment and then push it to staging for client approval. It also ensures that the staging environment and development environments are almost identical. I hope that this proves useful to you as well.