Wrox Home  
Search
Adam Kolawa - Co-Founder and CEO of Parasoft
Adam Kolawa is the co-founder and CEO of Parasoft. Kolawa, co-author of Bullet-
proofing Web Applications
(Wiley, 2001), has contributed to and written hundreds of commentary pieces and technical articles for publications, such as The Wall Street Journal, CIO, Computerworld, Dr. Dobb's Journal, and IEEE Computer. He has also authored numerous scientific papers on physics and parallel processing.

Articles by Adam Kolawa

Making Your Automated Build System Work for You: Example

By Adam Kolawa, Co-Founder of Parasoft

In a previous column, I introduced strategies for getting the most out of your automated build system. This month, I will provide examples of how those strategies can be applied to an actual build process.

Assume that you want to perform a full nightly build on a C++ application, and this application is divided into the following projects:

  • $BLDROOT/alex
  • $BLDROOT/mike
  • $BLDROOT/mike_Linux
  • $BLDROOT/paul
  • $BLDROOT/paul/test_suites
  • $BLDROOT/Makefile

$BLDROOT is the environment variable that represents the root location of the nightly build source. It is good practice to make this location a relative or parameterized project root path. This way, the build process will work on any machine that has the correct $BLDROOT environment variable — even on machines that do not have the same directory structure as the initial build machine. In other words, it allows the build process to be completely independent of the machine's directory structure.

$ARCH is another environment variable that sometimes must be set. If you are building on more than one platform, $ARCH will distinguish which configuration file to use and which projects to build. For example, to build the complete mike project from the Linux architecture, you need to build the mike_linux project.

Each project (alex, mike, paul) has multiple source code files. For example, assume that the alex project has the following files: Alex_a.cpp, Foo.cpp, Boo.cpp, Test.cpp. There is a directory of test suites for the paul project, but test suites are not yet available for the other two projects.

Each project also has its own Makefile, which is part of a larger hierarchy of Makefiles. At the top level is the application-wide Makefile (in $BLDROOT/Makefile), which sets macro variables used throughout this build process and coordinates the builds of all available projects by calling the lower-level project Makefiles in the designated order. This file is parameterized so that it will be portable in a multi-machine/multi-user environment. The lower-level Makefiles are the ones that are created for each project; when executed, they build one specific project, and nothing else.

The Makefile hierarchy is represented in the following figure.

Figure 1
Figure 1

Before you start implementing an automated build, it's a good idea to create a special account for running the build. If you run the build from a special account, you eliminate the possibility of developer configuration errors and make nightly builds portable. For this example, assume that you have created a special account named "nightly."

Once you have a special build account created, you can start implementing the details of the nightly build. The ideal automated build performs the following tasks:

  1. Cleaning: Cleaning involves removing all elements of the previous nightly build(s) — including sources, binaries, and temp files. Old files should always be removed before a new build begins.
  2. Shadowing or Cloning: If you have more than one machine using the same code for a nightly build, it is a good idea to shadow it on only one machine, then clone that shadowed code on the other machines. This way, the same code is used for all builds, even if changes are introduced into the source control system between the time the first machine shadows the code and the time the last machine accesses the code. If you create a source tar file to archive all latest sources, then the other machines can clone the build environment by getting the archive tar files.
    • Shadowing: Shadowing involves getting the latest project sources from the source control system. The sources should be stored in a directory that is accessible across the network. This way, if you have multiple machines running nightly builds, all nightly builds can access the archived source (or individual files) that were shadowed on the original machine.
    • Cloning: Cloning involves copying previously-shadowed source files over any existing files in the build directory. This process is called "cloning" because the same source archive is used for multiple machines (or multiple platforms).
  3. Building: Building is the process of actually constructing the application. It can be as simple as executing make on the $BLDROOT directory. In this case, the top-level Makefile in $BLDROOT can configure the build environment (for example, by creating a binary repository and setting environment variables) and call each project's Makefile to build the related project in the designated order.
  4. Testing: Testing involves automatically running the existing test suites for the application. Testing should only occur after a successful build.

The logistics of this procedure are illustrated in the following figure.

Figure 2
Figure 2

The following nightly.sh shell script orchestrates these steps for the sample application:

nightly.sh
#!/usr/bin/sh
# $BLDROOT must be set by user (nightly) environment or hard coded here
export BLDROOT=$HOME/build

# Set $DATE for source file
export DATE='date +%Y%m%d'

# Detect $ARCH if there is none available
if [ "x$ARCH" = "x" ]; then
	export ARCH = 'uname -s'
fi 

# clean
rm -rf $BLDROOT

# shadow (if it is the main building machine)
mkdir $BLDROOT
cd $BLDROOT; cvs get source_modules; tar cvzf  source-$date.tar.gz *

# clone (if it is a secondary building machine)	
mkdir $BLDROOT
cd $BLDROOT; cp /source/to/archive/source-$date.tar.gz $BLDROOT
tar xzvf source-$date.tar.gz

# build
make build ARCH=$ARCH MFLAG=-g

# test
make test ARCH=$ARCH MFLAG=-g

This shell script is parameterized so that it will be portable in a multi-machine/multi-user environment. Only one parameter comes in, then all build actions are performed automatically. The script says to make build and make test, and the script immediately knows what to do with this information. It also sends make flags and these flags are specifically from the include file, which has all the parameters.

This same script could easily be adapted to orchestrate the building of other projects. To make it work for your own projects, you would basically just need to modify the includes.

Note that after the script performs the necessary shadowing and cloning, it runs $BLDROOT/Makefile to launch the building process. For this example, the Makefile would look something like this:

$BLDROOT/Makefile:
# for multi-platform builds, read the platform
# specific config.def file to set up the build
# environment (config.def is not attached)
-include $(BLDROOT)/config/$(ARCH)/config.def

MAKE=make # you can override make with your build script name
all: setup build test
setup: 
	-mkdir $(BLDROOT)/$(BINDIR)
build: build_alex build_mike build_paul

test: test_alex test_mike test_paul

build_alex:
	-cd $(BLDROOT)/alex &&	$(MAKE)  $(MFLAGS)
build_mike:
	-cd $(BLDROOT)/mike_$(ARCH) && $(MAKE) $(MFLAGS)
	-cd $(BLDROOT)/mike &&	$(MAKE) $(MFLAGS)
build_paul:	
-cd $(BLDROOT)/paul && 	$(MAKE) $(MFLAGS)
test_alex:
	-cd $(BLDROOT)/alex/test_suites  &&	$(MAKE)  $(MFLAGS)
test_mike:
	-cd $(BLDROOT)/mike/test_suites &&	$(MAKE) $(MFLAGS)
test_paul:	
-cd $(BLDROOT)/paul/test_suites && 	$(MAKE) $(MFLAGS)

This sample Makefile could also be easily adapted for use with your own projects.

To ensure that this build is performed automatically each night, you would set up the following crontab for the "nightly" account:

0 18 * * * /usr/bin/sh $HOME/bin/nightly.sh > 
	$HOME/log/nightly-build.log 2>&1

This configures the machine to run the build process at 6:00 PM every day and to send all output to the nightly-build.log file.

If you wanted to build a Java application, the same principle steps still apply, and you have the option of using a Makefile, a shell script, or an Ant task to automate the required steps. For an idea of how this might be done by creating an Ant task and running that task with a script, consider the following sample Ant task and script:

$BLDROOT/build.xml
....
<property name="src.dir" value="${bldroot}"/>
	<property name="build.dir" value="${bldroot}/bin"/>
	<target name="init">
	<path id="classpath">
		<fileset dir="${src.dir}/">
			<include name="**/*.jar"/>
		</fileset>
	</path>
	<property name="classpath" refid="classpath"/>
</target>
<target name="clean">
	  <delete>
    		<fileset dir="${src.dir}" includes="**/**"/>
  	  </delete>
</target>
<target name=shadow>
      <cvs                     
		cvsRoot=":pserver:nightly@cvs.hello.com:/home/cvspublic"
         command="update -A -d"
         dest="${ws.dir}"/>
	    	<tar 	destfile="${src.dir}/source-$(date).tar.gz" 
compression="gzip">
			<tarfileset dir="${src.dir}/>
		</tar>

      </target>
<target name="build" depends="timestamp"
        description="build all java files in source directory">    
	<javac srcdir="${src.dir}"
           destdir="${build.dir}"
           classpath="${classpath}"
           source="1.4"
           deprecation="true"
           listfiles="true"
           optimize="off"
           debug="on">
    </javac>
</target>
<target name=test>
<junit printsummary="yes" haltonfailure="yes">
  <classpath>
    <pathelement location="${build.dir}"/>
    <pathelement path="${java.class.path}"/>
  </classpath>

  <formatter type="plain"/>
  <batchtest fork="yes" todir="${reports.tests}">
    <fileset dir="${src.dir}">
      <include name="**/*Test*.java"/>
      <exclude name="**/AllTests.java"/>
    </fileset>
  </batchtest>
</junit>
</target>
<target name="all" depends="init,clean,shadow,build,test"
       description="Build Nightly"/>

$HOME/bin nightly.sh
#!/usr/bin/bash

# Make sure JAVA_HOME is set and ant is set in PATH
cd $BLDROOT; ant all

As with the previous example, a crontab could be used to ensure that the build process is automatically run at the same time every day.