Programmers' Canvas Toolkit Build Robot Overview

Introduction

Automating the build process seems easy enough:

Scripting languages like Python, Perl, and JScript provide everything you need to checkout source code, run makefiles, execute scripts, and zip up the results.  It's a breeze.  You can get something working in an hour.

Now you've created a monster:

Compiler errors are detected but linker errors aren't.  QA wants the setup files for last week's build, and nobody can find them.  Developers want a list of changes made to each build.  An engineer informs you that his subproject uses Apache Ant, not makefiles.  Can your build script support vs.NET projects too?  A wise guy wants to see build status via his web browser.  Your team would really appreciate email notification when builds start and finish.

And reality hits you like a ton of code:

You're now the Build Guy, in addition to your "real" duties, unable to escape, for the rest of your foreseeable career.  Your noble attempt to save time for yourself and your team has backfired.

DevGuy's Build Robot^*, to the rescue!

* - Part of the DevGuy Programmers' Canvas Toolkit.

Features

1. Retrieves source code from CVS into a clean sandbox (optionally via tags)
2. Increments a build number.  The build number can be accessed by Java or C/C++ code.
3. Keeps track of build errors, CVS errors, and other types errors.   Error tracking is one of the build robot's greatest strengths.  This is usually the greatest weakness of home-grown tools.
4. The build robot writes extensive detail to log files which are accessible in real time via a web browser.  It is possible to follow the build robot every step of the way, remotely, even when executing long-running scripts, as long as they periodically output to STDOUT.  These log files are also available after builds for process debugging and other post-mortem tasks.
5. Builds Java, Apache Ant, Visual Studio .NET, C/C++, Visual C++ projects, etc.
   • Can build anything through 'make' or scripts; totally extensible
   • For Visual Studio projects, version resources can be automatically updated to contain the build number, company name, and other statistics
   • For .NET projects, assembly version number can be automatically updated
6. Executes scripts; for example, to update web site with latest files from CVS or run automated tests
7. Posts complete build status via a web-based application, including:
   • All build errors (if any)
   • List of all files that were built and their CVS revision numbers
   • All jars, DLLs, EXEs, etc. for a particular build
   • Check-in comments of files that were changed and added since the last build
   • Check-in comments can also be posted separately to a newsgroup, which is very handy for project managers, doc writers, etc.
   • As new builds are created, the new build is added to the list of available builds that can be downloaded
   • Builds Java source code documentation via javadoc
8. Sends condensed build status to a newsgroup or via email.  Notifications can be sent via "net send" (Windows command-line program) as well.
9. Tags source code files with a unique "incrementing" build stamp
10. Reads and writes XML files

System Requirements

Windows NT4, 2000, XP.  Server or workstation.  IIS 4.0 or better is required to host the build status web pages.

Important Note When Testing

When you're configuring the build robot for the first time, you should turn off CVS tagging by setting the element text in <ShouldTag> and <ShouldForceTag> in the build-options document to 0.  Otherwise you'll add a bunch of meaningless tags to the repository.

Quick Start

1. Follow the install guide
2. We'll assume you have installed the web application (pctkweb.tar.gz)
3. Modify the globals.xml file
4. Set <Domain>, <SMTPServer>
5. You should have created a sample CVS repository via the "clean" directory structure in clean.tar.gz (or installed via pctkexe.tar.gz).
6. Click here to view the directory tree
7. Read the development environment documentation
8. Check-out the sample repository
   cvs checkout build
9. Run a command-line shell (cmd.exe)
10. Change your current directory to the build module
11. Read about configurations

VC++ 6.0

1. Run a command-line shell (cmd.exe)
2. Change your current directory to the build module
3. Copy build/build.xml somewhere to your local disk (not in your CVS sandbox)
4. Modify build.xml on your local disk
   • Set <Sandbox>, <WebAddress>, <WebPath>, <WebSubdir>, <NetNotify>, and <SMTPNotify>
5. Change your current directory to the directory containing the local build.xml file
6. To use the executables, run
   dgbuild build.xml
7. If you use Perl, run
   perl -S dbuild.pl
8. This builds release and debug configurations.  Read about configurations

VS.NET

1. Check out the source code
   cvs co source/c/xmlacceptor
2. Run VS.NET
3. Open source/c/xmlacceptor/n_xmlacc/n_xmlacc.dsp
4. Instruct VS.NET to convert the project
5. Exit VS.NET and save everything
6. Check in the new files
   cvs add source/c/xmlacceptor/n_xmlacc/n_xmlacc.vcproj
cvs add source/c/xmlacceptor/n_xmlacc/n_xmlacc.sln
cvs commit
7. Modify build/build_all.xml
8. Change
<Build>build/all.dsw</Build>
   to
   <Build>source/c/xmlacceptor/n_xmlacc/n_xmlacc.vcproj</Build>
9. Run a command-line shell (cmd.exe)
10. Run
    %VSCOMNTOOLS%vsvars32
    (read about VSVARS.BAT)
11. Change your current directory to the directory containing the local build.xml file
12. To use the executables, run
    dgbuild build.xml
13. If you use Perl, run
    perl -S dgbuild.pl build.xml

Java

1. Make sure you have set environment variables
2. Run a command-line shell (cmd.exe)
3. Change your current directory to the build module
4. Copy build/build_java_options.xml somewhere to your local disk (not in your CVS sandbox)
5. Modify build_java_options.xml on your local disk
   • Set <Sandbox>, <WebAddress>, <WebPath>, <WebSubdir>, <NetNotify>, and <SMTPNotify>
6. Read about the makefile targets
7. Change your current directory to the directory containing the local build_java_options.xml file
8. To use the executables, run
   dgbuild build_java_options.xml
9. If you use Perl, run
   perl -S dgbuild.pl build_java_options.xml

Overview

The Build Robot can run on a scheduled and ad-hoc basis.  dgbuild.exe runs the robot.  If the Perl version is installed, use perl -S dgbuild.pl to invoke the Build Robot.

The Build Robot performs the following tasks:

1. Send notification that the build started
2. Remove all files from the sandbox
3. Download source code
   • Each module can be retrieved using different CVS tags
   • The build robot retrieves source as it goes.  For example, if it needs to build foo/bar/baf.mak and the foo/bar directory is missing, the robot will retrieve it (as well as any subdirectories).  
   • For Visual C++ projects and vs.net C++ projects, the build robot also retrieves the "include directories"
4. Increment the build number
5. Modify C++ resource files to add the version number, version name, build number, and configuration
6. Build source code
7. Tag files
8. Post build statistics to internal web site - including errors, warnings, and the CVS revision number of every file retrieved
9. Post the changes submitted by developers since the last build
10. Zip up files and post to internal web site
11. Send notification that the build finished

What the Build Robot Builds

1. The Build Robot builds Java and Oracle's SQLJ files via the "make" command.  See the development environment page regarding makefiles.
2. The Build Robot can build Java projects via Apache Ant
3. The Build Robot builds Microsoft Visual Studio projects (6.0 and .NET) packaged in workspace (DSW/SLN) and project (DSP/VCPROJ) files.  Developers can decide whether to put their projects into a common DSW/SLN file, or to build DSP/VCPROJ files individually.  The build robot inspects DSW/SLN files and "include directories" for CVS modules to check out.
4. The Build Robot's open architecture also supports makefiles, command-line scripts, and executables.

CVS Commit Log Generation

The build robot creates a file containing all CVS commits made since the last build.  The build robot uses a file called the "last build timestamp" to generate this log.  This file can be branched.

The CVS commit log will work properly only if the build robot tags files during each build.  This is because tracking CVS commits by date is tricky due to the fact that CVS stores all timestamps in GMT.  Furthermore, we've found that it's a good practice to tag files after each build because you never know when you might have to work backwards.  CVS seems to be able to handle a lot of tags gracefully.  If you don't like this behavior please send us a message asking for the change log generator to use timestamps instead.

The build robot creates the commit with the following rules:

1. If a file is not part of the build because it has been removed or it's not part of the branch, its change history is not output.  This means the build robot's change logs can not be used to see when files were deleted.   This decision was made because, in order to properly filter the change log, "cvs stat" is run against each file in the sandbox.  "cvs stat" doesn't work for files that aren't in the sandbox.  cvs stat" is used because the "cvs log" command's -r option outputs all revisions when two tags are provided and the file has been branched.
2. If a file is "new," all change history for the branch is output.  A "new" file is a file in which none of its revisions are tagged with the last build stamp.  This means that when files are "undeleted;" i.e., moved out of the attic, you might expect to only see commits since the file was moved out of the attic, but instead, you will see all change history for the file.

Scheduling Builds with the Windows Task Scheduler

1. Open the Scheduled Tasks applet in the control panel
2. Double-click Add Scheduled Task

To run the executable version:

1. Application to execute:  dgbuild
2. Command line parameters:  build.xml

To run the Perl version:

1. Application to execute:  perl
2. Command line parameters:  -S dgbuild.pl build.xml

Building from a Virtual Drive

It's a good idea to build from a virtual drive which points (via the 'subst' command) to the root of the sandbox.  This way, the debugger files (.PDB files) will work properly on all developers' machines as long as all developers have the same virtual drive defined.

One way to create a virtual drive is via the "subst" command which can be run via the Windows Task Scheduler.

Rebooting Before a Build

It's a good idea to reboot the build machine about five minutes before a scheduled build.  It removes any file locks that may be present in the sandbox directory; for example, on DLLs.

Use the dgshutdown.exe utility, which is included in the toolkit, to reboot, which is installed with the other tools.  Getting shutdown to work is tricky.  Executing it by itself doesn't seem to work, although your results may vary.  Use the administrator account on the local machine -- using a network account with administrator privileges doesn't seem to work.  Make sure to check "allow program to interact with desktop."

Configuration Files Used by the Build Robot

A substantial amount of documentation regarding the build robot's capabilities is contained in the XML descriptions listed below in this section.  The XML documentation also contains quite a bit of how-to information.  You must understand the documents in order to use the build robot.

The Build Robot uses files checked into the "build" module.  The Build Robot consumes four main XML documents:

• A globals document which is used by the toolkit in general
• A build options document provides parameters to the build script.  This XML document is a replacement for command-line arguments and switches.  It is not checked into the CVS repository but all other files are checked into CVS.
  • Here is an example for VC++ projects with debug configuration
  • Here is an example for Java projects with release and debug
• A build document specifies the source code to retrieve, the projects to build, and the files to distribute
  • Here is an example for VC++ projects
  • Here is an example for Java projects
• A package document specify which files to post to the web site for download, and also specify which files to upload to other network directories
  • Here is an example that copies all DLLs, EXEs, etc. created by a Visual C++ DSW project
  • Here is an example that copies all PDB (debugger database) files created by a Visual C++ DSW project
  • Here is an example that copies various files from the CVS repository.  These files are zipped up and a link to the zip file is added to the web page.
  • Here is an example that copies the JAR files
• A VC++ configuration document specifies various attributes of a DevStudio project
  • This document is needed if VC++ projects don't follow the default settings of basic projects -- for example, if the EXE name is not the same as the DSW file name.
  • Here is an example

Other Files Used by the Build Robot

1. "Build stats" (BuildNumberFile in Build-Options-Doc) files contain the current build number and build timestamp.  These files are intended to be used as "include" files in C++, Java, or other programming languages, but such usage is optional.

   The build number is incremented by the Build Robot.  A build number is a unique number for each build.  This file can be branched.  There are C++ and Java versions of this file that ship in the "default" installation. 

   This file must exist prior to running the build robot.

   Sample file:

   #pragma once

#define DG_BUILD_NUMBER /*CFGMGMT_BUILD_NUMBER*/ 1
#define DG_BUILD_STAMP /*CFGMGMT_BUILD_STAMP*/ "Q2003_0428_094300"
 

2. "Last build stamp" (ChangeLogInfo in Build-Options-Doc) files contain the last applied CVS tag.  This tag is used to automatically generate a document containing the changes that developers have submitted since the last build.  This file can be branched.  New last build stamp files should be checked into CVS as regular text files with no contents.

   Sample file:

   D2001_1119_000000

   This file is redundant compared to the "build stats" file above.  The difference between these files is that the build stamp in the "build stats" file always changes whereas the "last build stamp" file only changes if a tag is applied (tagging is optional).

   If you have different types of builds with different Build-Doc or Build-Options-Doc files that use the same branch, each build should have a different "last build stamp file."  This is because the build stamp is changed by the build robot each time it tags files.  Using the same file for both types of builds will likely result in lost change comments.
    

3. "Product version" (VersionFile in Build-Options-Doc) files contain a string representing the product version, the company name, and the copyright date.
   
   The Build Robot doesn't alter this file.  The robot reads this file in order to set the version resource appropriately for DLLs and EXEs.  Thus, this file is only applicable to VC++ and vs.NET C++ projects.
   
   
   1. The company name should be prefixed by CFGMGMT_COMPANY_NAME

      #define CFGMGMT_COMPANY_NAME "Acme Incorporated"

   2. The copyright year should be prefixed by CFGMGMT_COPYRIGHT_YEAR

      #define CFGMGMT_COPYRIGHT_YEAR "2004"

   3. The product version should be prefixed by CFGMGMT_VERSION and have the form M.N V where M and N are integers and V is an optional version description e.g., "Beta 2".

      #define PRODUCT_VERSION /*CFGMGMT_VERSION*/ "1.0 Beta 2"

Configurations

A "configuration" is a name for a type of build.  For example, most VC++ EXE projects have a configuration called "Win32 Release" whereas VC++ ATL projects use "Win32 Release MinSize" and "Win32 Release MinDependency."

The build robot supports configuration names directly.  When you start the build robot, you tell it which configurations to build.  Configuration names let you write one build XML file that supports multiple configurations and later choose which configurations to build.  The most common usage of configurations is to build Debug vs. Release builds.

Meta-Configurations

The configuration names given to the build robot are actually Meta-Configuration names.  These Meta-Configuration names are mapped to actual configuration names for VC++ and vs.net projects.  Meta-Configuration names are also mapped to makefile targets.

Meta-Configuration names are mapped to actual configuration names and makefile targets via the configuration mapping document.  For example, you can specify that a particular project should never be built using UNICODE by mapping the configuration "Win32 Release Unicode" to "Win32 Release."

You are free to choose your own Meta-Configuration names.  The configuration names that are used in the build options document must correspond to the names used in the build, package, and configuration mapping documents.

VC++ Configuration Names

Please refer to more information regarding VC++ configurations.

VS.NET Support

vs.NET SLN ("solution") files provide similar functionality to the build robot's meta-configurations.  The configuration document still applies, however, and is useful for overriding target file names.  Configuration name mapping only applies to the configuration names in the SLN file.  The build robot reads configuration name mappings in the SLN file.

Viewing Progress via a Web Browser

The build robot writes to a log file, log.txt, which resides on a web server.  The file is viewable before the build has finished.  If you navigate to the build page, the build page will output a page that contains a hyperlink to log.txt.

The build robot writes to additional log files as it builds each project.  Look for lines in log.txt containing the text "Sending build output to <URL>."  To read the output files, paste the URL to the Address field of your browser, and hit Enter.

Support for Visual C++ 6.0 and Visual Studio.NET

• The build robot lets developers work exclusively in their development environment without having to perform an excessive amount of work in order to accommodate the build robot.  For the most part, developers don't need to know that a build system is in place.  They just use the facilities provided by Visual Studio and the build robot automatically takes over from there.
• The build robot recognizes Visual C++ project (DSP) and workspace (DSW) files directly.  It also recognizes the VS.NET extensions SLN and VCPROJ.  These files can be put directly in <Build> elements.  Makefiles are not used.  This gives the fastest possible build times.
• All projects listed in DSW and SLN files are loaded from CVS - automatically!
• The build robot also inspects projects' "include directories" for C++ projects.  The directories are loaded from CVS before compilation begins.
• The robot reads project dependencies in DSW files to build projects in the correct order
• The build robot alters the Version resource for all C++ projects that have a resource file.  This information is read from the Product Version file which is specified in the build-options XML document.

1) The FileVersion is set to the form of "a.b.c.d" where d is the build number, c is 0, and a and b come from the version string in the Product Version file.

2) The following fields are added to the version resource, even if they are not present in the resource file:

• ProductVersion, which has the form of "a, Build b" where a is the version string from the Product Version file and b is the build number
• LegalCopyright, which has the form of Copyright © a, b
  Where a is the copyright year and b is the company name, both of which come from the Product Version file
• CompanyName, which contains the company name from the Product Version file
• PrivateBuild, which contains the configuration name
• BuildNumber, which contains the build number and build stamp
• SpecialBuild, which contains the version string from the Product Version file

DGMKTLB

The dgmktlb.pl script is handy for compiling multiple IDL files to TLB and C header files at once.  It processes files in the current directory and subdirectories.

For example:

<Build>dgmktlb
  <ScriptIsOutsideSandbox/>
  <Parameters>.</Parameters>
  <Once/>
  <Path>source/c/tlb</Path>
   <Filter>dggenericOutputParser</Filter>
</Build>

VSVARS.BAT

In order to build vs.NET projects from the command line, vs.NET environment variables must be loaded.  These environment variables can be installed when vs.NET is installed.  Alternatively, you can run the vsvars32 program.  One way to incorporate vsvars32 with the build robot is via the following custom script.  Change the following 2-line script to reflect the location of your vs.NET program files directory and save it as vsvars.bat.

call %VSCOMNTOOLS%vsvars32
%1 %2 %3 %4 %5 %6 %7 %8 %9

If VSCOMNTOOLS is not defined (which seems to be an unusual scenario), use:

call "c:\program files\Microsoft Visual Studio .NET\common7\tools\vsvars32"
%1 %2 %3 %4 %5 %6 %7 %8 %9

Invoke this batch file as:

vsvars.bat dgbuild "c:/sandbox/build.xml"

OR AS (if Perl version is installed):

vsvars.bat perl -S dgbuild.pl "c:/sandbox/build.xml"

Support for Java Projects

Programmers' Canvas toolkit has a built-in makefile infrastructure for building Java projects.  The makefile infrastructure is described in the build environment section.  In order to use the Build Robot, you must know how these makefiles work.

If you already have your own scripts or makefiles for building Java projects, the build robot can run them.  Refer to the scripts section.

The makefiles build all java projects under source/java.  The build robot then parses the output of the makefiles and summarizes errors and warnings on a per-module basis.

<Build>
    source/java/makefile
    <Sequence/>
    <Multiple/>
    <Targets>clean release</Targets>
    <Configuration>Java Release</Configuration>
</Build>

The above XML fragment builds everything.  You can build a particular module by changing <Build> element's text:

<Build>
    source/java/com/devguy/log/makefile
    <Sequence/>
    <Multiple/>
    <Targets>clean release</Targets>
    <Configuration>Java Release</Configuration>
</Build>

The <Multiple/> element causes the build robot to look for build errors and warnings on a per-module basis.

The <Sequence/> element tells the build robot to build and package each configuration separately.  Normally, the build robot builds everything and then packages everything.  However, this doesn't work when different configurations output their artifacts to the same directory regardless of the configuration.

See the example build XML file for Java projects.

Please refer to the <MakeTargets> element in configmap.xml.  This allows you to have one <Build> entry for one makefile.  Rather than specifying targets in the build XML document via <Targets>, targets are specified in configmap.xml via <MakeTargets>.  <Targets> in the build XML file overrides configmap.xml.

Support for Java via Apache Ant

While Apache Ant and the Build Robot have overlapping features, they can coexist and complement each other.  For example, Ant can be used to build Java and other projects and Build Robot can copy Jar files to a web site.

The toolkit was built before Apache Ant and therefore there are strategies in the toolkit for building Java projects without Ant.  However, these strategies do not need to exist.  You could just as easily have a <Build> instruction that specifies the name of a script that runs Ant and forgo all of the makefile crud.

Ant can be run via a script as mentioned in the scripts section.  Add an <Ant/> element to the <Build> tag of the build-options document.  If the script is not checked into CVS, use <ScriptIsOutsideSandbox/> in the <Build> tag of the build-options document.  Specify a relative sandbox directory in <Path>.  The build robot will get the source from CVS (per the <Build> and <Path> elements) and run the specified script.

The Build Robot only recognizes output related to compiling Java code.  If you want to capture other warnings and errors from the Ant script, add a <Filter/> element to the <Build> element in the build-options document.  This is described further below.

Support for NANT

Run NANT as a script.  See below.

PCTK 1.8.2.13 and later contains a filter called dgnantfilter, both as an executable and a Perl script.  Add a <Filter>dgnantfilter</Filter> or <Filter>dgnantfilter.pl</Filter> element to the <Build> element in the build-options document.  This is described further below.

Support for Incrementing Build Numbers and CVS Tags

The toolkit contains files (e.g., source/java/dgBuildNumber.java) which keep track of the current build number and CVS tag.  This file is referenced by the build options document.  The Build Robot automatically updates the information in the specified file(s) you specify in the build options document.  The Build Robot also commits the changes.

The sample files contain C and Java code.  You can just as easily use your own text file and put it in, say, the build directory.  The file doesn't have to be source code.

The Build Robot looks for the text CFGMGMT_BUILD_NUMBER and CFGMGMT_BUILD_STAMP.  It then replaces the text to the right of either word.  In the case of CFGMGMT_BUILD_STAMP, the text is placed in quotes.

import java.lang.String;

public class dgBuildNumber
{
    public static final int BUILD_NUMBER = /*CFGMGMT_BUILD_NUMBER*/ 1397;
    public static final String BUILD_STAMP = /*CFGMGMT_BUILD_STAMP*/ "D2001_1119_085106";
}

Support for Makefiles

The build robot runs 'make' or 'nmake' directly when:

1. The <Build> element contains a file named 'makefile'
2. The <Build> element contains a file whose extension is .mak
3. There is a <Make/> element present in <Build>
4. There is a <NMake/> element present in <Build>

See the scripts section regarding the makefile's expected outputs (bld.txt and bldlog.txt).

In most cases, when building via makefiles, add the <Multiple/> element which is described here.

Please refer to the <MakeTargets> element in configmap.xml.  This allows you to have one <Build> entry for one makefile.  Rather than specifying targets in the build XML document via <Targets>, targets are specified in configmap.xml via <MakeTargets>.  <Targets> in the build XML file overrides configmap.xml.

Support for Scripts and Executables

<Build>scriptname.bat
<Once/>
<Parameters>one two three</Parameters>
<ScriptIsOutsideSandbox>1</ScriptIsOutsideSandbox>
<Path>source/c/utils</Path>
</Build>

The text inside the <Build> element specifies the path of the script to run.  <ScriptIsOutsideSandbox> tells the build robot not to try to load the script from CVS, which it will do if the path is not found in the sandbox.

<Once>1</Once> instructs the build robot to run the script only once even if multiple configurations are specified in the build options document.  The script to execute is specified in the <Build> element's text.  The script can be relative to the sandbox directory, or you can specify a full path if the script isn't checked into CVS.

If you don't use <Once>, you may want to use <Sequence1></Sequence>.  This element tells the build robot to build and package each configuration separately.  Normally, the build robot builds everything and then packages everything.  However, this doesn't work when different configurations output their artifacts to the same directory regardless of the configuration.

If you don't use <Once>, you'll probably want to send the script different parameters based on the configuration.  Here's an example:

<Build>build/scriptname.bat
<Sequence/>
<Configuration>Win32 Release</Configuration>
<Parameters>release</Parameters>
</Build>

<Build>build/scriptname.bat
<Sequence/>
<Configuration>Win32 Debug</Configuration>
<Parameters>debug</Parameters>
</Build>

Note that instead of duplicating essentially the same XML in the build document, you can put the parameters in the configmap.xml file, specifying build/scriptname.bat as the project.

Current Directory

When the script is executed, the current directory will be one of the following:

1. The directory specified in the <Path> (which is relative to the sandbox) element takes precedence
2. If the script or executable is relative to the sandbox (it does not start with a drive letter or UNC name), and <ScriptIsOutsideSandbox>1</ScriptIsOutsideSandbox> is not specified, the script's directory is used
3. The sandbox directory

Output Files

Scripts and executables can optionally write to a file called bld.txt that contains at least one line of the form:

Errors, Warnings

Where Errors is the number of errors encountered and Warnings is the number of warnings encountered.

The script may also write to bldlog.txt.  If Errors or Warnings is greater than zero, bldlog.txt will be made available on the build status web page.  If bldlog.txt does not exist, the build robot uses the text that the script writes to STDOUT.

bld.txt and bldlog.txt must be written to the current directory when the script is executed. 

Filters

The <Filter> element allows you to kick-off an external script after the build robot executes the task.  Place the name of the script (full command line) inside <Filter>. 

• The script parameters are file names.  Each file name should be opened and processed.
• The script should write to bld.txt and bldlog.txt in the current directory.

dggenericOutputParser

dggenericOutputParser is a custom filter.  It reads a set of input files and creates bld.txt and bldlog.txt. The script counts lines starting with "Error" or "Warning". The error and warning counts are written to bld.txt and all of the processed lines are written to bldlog.txt.

Usage:

dggenericoutputparser file1 file2 file3...

Or

dggenericoutputparser.pl file1 file2 file3...

Here is a <Build> fragment that uses it:

<Build>dgmktlb
  <ScriptIsOutsideSandbox>1</ScriptIsOutsideSandbox>
  <Parameters>.</Parameters>
  <Once>1</Once>
  <Path>source/c/tlb</Path>
   <Filter>dggenericOutputParser</Filter>
</Build>

dgnantfilter

dggenericOutputParser is a custom filter that processes NANT output.  It reads a set of input files and creates bld.txt and bldlog.txt.

Building Multiple Projects with one Makefile or Script

If a script or makefile builds multiple 'projects', specify <Multiple>1</Multiple> in the <Build> element.  bld.txt is not used in this case.  However, bldlog.txt is read if it exists (otherwise STDOUT is used).  The <Filter> element allows external scripts to create bldlog.txt from the build task's output.

<Multiple> is a convenience for the end-user.  Errors and warnings are reported on a per-project basis, and these errors appear in different 'project' sections in the build summary web page.  Each project also has its own log file containing the actual text of warnings and errors.  Without this feature, the end-user would be forced to trawl through a large amount of script output in order to find error and warning messages.

The only accepted output format is as follows:

1. Each project is denoted by lines like:
   Entering directory 'source/java/com/devguy'
Whatever appears between the quotes (single, double, or backticks) is used as the project name.
   The directory can be relative to the sandbox or an absolute path name.  Absolute path names must be 'beneath' the sandbox.
   When these lines are encountered, the build robot resets the error and warning counters, captured log output, etc.
2. Errors are denoted by lines like:
   56 errors
The lines must begin with a number or whitespace 
3. Warnings are denoted by lines like:
   10 warnings
The lines must begin with a number or whitespace 

Note that it is not possible to specify the configuration name (for example, devenv.exe output allows this).  That will be added at a later date.

Build Script Return Value

The build robot script dgbuild returns 0 on success and 1 if a runtime or build error occurs.

To-Do Text

It is not uncommon to check-in code that works but needs refinement.  Typically developers add comments to their code noting its deficiencies such as "add error handling," etc..  These comments often disappear into the "ether," never to be addressed.  Sometimes these issues are entered into a bug tracking system, but in many cases they are not.  Wouldn't it be nice if these reminders were brought to the teams' attention until they were resolved?  The build robot does just that without the use of a bug database.  In the case of C code, the reminders can appear in the source code itself, where they belong.

The build web pages have a link called "To-Do."  This links to a page that contains a list of to-do notes that were output during build time.  The build robot creates this page by inspecting the output of VC++ projects and makefiles looking for any line that contains the text TODO (all caps).  For example:

V:\source\c\IPResolver\n_iprslv\UKAddressFilter.cpp(505) : TODO - devguy: Create StoredProcedure get_post_towns

Outputting text at compilation time is easily done in C using #pragma message.  There is no Java counterpart that we know of, except for adding echo statements to a makefile.

Here's a C macro called TODO that outputs TODO text containing a file name, line number, and the name of the developer who created the message.

Usage:

#pragma TODO(devguy, "Create StoredProcedure get_post_towns")

todo.h:

#define DGTODO_INT2STR(intVal) "(" ###intVal ")"
#define DGTODO_LINE_STR(cLine) DGTODO_INT2STR(cLine)

#ifndef DGTODO
#ifdef DGTODO_OFF
#define DGTODO(idReviewer, szMsg )
#else
#define DGTODO( idReviewer, szMsg ) message( __FILE__ DGTODO_LINE_STR(__LINE__) " : TODO - " #idReviewer ": " szMsg )
#endif
#endif

TODO messages can be disabled by defining the preprocessor variable DGTODO_OFF.

"Missing" Files

The build web pages contain a link called "Missing Files."  This links to a page containing the files that were specified in the package configuration files but were not found.  This indicates that either a project was not built, it failed, or the files weren't retrieved from CVS.

The file "mktlb1.dll" is always listed because this VC++ project has no build targets.

Concurrency

Packaging

Once the build robot has built all projects for all specified configurations, the build artifacts are packaged up and posted to the build statistics web page.

The build XML file contains <Package> tags which refer to external <Package> documents.  Thus, in order to package files, you must alter the <Build> document and create new <Package> documents.

Please see the packager documentation for examples.

Stopping Builds

To stop a build, run the following commands from the command line:

1. dgkill -f perl
2. dgkill -f dgbuild
3. dgkill -f msdev
4. dgkill -f cl.exe
5. dgkill -f devenv

How to Remove a Build From the Web Application

Archived builds must be deleted by hand.

1. Go to a build folder on the IIS server such as c:\inetpub\wwwroot\build\nightly
2. Delete the folder(s) for the build(s) you want to remove - each folder is named using the date and time on which the build robot was started
3. Edit the file builds.xml

Troubleshooting

Sample Build Files - C++ Project 

• build.xml

• package.xml

Sample Build Files - Java Project

• build-options.xml

• build.xml

• package.xml