Automating Builds with PDE BUILD

Last Updated: June16, 2005

Introduction

Preparing the infrastrucure

Preparing Build Configuration Files

Build Execution

 

Introduction

This document describes how to automate the building of Eclipse-based features and their plug-ins using script generators in the org.eclipse.pde.build plug-in in Eclipse 3.2 stream SDK.

Either an existing Eclipse 3.2 stream SDK or the org.eclipse.releng.basebuilder project on dev.eclipse.org:/home/eclipse can be used in this automated build process. The org.eclipse.releng.basebuilder project contains the minimal set of plug-ins extracted from the latest stable Eclipse 3.2 stream SDK build (a milestone or release) needed to run applications and custom Ant tasks in org.eclipse.ant.core, org.eclipse.pde.build, org.eclipse.pde, and org.eclipse.help.base. This project is used by the Eclipse release engineering team to build Eclipse itself.

It is assumed the reader is starting with a set of pre-existing features and plug-ins, their build.properties, and has a working knowledge of Apache Ant. In this document, features, plug-ins and fragments are also referred to as "elements". The word "distribution" is used to describe the end result of building a feature, a functional unit comprised of one or more groupings of plug-ins. The result of building a feature results in a zip or tar.gz file which contains the binary version of the feature and it's plug-ins. If the feature contains nested features (i.e. <includes> elements), the nested features will be built recursively and included in the distribution.

For example, the Eclipse SDK distributions are built from the org.eclipse.sdk feature which is comprised of features org.eclipse.rcp, org.eclipse.rcp.source, org.eclipse.platform, org.eclipse.platform.source, org.eclipse.jdt, org.eclipse.jdt.source, org.eclipse.pde, org.eclipse.pde.source and the org.eclipse.sdk plug-in. The distribution built from the org.eclipse.sdk feature will therefore contain the binary version of the org.eclipse.sdk feature and its one plug-in plus the binary versions of its eight nested features and all their plug-ins.

Preparing the infrastructure

Commit feature and plug-in projects to CVS repository

Since Eclipse 3.0, PDE Build allows a very flexible organization of feature, plug-in and fragment projects in a CVS repository where:

This flexibility was not present for the Eclipse process prior to version 3.0 which is one reason for the very flat organization of projects in the dev.eclipse.org:/home/eclipse repository. It is recommended that this flat structure not be used as an example. Rather, using a structure similar to the one used for the Equinox or Stellation project should be considered where all files and directories for a given product are stored under a single directory or module at the cvs root level:

<cvs root>

     /org.eclipse.equinox

          /plugins

               /all plug-ins at this level
               

Once the source for all elements is committed to a repository, the next step consists of recording the location and access method for each feature, plug-in and fragment in one or more map files.

 

Create map file project

A .map file is a java property file which contains mappings of elements to their CVS locations and access methods. Map files are used by PDE Build early in the build process to generate Ant scripts which use the Ant <cvs> task to export source to a directory. This is described further below.

Map file entries use the following format:

feature|fragment|plugin@element.Id=<cvs tag>,<access method>:<cvs user>@<cvs repository>,<cvs password>[,<repository path> (no starting slash) ]

The <repository path> is only required when the module (or directory) containing the source for the element does not match the element.Id or if the directory is not at the root of the repository.

A map file entry must exist for each feature being built, it's <plugin> elements and it's <includes> elements (ie. nested features and their plug-ins). Adding a plug-in or fragment to a feature therefore requires updating the map files with the new element.


Map File Entry Examples

One or more map files can be used to list the elements. The map files can be kept under version control. Some examples of map file projects include org.eclipse.releng, org.eclipse.gef.releng, org.eclipse.ve.releng.

Generating source features and plug-ins at build time

Source features and plug-ins can be generated at build time by PDE Build. Source features and associated source plug-ins are typically generated for a development kit distributions (i.e. SDK). It is also possible to generate a source plug-in only. This is typically the case for example features or JUnit testing features.

To generate a source feature and associated source plug-in at build time, you will need to do the following:

  1. Add an entry to the build.properties file in the feature project for which you wish to include the source feature and plug-in. The generated source feature should also be listed in the feature.xml as an <includes> element.

    The build.properties entry should use the following format:

         generate.feature@<source.feature.id to generate>=<feature.id from which to which to collect source>, plugin@<plugin.id>

    Example taken from org.eclipse.sdk-feature/build.properties:
         generate.feature@org.eclipse.jdt.source=org.eclipse.jdt, plugin@org.eclipse.jdt.doc.isv

    In this example, a source feature and a plug-in, both with id "org.eclipse.jdt.source" will be generated and will contain source from plug-ins listed in the org.eclipse.jdt feature and will also include the plug-in org.eclipse.jdt.doc.isv. The generated org.eclipse.jdt.source plug-in will be automatically listed in the org.eclipse.jdt.source feature.xml.


  2. In the feature project from which the source feature will be generated, a directory called "sourceTemplateFeature" and a directory called "sourceTemplatePlugin" will be required. These directories should contain the files that are included in the root of the generated source feature and plug-in. The feature.xml and plugin.xml files are not required since these are generated. A build.properties is required in the sourceTemplatePlugin directory. This should contain a "bin.includes" setting as well as the entry "sourcePlugin = true". The plugin.xml file and src/ directory should be listed in the bin.includes.

    See org.eclipse.jdt-feature and org.eclipse.platform-feature for examples.

To generate a source plug-in only at build time, you will need to do the following:

  1. Add an entry to the build.properties file in the feature project for which you wish to include the source plug-in. The generated source plug-in should also be listed in the feature.xml as a <plugin> element..

    The build.properties entry should use the following format:

    generate.plugin@<source.plugin.id to generate>=<feature.id from which to which to collect source>, plugin@<plugin.id>

    Example taken from org.eclipse.ve.tests-feature/build.properties:
         generate.plugin@org.eclipse.ve.tests.source=org.eclipse.ve.tests


  2. In the runtime feature project from which the source plug-in will be generated, create a directory called "sourceTemplatePlugin" which must contain a build.properties with a "bin.includes" setting and "sourcePlugin=true". The plugin.xml file and src/ directory should be listed in the bin.includes.

    See org.eclipse.ve.examples-feature/sourceTemplatePlugin for an example.

Preparing Build Configuration Files

The distilled build process consists of the following four steps:

  1. build environment setup
  2. check out source from one or more CVS repositories
  3. compilation
  4. assembly of distribution

The script which controls the build sequence is the build.xml Ant script in org.eclipse.pde.build. However this script requires two user-implemented build configuration files,build.properties and customTargets.xml. These two files provide the information on the "where and how" to build specific elements.

Templates of these files are provided in the org.eclipse.pde.build/templates directory and examples are available in org.eclipse.releng.eclipsebuilder, org.eclipse.releng.gefbuilder and org.eclipse.ve.releng.builder for building Eclipse, GEF and VE runtimes, respectively.

build.properties

The build.properties file defines a number of properties that are used as Ant properties at build time and as arguments to script generators in org.eclipse.pde.build to describe how and where to execute the build. The values for properties listed in this file override any values set in the generated build.xml files. See "Generating Ant scripts from the command line" in the PDE Guide in Eclipse 3.2 stream Help for a description of required and optional properties.

customTargets.xml

customTargets.xml is an Ant script containing targets called by PDE Build scripts to provide the following information:

  1. the list of elements for which to generate scripts
  2. instruction on retrieval of map file projects
  3. steps to execute before and after the following: retrieving map files, checking out source, generating build.xml scripts, executing build.xml scripts, and assembling the binary distributions.
  4. instruction on things to do after the build is done.

The table below lists the targets that are used to provide this information.

TargetDescription
allElements

This target lists all features that will be packaged into a binary distribution where each listing comes in the form of an <ant> call to org.eclipse.pde.build/scripts/genericTargets.xml:

<ant antfile="${genericTargets}" target="${target}" >
     <property name="type" value="feature" />
     <property name="id" value="<element.id>" />
</ant>

The user is only required to specify a value for properties "type" and id (the value for the id attribute in the feature.xml) for each listing. At this time, only the type "feature" is supported.

Example from org.eclipse.releng.eclipsebuilder/jdt/customTargets.xml:

<target name="allElements">
     <ant antfile="${genericTargets}" target="${target}" >
          <property name="type" value="feature" />
          <property name="id" value="org.eclipse.jdt" />
     </ant>
</target>

The genericTargets.xml script is an Ant script in the org.eclipse.pde.build plug-in containing targets which call PDE Build custom Ant tasks to generate scripts for the specified elements at various stages of the build. This script also executes the generated scripts at various build stages. The target property is set by it's calling script, org.eclipse.pde.build/scripts/build.xml.

For example, the fetch target in the build.xml calls allElements and sets the target to property to "fetchElement":

<ant antfile="${customTargets}" target="allElements">
      <property name="target" value="fetchElement" />
</ant>

The result of this is that the fetchElement target in genericTargets.xml is executed using arguments type and id set in allElements.

assemble.<element.id>[.config.spec]

For every configuration specified in the build.properties for the distribution (see configs above), a target named "assemble.<element.id>.<config.spec>" is required. If the distribution is not platform-specific, the ".<config.spec>" section of the target name is not required.

Providing the target name should be all that is required unless you wish to give the produced binary distributable file a name different from the default "<elment-id>-<buildid>-<config.spec>.zip". In this case, an explicit value for the property "archiveName" should be specified at the beginning of the target.

Example from org.eclipse.releng.eclipsebuilder/jdt/customTargets.xml:

Since two configurations for building the org.eclipse.jdt distribution are specified in it's build.properties "configs=*,*,* & macosx,carbon,ppc", the following two targets are provided in the customTargets.xml script.

<target name="assemble.org.eclipse.jdt">
     <property name="archiveName" value="eclipse-JDT-${buildId}.zip"/>
     <ant antfile="${assembleScriptName}"/>
</target>

<target name="assemble.org.eclipse.jdt.macosx.carbon.ppc">
     <property name="archiveName" value="eclipse-JDT-${buildId}-macosx-carbon.tar.gz"/>
     <ant antfile="${assembleScriptName}"/>
</target>

getMapFiles

The result of executing this target should be to place *.map files in any directory or subdirectory under ${buildDirectory}/maps. All .map files found here are concatenated into a single file ${buildDirectory}/directory.txt. Map file projects are typically kept under version control in a CVS repository.

In the following example from org.eclipse.releng.eclipsebuilder/sdk/customTargets.xml, the implementation of this target and it's helper targets are provided to illustrate how map files for Eclipse builds are checked out from a CVS repository and then tagged with the build timestamp to capture the versions of all projects used in the build.

<target name="getMapFiles" depends="checkLocal" unless="mapsLocal">
     <property name="mapCvsRoot" value=":pserver:anonymous@dev.eclipse.org:/home/eclipse" />
     <property name="mapVersionTag" value="HEAD" />
     <cvs cvsRoot="${mapCvsRoot}"
          package="org.eclipse.releng"
          dest="${buildDirectory}/maps"
          tag="${mapVersionTag}"
     />
    <antcall target="tagMapFiles" />
</target>

<!--helper targets--->

<target name="checkLocal">
     <available property="mapsLocal" file="${buildDirectory}/maps/org.eclipse.releng" />
</target>

<target name="tagMapFiles" if="tagMaps">
     <cvs dest="${buildDirectory}/maps/org.eclipse.releng" command="tag v${timestamp}" />
</target>

preSetup and postSetup

Used to run operations before and after retrieving the map files.

Example from org.eclipse.releng.gefbuilder/sdk. This example demonstrates how the postSetup target (and a helper target) is used to download and install an Eclipse SDK to compile against.

<target name="postSetup">
     <available file="${buildDirectory}/../eclipse-SDK.zip" property="baseExists" />
     <antcall target="getBaseEclipse" />
</target>

<target name="getBaseEclipse" unless="baseExists">

     <!--this task definition is available in org.eclipse.releng.basebuilder/plugins/org.eclipse.build.tools. It removes the             _ <version> from all directories specified as a value for the directory attribute. Not really necessary, but helpful in        this case to avoid having/needing hard-coded versions in GEF javadoc scripts.-->
     <taskdef name="stripVersions" classname="org.eclipse.releng.VersionNumberStripper" />

     <!--this property file contains the values for ${eclipseURL} and ${eclipseBuildID}-->
     <property file="${buildDirectory}/maps/org.eclipse.gef.releng/maps/build.cfg" />
     <get src="${eclipseURL}/eclipse-SDK-${eclipseBuildID}-win32.zip" dest="${buildDirectory}/../eclipse-SDK.zip" />
     <exec dir="${buildDirectory}/.." executable="unzip">
          <arg line="-o -qq eclipse-SDK.zip" />
     </exec>
     
     <stripVersions directory="${buildDirectory}/plugins" />

     <!-- Extract doc.zip so we can create links in GEF java doc -->
     <exec dir="${buildDirectory}/plugins/org.eclipse.platform.doc.isv" executable="unzip">
          <arg line="-o -qq doc.zip" />
     </exec>
</target>

preFetch and postFetch

Used to run operations before and after fetching source for the build.

Example from org.eclipse.releng.gefbuilder/sdk. This example demonstrates how the postFetch target can be used to set the build timestamp as a value for "0" in about.mappings files.

<target name="postFetch">
     <replace dir="${buildDirectory}/plugins" value="${timestamp}" token="@buildid@">
          <include name="**/about.mappings" />
     </replace>
</target>

preGenerate and postGenerate

Used to run operations before and after generating build.xml files for features, plug-ins and fragments.

Example from org.eclipse.releng.gefbuilder/sdk. This example demonstrates how the postGenerate target (and a helper target) is used to run the build.xml scripts to clean the source of any stale, pre-compiled jars that might exist in the source directories. jars are not recompiled if they already exist in plug-in or fragment directories.

<target name="postGenerate">
     <antcall target="clean" />
</target>

<target name="clean" unless="noclean">
     <antcall target="allElements">
          <param name="target" value="cleanElement" />
     </antcall>
</target>

preProcess and postProcess Used to run operations before and after compiling the source.
preAssemble and postAssembleUsed to run operations before and after assembling the binary distributables.
postBuild

Used to run operations at the end of the build.

Example from org.eclipse.releng.gefbuilder/sdk. This example demonstrates how the postBuild target (and a helper target) is used to gather and place the compile logs in the ${buildLabel} directory. These files are used in the publishing of the GEF build (see below under Publishing the Build section).

<target name="postBuild">
     <antcall target="gatherLogs" />
</target>

<target name="gatherLogs">
     <mkdir dir="${buildDirectory}/${buildLabel}/compilelogs" />
          <antcall target="allElements">
               <param name="target" value="gatherLogs" />
          </antcall>
     <unzip dest="${buildDirectory}/${buildLabel}/compilelogs" overwrite="true">
          <fileset dir="${buildDirectory}/features/org.eclipse.gef.sdk">
               <include name="*.log.zip" />
          </fileset>
     </unzip>
</target>

 


Build Execution

Build machine setup

This build process can be executed on any of the Eclipse 3.2 Reference Platforms plus the following:

The cvs, zip and unzip executables should be placed on the system path.

On Windows systems, the HOME environement variable should be set to "c:" (no slash) for CVS operations.

The org.eclipse.releng.basebuilder project can be placed in any directory.


Running the Build

To run the build, execute the following command from ../plugins/org.eclipse.pde.build/scripts to build a single distribution:

java -jar <path>/startup.jar -application org.eclipse.ant.core.antRunner [-buildfile build.xml] -Dbuilder=<path to directory containing build.properties and customTargets.xml> [Ant property settings to override those in build.properties]