README

last updated: October 26, 2005

Introduction

This readme describes how to create performance charts and tables similar to those available on the eclipse platform download pages using the stand-alone java program org.eclipse.test.performance.ui.Main stored in this project. Although the program is currently geared to generating results for the Eclipse Platform builds, the program may work for you given the following conditions:

• The "eclipse.perf.config" system property used to run tests used three keys config, build and jvm
  • for example, "eclipse.perf.config=config=relengbuildwin2;build=I20050219-1500;jvm=sun1.4.2_06".
    
    
• Values for the "build" key begin with either an 'I' or 'N' with the exception of the reference builds (for example, 3.0).
  • The line graphs plot the reference build, all builds starting with 'I' and the seven 'N' builds preceding the current build.
  • This can be overriden with the use of the "-current.prefix" parameter where you can specify a comma-separated list of prefixes for build ids to include in the main line graph.
    
    
• Scenario names all contain a prefix "<component>.test"
  • For example, it is assumed that the scenario "org.eclipse.help.tests.performance.appserver.TestAppserver#testAppserverCycle()" belongs to the "org.eclipse.help" component.
  • Unique scenario prefixes are derived from all the scenarios in the database and are used to group results.

For purposes of this document, the term "fingerprint" will be used to refer to a bar graph which represents the performance summary at a global or component level for each test configuration.

The term "scenario status table" will be used to refer to a table of scenarios with green checks and red x indicators below a component fingerprint. This table gives an overall view of the status for all scenarios for a component for all configurations.

The term "scenario results" will be used to refer to a web page comparing current performance measurements to the baseline measurements for one scenario on one configuration. The web page displays the raw data for the current and baseline measurements and also displays line graphs showing the measured values over the course of builds between the current and baseline.

Requirements

• Windows or Linux (x86) operating system
• Cloudscape or Derby plug-in
• org.eclipse.releng.basebuilder
• 1.4 level jvm installed and on path

Setup

• Checkout org.eclipse.releng.basebuilder from cvs repository dev.eclipse.org:/home/eclipse (HEAD should be fine).
• Create a Cloudscape or derby plug-in as described here.
• Set the environment variable "dbloc" to the location of the database containing the performance data
  For example:
  
  • export dbloc=net://minsky.ottawa.ibm.com (Linux)
  • set dbloc==net://minsky.ottawa.ibm.com (Windows)
    

Running the program

From a terminal or dos prompt,

1. Set the environment variable "dbloc" to the location of the database containing the performance data
   For example:
   • export dbloc=net://minsky.ottawa.ibm.com (Linux)
   • set dbloc==net://minsky.ottawa.ibm.com (Windows)
     
     
2. cd to org.eclipse.releng.basebuilder\plugins\org.eclipse.test.performance.ui\scripts
   
3. If running on Linux, execute "chmod 755 genresults"
4. Execute the following command:
   
   genresults -baseline <baseline build id> -current <current build id> -config <comma-separated list of configs> -jvm <jvm name> -output <output dir> [-config.properties <properties>] [highlight <buildid patterns>] [scenario.pattern <scenario prefix patterns>] [-fingerprints][-scenarioresults][-current.prefix <comma-separated list of build id prefixes>][-baseline.prefix <comma-separated list of build id prefixes>]

-baseline <baseline build id. A previous value used in the "build" key in the eclipse.perf.config system property or the eclipse.perf.assertAgainst property. >
-current <current build id. The value used in the "build" key in the eclipse.perf.config system property.>
-config <a comma-separated list of names of configurations for which to produce results. The values used should be the same as values specified for the "config" key in the eclipse.perf.config system property.>
-jvm <jvm description. The value used in the "jvm" key in the eclipse.perf.config system property.>
-output <path to an output directory for fingerprints and scenarios for each configuration specified in -config parameter. Fingerprint gifs and html files are generated in the directory specified, configuration specific subdirectories are created to store scenario results and line graphs.>

Optional arguments:

-config.properties <semi-colon separated list of: config, alternate config description. The value should be quoted if there are spaces in the value.>
of builds which match any prefix in this list. When omitted, a magenta line is draw on the graph which represents the baseline value specified in the -baseline parameter.>
-highlight <Comma-separated list of build Id prefixes(excluding 'I' and 'N'), which are used to add and highlight the most recent matching buildId on line graphs.">
-scenario.pattern <Scenario prefix pattern used to query performance results database. Can be used to generate results for a single component or subset of scenarios.>
-fingerprints or -scenarioresults <use one or the other to generate fingerprints or scenario results only. Not specifying either will execute both.>
-baseline.prefix <semi-colon separated list of: build id prefixes used in eclipse.perf.assertAgainst property or eclipse.perf.config system property. Values used to generate a second line graph representing changes in repeated baseline test runs>.
-current.prefix <semi-colon separated list of: build id prefixes used in eclipse.perf.config system property. Values here used to override default of "N,I". Used to select build id's to include in main line graph.>