<#--

============================================================================ 

Name        : 

Part of     : Helium 

Copyright (c) 2009 Nokia Corporation and/or its subsidiary(-ies).

All rights reserved.

This component and the accompanying materials are made available

under the terms of the License "Eclipse Public License v1.0"

which accompanies this distribution, and is available

at the URL "http://www.eclipse.org/legal/epl-v10.html".

Initial Contributors:

Nokia Corporation - initial contribution.

Contributors:

Description:

============================================================================

-->

.. index::

  module: Stages

=============

Helium stages

=============

.. contents::

This section gives details of each of the phases of a Helium build and what they do.

.. index::

  single: Stage - startup

Stage: Startup

==============

.. index::

  single: Remote builds

Remote builds

-------------

.. index::

  single: Remote build Commands

Remote build Commands

:::::::::::::::::::::

Remote builds are used when a number of build configurations need to be run on several machines from a single work area. For a remote machine to receive the commands, a build manager must login and start the antserver process by running ``c:\apps\antserver\run_ant_server.bat``.

Two commands are supported::

  hlm distribute-work-area

Tars up the work area, copies it to a network location defined by the work.area.temp.dir property, and sends a command to the remote servers to untar the work area. On the remote servers the basedir is deleted before the work area is untarred. ::

  hlm start-remote-builds

Sends commands to start the builds based on the remote builds configuration file entries.

.. index::

  single: Remote build configuration

Remote build configuration

::::::::::::::::::::::::::

The configuration file format defines one or more builds::

  <BuildProcessDefinition>

      <remoteBuilds>

          <build machine="vcbldsrv12" ccmhomedir="${r'$'}{ccm.home.dir}" basedir="${r'$'}{ccm.base.dir}" executable="hlm" dir="${r'$'}{_build.dir}\PRODUCT" args="product-build -Dbuild.number=${r'$'}{build.number} -Dprep.root.dir=d:\"/>

      </remoteBuilds>

  </BuildProcessDefinition>

Each ``<build>`` element has a number of attributes:

machine

  The name of the remote build machine. The commands will only work if an Ant server instance is running, so be careful not to run the server on the local machine!

ccmhomedir

  This should match to the ``ccm.home.dir`` property.

basedir

  This defines the directory in which the current work area (under ``ccm.home.dir``) is located.

executable

  The file to be executed when starting a build. Typically this can be left as ``hlm``.

dir

  The directory where the executable should be found and where the command will be run from.

args

  The arguments passed to the executable. These should consist of Ant arguments, as the build is run using Ant. Note that this attribute value is treated in the same way as the line attribute in the Ant ``exec`` task - spaces are interpreted as separating the arguments.

The ``remote.builds.config.file`` property defines the location of the configuration file. This should be defined in a team Ant configuration file.

.. Commented out because we will not use this for our releases

   Subcon bootstrap

   ----------------

   The subcon edition of Helium does not include any 3rd party libraries due to licensing restrictions.

   Before you start using a copy of helium for the first time you need to call ``hlm-bootstrap.bat``.

   Run like this if you get timeout errors and set to the values of your proxy server::

     hlm-bootstrap.bat -Dproxy.host=172.16.42.137 -Dproxy.port=8080

   Or if you have no proxy server::

     hlm-bootstrap.bat -Dproxy.disabled=y

   The bootstrap process is:

    * Download Ivy jars.

    * Use Ivy to download dependencies.

    * Extract and install dependencies.

.. index::

  single: Stage - Preparation

Stage: Preparation

==================

At the start of preparation a new directory is created for the build and subst'ed to ``build.drive``. If a directory with this name already exists, it is renamed to have a current timestamp on the end.

.. index::

  single: How to prepare the build area?

How to prepare the build area?

------------------------------

A key part of build preparation is initialising the build drive by copying or unzipping the input files. The ``build.prep.config.file`` should reference a file that follows the prep XML file format (e.g. build/_build/prep.xml). A suggestion is that this file is called prep.xml by default.

The XML format of the prep file is as follows:

Beginning of the file and config

:::::::::::::::::::::::::::::::::

.. code-block:: xml

    <?xml version="1.0" encoding="UTF-8"?>

    <prepSpec>

        <config>

            <exclude name="_ccmwaid.inf"/>

            <exclude name="abld.bat"/>

            <exclude name=".static_wa"/> 

            <exclude name="documentation/*"/> 

            <exclude name="documents/*"/> 

            <exclude name="doc/*"/>

        </config>

Each exclude under config defines which files are NOT extracted during unzip phases.

Unzip

:::::

.. code-block:: xml

    <source label="Symbian" basedir="${r'$'}{symbian.release.dir}">

        <unzip name="${r'$'}{symbian.zip.prefix}${r'$'}{symbian.release}_src_generic_part1.zip"/>

        <unzip name="${r'$'}{symbian.zip.prefix}${r'$'}{symbian.release}_epoc32.zip">

            <include name="epoc32/tools/*"/>

        </unzip>

    </source>

Each separate unzip phase is defined beginning with source. Each file that needs to be unzipped is specified with unzip. It is also possible to define what parts of the zips are extracted using include tags.

ICD/ICF Unzip

::::::::::::::

.. code-block:: xml

    <source name="icds" basedir="">

        <unzipicds dest="Z:\">

            <location name="${r'$'}{ccm.base.dir}/S60_3_1/S60_3_1/Symbian_ICD_ICF/${r'$'}{symbian.release}" />

            <include name="src/*" />

        </unzipicds>

    </source>

ICDs/ICFs (Intermediate Code Fix\Drop) are extracted using unzipicds tag. Using this command ensures that the files are extracted in the correct order. Multiple ICD/ICF directories can be given to be extracted by giving multiple locations.

Copy

::::

.. code-block:: xml

    <source label="S60" basedir="${r'$'}{ccm.base.dir}/S60_3_1/S60_3_1">

        <copy name="adaptation" dest="adaptation"/>

        <copy name="S60" dest="S60">

            <exclude name="S60/bldvariant/Series_60_3_1_common/ProductVariant.hrh"/>

        </copy>

    </source>

Each separate copy phase is defined beginning with source. The dest parameter for each copy command defines to what folder the files are copied starting fron the given basedir. It is also possible to exclude files using the exclude tag.

File end

::::::::

::

    </prepSpec>

ICF/ICD:

````````

This information is extracted from the filesystem, it uses the sources defined in the preparation xml configuration file to retreive the ICF/ICD content.

.. Note::

    This means that all ICD/ICF extracted from any other source will not be considered.

    So you MUST use the <unzipicds dest=""> statement of the preparation configuration file.

Flags:

``````

Flags are extracted from the ProductVariant.hrh (config/${r'$'}{build.configuration}_config/${r'$'}{product.name}/include/ProductVariant.hrh).

.. index::

  single: Stage - Source preparation

Stage: Source preparation

=========================

The build preparation consists in two parts:

 * Getting delivery content (SCM, zips...),

 * Preparing the build area.

To get SCM source you just have to run::

  hlm prep-work-area

To create 'build of materials'::

  hlm create-bom

Synergy

-------

In order for the synergy commands to be executed you must define the property ccm.enabled=true in one of the your config files or on the command line. e.g. 

.. code-block:: xml

    <property name="ccm.enabled" value="true" />

It is possible to automatically get content from Synergy using the Helium framework.

To handle that you have to configure the delivery.xml file from your family build configuration folder and reference by the property prep.delivery.file.

Example configurations like a minibuild can be found under the Helium source tree.

Example of configuration:

.. code-block:: xml

    <build>

        <config name="" abstract="true">

            <set name="database" value="fa1f"/>

            <set name="host" value="${r'$'}{ccm.engine.host}" />

            <set name="dbpath" value="${r'$'}{ccm.database.path}" />

            <set name="dir" value="Z:\some\location"/>

            <set name="threads" value="4"/> 

            <set name="sync" value="true"/> 

            <set name="use.reconfigure.template" value="false"/> 

            <set name="release" value="${r'$'}{release.tag}" />

            <config name="ppd_sw-PPD51.32_200810:project:sa1spp#1" type="checkout" >

               <set name="folders" value="jk1f#1820" />

            </config>

            <config name="WLANSniffer2-2007_wk21:project:e002sa08#1" type="snapshot" />

            <config name="NSeries08_Themes-1:project:fa1f#1" type="checkout" >

               <set name="tasks" value="jk1f#1763" />

               <set name="skip.ci" value="true"/> 

            </config>

            <config name="NSeries08_Themes-1:project:fa1f5133#2" type="checkout" >               

               <set name="skip.ci" value="false"/> 

               <set name="ci.custom.query" value="(release='MinibuildDomain/next')"/>

            </config>

            <config name="S60-S60.32_200810:project:sa1spp#1" type="checkout" >

               <set name="folders" value="jk1f#1983" />

            </config>

            <config name="ppd_sw-username:project:sa1spp#1" type="update"/>

            <config name="cellmo" abstract="true">

                <set name="dir" value="${r'$'}{ccm.base.dir}\cellmo" />

                <set name="threads" value="1" />

                <config name="cellmo_bins_rm235_PRODUCT-ncpp.ICPR71_08w24.2:project:tr1cmtsw#1" type="snapshot" />

                <config name="cellmo_bins_rm236_PRODUCT_chn-ncpp.ICPR71_08w24.2:project:tr1cmtsw#1" type="snapshot" />

                <config name="cellmo_bins_rm342_PRODUCT_lta-ncpp.ICPR71_08w24.2:project:tr1cmtsw#1" type="snapshot" />

            </config>

        </config>

    <build>

Checkout: only need to define this when extra tasks are required on top of the listed project, otherwise use the snapshot type.

    The following properties are required:

        - release : synergy release to use.

        - dir     : the location of your target snapshot.

        - database: the name of the synergy database you want to use.

    The following properties are optional:

        - thread  : optional parameter, this define the number of process to run for parallel snapshots.

        - purpose : Purpose to check out with.

        - sync : Force a sync step after the work area update.

        - version : the version to check out toward to.

        - tasks : add additional tasks to the reconfigure properties.

        - folders : add additional folders to the reconfigure properties.

        - use.reconfigure.template: enable the usage of the reconfigure templates, this means the project will just be reconfigured, the reconfigure properties will not be modified.

        - fix.missing.baselines: automatically detect new projects and check them out.

        - replace.subprojects: boolean value to enable/disable project replacement during update (default: true).

        - skip.ci: boolean value to include/exclude the project from CC modificationset checking.

        - ci.custom.query: Extend the synergy query for CC modificationset checking eg.(release='MinibuildDomain/next').

        - show.conflicts: boolean value to check for task conflicts.

        - show.conflicts.objects: boolean value to check for object conflicts.

Snapshot: define type of the spec as snapshot and name as the baseline name.

    The following properties are required:

        - dir     : the location of your target snapshot.

        - database: the name of the synergy database you want to use.

    The following properties are optional:

        - thread  : optional parameter, this define the number of process to run for parallel snapshots.

Update: define type of the spec as update and name as the project to update.

    The following properties are required:

        - database: the name of the synergy database you want to use.

Mercurial

---------

Add to ant configuration:

.. code-block:: xml

    <target name="prep-work-area">

        <hlm:scm scmUrl="scm:hg:C:/Build_C/master"> 

            <hlm:checkout baseDir="${r'$'}{ccm.project.wa_path}/GraphicsDomain"/>

            <hlm:changelog baseDir="${r'$'}{ccm.project.wa_path}/GraphicsDomain" xmlbom="${r'$'}{build.log.dir}/${r'$'}{build.id}_bom.xml" />

        </hlm:scm>

    </target>

For more information see API_

.. _API: ../helium-antlib/api/doclet/index.SCM.html

.. index::

  single: Stage - Compilation

Stage: Compilation

==================

Compilation is based on configuration using Symbian System Definition XML files.

See http://developer.symbian.org/wiki/index.php/System_Definition

Every System Definition file can contain content for two separate sections:

System model

    A definition of the system describing the components that exist, broken into layers, modules, etc.

Build model

    Build configurations that define what is to be built and how it will be built. Separate unitLists define groups of components.

The steps to configure a Helium build for main compilation are as follows:

1. Put together a list of the System Definition files that define the components needing to be built in the system model sections. This could be one or several files depending on what components need building. They should be defined in an Ant ``<path>`` type with an ``id`` atttribute set to ``system.definition.files``, e.g:

.. code-block:: xml

    <path id="system.definition.files">

        <fileset dir="${r'$'}{build.drive}/src/common/generic/tools/build" includes="System_Definition*.xml"/>

        <pathelement path="${r'$'}{build.drive}/build/ibusal_40_build/ibusal_40/IBUSAL40_System_Definition.xml"/>

        <fileset dir="${r'$'}{build.drive}/s60/tools/build_platforms/build/data" includes="S60_System*.xml"/>

        <pathelement path="${r'$'}{build.drive}/me/me_scd_desw/sysdef/System_Definition_PRODUCT.xml"/>

        <pathelement path="${r'$'}{build.drive}/build/${r'$'}{product.family}_build/_System_Definition.xml"/>

    </path>

The order of the files is significant. If building Symbian OS, the Symbian System Definition file must come first. Here both ``fileset`` and ``pathelement`` are used. ``pathelement`` selects just one file whereas a ``fileset`` can use wildcards to select multiple files or handle problems of filenames changing across different platform releases.

2. Determine if an existing build configuration in any of the build model sections of the files are suitable for what needs to be built. A build configuration typically looks something like this:

.. code-block:: xml

    <configuration name="foo_config" description="Build foo">

         <unitListRef unitList="foo_list"/>

         <task><buildLayer command="bldmake bldfiles" unitParallel="Y"/></task>

         <task><buildLayer command="abld export" unitParallel="Y"/></task>

         <task><buildLayer command="abld makefile" targetList="default" unitParallel="Y" targetParallel="N"/></task>

         <task><buildLayer command="abld resource" targetList="default" unitParallel="N" targetParallel="N"/></task>

         <task><buildLayer command="abld library" targetList="default" unitParallel="N" targetParallel="N"/></task>

         <task><buildLayer command="abld target" targetList="default" unitParallel="Y" targetParallel="Y"/></task>

         <task><buildLayer command="abld final" targetList="default" unitParallel="N" targetParallel="N"/></task>

         <task><buildLayer command="abld -what export" unitParallel="Y"/></task>

         <task><buildLayer command="abld -what target" targetList="default" unitParallel="Y" targetParallel="Y"/></task>

         <task><buildLayer command="abld help" unitParallel="Y"/></task>

         <task><buildLayer command="abld -check build" targetList="default" unitParallel="Y" targetParallel="Y"/></task>

    </configuration>

A ``unitListRef`` includes a ``unitList`` defined somewhere else as part of this configuration. The ``buildLayer`` elements define ``abld`` steps to run on each component. If an existing configuration is not sufficient a new one must be defined in a separate file (which should be included in the ``path`` type).

3. Define the ``sysdef.configurations`` Ant property to contain a comma-separated list of build configuration names that must match the ``name`` attribute of the ``configuration``. Each configuration will be built in turn in the ``compile-main`` Ant target.

Note: Build will fail if compilation error exceeds the number specified in ``build.errors.limit``. Default value is ``0`` and set it to ``-1`` to ignore this.

.. index::

  triple: Builds; EBS; EC

EBS and EC compilation

----------------------

If you want to get Helium to switch compiler version you need to define the HLM_RVCT_VERSION environment variable (Nokia specific feature)::

    set HLM_RVCT_VERSION=22_593

The setting could be mentioned under the configuration's Helium bootstrapper.

By default Helium is configured to run EBS builds. The ``build.system`` property determines what build system to use. An EC build can be run from the command line using::

    hlm <build target> -Dbuild.system=ec-helium

EC build could be configured to be running in parallel (default) or in serial mode (1 node build)::

    hlm <build target> -Dbuild.system=ec-helium -Dec.mode=serial

Also the --emake-debug could be configured either by the environment using the EMAKE_DEBUG variable or using the ``emake_debug_flag`` property. Its default value is 'g'.

.. index::

  single: Stage - Post Build

Stage: Post Build

=================

.. index::

  single: SIS Files

Sis Files

---------

SIS files can be built during the postbuild stage. The ``sis.config.file`` property defines the path to a :ref:`common-configuration-format-label` file, e.g.

.. code-block:: xml

    <build>

        <config abstract="true">

            <set name="makesis.tool" value="${r'$'}{build.drive}\epoc32\tools\makesis.exe"/>

            <set name="signsis.tool" value="${r'$'}{build.drive}\epoc32\tools\signsis.exe"/>

            <set name="build.sisfiles.dir" value="${r'$'}{build.sisfiles.dir}"/>

            <set name="key" value="${r'$'}{build.drive}\s60\tools\taskmgr\internal\sis\RDTest_02.key"/>

            <set name="cert" value="${r'$'}{build.drive}\s60\tools\taskmgr\internal\sis\RDTest_02.der"/>

            <config>

                <set name="name" value="ScreenGrabber_3"/>

                <set name="path" value="${r'$'}{build.drive}\s60\tools\screengrabber\sis"/>

            </config>

            <config>

                <set name="name" value="app_trk"/>

                <set name="path" value="${r'$'}{build.drive}\s60\tools\trk\sis"/>

            </config>

        </config>

    </build>

The propertes are:

.. csv-table:: Property descriptions

   :header: "Property", "Description", "Values"

   "``makesis.tool``", "The path for the makesis tool that builds a .sis file.", ""

   "``signsis.tool``", "The path for the signsis tool that signs a .sis file to create a .sisx file.", ""

   "``publish.unsigned``", "This will copy .sis files into ${r'$'}{build.output.dir}/sisfiles.", "true, false"

   "``build.sisfiles.dir``", "The directory where the .sis file should be put.", ""

   "``key``", "The key to use for signing.", ""

   "``cert``", "The certificate to use for signing.", ""

   "``name``", "The name of the .pkg file to parse.", ""

   "``sis.name``", "The name of the .sis file to create. If omitted it will default to the name of the .pkg file.", ""

   "``path``", "The path where the .pkg file exists as input to building the .sis file.", ""

Configuration enhancements

::::::::::::::::::::::::::

*Since Helium 7.0.*

The configuration method above will be replaced by a more flexible approach:

.. csv-table:: Property descriptions

   :header: "Property", "Description", "Values"

   "``makesis.tool``", "The path for the makesis tool that builds a .sis file.", ""

   "``signsis.tool``", "The path for the signsis tool that signs a .sis file to create a .sisx file.", ""

   "``build.sisfiles.dir``", "The directory where the .sis file should be put.", ""

   "``key``", "The key to use for signing.", ""

   "``cert``", "The certificate to use for signing.", ""

   "``input``", "The full path and filename of the input file. This can be a .pkg file, for generating a SIS file, a .sis file for signing, or a .sisx file for multiple signing.", ""

   "``output``", "The full path and filename of the output file. This is only needed if the location or name needs to be different from the default, which is that the file extension changes appropriately.", ""

Also a ``sis.config.name`` property is added that allows the name of a <config> block to be supplied. This can be overridden to allow particular subsets of configurations to be built.

Checking Stub SIS files

-----------------------

This step involves checking stub sis files published to ``z:/epoc32/data/z/system/install`` and it ensures that only valid stub sis files are published.The target is included in postbuild and using ParseStubSis.pl script to do the work, it will run automatically and save the output in stubresult.log.xml. It can also be run from the command line by using::

    hlm check-stub-sis

It checks all files in the target folder and renames the invalid sis files with .bak extention. 

.. index::

  single: Stage - Publishing

Stage: Publishing

=================

.. index::