buildframework/helium/sf/java/diamonds/diamonds.rst
author Bob Rosenberg <bob.rosenberg@nokia.com>
Mon, 13 Sep 2010 13:52:18 +0100
changeset 656 88e01c8f013e
parent 628 7c4a911dc066
permissions -rw-r--r--
Fix merging bug where downstream items somtimes got ignored if they appear after an upstream-only item. Also, reduce the severity of the validation error if a tech-domain is not recognised on a non-Foundation package.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
     1
.. index::
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
     2
  module: Configuring Diamonds
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
     3
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
     4
====================
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
     5
Configuring Diamonds
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
     6
====================
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
     7
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
     8
.. contents::
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
     9
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    10
Introduction
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    11
------------
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    12
Diamonds is web application that can collect all build related information and categorize
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    13
builds. It can represent build information in different metrics. This document describes how
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    14
to configure diamonds in helium and minimum set of properties required.
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    15
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    16
Diamonds Server setup
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    17
---------------------
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    18
These are the minimum set of properties required in order to start the diamonds. All the properties are
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    19
defined automatically with already defined set of properties. The end user would not be required to change
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    20
any thing. As these are configured once for different vendors (symbian foundation, nokia, others.)
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    21
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    22
    <!-- Diamonds server details -->
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    23
    
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    24
    <property name="diamonds.host" value="diamonds.xxx.com"/>
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    25
    <property name="diamonds.port" value="9900"/>
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    26
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    27
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    28
Initialize diamonds
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    29
-------------------
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    30
`diamonds` target is the initialize target for diamonds logging. Call diamonds target in build target sequence
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    31
and this will log the already available data to diamonds and continue to log data onward as soon as they are available.
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    32
This is done already in helium build target sequence. So user can ignore this section. Earlier the diamonds
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    33
target needs to be called once the build area is initialized, but now this could be called even
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    34
before, as the output for diamonds files are generated in the cache location.
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    35
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    36
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    37
Disable diamonds reporting
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    38
--------------------------
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    39
Diamonds reporting can be skipped by defining the property ``diamonds.enabled`` to false.
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    40
e.g.::
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    41
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    42
    hlm -Ddiamonds.enabled=false 
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    43
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    44
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    45
Diamonds Configuration details
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    46
------------------------------
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    47
Diamonds configurations are extendable now. The default diamonds configuration is there under
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    48
${helium.dir}/config/diamonds_config_default.ant.xml. The configuration is based on ant properties
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    49
and references. So if the user wants to add process and report for new information, they can add
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    50
the details in their configurations. There are three types of information being provided using the 
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    51
configurations and are below.
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    52
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    53
Properties Required:
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    54
====================
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    55
Below are the properties requried for processing diamonds. But these are mapped to predifined properties
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    56
in helium and no action required for the user. The end user would not be required to change
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    57
any thing. As these are configured once for different vendors (symbian foundation, nokia, others.)
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    58
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    59
    <property name="diamonds.smtp.server" value="${email.smtp.server}" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    60
    <property name="diamonds.ldap.server" value="${email.ldap.server}" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    61
    <property name="diamonds.initializer.targetname" value="diamonds" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    62
    <property name="diamonds.tstamp.format" value="yyyy-MM-dd'T'HH:mm:ss" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    63
    <property name="diamonds.category" value="${build.family}" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    64
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    65
Stage Configurations:
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    66
=====================
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    67
Stages are to record information specific to stages. Stage information is used for both logging and
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    68
diamonds reporting. The build process needs to define stages clearly and map it with the configurations
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    69
as below.
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    70
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    71
.. code-block:: xml
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
    72
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    73
      <hlm:stage id="get-baseline" startTarget="check-free-space" endTarget="enable-abiv2" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    74
      <hlm:stage id="get-source" startTarget="do-prep-work-area" endTarget="create-bom" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    75
      <hlm:stage id="clean-and-prep" startTarget="ido-prep-clean-dfs" endTarget="ido-pre-compile" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    76
      <hlm:stage id="build" startTarget="ido-build-parallel-dfs" endTarget="compile-ctc" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    77
      <hlm:stage id="rombuild" startTarget="image-creation" endTarget="image-creation" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    78
      <hlm:stage id="create-ATS-drop" startTarget="ats-test" endTarget="ats-aste" />        
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    79
      <hlm:stage id="post-build" startTarget="image-creation" endTarget="archive" />
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    80
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    81
The stage configuration provides the information about the stage starting and ending target sequence.
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    82
There should be a corresponding stagerecord for each stage, which is to store the log information
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    83
for that specific stages, please refer to logging module for more information.
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    84
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    85
Both the stages / target reporting using messaging type to provide details to be sent to diamonds
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    86
reporting. See details in messaging sections for further details.
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    87
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    88
Currently the diamonds reporting just records the start / end time using the following configuration.
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    89
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    90
.. code-block:: xml
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    91
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    92
    <hlm:fmppMessage id="stage.time.message" sourceFile="${diamonds.template-dir}/diamonds_stage.xml.ftl">
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    93
        <data expandProperties="yes">
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    94
            ant: antProperties()
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    95
        </data>
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    96
    </hlm:fmppMessage>
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    97
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    98
The config takes a template to be used to convert, the template is converted using fmpp and all 
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
    99
the output files are processed and sent to diamonds. All the input to fmpp task could be used here.
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   100
The template diamonds_stage.xml.ftl just reports the start / end time. In addition to duration, if 
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   101
the user wants to send more information for the stages it could be done by overriding the 
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   102
configuration as below and controlling using the user defined template.
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   103
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   104
.. code-block:: xml
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   105
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   106
    <hlm:fmppMessage id="stage.time.message" ${diamonds.custom.template.dir}/diamonds_stage_custom.xml.ftl>
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   107
        <data expandProperties="yes">
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   108
            ant: antProperties()
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   109
        </data>
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   110
    </hlm:fmppMessage>
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
   111
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
   112
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   113
Reporting based on target execution:
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   114
====================================
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   115
If some data needs to be sent at the end of target execution, this can be defined with below configuration.
587
85df38eb4012 helium_9.0-a7879c935424
wbernard
parents:
diff changeset
   116
628
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   117
    <hlm:targetMessage id="diamonds.id" target="diamonds">
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   118
        <hlm:fmppMessage sourceFile="${helium.dir}/tools/common/templates/diamonds/tool.xml.ftl" >
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   119
            <data expandProperties="yes">
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   120
                ant: antProperties()
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   121
            </data>
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   122
        </hlm:fmppMessage>
7c4a911dc066 helium_11.0.0-e00f171ca185
wbernard
parents: 587
diff changeset
   123
    </hlm:targetMessage>