--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/buildframework/helium/sf/java/metadata/doc/metadata.rst Fri Aug 13 14:59:05 2010 +0300
@@ -0,0 +1,243 @@
+.. index::
+ module: Configuring Metadata
+
+====================
+Configuring Metadata
+====================
+
+.. contents::
+
+This document describes the purpose of metadata and how is being used in helium and
+how it can be used by the customer.
+
+Overview
+========
+
+Metadata is process to find the errors, warnings from the output log differnt section of the build
+and store it to the database, which could be used during each stage of the build to process efficiently
+and send details to diamonds, used for signaling, and generating summary file.
+
+
+Metadata Details
+================
+
+1. Metadatarecord : Which is used to store the errors, warnings information to the database
+
+ 1.1. It takes the metadatainput (the type of log parsing to be used), currently supported parsing are
+
+ a. sbsmetadatainput - sbs log processing (based on xml processing)
+
+ b. textmetadatainput - general text log processing (based on text processing)
+
+ c. policylogmetadatainput - policy log output processing (based on xml processing)
+
+ d. antlogmetadatainput - ant log output processing (based on text processing)
+
+ e. abldlogmetadatainput - abld log output processing (based on text processing)
+
+ Please see ant doc for more details on metadatarecord.
+
+ 1.2 It takes the fileset containing list of log files
+
+ 1.3 It takes the metadata filter, list of regular expression for searching strings.
+
+Metadata Filters
+================
+
+This document describes the usage of metadata filter to change the severity level during different stages of the build.
+
+Overview
+--------
+
+Metadata filters are set of regular expressions used to match the text of the build output and process the errors, categorize it,
+and used to generate the output for diamonds, summary file, email output. Predefined set of ids are defined for each stage of the
+build. For example for raptor compilation filter is defined as below,
+
+The default definition of filterset.sbs is
+
+.. code-block:: xml
+
+ <hlm:metadatafilterset id="filterset.sbs">
+ <metadatafilterset refid="filterset.common" />
+ </hlm:metadatafilterset>
+
+
+which is using the common definition which is,
+
+.. code-block:: xml
+
+ <hlm:metadatafilterset id="filterset.common">
+ <metadatafilterset filterfile="${helium.dir}/config/metadata_regex.csv" />
+ </hlm:metadatafilterset>
+
+
+The complete list of predefined ids for various stages of the build are defined in this file,
+
+helium/config/metadata_filter_config_default.xml
+
+Each ID can be overridden to provide additional regular expression to control the results of the build for different stages.
+
+Two ways to add the regular expressions
+---------------------------------------
+
+ - Adding more than one regular expression
+
+Define your own csv file and override it in your configuration as below (add this after importing helium.ant.xml file),
+
+.. code-block:: xml
+
+ <hlm:metadatafilterset id="filterset.sbs">
+ <metadatafilterset filterfile="${s60.config}/config/metadata_regex.csv" />
+ </hlm:metadatafilterset>
+
+ - Adding just one regular expression
+
+This can be done as below,
+
+.. code-block:: xml
+
+ <hlm:metadatafilterset id="filterset.sbs">
+ <metadatafilter severity="error" regex=".*Error\s*:\s+.*" description="sbs compilation error" />
+ <metadatafilterset filterfile="${helium.dir}/config/metadata_regex.csv" />
+ </hlm:metadatafilterset>
+
+Note
+----
+
+1. The order of metadatafilter / metadatafilterset is important, so the first one takes precedence than the second one.
+
+2. Order is also preserved in the csv file, the expressions which are defined first get precedence than the later one.
+
+3. All the regular expressions are JAVA patterns.
+
+
+Usage in Helium
+===============
+
+Different build stages were processed and identified the type of output and added the metadatarecord
+task for each build stage and captured the output in the database. And after storing it, using fmpp
+template the error information from database are processed to send to diamonds, raised signal accordingly.
+
+Usage
+=====
+
+Examples:
+ SBS comilation output in db:
+
+.. code-block:: xml
+
+ <hlm:metadatarecord database="${metadata.dbfile}">
+ <hlm:sbsmetadatainput cleanLogFile="${sbs.clean.log.file}">
+ <fileset casesensitive="false" file="${sbs.log.file}"/>
+ <metadatafilterset refid="filterset.sbs" />
+ </hlm:sbsmetadatainput>
+ </hlm:metadatarecord>
+
+This example is to process sbs output. The metadatainput is sbsmetadatainput to process the sbs log file, takes the sbs.log.file
+uses the regular expression defined by filterset.sbs
+
+.. code-block:: xml
+
+ <hlm:metadatarecord database="${metadata.dbfile}">
+ <hlm:abldmetadatainput>
+ <fileset casesensitive="false" file="${build.log.dir}/${build.id}${cmaker.log.label}.export.cmaker.log" />
+ <metadatafilterset refid="filterset.compile" />
+ </hlm:abldmetadatainput>
+ </hlm:metadatarecord>
+
+This example process the cmaker output as abld output log. It takes abldmetadatainput as metadatainput
+and the logfile as ${build.log.dir}/${build.id}${cmaker.log.label}.export.cmaker.log and the regular
+expression is used from the reference filterset.compile.
+
+Similarly any of the log output file can be easily processed in a similar way.
+
+Database schema
+===============
+
+The following diagram describes the current database schema (for SQL based queries).
+
+.. image:: metadata_schema.png
+ :align: center
+
+
+It is also possible to use the JPQL language which allows the usage of the Java ORM mapping class. This means that database will be represented
+by their Java model class, table fields by the class attributes. This diagrams describes the JPQL diagram:
+
+.. image:: metadata_jpql_schema.png
+ :align: center
+
+
+Example of queries:
+
+SQL::
+
+ select * from metadataentry as e, severity as s where e.severity_id = s.severity_id and s.severity = 'error'
+
+JPQL::
+
+ select e from MetadataEntry e JOIN e.severity s WHERE s.severity = 'error'
+
+
+Using the Metadata framework with FMPP
+======================================
+
+The Metadata framework gives an efficient opportunity to record huge amount or data in a fast and reliable way (timewise and memory consumption-wise).
+Thanks to the ORMFMPPLoader database loader it is really simple to access those data and render then in any other format: HTML for easy to read build summary,
+XML to communicated with other tools, text file...
+
+Loading a database
+------------------
+
+A database can be load and assigned to a template variable using the pp.loadData functionnality from the FMPP task. The 'com.nokia.helium.metadata.ORMFMPPLoader'
+accept one argument which is the path to the database.
+
+Example::
+
+ <#assign database = pp.loadData('com.nokia.helium.metadata.ORMFMPPLoader', "C:/path/to/database_db") >
+
+
+Then the database variable can be used to access the database the following different ways:
+
+ - jpasingle: Query with single result e.g: select count(s) from Severity s
+ - jpa: allow iteration on the JPA object results: select s from Severity s
+ - native:<type>: Native SQL format query, type is use to determine the object to use in the model
+
+Accessing data using a JPA single query
+---------------------------------------
+
+The 'jpasingle' is the best way to access results from single values like count of entities. The jpasingle queries must be written in JPQL,
+please check the valid database schema in the previous section (case matter!).
+
+Example of a template that will return the number of log files recorded in the database::
+
+ <#assign database = pp.loadData('com.nokia.helium.metadata.ORMFMPPLoader', "C:/path/to/database_db") >
+ Number of logfiles: ${database['jpasingle']['select l from LogFile l'][0]}
+
+Accessing data using a JPA query
+--------------------------------
+
+The JPA query allows you to perform query and directly use JPA entity object directly inside the template. The jpa queries must be written in JPQL,
+please check the valid database schema in the previous section (case matter!).
+
+In the following example the query loop through the available log files::
+
+ <#assign database = pp.loadData('com.nokia.helium.metadata.ORMFMPPLoader', "C:/path/to/database_db") >
+ <#list database['jpasingle']['select l from LogFile l'] as l>
+ ${l.id}: ${l.path}
+ </#list>
+
+
+Accessing data using a native query
+-----------------------------------
+
+The native query enables you to perform SQL queries through the JDBC interface of the database. If native is used then make sure you use
+the SQL schema.
+
+In the following example the query loop through the available log files path::
+
+ <#assign database = pp.loadData('com.nokia.helium.metadata.ORMFMPPLoader', "C:/path/to/database_db") >
+ <#list table_info['native:java.lang.String']['SELECT l.PATH FROM LOGFILE as l'] as l>
+ ${l}
+ </#list>
+
+
\ No newline at end of file