1 .. index:: |
|
2 module: Configuring Metadata |
|
3 |
|
4 ==================== |
|
5 Configuring Metadata |
|
6 ==================== |
|
7 |
|
8 .. contents:: |
|
9 |
|
10 This document describes the purpose of metadata and how is being used in helium and |
|
11 how it can be used by the customer. |
|
12 |
|
13 Overview |
|
14 ======== |
|
15 |
|
16 Metadata is process to find the errors, warnings from the output log differnt section of the build |
|
17 and store it to the database, which could be used during each stage of the build to process efficiently |
|
18 and send details to diamonds, used for signaling, and generating summary file. |
|
19 |
|
20 |
|
21 Metadata Details |
|
22 ================ |
|
23 |
|
24 1. Metadatarecord : Which is used to store the errors, warnings information to the database |
|
25 |
|
26 1.1. It takes the metadatainput (the type of log parsing to be used), currently supported parsing are |
|
27 |
|
28 a. sbsmetadatainput - sbs log processing (based on xml processing) |
|
29 |
|
30 b. textmetadatainput - general text log processing (based on text processing) |
|
31 |
|
32 c. policylogmetadatainput - policy log output processing (based on xml processing) |
|
33 |
|
34 d. antlogmetadatainput - ant log output processing (based on text processing) |
|
35 |
|
36 e. abldlogmetadatainput - abld log output processing (based on text processing) |
|
37 |
|
38 Please see ant doc for more details on metadatarecord. |
|
39 |
|
40 1.2 It takes the fileset containing list of log files |
|
41 |
|
42 1.3 It takes the metadata filter, list of regular expression for searching strings. |
|
43 |
|
44 Metadata Filters |
|
45 ================ |
|
46 |
|
47 This document describes the usage of metadata filter to change the severity level during different stages of the build. |
|
48 |
|
49 Overview |
|
50 -------- |
|
51 |
|
52 Metadata filters are set of regular expressions used to match the text of the build output and process the errors, categorize it, |
|
53 and used to generate the output for diamonds, summary file, email output. Predefined set of ids are defined for each stage of the |
|
54 build. For example for raptor compilation filter is defined as below, |
|
55 |
|
56 The default definition of filterset.sbs is |
|
57 |
|
58 .. code-block:: xml |
|
59 |
|
60 <hlm:metadatafilterset id="filterset.sbs"> |
|
61 <metadatafilterset refid="filterset.common" /> |
|
62 </hlm:metadatafilterset> |
|
63 |
|
64 |
|
65 which is using the common definition which is, |
|
66 |
|
67 .. code-block:: xml |
|
68 |
|
69 <hlm:metadatafilterset id="filterset.common"> |
|
70 <metadatafilterset filterfile="${helium.dir}/config/metadata_regex.csv" /> |
|
71 </hlm:metadatafilterset> |
|
72 |
|
73 |
|
74 The complete list of predefined ids for various stages of the build are defined in this file, |
|
75 |
|
76 helium/config/metadata_filter_config_default.xml |
|
77 |
|
78 Each ID can be overridden to provide additional regular expression to control the results of the build for different stages. |
|
79 |
|
80 Two ways to add the regular expressions |
|
81 --------------------------------------- |
|
82 |
|
83 - Adding more than one regular expression |
|
84 |
|
85 Define your own csv file and override it in your configuration as below (add this after importing helium.ant.xml file), |
|
86 |
|
87 .. code-block:: xml |
|
88 |
|
89 <hlm:metadatafilterset id="filterset.sbs"> |
|
90 <metadatafilterset filterfile="${s60.config}/config/metadata_regex.csv" /> |
|
91 </hlm:metadatafilterset> |
|
92 |
|
93 - Adding just one regular expression |
|
94 |
|
95 This can be done as below, |
|
96 |
|
97 .. code-block:: xml |
|
98 |
|
99 <hlm:metadatafilterset id="filterset.sbs"> |
|
100 <metadatafilter priority="error" regex=".*Error\s*:\s+.*" description="sbs compilation error" /> |
|
101 <metadatafilterset filterfile="${helium.dir}/config/metadata_regex.csv" /> |
|
102 </hlm:metadatafilterset> |
|
103 |
|
104 Note |
|
105 ---- |
|
106 |
|
107 1. The order of metadatafilter / metadatafilterset is important, so the first one takes precedence than the second one. |
|
108 |
|
109 2. Order is also preserved in the csv file, the expressions which are defined first get precedence than the later one. |
|
110 |
|
111 3. All the regular expressions are JAVA patterns. |
|
112 |
|
113 |
|
114 Usage in Helium |
|
115 =============== |
|
116 |
|
117 Different build stages were processed and identified the type of output and added the metadatarecord |
|
118 task for each build stage and captured the output in the database. And after storing it, using fmpp |
|
119 template the error information from database are processed to send to diamonds, raised signal accordingly. |
|
120 |
|
121 Usage |
|
122 ===== |
|
123 |
|
124 Examples: |
|
125 SBS comilation output in db: |
|
126 |
|
127 .. code-block:: xml |
|
128 |
|
129 <hlm:metadatarecord database="${metadata.dbfile}"> |
|
130 <hlm:sbsmetadatainput cleanLogFile="${sbs.clean.log.file}"> |
|
131 <fileset casesensitive="false" file="${sbs.log.file}"/> |
|
132 <metadatafilterset refid="filterset.sbs" /> |
|
133 </hlm:sbsmetadatainput> |
|
134 </hlm:metadatarecord> |
|
135 |
|
136 This example is to process sbs output. The metadatainput is sbsmetadatainput to process the sbs log file, takes the sbs.log.file |
|
137 uses the regular expression defined by filterset.sbs |
|
138 |
|
139 .. code-block:: xml |
|
140 |
|
141 <hlm:metadatarecord database="${metadata.dbfile}"> |
|
142 <hlm:abldmetadatainput> |
|
143 <fileset casesensitive="false" file="${build.log.dir}/${build.id}${cmaker.log.label}.export.cmaker.log" /> |
|
144 <metadatafilterset refid="filterset.compile" /> |
|
145 </hlm:abldmetadatainput> |
|
146 </hlm:metadatarecord> |
|
147 |
|
148 This example process the cmaker output as abld output log. It takes abldmetadatainput as metadatainput |
|
149 and the logfile as ${build.log.dir}/${build.id}${cmaker.log.label}.export.cmaker.log and the regular |
|
150 expression is used from the reference filterset.compile. |
|
151 |
|
152 Similarly any of the log output file can be easily processed in a similar way. |
|