|
1 .. ============================================================================ |
|
2 Name : stage_ats.rst.ftl |
|
3 Part of : Helium |
|
4 |
|
5 Copyright (c) 2009 Nokia Corporation and/or its subsidiary(-ies). |
|
6 All rights reserved. |
|
7 This component and the accompanying materials are made available |
|
8 under the terms of the License "Eclipse Public License v1.0" |
|
9 which accompanies this distribution, and is available |
|
10 at the URL "http://www.eclipse.org/legal/epl-v10.html". |
|
11 |
|
12 Initial Contributors: |
|
13 Nokia Corporation - initial contribution. |
|
14 |
|
15 Contributors: |
|
16 |
|
17 Description: |
|
18 |
|
19 ============================================================================ |
|
20 |
|
21 .. index:: |
|
22 module: Testing |
|
23 |
|
24 ======= |
|
25 Testing |
|
26 ======= |
|
27 |
|
28 This is a good start for Helium users who want to setup test automation using ATS4. (**ATS3 users**, please `read here`_ .) |
|
29 |
|
30 .. _`read here`: stage_ats_old.html |
|
31 |
|
32 |
|
33 |
|
34 .. contents:: |
|
35 |
|
36 |
|
37 |
|
38 Helium Test Automation |
|
39 ====================== |
|
40 |
|
41 Helium can be used to auomate testing. For this purpose, test asset must be alligned with the standard guidelines for writing the tests. |
|
42 |
|
43 Helium supports several test frameworks including `STIF`_, `TEF`_, RTest, MTF, `SUT`_, QtTest `EUnit`_, TDriver and ASTE. (`Description of test frameworks`_) |
|
44 |
|
45 Most of the above mentioned test frameworks share common configuration to setup TA environemnet. However, there are a few exceptions, which are discussed below under the headings of every test framework. |
|
46 |
|
47 |
|
48 |
|
49 <#if !(ant?keys?seq_contains("sf"))> |
|
50 .. _`STIF`: http://s60wiki.nokia.com/S60Wiki/STIF |
|
51 .. _`TEF`: http://s60wiki.nokia.com/S60Wiki/TEF_%28TestExecute_Framework%29 |
|
52 .. _`EUnit`: http://s60wiki.nokia.com/S60Wiki/EUnit |
|
53 </#if> |
|
54 |
|
55 .. _`SUT`: http://developer.symbian.org/wiki/index.php/Symbian_Test_Tools#SymbianUnitTest |
|
56 .. _`Description of test frameworks`: http://developer.symbian.org/wiki/index.php/Symbian_Test_Tools |
|
57 |
|
58 |
|
59 |
|
60 Prerequisites |
|
61 ------------- |
|
62 |
|
63 * `Harmonized Test Interface (HTI)`_ needs to be compiled and into the image. |
|
64 * The reader is expected to already have a working ATS setup in which test cases can be executed. ATS server names, |
|
65 access rights and authentication etc. is supposed to be already taken care of. |
|
66 |
|
67 <#if !(ant?keys?seq_contains("sf"))> |
|
68 .. _`Harmonized Test Interface (HTI)`: http://s60wiki.nokia.com/S60Wiki/HTI |
|
69 <#else> |
|
70 .. _`Harmonized Test Interface (HTI)`: http://developer.symbian.org/wiki/index.php/HTI_Tool |
|
71 </#if> |
|
72 |
|
73 |
|
74 Setting up a Test Automation Environment with Helium |
|
75 ==================================================== |
|
76 |
|
77 Basic Test Automation step-by-step setup guide. |
|
78 |
|
79 |
|
80 Step 0: Structuring Test-source/test-asset |
|
81 ------------------------------------------ |
|
82 Test source usually lives in a component's ``tsrc`` directory. Test source components are created like any other Symbian SW component; |
|
83 there is a ``group`` directory with a ``bld.inf`` file for building, ``.mmp`` files for defining the targets, and so on. |
|
84 |
|
85 The test generation code expects ``.pkg`` file in the ``group`` directory of test component to be compiled, to get the paths of the files |
|
86 (can be data, configuration, initialization, etc) to be installed and where to install on the phone. |
|
87 |
|
88 **Please note** that not all components have ``tsrc`` and ``group`` directories. For instance, Qt, ASTE and TDriver do not have similar test asset structure as STIF, TEF and other test components. It is recommended to follow the test asset guidelines prior to setting up test automation with Helium. |
|
89 |
|
90 |
|
91 Step 1: Setting up system definition file |
|
92 ----------------------------------------- |
|
93 **System Definition Files supporting layers.sysdef.xml** |
|
94 **layers** in ``layers.sysdef.xml`` file and **configuration** in ``build.sysdef.xml`` file (`Structure of System Definition files version 1.4`_). |
|
95 |
|
96 <#if !(ant?keys?seq_contains("sf"))> |
|
97 .. _`new API test automation guidelines`: http://s60wiki.nokia.com/S60Wiki/Test_Asset_Guidelines |
|
98 .. _`Structure of System Definition files version 1.4`: http://delivery.nmp.nokia.com/trac/helium/wiki/SystemDefinitionFiles |
|
99 </#if> |
|
100 |
|
101 A template of layer in layers.sysdef.xml for system definition files |
|
102 |
|
103 .. code-block:: xml |
|
104 |
|
105 <layer name="name_test_layer"> |
|
106 <module name="module_name_one"> |
|
107 <unit unitID="unit_id1" name="unit_name1" bldFile="path_of_tsrc_folder_to_be_built" mrp="" /> |
|
108 </module> |
|
109 |
|
110 <module name="module_name_two"> |
|
111 <unit unitID="unit_id2" name="unit_name2" bldFile="path_of_tsrc_folder_to_be_built" mrp="" /> |
|
112 </module> |
|
113 </layer> |
|
114 |
|
115 |
|
116 * Layer name should end with **_test_layer** |
|
117 * Two standard names for ATS test layers are being used; ``unit_test_layer`` and ``api_test_layer``. Test components (the``unit`` tags) |
|
118 should be specified under these layers and grouped by ``module`` tag(s). |
|
119 * In the above, two modules means two drop files will be created; ``module`` may have one or more ``unit`` |
|
120 * By using property ``exclude.test.layers``, complete layers can be excluded and the components inside that layer will not be included in the AtsDrop. This property is a comma (,) separated list |
|
121 |
|
122 **System Definition Files version 3.0 (SysDefs3)** (new Helium v.10.79) |
|
123 The `structure of System Definition files version 3.0`_ is different than previous versions of system definition files. In SysDefs3, package definition files are used for components specification. Instead of layers naming conventions, filters are used to identify test components and test types, for example: "test, unit_test, !api_test" etc. |
|
124 |
|
125 <#if !(ant?keys?seq_contains("sf"))> |
|
126 .. _`structure of System Definition files version 3.0`: http://wikis.in.nokia.com/view/SWManageabilityTeamWiki/PkgdefUse |
|
127 <#else> |
|
128 .. _`structure of System Definition files version 3.0`: sysdef3.html |
|
129 </#if> |
|
130 |
|
131 An example template for defining test components in a package definition file. |
|
132 |
|
133 .. code-block:: xml |
|
134 |
|
135 <package id="dummytest" name="dummytest" levels="demo"> |
|
136 <collection id="test_nested" name="test_nested" level="demo"> |
|
137 |
|
138 <component id="tc1" name="tc1" purpose="development" filter="test, unit_test"> |
|
139 <unit bldFile="test_nested/tc1/group" mrp="" /> |
|
140 </component> |
|
141 |
|
142 <component id="tc2" name="tc2" purpose="development" filter="test"> |
|
143 <meta rel="testbuild"> |
|
144 <group name="drop_tc2_and_tc3" /> |
|
145 </meta> |
|
146 <unit bldFile="test_nested/tc2/group" mrp="" /> |
|
147 </component> |
|
148 |
|
149 <component id="tc3" name="tc3" purpose="development" filter="test"> |
|
150 <meta rel="testbuild"> |
|
151 <group name="drop_tc2_and_tc3" /> |
|
152 </meta> |
|
153 <unit bldFile="test_nested/tc3/group" mrp="" /> |
|
154 </component> |
|
155 |
|
156 </collection> |
|
157 </package> |
|
158 |
|
159 |
|
160 * Filter "test" must be specified for every test component. If it is not specified, the component will not be considered as a test component. |
|
161 * <meta>/<group> are now used to group test components, it work in the same way as <module>...<module> in sysdef v1.4 works. The components having same group name are grouped together. |
|
162 Separate drop files are created for different groups. In the above example, if only 'test' is selected, then two drop files will be created, one with tc1 and the other one with tc2 and tc3. |
|
163 |
|
164 |
|
165 Step 2: Configure ATS Ant properties |
|
166 --------------------------------------- |
|
167 The properties are categorized as |
|
168 |
|
169 * **Common** - Valid for all test frameworks (Table-1). |
|
170 * **API/Module** - Valid for only API/Module tests like STIF, STF, EUNit etc., and hence, are shared among many test frameworks (Table-2). |
|
171 |
|
172 |
|
173 Also, the edit status of the properties can be described as |
|
174 |
|
175 * [must] - must be set by user |
|
176 * [recommended] - should be set by user but not mandatory |
|
177 * [allowed] - should **not** be set by user however, it is possible. |
|
178 |
|
179 .. csv-table:: Table-1: ATS - Common Properties |
|
180 :header: "Property name", "Edit status", "Description" |
|
181 |
|
182 ":hlm-p:`ats.server`", "[must]", ":hlm-p:`ats.server[documentation]`" |
|
183 ":hlm-p:`ats.drop.location`", "[allowed]", ":hlm-p:`ats.drop.location[documentation]`" |
|
184 ":hlm-p:`ats.product.name`", "[must]", ":hlm-p:`ats.product.name[documentation]`" |
|
185 ":hlm-p:`ats.email.list`", "[allowed]", ":hlm-p:`ats.email.list[documentation]`" |
|
186 ":hlm-p:`ats.report.type`", "[allowed]", ":hlm-p:`ats.report.type[documentation]`" |
|
187 ":hlm-p:`ats.flashfiles.minlimit`", "[allowed]", ":hlm-p:`ats.flashfiles.minlimit[documentation]`" |
|
188 ":hlm-p:`ats.plan.name`", "[allowed]", ":hlm-p:`ats.plan.name[documentation]`" |
|
189 ":hlm-p:`ats.product.hwid`", "[allowed]", ":hlm-p:`ats.product.hwid[documentation]`" |
|
190 ":hlm-p:`ats.script.type`", "[allowed]", ":hlm-p:`ats.script.type[documentation]`" |
|
191 ":hlm-p:`ats.test.timeout`", "[allowed]", ":hlm-p:`ats.test.timeout[documentation]`" |
|
192 ":hlm-p:`ats.testrun.name`", "[allowed]", ":hlm-p:`ats.testrun.name[documentation]`" |
|
193 ":hlm-p:`ats.report.location`", "[allowed]", ":hlm-p:`ats.report.location[documentation]`" |
|
194 ":hlm-p:`ats.diamonds.signal`", "[allowed]", ":hlm-p:`ats.diamonds.signal[documentation]`" |
|
195 |
|
196 |
|
197 An example of setting up the common properties as in table-1: |
|
198 |
|
199 .. code-block:: xml |
|
200 |
|
201 <property name="ats.server" value="4fio00105" /> |
|
202 <property name="ats.drop.location" location="\\trwsimXX\ATS_TEST_SHARE\" /> |
|
203 <property name="ats.product.name" value="PRODUCT" /> |
|
204 <property name="ats.email.list" value="temp.user@company.com; another.email@company.com" /> |
|
205 <property name="ats.report.type" value="simplelogger" /> |
|
206 <property name="ats.flashfiles.minlimit" value="2" /> |
|
207 <property name="ats.plan.name" value="plan" /> |
|
208 <property name="ats.product.hwid" value="" /> |
|
209 <property name="ats.script.type" value="runx" /> |
|
210 <property name="ats.test.timeout" value="60" /> |
|
211 <property name="ats.testrun.name" value="${r'$'}{build.id}_${r'$'}{ats.product.name}_${r'$'}{major.version}.${r'$'}{minor.version}" /> |
|
212 <property name="ats.report.location" value="${r'$'}{publish.dir}/${r'$'}{publish.subdir}" /> |
|
213 <property name="ats.diamonds.signal" value="false" /> |
|
214 |
|
215 |
|
216 .. csv-table:: Table-2: ATS - API/Module properties |
|
217 :header: "Property name", "Edit status", "Description" |
|
218 |
|
219 ":hlm-p:`ats.target.platform`", "[allowed]", ":hlm-p:`ats.target.platform[documentation]`" |
|
220 ":hlm-p:`ats.obey.pkgfiles.rule`", "[allowed]", ":hlm-p:`ats.obey.pkgfiles.rule[documentation]`" |
|
221 ":hlm-p:`ats.specific.pkg`", "[allowed]", ":hlm-p:`ats.specific.pkg[documentation]`" |
|
222 ":hlm-p:`ats.test.filterset`", "[allowed]", ":hlm-p:`ats.test.filterset[documentation]`" |
|
223 |
|
224 |
|
225 An example of setting up API/Module testing properties as in table-2: |
|
226 |
|
227 .. code-block:: xml |
|
228 |
|
229 <property name="ats.target.platform" value="armv5 urel" /> |
|
230 <property name="ats.obey.pkgfiles.rule" value="false" /> |
|
231 <property name="ats.specific.pkg" value="sanity" /> |
|
232 <property name="ats.test.filterset" value="sysdef.filters.tests" /> |
|
233 |
|
234 <hlm:sysdefFilterSet id="sysdef.filters.tests"> |
|
235 <filter filter="test, " type="has" /> |
|
236 <config file="bldvariant.hrh" includes="" /> |
|
237 </hlm:sysdefFilterSet> |
|
238 |
|
239 |
|
240 Step 3: Configure or select ROM images (Optional) |
|
241 ------------------------------------------------- |
|
242 Since helium 10 images are picked up using :hlm-p:`ats.product.name` and Imaker iconfig.xml files. Property ``release.images.dir`` is searched for iconfig.xml files, the ones where the product name is part of :hlm-p:`ats.product.name` is used. |
|
243 |
|
244 You should only build the images for each product you want to include in ats. See `Imaker`_ docs for more info. Eg. |
|
245 |
|
246 .. _`Imaker`: ../helium-antlib/imaker.html |
|
247 |
|
248 .. code-block:: xml |
|
249 |
|
250 <hlm:imakerconfigurationset id="configname"> |
|
251 <imakerconfiguration> |
|
252 <hlm:product list="${r'$'}{product.list}" ui="true"/> |
|
253 <targetset> |
|
254 <include name="^core${r'$'}"/> |
|
255 <include name="^langpack_01${r'$'}"/> |
|
256 <include name="^custvariant_01_tools${r'$'}"/> |
|
257 <include name="^udaerase${r'$'}"/> |
|
258 </targetset> |
|
259 <variableset> |
|
260 <variable name="TYPE" value="rnd"/> |
|
261 </variableset> |
|
262 </imakerconfiguration> |
|
263 </hlm:imakerconfigurationset> |
|
264 |
|
265 |
|
266 For older products where there are no iconfig.xml, ``reference.ats.flash.images`` is used: |
|
267 |
|
268 .. code-block:: xml |
|
269 |
|
270 <fileset id="reference.ats.flash.images" dir="${r'$'}{release.images.dir}"> |
|
271 <include name="**/${r'$'}{build.id}*.core.fpsx"/> |
|
272 <include name="**/${r'$'}{build.id}*.rofs2.fpsx"/> |
|
273 <include name="**/${r'$'}{build.id}*.rofs3.fpsx"/> |
|
274 </fileset> |
|
275 |
|
276 |
|
277 .. Note:: |
|
278 |
|
279 Always declare *Properties* before and *filesets* after importing helium.ant.xml in order to overwrite the default values during the build. |
|
280 |
|
281 |
|
282 Step 4: Enabling or disabling test automation features |
|
283 ------------------------------------------------------ |
|
284 Helium supports a number of test automation features, which are discussed below. These features can be enabled or disabled by switching the values of the following properties to either *true* or *false*. |
|
285 |
|
286 |
|
287 .. csv-table:: Table-3: ATS - Switches/enablers |
|
288 :header: "Property name", "Edit status", "Description" |
|
289 |
|
290 ":hlm-p:`ats.enabled`", "[allowed]", ":hlm-p:`ats.enabled[documentation]`" |
|
291 ":hlm-p:`ats4.enabled`", "[allowed]", ":hlm-p:`ats4.enabled[documentation]`" |
|
292 ":hlm-p:`ats.stf.enabled`", "[allowed]", ":hlm-p:`ats.stf..enabled[documentation]`" |
|
293 ":hlm-p:`aste.enabled`", "[allowed]", ":hlm-p:`aste.enabled[documentation]`" |
|
294 ":hlm-p:`ats.ctc.enabled`", "[allowed]", ":hlm-p:`ats.ctc.enabled[documentation]`" |
|
295 ":hlm-p:`ats.trace.enabled`", "[allowed]", ":hlm-p:`ats.trace.enabled[documentation]`" |
|
296 ":hlm-p:`ats.emulator.enable`", "[allowed]", ":hlm-p:`ats.emulator.enable[documentation]`" |
|
297 ":hlm-p:`ats.singledrop.enabled`", "[allowed]", ":hlm-p:`ats.singledrop.enabled[documentation]`" |
|
298 ":hlm-p:`ats.multiset.enabled`", "[allowed]", ":hlm-p:`ats.multiset.enabled[documentation]`" |
|
299 ":hlm-p:`ats.delta.enabled`", "[allowed]", ":hlm-p:`ats.delta.enabled[documentation]`" |
|
300 ":hlm-p:`ats.java.importer.enabled`", "[allowed]", ":hlm-p:`ats.java.importer.enabled[documentation]`" |
|
301 ":hlm-p:`ats.tdriver.enabled`", "[allowed]", ":hlm-p:`ats.tdriver.enabled[documentation]`" |
|
302 |
|
303 |
|
304 For example: |
|
305 |
|
306 .. code-block:: xml |
|
307 |
|
308 <property name="ats.enabled" value="true" /> |
|
309 |
|
310 |
|
311 Supported Test Frameworks |
|
312 ========================= |
|
313 In this section only Helium specific properties, targets or other related issues are discussed to configure the following test frameworks. However, as mentioned earlier, there are test asset guidelines to setup test components for different test frameworks. |
|
314 |
|
315 ASTE |
|
316 ---- |
|
317 * ASTE tests can be enabled by setting :hlm-p:`aste.enabled` (see table-3). |
|
318 * `SW Test Asset`_ location and type of test should be known as a prerequisite. |
|
319 * To configure the ASTE tests, aste specific properties are required in addition to those in table-1 |
|
320 |
|
321 <#if !(ant?keys?seq_contains("sf"))> |
|
322 .. _`SW Test Asset`: http://s60wiki.nokia.com/S60Wiki/MC_SW_Test_Asset_documentation |
|
323 </#if> |
|
324 |
|
325 .. csv-table:: Table: ATS - ASTE properties |
|
326 :header: "Property name", "Edit status", "Description" |
|
327 |
|
328 ":hlm-p:`ats.aste.testasset.location`", "[must]", ":hlm-p:`ats.aste.testasset.location[documentation]`" |
|
329 ":hlm-p:`ats.aste.software.release`", "[must]", ":hlm-p:`ats.aste.software.release[documentation]`" |
|
330 ":hlm-p:`ats.aste.software.version`", "[must]", ":hlm-p:`ats.aste.software.version[documentation]`" |
|
331 ":hlm-p:`ats.aste.testasset.caseids`", "[recommended]", ":hlm-p:`ats.aste.testasset.caseids[documentation]`" |
|
332 ":hlm-p:`ats.aste.language`", "[recommended]", ":hlm-p:`ats.aste.language[documentation]`" |
|
333 ":hlm-p:`ats.aste.test.type`", "[recommended]", ":hlm-p:`ats.aste.test.type[documentation]`" |
|
334 ":hlm-p:`ats.aste.plan.name`", "[recommended]", ":hlm-p:`ats.aste.plan.name[documentation]`" |
|
335 ":hlm-p:`ats.aste.testrun.name`", "[recommended]", ":hlm-p:`ats.aste.testrun.name[documentation]`" |
|
336 ":hlm-p:`ats.aste.email.list`", "[recommended]", ":hlm-p:`ats.aste.email.list[documentation]`" |
|
337 |
|
338 |
|
339 An example of setting up ASTE properties: |
|
340 |
|
341 .. code-block:: xml |
|
342 |
|
343 <property name="ats.aste.testasset.location" value="" /> |
|
344 <property name="ats.aste.software.release" value="SPP 51.32" /> |
|
345 <property name="ats.aste.software.version" value="W810" /> |
|
346 <property name="ats.aste.testasset.caseids" value="100,101,102,104,106," /> |
|
347 <property name="ats.aste.language" value="English" /> |
|
348 <property name="ats.aste.test.type" value="smoke" /> |
|
349 <property name="ats.aste.plan.name" value="plan" /> |
|
350 <property name="ats.aste.testrun.name" value="${r'$'}{build.id}_${r'$'}{ats.product.name}_${r'$'}{major.version}.${r'$'}{minor.version}" /> |
|
351 <property name="ats.aste.email.list" value="temp.user@company.com; another.email@company.com" /> |
|
352 |
|
353 |
|
354 EUnit |
|
355 ----- |
|
356 * Test framework is selected if there is a library ``eunit.lib`` in the ``.mmp`` file of a test component |
|
357 * Following EUnit specific properties are required in addition to those in table-1 and table-2. |
|
358 |
|
359 .. csv-table:: Table: ATS - ASTE properties |
|
360 :header: "Property name", "Edit status", "Description" |
|
361 |
|
362 ":hlm-p:`eunit.test.package`", "[allowed]", ":hlm-p:`eunit.test.package[documentation]`" |
|
363 ":hlm-p:`eunitexerunner.flags`", "[allowed]", ":hlm-p:`eunitexerunner.flags[documentation]`" |
|
364 |
|
365 |
|
366 An example of setting up ASTE properties as in the above table: |
|
367 |
|
368 .. code-block:: xml |
|
369 |
|
370 <property name="eunit.test.package" value="" /> |
|
371 <property name="eunitexerunner.flags" value="/E S60AppEnv /R Off" /> |
|
372 |
|
373 |
|
374 MTF |
|
375 --- |
|
376 * The test framework is selected if there is a library ``testframeworkclient.lib`` in the ``.mmp`` file of a test component |
|
377 * There is no MTF specific configuration for Helium in addition to those in table-1 and table-2. |
|
378 |
|
379 |
|
380 QtTest |
|
381 ------ |
|
382 * The test framework is selected if there is a library ``QtTest.lib`` in the ``.mmp`` file of a test component |
|
383 * There are several ``.PKG`` files created after executing ``qmake``, but only one is selected based on a set target platform. See (:hlm-p:`ats.target.platform`) description in table-2. |
|
384 * Properties in table-1 and table-2 should also be configured. |
|
385 |
|
386 |
|
387 RTest |
|
388 ----- |
|
389 * The test framework is selected if there is a library ``euser.lib`` and a comment ``//RTEST``in the ``.mmp`` file of a test component. |
|
390 * There is no RTest specific configuration for Helium in addition to those in table-1 and table-2. |
|
391 |
|
392 |
|
393 STF |
|
394 --- |
|
395 * The test framework is selected if there is ``ModuleName=TEFTESTMODULE`` in ``.ini`` file of a component. |
|
396 * There is no STF specific configuration for Helium in addition to those in table-1 and table-2. |
|
397 * To enable STF for ATS set, :hlm-p:`ats.stf.enabled` (see table-3). By default this is not enabled. |
|
398 |
|
399 |
|
400 STIF |
|
401 ---- |
|
402 * The test framework is selected if there is a library ``stiftestinterface.lib`` in the ``.mmp`` file of a test component |
|
403 * There is no STIF specific configuration for Helium in addition to those in table-1 and table-2. |
|
404 |
|
405 |
|
406 SUT |
|
407 --- |
|
408 * The test framework is selected if there is a library ``symbianunittestfw.lib`` in the ``.mmp`` file of a test component |
|
409 * There is no SUT specific configuration for Helium in addition to those in table-1 and table-2. |
|
410 |
|
411 |
|
412 TEF |
|
413 --- |
|
414 * The test framework is selected if there is a library ``testframeworkclient.lib`` in the ``.mmp`` file of a test component |
|
415 * There is no TEF specific configuration for Helium in addition to those in table-1 and table-2. |
|
416 |
|
417 |
|
418 TDriver |
|
419 ------- |
|
420 * TDriver tests can be enabled by setting :hlm-p:`ats.tdriver.enabled` (see table-3). |
|
421 * TDriver Test Asset location should be known as a prerequisite. |
|
422 * Following TDriver specific properties are required in addition to those in table-1. |
|
423 |
|
424 |
|
425 .. csv-table:: Table: ATS Ant properties |
|
426 :header: "Property name", "Edit status", "Description" |
|
427 |
|
428 ":hlm-p:`ats.tdriver.enabled`", "[must]", ":hlm-p:`ats.tdriver.enabled[documentation]`" |
|
429 ":hlm-p:`tdriver.asset.location`", "[must]", ":hlm-p:`tdriver.asset.location[documentation]`" |
|
430 ":hlm-p:`tdriver.test.profiles`", "[must]", ":hlm-p:`tdriver.test.profiles[documentation]`" |
|
431 ":hlm-p:`tdriver.tdrunner.enabled`", "[must]", ":hlm-p:`tdriver.tdrunner.enabled[documentation]`" |
|
432 ":hlm-p:`tdriver.test.timeout`", "[must]", ":hlm-p:`tdriver.test.timeout[documentation]`" |
|
433 ":hlm-p:`tdriver.parameters`", "[must]", ":hlm-p:`tdriver.parameters[documentation]`" |
|
434 ":hlm-p:`tdriver.sis.files`", "[must]", ":hlm-p:`tdriver.sis.files[documentation]`" |
|
435 ":hlm-p:`tdriver.tdrunner.parameters`", "[must]", ":hlm-p:`tdriver.tdrunner.parameters[documentation]`" |
|
436 ":hlm-p:`tdriver.template.file`", "[allowed]", ":hlm-p:`tdriver.template.file[documentation]`" |
|
437 |
|
438 |
|
439 An example of setting up TDriver properties: |
|
440 |
|
441 .. code-block:: xml |
|
442 |
|
443 <property name="ats.tdriver.enabled" value="true" /> |
|
444 <property name="tdriver.asset.location" value="\\server\share\tdriver_testcases, x:\dir\tdriver_testcases," /> |
|
445 <property name="tdriver.test.profiles" value="bat, fute" /> |
|
446 <property name="tdriver.tdrunner.enabled" value="true" /> |
|
447 <property name="tdriver.test.timeout" value="1200" /> |
|
448 <property name="tdriver.parameters" value="x:\dir\tdriverparameters\tdriver_parameters.xml" /> |
|
449 <property name="tdriver.sis.files" value="x:\sisfiles\abc.sis#f:\data\abc.sis#C:\abc.sis, x:\sisfiles\xyz.sis#f:\data\xyz.sis#F:\xyz.sis" /> |
|
450 <property name="tdriver.tdrunner.parameters" value="--ordered" /> |
|
451 <property name="tdriver.template.file" value="x:\dir\templates\tdriver_template_2.xml" /> |
|
452 |
|
453 |
|
454 * To execute the tests, :hlm-t:`tdriver-test` target should be called. |
|
455 * To create custom templates for TDriver, read `Instructions for creating TDriver custom template`_. |
|
456 |
|
457 |
|
458 .. _`Instructions for creating TDriver custom template`: tdriver_template_instructions.html |
|
459 |
|
460 |
|
461 |
|
462 Test Automation Features |
|
463 ======================== |
|
464 |
|
465 CTC (Code Coverage) |
|
466 ------------------- |
|
467 |
|
468 * To enable ctc for ATS set, :hlm-p:`ats.ctc.enabled` (see table-3). |
|
469 * To compile components for CTC see `configure CTC for SBS`_ |
|
470 |
|
471 .. _`configure CTC for SBS`: ../helium-antlib/sbsctc.html |
|
472 |
|
473 * Once ATS tests have finished results for CTC will be shown in Diamonds. |
|
474 * The following are optional CTC properties |
|
475 |
|
476 .. csv-table:: Table: ATS Ant properties |
|
477 :header: "Property name", "Edit status", "Description" |
|
478 |
|
479 "``ctc.instrument.type``", "[allowed]", "Sets the instrument type" |
|
480 "``ctc.build.options``", "[allowed]", "Enables optional extra arguments for CTC, after importing a parent ant file." |
|
481 |
|
482 |
|
483 For example, |
|
484 |
|
485 .. code-block:: xml |
|
486 |
|
487 <property name="ctc.instrument.type" value="m" /> |
|
488 |
|
489 <import file="../../build.xml" /> |
|
490 |
|
491 <hlm:argSet id="ctc.build.options"> |
|
492 <arg line="-C OPT_ADD_COMPILE+-DCTC_NO_START_CTCMAN" /> |
|
493 </hlm:argSet> |
|
494 |
|
495 Or |
|
496 |
|
497 .. code-block:: xml |
|
498 |
|
499 <hlm:argSet id="ctc.build.options"> |
|
500 <arg line='-C "EXCLUDE+*\sf\os\xyz\*,*\tools\xyz\*"'/> |
|
501 </hlm:argSet> |
|
502 |
|
503 |
|
504 See `more information on code coverage`_. |
|
505 |
|
506 <#if !(ant?keys?seq_contains("sf"))> |
|
507 .. _`more information on code coverage`: http://s60wiki.nokia.com/S60Wiki/CTC |
|
508 <#else> |
|
509 .. _`more information on code coverage`: http://developer.symbian.org/wiki/index.php/Testing_Guidelines_for_Package_Releases#Code_coverage |
|
510 </#if> |
|
511 |
|
512 |
|
513 |
|
514 Customized test XML files |
|
515 ------------------------- |
|
516 |
|
517 The user can customize the generated test.xml with files: |
|
518 |
|
519 * **preset_custom.xml** goes before first set |
|
520 * **postset_custom.xml** goes after last set |
|
521 * **precase_custom.xml** goes before first case |
|
522 * **postcase_custom.xml** goes after last case |
|
523 * **prestep_custom.xml** goes before first step |
|
524 * **poststep_custom.xml** goes after last step |
|
525 * **prerun_custom.xml** goes before first run or execute step |
|
526 * **postrun_custom.xml** goes after last run or execute step |
|
527 * **prepostaction.xml** goes before first postaction |
|
528 * **postpostaction.xml** goes after last postaction |
|
529 |
|
530 The files must be in the directory 'custom' under the 'tsrc' or 'group' folder to be processed. |
|
531 |
|
532 The files need to be proper XML snippets that fit to their place. In case of an error an error is logged and a comment inserted to the generated XML file. |
|
533 |
|
534 A postaction section customization file (prepostaction.xml or postpostaction.xml) could look like this |
|
535 |
|
536 .. code-block:: xml |
|
537 |
|
538 <action> |
|
539 <type>RunProcessAction</type> |
|
540 <parameters> |
|
541 <parameter value="java" name="command"/> |
|
542 <parameter value="-version" name="parameters"/> |
|
543 </parameters> |
|
544 </action> |
|
545 |
|
546 The ``prestep_custom.xml`` can be used to flash and unstall something custom. |
|
547 |
|
548 .. code-block:: xml |
|
549 |
|
550 <task> |
|
551 <type>FileUploadTask</type> |
|
552 <parameters> |
|
553 <parameter name="src" value="Nokia_Energy_Profiler_1_1.sisx"/> |
|
554 <parameter name="dst" value="c:\data\Nokia_Energy_Profiler_1_1.sisx"/> |
|
555 <parameter name="reboot-retry-count" value="1"/> |
|
556 <parameter name="retry-count" value="1"/> |
|
557 </parameters> |
|
558 </task> |
|
559 |
|
560 |
|
561 |
|
562 And then the ``prerun_custom.xml`` can be used to execute a task. |
|
563 |
|
564 .. code-block:: xml |
|
565 |
|
566 <task> |
|
567 <type>NonTestExecuteTask</type> |
|
568 <parameters> |
|
569 <parameter value="true" name="local"/> |
|
570 <parameter value="daemon.exe" name="file"/> |
|
571 <parameter value="test.cfg" name="parameters"/> |
|
572 <parameter value="true" name="async"/> |
|
573 <parameter value="my_daemon" name="pid"/> |
|
574 </parameters> |
|
575 </task> |
|
576 |
|
577 **Note:** The users is expected to check the generated test.xml manually, as there is no validation. Invalid XML input files will be disregarded and a comment will be inserted to the generated XML file. |
|
578 |
|
579 |
|
580 Custom templates/drops |
|
581 ---------------------- |
|
582 * If you need to send a static drop to ATS then you can call the target :hlm-t:`ats-custom-drop`. |
|
583 * An example template is in helium/tools/testing/ats/templates/ats4_naviengine_template.xml |
|
584 * Then set a property to your own template, as follows. |
|
585 |
|
586 .. code-block:: xml |
|
587 |
|
588 <property name="ats.custom.template" value="path/to/mytemplate.xml" /> |
|
589 |
|
590 |
|
591 Overriding XML values |
|
592 --------------------- |
|
593 * Set the property ``ats.config.file`` to the location of the config file. |
|
594 |
|
595 Example configuration: |
|
596 |
|
597 .. code-block:: xml |
|
598 |
|
599 <ATSConfigData> |
|
600 <config name="common" abstract="true"> |
|
601 |
|
602 <!-- Properties to add/ modify --> |
|
603 <config type="properties"> |
|
604 <set name="HARNESS" value="STIF" /> |
|
605 <set name="2" value="3" /> |
|
606 </config> |
|
607 |
|
608 <!-- Settings to add/ modify --> |
|
609 <config type="settings"> |
|
610 <set name="HARNESS" value="STIF" /> |
|
611 <set name="2" value="3" /> |
|
612 </config> |
|
613 |
|
614 <!-- Attributes to modify --> |
|
615 <config type="attributes"> |
|
616 <set name="xyz" value="2" /> |
|
617 <set name="significant" value="true" /> |
|
618 </config> |
|
619 </config> |
|
620 </ATSConfigData> |
|
621 |
|
622 |
|
623 Delta testing |
|
624 ------------- |
|
625 |
|
626 |
|
627 Multiset support |
|
628 ---------------- |
|
629 * Enable the feature by setting property :hlm-p:`ats.multiset.enabled` to ``true``. |
|
630 * If enabled, a 'set' in test.xml, is used for each pkg file in a component, this allows tests to run in parallel on several devices. |
|
631 |
|
632 ROM Bootup Tests |
|
633 ---------------- |
|
634 * ROM images can be tested on ATS by executing target ":hlm-t:`ats-bootup-test`". This feature is useful to test whther the created ROM images boot-up a device or not . |
|
635 * To enable this feature, set a property ":hlm-p:`ats.bootuptest.enabled`" (see table-3) |
|
636 * In addition to enable the feature, properties in the table-1 are also required. |
|
637 |
|
638 |
|
639 Single/Multiple test drops creation |
|
640 ----------------------------------- |
|
641 * It is mentioned earlier in Step 1, that components can be grouped together. |
|
642 * During automation, separate TestDrops are created based on these groups. |
|
643 * This grouping can be neglected and a single test drop can be created by setting a property :hlm-p:`ats.singledrop.enabled` By default the value is 'false'. For example, |
|
644 |
|
645 |
|
646 .. code-block:: xml |
|
647 |
|
648 <property name="ats.singledrop.enabled" value="true" /> |
|
649 |
|
650 |
|
651 |
|
652 Skip uploading test drops |
|
653 ------------------------- |
|
654 * ``ats-test`` target can only create a drop file, and does not send the drop (or package) to ATS server. |
|
655 * To use the feature, set the following property to ``flase``. |
|
656 |
|
657 .. code-block:: xml |
|
658 |
|
659 <property name="ats.upload.enabled" value="false" /> |
|
660 |
|
661 |
|
662 <#if !(ant?keys?seq_contains("sf"))> |
|
663 |
|
664 Support for multiple products (ROM images) |
|
665 ------------------------------------------ |
|
666 |
|
667 See: `Instructions for setting up multiple roms and executing specific tests`_. |
|
668 |
|
669 .. _`Instructions for setting up multiple roms and executing specific tests`: http://helium.nmp.nokia.com/doc/ido/romandtest.html |
|
670 |
|
671 |
|
672 </#if> |
|
673 |
|
674 |
|
675 Testing with Winscw Emulator |
|
676 ---------------------------- |
|
677 * If enabled, ``ats-test`` target creates a zip of build area instead of images for use by emulator on ATS server. |
|
678 * Set a property as follows. |
|
679 |
|
680 .. code-block:: xml |
|
681 |
|
682 <property name="ats.emulator.enable" value="true" /> |
|
683 |
|
684 |
|
685 <#if !(ant?keys?seq_contains("sf"))> |
|
686 |
|
687 Tracing |
|
688 ------- |
|
689 * Currently there isn't a single standard method of doing tracing in Symbian Platform. |
|
690 * Application, middleware, driver and kernel developers have used different methods for instrumenting their code. Due to the different methods used, it is inherently difficult to get a coherent overview of the whole platform when debugging and testing sw. |
|
691 * Current implementation of Tracing in Helium is based on the instruction given `here`_. |
|
692 * Tracing can be enabled by setting :hlm-p:`ats.trace.enabled` to ``true`` (see table-3). |
|
693 |
|
694 .. _`here`: http://s60wiki.nokia.com/S60Wiki/Tracing |
|
695 |
|
696 </#if> |
|
697 |
|
698 |
|
699 Troubleshooting TA |
|
700 ================== |
|
701 |
|
702 .. csv-table:: Table: Trouble shooting test automation |
|
703 :header: "Type", "Description", "Possible solution" |
|
704 |
|
705 "Error", "'*<path>*' not found", "Either the PKG file does not exist or incorrect filename." |
|
706 "Error", "No test modules found in '*<path>*'", "This error is raised when there is no test components available. Check that your components are in the SystemDefinition files, and that the filters are set accordingly to the test asset documentation and that the components actually exists in the asset." |
|
707 "Error", "'*<path>*' - test source not found", "Path in the bld.inf file is either incorrect or the component does not exist." |
|
708 "Error", "Not enough flash files: # defined, # needed", "Check property :hlm-p:`ats.flashfiles.minlimit`. Selected ROM images files # is lesser than the required no. of files. This error can also be eliminated by reducing the value of the property." |
|
709 "Error", "'CPP failed: '<command>' in: '*<path>*'", "Check the path and/or the file. There can be broken path in the file or mising directives and macros." |
|
710 "Error", "*<path>* - test sets are empty", "missing/invalid deirectives and/or project macros. The .mmp file ca be missing." |
|
711 |
|
712 |