|
1 .. ============================================================================ |
|
2 Name : stage_ats_old.rst.ftl |
|
3 Part of : Helium |
|
4 |
|
5 Copyright (c) 2009 Nokia Corporation and/or its subsidiary(-ies). |
|
6 All rights reserved. |
|
7 This component and the accompanying materials are made available |
|
8 under the terms of the License "Eclipse Public License v1.0" |
|
9 which accompanies this distribution, and is available |
|
10 at the URL "http://www.eclipse.org/legal/epl-v10.html". |
|
11 |
|
12 Initial Contributors: |
|
13 Nokia Corporation - initial contribution. |
|
14 |
|
15 Contributors: |
|
16 |
|
17 Description: |
|
18 |
|
19 ============================================================================ |
|
20 |
|
21 .. index:: |
|
22 module: TestingOld |
|
23 |
|
24 |
|
25 =========================== |
|
26 Testing (ATS3/Old Document) |
|
27 =========================== |
|
28 |
|
29 This is an old version of document for test automation for the **Helium users using ATS3**. |
|
30 |
|
31 **ATS4 users** should read `Helium Test Automation User Guide`_ (revised). |
|
32 |
|
33 .. _`Helium Test Automation User Guide`: stage_ats.html |
|
34 |
|
35 .. contents:: |
|
36 |
|
37 |
|
38 Stage: ATS - STIF, TEF, RTEST, MTF, SUT and EUnit (also Qt) |
|
39 =========================================================== |
|
40 |
|
41 ATS testing is the automatic testing of the phone code once it has been compiled and linked to create a ROM image. |
|
42 |
|
43 Explanation of the process for getting ATS (`STIF`_ and `EUnit`_) tests compiled and executed by Helium, through the use of the :hlm-t:`ats-test` target. |
|
44 |
|
45 http://developer.symbian.org/wiki/index.php/Symbian_Test_Tools |
|
46 |
|
47 <#if !(ant?keys?seq_contains("sf"))> |
|
48 .. _`STIF`: http://s60wiki.nokia.com/S60Wiki/STIF |
|
49 .. _`EUnit`: http://s60wiki.nokia.com/S60Wiki/EUnit |
|
50 </#if> |
|
51 |
|
52 .. image:: ats.dot.png |
|
53 |
|
54 Prerequisites |
|
55 ------------- |
|
56 |
|
57 * `Harmonized Test Interface (HTI)`_ needs to be compiled and into the image. |
|
58 * The reader is expected to already have a working ATS setup in which test cases can be executed. ATS server names, |
|
59 access rights and authentication etc. is supposed to be already taken care of. |
|
60 |
|
61 <#if !(ant?keys?seq_contains("sf"))> |
|
62 .. _`Harmonized Test Interface (HTI)`: http://s60wiki.nokia.com/S60Wiki/HTI |
|
63 <#else> |
|
64 .. _`Harmonized Test Interface (HTI)`: http://developer.symbian.org/wiki/index.php/HTI_Tool |
|
65 </#if> |
|
66 |
|
67 Test source components |
|
68 ---------------------- |
|
69 |
|
70 Test source usually lives in a component's ``tsrc`` directory. Test source components are created like any other Symbian SW component; |
|
71 there is a ``group`` directory with a ``bld.inf`` file for building, ``.mmp`` files for defining the targets, and so on. |
|
72 |
|
73 The test generation code expects ``.pkg`` file in the ``group`` directory of test component to be compiled, to get the paths of the files |
|
74 (can be data, configuration, initialization, etc files) to be installed and where to install on the phone. |
|
75 |
|
76 |
|
77 Three STEPS to setup ATS with Helium |
|
78 ------------------------------------ |
|
79 |
|
80 **Step 1: Configure System Definition Files** |
|
81 If the tsrc directory structure meets the criteria defined in the `new API test automation guidelines`_, then test components |
|
82 should be included in the System Definition files. |
|
83 |
|
84 **System Definition Files supporting layers.sysdef.xml** |
|
85 **layers** in ``layers.sysdef.xml`` file and **configuration** in ``build.sysdef.xml`` file (`Structure of System Definition files version 1.4`_). |
|
86 |
|
87 <#if !(ant?keys?seq_contains("sf"))> |
|
88 .. _`new API test automation guidelines`: http://s60wiki.nokia.com/S60Wiki/Test_Asset_Guidelines |
|
89 .. _`Structure of System Definition files version 1.4`: http://delivery.nmp.nokia.com/trac/helium/wiki/SystemDefinitionFiles |
|
90 </#if> |
|
91 |
|
92 A template of layer in layers.sysdef.xml for system definition files |
|
93 |
|
94 .. code-block:: xml |
|
95 |
|
96 <layer name="name_test_layer"> |
|
97 <module name="module_name_one"> |
|
98 <unit unitID="unit_id1" name="unit_name1" bldFile="path_of_tsrc_folder_to_be_built" mrp="" /> |
|
99 </module> |
|
100 |
|
101 <module name="module_name_two"> |
|
102 <unit unitID="unit_id2" name="unit_name2" bldFile="path_of_tsrc_folder_to_be_built" mrp="" /> |
|
103 </module> |
|
104 </layer> |
|
105 |
|
106 * Layer name should end with **_test_layer** |
|
107 * Two standard names for ATS test layers are being used; ``unit_test_layer`` and ``api_test_layer``. Test components (the``unit`` tags) |
|
108 should be specified under these layers and grouped by ``module`` tag(s). |
|
109 * In the above, two modules means two drop files will be created; ``module`` may have one or more ``unit`` |
|
110 * By using property ``exclude.test.layers``, complete layers can be excluded and the components inside that layer will not be included in the AtsDrop. This property is a comma (,) separated list |
|
111 |
|
112 **System Definition Files version 3.0 (SysDefs3)** (new Helium v.10.79) |
|
113 The `structure of System Definition files version 3.0`_ is different than previous versions of system definition files. In SysDefs3, package definition files are used for components specification. Instead of layers naming conventions, filters are used to identify test components and test types, for example: "test, unit_test, !api_test" etc. |
|
114 |
|
115 <#if !(ant?keys?seq_contains("sf"))> |
|
116 .. _`structure of System Definition files version 3.0`: http://wikis.in.nokia.com/view/SWManageabilityTeamWiki/PkgdefUse |
|
117 <#else> |
|
118 .. _`structure of System Definition files version 3.0`: sysdef3.html |
|
119 </#if> |
|
120 |
|
121 An example template for defining test components in a package definition file. |
|
122 |
|
123 .. code-block:: xml |
|
124 |
|
125 <package id="dummytest" name="dummytest" levels="demo"> |
|
126 <collection id="test_nested" name="test_nested" level="demo"> |
|
127 |
|
128 <component id="tc1" name="tc1" purpose="development" filter="test, unit_test"> |
|
129 <unit bldFile="test_nested/tc1/group" mrp="" /> |
|
130 </component> |
|
131 |
|
132 <component id="tc2" name="tc2" purpose="development" filter="test"> |
|
133 <meta rel="testbuild"> |
|
134 <group name="drop_tc2_and_tc3" /> |
|
135 </meta> |
|
136 <unit bldFile="test_nested/tc2/group" mrp="" /> |
|
137 </component> |
|
138 |
|
139 <component id="tc3" name="tc3" purpose="development" filter="test"> |
|
140 <meta rel="testbuild"> |
|
141 <group name="drop_tc2_and_tc3" /> |
|
142 </meta> |
|
143 <unit bldFile="test_nested/tc3/group" mrp="" /> |
|
144 </component> |
|
145 |
|
146 </collection> |
|
147 </package> |
|
148 |
|
149 * Filter "test" must be specified for every test component. If it is not specified, the component will not be considered as a test component. |
|
150 * <meta>/<group> are now used to group test components, it work in the same way as <module>...<module> in sysdef v1.4 works. The components having same group name are grouped together. |
|
151 Separate drop files are created for different groups. In the above example, if only 'test' is selected, then two drop files will be created, one with tc1 and the other one with tc2 and tc3. |
|
152 |
|
153 |
|
154 **Step 2: Configure ATS properties in build.xml** |
|
155 |
|
156 **(A)** Username and Password for the ATS should be set in the `.netrc file`_:: |
|
157 |
|
158 machine ats login ats_user_name password ats_password |
|
159 |
|
160 Add the above line in the ``.netrc`` file and replace ``ats_user_name`` with your real ATS username and ``ats_password`` with ATS password. |
|
161 |
|
162 **(B)** The following properties are ATS dependent with their edit status |
|
163 |
|
164 * [must] - must be set by user |
|
165 * [recommended] - should be set by user but not mandatory |
|
166 * [allowed] - should **not** be set by user however, it is possible. |
|
167 |
|
168 .. csv-table:: ATS Ant properties |
|
169 :header: "Property name", "Edit status", "Description" |
|
170 |
|
171 ":hlm-p:`ats.server`", "[must]", "For example: ``4fix012345`` or ``catstresrv001.company.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as 80. The host can be different depending on site and/or product." |
|
172 ":hlm-p:`ats.drop.location`", "[allowed]", "Server location (UNC path) to save the ATSDrop file, before sending to the ATS Server. For example: ``\\\\trwsem00\\some_folder\\``. In case, :hlm-p:`ats.script.type` is set to ``import``, ATS doesn't need to have access to :hlm-p:`ats.drop.location`, its value can be any local folder on build machine, for example ``c:/temp`` (no network share needed)." |
|
173 ":hlm-p:`ats.product.name`", "[must]", "Name of the product to be tested." |
|
174 ":hlm-p:`eunit.test.package`", "[allowed]", "The EUnit package name to be unzipped on the environment, for executing EUnit tests." |
|
175 ":hlm-p:`eunitexerunner.flags`", "[allowed]", "Flags for EUnit exerunner can be set by setting the value of this variable. The default flags are set to ``/E S60AppEnv /R Off``." |
|
176 ":hlm-p:`ats.email.list`", "[allowed]", "The property is needed if you want to get an email from ATS server after the tests are executed. There can be one to many semicolon-separated email addresses." |
|
177 ":hlm-p:`ats.report.type`", "[allowed]", "Value of the ats email report, for ATS4 set to 'no_attachment' so email size is reduced" |
|
178 ":hlm-p:`ats.flashfiles.minlimit`", "[allowed]", "Limit of minimum number of flash files to execute :hlm-t:`ats-test` target, otherwise ``ATSDrop.zip`` will not be generated. Default value is 2 files." |
|
179 ":hlm-p:`ats.plan.name`", "[allowed]", "Modify the plan name if you have understanding of ``test.xml`` file or leave it as it is. Default value is ``plan``." |
|
180 ":hlm-p:`ats.product.hwid`", "[allowed]", "Product HardWare ID (HWID) attached to ATS. By default the value of HWID is not set." |
|
181 ":hlm-p:`ats.script.type`", "[allowed]", "There are two types of ats script files to send drop to ATS server, ``runx`` and ``import``; only difference is that with ``import`` ATS doesn't have to have access rights to ``testdrop.zip`` file, as it is sent to the system over http and import doesn't need network shares. If that is not needed ``import`` should not be used. Default value is ``runx`` as ``import`` involves heavy processing on ATS server." |
|
182 ":hlm-p:`ats.target.platform`", "[allowed]", "Sets target platform for compiling test components. Default value is ``armv5 urel``." |
|
183 ":hlm-p:`ats.test.timeout`", "[allowed]", "To set test commands execution time limit on ATS server, in seconds. Default value is ``60``." |
|
184 ":hlm-p:`ats.testrun.name`", "[allowed]", "Modify the test-run name if you have understanding of ``test.xml`` file or leave it as it is. Default value is a string consist of build id, product name, major and minor versions." |
|
185 ":hlm-p:`ats.trace.enabled`", "[allowed]", "Should be ``true`` if tracing is needed during the tests running on ATS. Default value is ``false``, the values are case-sensitive. See http://s60wiki.nokia.com/S60Wiki/CATS/TraceTools." |
|
186 ":hlm-p:`ats.ctc.enabled`", "[allowed]", "Should be ``true`` if coverage measurement and dynamic analysis (CTC) tool support is to be used by ATS. Default value is ``false``. The values are case-sensitive." |
|
187 ":hlm-p:`ats.ctc.host`", "[allowed]", "ATS3 only CTC host, provided by CATS used to create coverage measurement reports. MON.sym files are copied to this location, for example ``10.0.0.1``. If not given, code coverage reports are not created" |
|
188 ":hlm-p:`ats.obey.pkgfiles.rule`", "[allowed]", "If the property is set to ``true``, then the only test components which will have PKG files, will be included into the ``test.xml`` as a test-set. Which means, even if there's a test component (executable) but there's no PKG file, it should not be considered as a test component and hence not included into the test.xml as a separate test. By default the property value is ``false``." |
|
189 ":hlm-p:`tsrc.data.dir`", "[allowed]", "The default value is ``data`` and refers to the 'data' directory under 'tsrc' directory." |
|
190 ":hlm-p:`tsrc.path.list`", "[allowed]", "Contains list of the tsrc directories. Gets the list from system definition layer files. Assuming that the test components are defined already in te ``layers.sysdef.xml`` files to get compiled. Not recommended, but the property value can be set if there are no System Definition file(s), and tsrc directories paths to set manually." |
|
191 ":hlm-p:`ats.report.location`", "[allowed]", "Sets ATS reports store location. Default location is ``${r'$'}{publish.dir}/${r'$'}{publish.subdir}``." |
|
192 ":hlm-p:`ats.multiset.enabled`", "[allowed]", "Should be ``true`` so a set is used for each pkg file in a component, this allows tests to run in parallel on several devices." |
|
193 ":hlm-p:`ats.diamonds.signal`", "[allowed]", "Should be ``true`` so at end of the build diamonds is checked for test results and Helium fails if tests failed." |
|
194 ":hlm-p:`ats.delta.enabled`", "[allowed]", "Should be ``true`` so only ADOs changed during :hlm-t:`do-prep-work-area` are tested by ATS." |
|
195 ":hlm-p:`ats4.enabled`", "[allowed]", "Should be ``true`` if ATS4 is to be used." |
|
196 ":hlm-p:`ats.emulator.enable`", "[allowed]", "Should be ``true`` if ``WINSCW`` emulator is to be used." |
|
197 ":hlm-p:`ats.specific.pkg`", "[allowed]", "Text in name of PKG files to use eg. 'sanity' would only use xxxsanity.pkg files from components." |
|
198 ":hlm-p:`ats.singledrop.enabled`", "[allowed]", "If present and set to 'true', it will create one drop file, if set to any other value or not present it will create multiple drop files (defined by the sysdef file). This is to save traffic to the server." |
|
199 ":hlm-p:`ats.java.importer.enabled`", "[allowed]", "If set to 'true', for older uploader is used for ats3 which shows improved error message." |
|
200 ":hlm-p:`ats.test.filterset`", "[allowed]", "(new Helium v.10.79)Contains a name of test filterset (see example below). A filterset is used to select/unselect test components. The filter(s) is/are effective when the same filters are defined in the package definition file for component(s)." |
|
201 |
|
202 An example of setting up properties: |
|
203 |
|
204 .. code-block:: xml |
|
205 |
|
206 <property name="ats.server" value="4fio00105" /> |
|
207 <property name="ats.drop.location" location="\\trwsimXX\ATS_TEST_SHARE\" /> |
|
208 <property name="ats.email.list" value="temp.user@company.com; another.email@company.com" /> |
|
209 <property name="ats.flashfiles.minlimit" value="2" /> |
|
210 <property name="ats.product.name" value="PRODUCT" /> |
|
211 <property name="ats.plan.name" value="plan" /> |
|
212 <property name="ats.product.hwid" value="" /> |
|
213 <property name="ats.script.type" value="runx" /> |
|
214 <property name="ats.target.platform" value="armv5 urel" /> |
|
215 <property name="ats.test.timeout" value="60" /> |
|
216 <property name="ats.testrun.name" value="${r'$'}{build.id}_${r'$'}{ats.product.name}_${r'$'}{major.version}.${r'$'}{minor.version}" /> |
|
217 <property name="ats.trace.enabled" value="false" /> |
|
218 <property name="ats.ctc.enabled" value="false" /> |
|
219 <property name="ats.obey.pkgfiles.rule" value="false" /> |
|
220 <property name="ats.report.location" value="${r'$'}{publish.dir}/${r'$'}{publish.subdir}" /> |
|
221 <property name="eunit.test.package" value="" /> |
|
222 <property name="eunitexerunner.flags" value="/E S60AppEnv /R Off" /> |
|
223 <property name="ats.test.filterset" value="sysdef.filters.tests" /> |
|
224 |
|
225 <hlm:sysdefFilterSet id="sysdef.filters.tests"> |
|
226 <filter filter="test, " type="has" /> |
|
227 <config file="bldvariant.hrh" includes="" /> |
|
228 </hlm:sysdefFilterSet> |
|
229 |
|
230 .. Note:: |
|
231 |
|
232 Always declare *Properties* before and *filesets* after importing helium.ant.xml. |
|
233 |
|
234 **STEP 3: Call target ats-test** |
|
235 |
|
236 To execute the target, a property should be set(``<property name="ats.enabled" value="true" />``). |
|
237 |
|
238 Then call :hlm-t:`ats-test`, which will create the ATSDrop.zip (test package). |
|
239 |
|
240 If property *ats.email.list* is set, an email (test report) will be sent when the tests are ready on ATS. |
|
241 |
|
242 CTC |
|
243 --- |
|
244 |
|
245 * To enable ctc for ATS set, :hlm-p:`ats.ctc.enabled`. |
|
246 |
|
247 For ATS3 only, set the ftp hostname for the ATS server: |
|
248 |
|
249 .. code-block:: xml |
|
250 |
|
251 <property name="ats.ctc.host" value="1.2.3"/> |
|
252 |
|
253 |
|
254 * To compile components for CTC see `configure CTC for SBS`_ |
|
255 |
|
256 .. _`configure CTC for SBS`: ../helium-antlib/sbsctc.html |
|
257 |
|
258 |
|
259 * Once ATS tests have finished results for CTC will be shown in Diamonds. |
|
260 * The following are optional CTC properties |
|
261 |
|
262 .. csv-table:: Table: ATS Ant properties |
|
263 :header: "Property name", "Edit status", "Description" |
|
264 |
|
265 "``ctc.instrument.type``", "[allowed]", "Sets the instrument type" |
|
266 "``ctc.build.options``", "[allowed]", "Enables optional extra arguments for CTC, after importing a parent ant file." |
|
267 |
|
268 |
|
269 For example, |
|
270 |
|
271 .. code-block:: xml |
|
272 |
|
273 <property name="ctc.instrument.type" value="m" /> |
|
274 |
|
275 <import file="../../build.xml" /> |
|
276 |
|
277 <hlm:argSet id="ctc.build.options"> |
|
278 <arg line="-C OPT_ADD_COMPILE+-DCTC_NO_START_CTCMAN" /> |
|
279 </hlm:argSet> |
|
280 |
|
281 Or |
|
282 |
|
283 .. code-block:: xml |
|
284 |
|
285 <hlm:argSet id="ctc.build.options"> |
|
286 <arg line='-C "EXCLUDE+*\sf\os\xyz\*,*\tools\xyz\*"'/> |
|
287 </hlm:argSet> |
|
288 |
|
289 |
|
290 See `more information on code coverage`_ |
|
291 |
|
292 <#if !(ant?keys?seq_contains("sf"))> |
|
293 .. _`more information on code coverage`: http://s60wiki.nokia.com/S60Wiki/CTC |
|
294 <#else> |
|
295 .. _`more information on code coverage`: http://developer.symbian.org/wiki/index.php/Testing_Guidelines_for_Package_Releases#Code_coverage |
|
296 </#if> |
|
297 |
|
298 |
|
299 |
|
300 |
|
301 |
|
302 |
|
303 |
|
304 Qt Tests |
|
305 -------- |
|
306 |
|
307 QtTest.lib is supported and the default harness is set to EUnit. If ``QtTest.lib`` is there in ``.mmp`` file, Helium sets the Harness to Eunit and ATS supported Qt steps are added to test.xml file |
|
308 |
|
309 In ``layers.sysdef.xml`` file, the layer name should end with "_test_layer" e.g. "qt_unit_test_layer". |
|
310 |
|
311 There are several ``.PKG`` files created after executing ``qmake``, but only one is selected based on which target platform is set. Please read the property (:hlm-p:`ats.target.platform`) description above. |
|
312 |
|
313 .. _`Skip-Sending-AtsDrop-label`: |
|
314 |
|
315 Skip Sending AtsDrop to ATS |
|
316 --------------------------- |
|
317 |
|
318 By setting property of :hlm-p:`ats.upload.enabled` to ``false``, ``ats-test`` target only creates a drop file, and does not send the drop (or package) to ATS server. |
|
319 |
|
320 Choosing images to send to ATS |
|
321 ------------------------------ |
|
322 |
|
323 Since helium 10 images are picked up using :hlm-p:`ats.product.name` and Imaker iconfig.xml files. ``release.images.dir`` is searched for iconfig.xml files, the ones where the product name is part of :hlm-p:`ats.product.name` is used. |
|
324 |
|
325 You should only build the images for each product you want to include in ats. Eg. |
|
326 |
|
327 .. code-block:: xml |
|
328 |
|
329 <hlm:imakerconfigurationset id="configname"> |
|
330 <imakerconfiguration> |
|
331 <hlm:product list="${r'$'}{product.list}" ui="true"/> |
|
332 <targetset> |
|
333 <include name="^core${r'$'}"/> |
|
334 <include name="^langpack_01${r'$'}"/> |
|
335 <include name="^custvariant_01_tools${r'$'}"/> |
|
336 <include name="^udaerase${r'$'}"/> |
|
337 </targetset> |
|
338 <variableset> |
|
339 <variable name="TYPE" value="rnd"/> |
|
340 </variableset> |
|
341 </imakerconfiguration> |
|
342 </hlm:imakerconfigurationset> |
|
343 |
|
344 For older products where there are no iconfig.xml, ``reference.ats.flash.images`` is used: |
|
345 |
|
346 .. code-block:: xml |
|
347 |
|
348 <fileset id="reference.ats.flash.images" dir="${r'$'}{release.images.dir}"> |
|
349 <include name="**/${r'$'}{build.id}*.core.fpsx"/> |
|
350 <include name="**/${r'$'}{build.id}*.rofs2.fpsx"/> |
|
351 <include name="**/${r'$'}{build.id}*.rofs3.fpsx"/> |
|
352 </fileset> |
|
353 |
|
354 Customizing the test.xml in ATS |
|
355 ------------------------------- |
|
356 |
|
357 The user can customize the generated test.xml with files: |
|
358 |
|
359 * **preset_custom.xml** goes before first set |
|
360 * **postset_custom.xml** goes after last set |
|
361 * **precase_custom.xml** goes before first case |
|
362 * **postcase_custom.xml** goes after last case |
|
363 * **prestep_custom.xml** goes before first step |
|
364 * **poststep_custom.xml** goes after last step |
|
365 * **prerun_custom.xml** goes before first run or execute step |
|
366 * **postrun_custom.xml** goes after last run or execute step |
|
367 * **prepostaction.xml** goes before first postaction |
|
368 * **postpostaction.xml** goes after last postaction |
|
369 |
|
370 The files must be in the directory 'custom' under the 'tsrc' or 'group' folder to be processed. |
|
371 |
|
372 The files need to be proper XML snippets that fit to their place. In case of an error an error is logged and a comment inserted to the generated XML file. |
|
373 |
|
374 A postaction section customization file (prepostaction.xml or postpostaction.xml) could look like this |
|
375 |
|
376 .. code-block:: xml |
|
377 |
|
378 <postAction> |
|
379 <type>Pre PostAction from custom file</type> |
|
380 <params> |
|
381 <param name="foo2" value="bar2" /> |
|
382 </params> |
|
383 </postAction> |
|
384 |
|
385 |
|
386 |
|
387 The ``prestep_custom.xml`` can be used to flash and unstall something custom. |
|
388 |
|
389 .. code-block:: xml |
|
390 |
|
391 <step name="Install measurement tools" harness="STIF" significant="false"> |
|
392 <!-- Copy SIS-packages to DUT --> |
|
393 <command>install</command> |
|
394 <params> |
|
395 <param src="Nokia_Energy_Profiler_1_1.sisx"/> |
|
396 <param dst="c:\data\Nokia_Energy_Profiler_1_1.sisx"/> |
|
397 </params> |
|
398 ... |
|
399 </step> |
|
400 |
|
401 |
|
402 And then the ``prerun_custom.xml`` can be used to start measuring. |
|
403 |
|
404 .. code-block:: xml |
|
405 |
|
406 <step name="Start measurement" harness="STIF" significant="false"> |
|
407 <!-- Start measurement --> |
|
408 <command>execute</command> |
|
409 <params> |
|
410 <param file="neplauncher.exe"/> |
|
411 <param parameters="start c:\data\nep.csv"/> |
|
412 <param timeout="30"/> |
|
413 </params> |
|
414 </step> |
|
415 |
|
416 |
|
417 |
|
418 **Note:** The users is expected to check the generated test.xml manually, as there is no validation. Invalid XML input files will be disregarded and a comment will be inserted to the generated XML file. |
|
419 |
|
420 Overriding Test xml values |
|
421 -------------------------- |
|
422 |
|
423 Set the property ``ats.config.file`` to the location of the config file. |
|
424 |
|
425 Example configuration: |
|
426 |
|
427 .. code-block:: xml |
|
428 |
|
429 <ATSConfigData> |
|
430 <config name="common" abstract="true"> |
|
431 |
|
432 <!-- Properties to add/ modify --> |
|
433 <config type="properties"> |
|
434 <set name="HARNESS" value="STIF" /> |
|
435 <set name="2" value="3" /> |
|
436 </config> |
|
437 |
|
438 <!-- Settings to add/ modify --> |
|
439 <config type="settings"> |
|
440 <set name="HARNESS" value="STIF" /> |
|
441 <set name="2" value="3" /> |
|
442 </config> |
|
443 |
|
444 <!-- Attributes to modify --> |
|
445 <config type="attributes"> |
|
446 <set name="xyz" value="2" /> |
|
447 <set name="significant" value="true" /> |
|
448 </config> |
|
449 </config> |
|
450 </ATSConfigData> |
|
451 |
|
452 |
|
453 .. index:: |
|
454 single: ATS - ASTE |
|
455 |
|
456 Stage: ATS - ASTE |
|
457 ================= |
|
458 |
|
459 Explanation of the process for getting ATS `ASTE`_ tests compiled and executed by Helium, through the use of the :hlm-t:`ats-aste` target. |
|
460 |
|
461 <#if !(ant?keys?seq_contains("sf"))> |
|
462 .. _`ASTE`: http://s60wiki.nokia.com/S60Wiki/ASTE |
|
463 </#if> |
|
464 |
|
465 Prerequisites |
|
466 ------------- |
|
467 |
|
468 * `Harmonized Test Interface (HTI)`_ needs to be compiled and into the image. |
|
469 * The reader is expected to already have a working ATS setup in which test cases can be executed. ATS server names, access rights and authentication etc. is supposed to be already taken care of. |
|
470 * `SW Test Asset`_ location and type of test should be known. |
|
471 |
|
472 <#if !(ant?keys?seq_contains("sf"))> |
|
473 .. _`Harmonized Test Interface (HTI)`: http://s60wiki.nokia.com/S60Wiki/HTI |
|
474 .. _`SW Test Asset`: http://s60wiki.nokia.com/S60Wiki/MC_SW_Test_Asset_documentation |
|
475 </#if> |
|
476 |
|
477 Test source components |
|
478 ---------------------- |
|
479 |
|
480 Unlike STIF, EUnit etc tests, test source components (or ``tsrc`` structure) is not needed for `ASTE`_ tests. |
|
481 |
|
482 Two STEPS to setup ASTE with Helium |
|
483 ----------------------------------- |
|
484 |
|
485 **STEP 1: Configure ASTE properties in build.xml** |
|
486 |
|
487 **(A)** Username and Password for the ATS should be set in the `.netrc file`_ |
|
488 |
|
489 .. code-block:: text |
|
490 |
|
491 machine ats login ats_user_name password ats_password |
|
492 |
|
493 Add the above line in the .netrc file and replace *ats_user_name* with your real ats username and "ats_password" with ats password. |
|
494 |
|
495 .. _`.netrc file`: configuring.html?highlight=netrc#passwords |
|
496 |
|
497 |
|
498 **(B)** The following properties are ASTE dependent with their edit status |
|
499 |
|
500 * [must] - must be set by user |
|
501 * [recommended] - should be set by user but not mandatory |
|
502 * [allowed] - should **not** be set by user however, it is possible. |
|
503 |
|
504 .. csv-table:: ATS Ant properties |
|
505 :header: "Property name", "Edit status", "Description" |
|
506 |
|
507 ":hlm-p:`ats.server`", "[must]", "For example: ``4fio00105`` or ``catstresrv001.company.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as ``80``. The host can be different depending on site and/or product." |
|
508 ":hlm-p:`ats.drop.location`", "[must]", "Server location (UNC path) to save the ATSDrop file, before sending to the ATS. For example: ``\\\\trwsem00\\some_folder\\``. In case, ``ats.script.type`` is set to ``import``, ATS doesn't need to have access to :hlm-p:`ats.drop.location`, its value can be any local folder on build machine, for example ``c:/temp`` (no network share needed)." |
|
509 ":hlm-p:`ats.product.name`", "[must]", "Name of the product to be tested." |
|
510 ":hlm-p:`ats.aste.testasset.location`", "[must]", "Location of SW Test Assets, if the TestAsset is not packaged then it is first compressed to a ``.zip`` file. It should be a UNC path." |
|
511 ":hlm-p:`ats.aste.software.release`", "[must]", "Flash images releases, for example 'SPP 51.32'." |
|
512 ":hlm-p:`ats.aste.software.version`", "[must]", "Version of the software to be tested. For example: 'W810'" |
|
513 ":hlm-p:`ats.aste.email.list`", "[recommended]", "The property is needed if you want to get an email from ATS server after the tests are executed. There can be one to many semicolon(s) ";" separated email addresses." |
|
514 ":hlm-p:`ats.flashfiles.minlimit`", "[recommended]", "Limit of minimum number of flash files to execute ats-test target, otherwise ATSDrop.zip will not be generated. Default value is "2" files." |
|
515 ":hlm-p:`ats.aste.plan.name`", "[recommended]", "Modify the plan name if you have understanding of test.xml file or leave it as it is. Default value is "plan"." |
|
516 ":hlm-p:`ats.product.hwid`", "[recommended]", "Product HardWare ID (HWID) attached to ATS. By default the value of HWID is not set." |
|
517 ":hlm-p:`ats.test.timeout`", "[recommended]", "To set test commands execution time limit on ATS server, in seconds. Default value is '60'." |
|
518 ":hlm-p:`ats.aste.testrun.name`", "[recommended]", "Modify the test-run name if you have understanding of ``test.xml`` file or leave it as it is. Default value is a string consists of build id, product name, major and minor versions." |
|
519 ":hlm-p:`ats.aste.test.type`", "[recommended]", "Type of test to run. Default is 'smoke'." |
|
520 ":hlm-p:`ats.aste.testasset.caseids`", "[recommended]", "These are the cases that which tests should be run from the TestAsset. For example, value can be set as ``100,101,102,103,105,106,``. A comma is needed to separate case IDs" |
|
521 ":hlm-p:`ats.aste.language`", "[recommended]", "Variant Language to be tested. Default is 'English'" |
|
522 |
|
523 |
|
524 An example of setting up properties: |
|
525 |
|
526 .. code-block:: xml |
|
527 |
|
528 <property name="ats.server" value="4fio00105" /> |
|
529 <property name="ats.drop.location" value="\\trwsimXX\ATS_TEST_SHARE\" /> |
|
530 <property name="ats.aste.email.list" value="temp.user@company.com; another.email@company.com" /> |
|
531 <property name="ats.flashfiles.minlimit" value="2" /> |
|
532 <property name="ats.product.name" value="PRODUCT" /> |
|
533 <property name="ats.aste.plan.name" value="plan" /> |
|
534 <property name="ats.product.hwid" value="" /> |
|
535 <property name="ats.test.timeout" value="60" /> |
|
536 <property name="ats.aste.testrun.name" value="${r'$'}{build.id}_${r'$'}{ats.product.name}_${r'$'}{major.version}.${r'$'}{minor.version}" /> |
|
537 <property name="ats.aste.testasset.location" value="" /> |
|
538 <property name="ats.aste.software.release" value="SPP 51.32" /> |
|
539 <property name="ats.aste.test.type" value="smoke" /> |
|
540 <property name="ats.aste.testasset.caseids" value="100,101,102,104,106," /> |
|
541 <property name="ats.aste.software.version" value="W810" /> |
|
542 <property name="ats.aste.language" value="English" /> |
|
543 |
|
544 |
|
545 *PLEASE NOTE:* Always declare *Properties* before and *filesets* after importing helium.ant.xml. |
|
546 |
|
547 **STEP 2: Call target ats-aste** |
|
548 |
|
549 To execute the target, a property should be set(``<property name="enabled.aste" value="true" />``). |
|
550 |
|
551 Then call :hlm-t:`ats-aste`, which will create the ATSDrop.zip (test package). |
|
552 |
|
553 If property ``ats.aste.email.list`` is set, an email (test report) will be sent when the tests are ready on ATS/ASTE. |
|
554 |
|
555 |
|
556 Skip Sending AtsDrop to ATS |
|
557 --------------------------- |
|
558 |
|
559 click :ref:`Skip-Sending-AtsDrop-label`: |
|
560 |
|
561 |
|
562 |
|
563 |