--- a/buildframework/helium/doc/src/manual/stage_ats.rst.inc.ftl Wed Jun 16 16:51:40 2010 +0300
+++ b/buildframework/helium/doc/src/manual/stage_ats.rst.inc.ftl Fri Aug 13 14:59:05 2010 +0300
@@ -63,20 +63,23 @@
The test generation code expects ``.pkg`` file in the ``group`` directory of test component to be compiled, to get the paths of the files
(can be data, configuration, initialization, etc files) to be installed and where to install on the phone.
+
Three STEPS to setup ATS with Helium
--------------------------------------
**Step 1: Configure System Definition Files**
If the tsrc directory structure meets the criteria defined in the `new API test automation guidelines`_, then test components
- should be included in the System Definition files; **layers** in ``layers.sysdef.xml`` file and **configuration** in ``build.sysdef.xml``
- file (`Structure of System Definition files`_).
+ should be included in the System Definition files.
+
+**System Definition Files supporting layers.sysdef.xml**
+ **layers** in ``layers.sysdef.xml`` file and **configuration** in ``build.sysdef.xml`` file (`Structure of System Definition files version 1.4`_).
<#if !(ant?keys?seq_contains("sf"))>
.. _`new API test automation guidelines`: http://s60wiki.nokia.com/S60Wiki/Test_Asset_Guidelines
-.. _`Structure of System Definition files`: http://delivery.nmp.nokia.com/trac/helium/wiki/SystemDefinitionFiles
+.. _`Structure of System Definition files version 1.4`: http://delivery.nmp.nokia.com/trac/helium/wiki/SystemDefinitionFiles
</#if>
-A template of layer in layers.sysdef.xml
+A template of layer in layers.sysdef.xml for system definition files
.. code-block:: xml
@@ -96,6 +99,47 @@
* In the above, two modules means two drop files will be created; ``module`` may have one or more ``unit``
* By using property ``exclude.test.layers``, complete layers can be excluded and the components inside that layer will not be included in the AtsDrop. This property is a comma (,) separated list
+**System Definition Files version 3.0 (SysDefs3)** (new Helium v.10.79)
+ The `structure of System Definition files version 3.0`_ is different than previous versions of system definition files. In SysDefs3, package definition files are used for components specification. Instead of layers naming conventions, filters are used to identify test components and test types, for example: "test, unit_test, !api_test" etc.
+
+<#if !(ant?keys?seq_contains("sf"))>
+.. _`structure of System Definition files version 3.0`: http://wikis.in.nokia.com/view/SWManageabilityTeamWiki/PkgdefUse
+<#else>
+.. _`structure of System Definition files version 3.0`: sysdef3.rst
+</#if>
+
+An example template for defining test components in a package definition file.
+
+.. code-block:: xml
+
+ <package id="dummytest" name="dummytest" levels="demo">
+ <collection id="test_nested" name="test_nested" level="demo">
+
+ <component id="tc1" name="tc1" purpose="development" filter="test, unit_test">
+ <unit bldFile="test_nested/tc1/group" mrp="" />
+ </component>
+
+ <component id="tc2" name="tc2" purpose="development" filter="test">
+ <meta rel="testbuild">
+ <group name="drop_tc2_and_tc3" />
+ </meta>
+ <unit bldFile="test_nested/tc2/group" mrp="" />
+ </component>
+
+ <component id="tc3" name="tc3" purpose="development" filter="test">
+ <meta rel="testbuild">
+ <group name="drop_tc2_and_tc3" />
+ </meta>
+ <unit bldFile="test_nested/tc3/group" mrp="" />
+ </component>
+
+ </collection>
+ </package>
+
+* Filter "test" must be specified for every test component. If it is not specified, the component will not be considered as a test component.
+* <meta>/<group> are now used to group test components, it work in the same way as <module>...<module> in sysdef v1.4 works. The components having same group name are grouped together.
+ Separate drop files are created for different groups. In the above example, if only 'test' is selected, then two drop files will be created, one with tc1 and the other one with tc2 and tc3.
+
**Step 2: Configure ATS properties in build.xml**
@@ -114,35 +158,37 @@
.. csv-table:: ATS Ant properties
:header: "Property name", "Edit status", "Description"
- ":hlm-p:`ats.server`", "[must]", "For example: ``4fix012345`` or ``catstresrv001.cats.noklab.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as 80. The host can be different depending on site and/or product."
+ ":hlm-p:`ats.server`", "[must]", "For example: ``4fix012345`` or ``catstresrv001.company.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as 80. The host can be different depending on site and/or product."
":hlm-p:`ats.drop.location`", "[allowed]", "Server location (UNC path) to save the ATSDrop file, before sending to the ATS Server. For example: ``\\\\trwsem00\\some_folder\\``. In case, :hlm-p:`ats.script.type` is set to ``import``, ATS doesn't need to have access to :hlm-p:`ats.drop.location`, its value can be any local folder on build machine, for example ``c:/temp`` (no network share needed)."
":hlm-p:`ats.product.name`", "[must]", "Name of the product to be tested."
":hlm-p:`eunit.test.package`", "[allowed]", "The EUnit package name to be unzipped on the environment, for executing EUnit tests."
":hlm-p:`eunitexerunner.flags`", "[allowed]", "Flags for EUnit exerunner can be set by setting the value of this variable. The default flags are set to ``/E S60AppEnv /R Off``."
":hlm-p:`ats.email.list`", "[allowed]", "The property is needed if you want to get an email from ATS server after the tests are executed. There can be one to many semicolon-separated email addresses."
- ":hlm-p:`ats.report.type`", "[allowed]", "Value of the ats email report"
- ":hlm-p:`ats.flashfiles.minlimit`", "[recommended]", "Limit of minimum number of flash files to execute :hlm-t:`ats-test` target, otherwise ``ATSDrop.zip`` will not be generated. Default value is 2 files."
+ ":hlm-p:`ats.report.type`", "[allowed]", "Value of the ats email report, for ATS4 set to 'no_attachment' so email size is reduced"
+ ":hlm-p:`ats.flashfiles.minlimit`", "[allowed]", "Limit of minimum number of flash files to execute :hlm-t:`ats-test` target, otherwise ``ATSDrop.zip`` will not be generated. Default value is 2 files."
":hlm-p:`ats.plan.name`", "[allowed]", "Modify the plan name if you have understanding of ``test.xml`` file or leave it as it is. Default value is ``plan``."
":hlm-p:`ats.product.hwid`", "[allowed]", "Product HardWare ID (HWID) attached to ATS. By default the value of HWID is not set."
":hlm-p:`ats.script.type`", "[allowed]", "There are two types of ats script files to send drop to ATS server, ``runx`` and ``import``; only difference is that with ``import`` ATS doesn't have to have access rights to ``testdrop.zip`` file, as it is sent to the system over http and import doesn't need network shares. If that is not needed ``import`` should not be used. Default value is ``runx`` as ``import`` involves heavy processing on ATS server."
":hlm-p:`ats.target.platform`", "[allowed]", "Sets target platform for compiling test components. Default value is ``armv5 urel``."
":hlm-p:`ats.test.timeout`", "[allowed]", "To set test commands execution time limit on ATS server, in seconds. Default value is ``60``."
":hlm-p:`ats.testrun.name`", "[allowed]", "Modify the test-run name if you have understanding of ``test.xml`` file or leave it as it is. Default value is a string consist of build id, product name, major and minor versions."
- ":hlm-p:`ats.trace.enabled`", "[allowed]", "Should be ``True`` if tracing is needed during the tests running on ATS. Default value is ``False``, the values are case-sensitive. See http://s60wiki.nokia.com/S60Wiki/CATS/TraceTools."
- ":hlm-p:`ats.ctc.enabled`", "[allowed]", "Should be ``True`` if coverage measurement and dynamic analysis (CTC) tool support is to be used by ATS. Default value is ``False``. The values are case-sensitive."
+ ":hlm-p:`ats.trace.enabled`", "[allowed]", "Should be ``true`` if tracing is needed during the tests running on ATS. Default value is ``false``, the values are case-sensitive. See http://s60wiki.nokia.com/S60Wiki/CATS/TraceTools."
+ ":hlm-p:`ats.ctc.enabled`", "[allowed]", "Should be ``true`` if coverage measurement and dynamic analysis (CTC) tool support is to be used by ATS. Default value is ``false``. The values are case-sensitive."
":hlm-p:`ats.ctc.host`", "[allowed]", "CTC host, provided by CATS used to create coverage measurement reports. MON.sym files are copied to this location, for example ``10.0.0.1``. If not given, code coverage reports are not created"
- ":hlm-p:`ats.obey.pkgfiles.rule`", "[allowed]", "If the property is set to ``True``, then the only test components which will have PKG files, will be included into the ``test.xml`` as a test-set. Which means, even if there's a test component (executable) but there's no PKG file, it should not be considered as a test component and hence not included into the test.xml as a separate test. By default the property value is ``False``."
+ ":hlm-p:`ats.obey.pkgfiles.rule`", "[allowed]", "If the property is set to ``true``, then the only test components which will have PKG files, will be included into the ``test.xml`` as a test-set. Which means, even if there's a test component (executable) but there's no PKG file, it should not be considered as a test component and hence not included into the test.xml as a separate test. By default the property value is ``false``."
"``reference.ats.flash.images``", "[allowed]", "Fileset for list of flash images (can be .fpsx, .C00, .V01 etc) It is recommended to set the fileset, default filset is given below which can be overwritten. set *dir=""* attribute of the filset to ``${r'$'}{build.output.dir}/variant_images`` if hlm-t:`variant-image-creation` target is being used."
":hlm-p:`tsrc.data.dir`", "[allowed]", "The default value is ``data`` and refers to the 'data' directory under 'tsrc' directory."
":hlm-p:`tsrc.path.list`", "[allowed]", "Contains list of the tsrc directories. Gets the list from system definition layer files. Assuming that the test components are defined already in te ``layers.sysdef.xml`` files to get compiled. Not recommended, but the property value can be set if there are no System Definition file(s), and tsrc directories paths to set manually."
":hlm-p:`ats.report.location`", "[allowed]", "Sets ATS reports store location. Default location is ``${r'$'}{publish.dir}/${r'$'}{publish.subdir}``."
- ":hlm-p:`ats.multiset.enabled`", "[allowed]", "Should be ``True`` so a set is used for each pkg file in a component, this allows tests to run in parallel on several devices."
+ ":hlm-p:`ats.multiset.enabled`", "[allowed]", "Should be ``true`` so a set is used for each pkg file in a component, this allows tests to run in parallel on several devices."
":hlm-p:`ats.diamonds.signal`", "[allowed]", "Should be ``true`` so at end of the build diamonds is checked for test results and Helium fails if tests failed."
":hlm-p:`ats.delta.enabled`", "[allowed]", "Should be ``true`` so only ADOs changed during :hlm-t:`do-prep-work-area` are tested by ATS."
":hlm-p:`ats4.enabled`", "[allowed]", "Should be ``true`` if ATS4 is to be used."
":hlm-p:`ats.emulator.enable`", "[allowed]", "Should be ``true`` if ``WINSCW`` emulator is to be used."
":hlm-p:`ats.specific.pkg`", "[allowed]", "Text in name of PKG files to use eg. 'sanity' would only use xxxsanity.pkg files from components."
- ":hlm-p:`ats.create.singledrop.file`", "[recommended]", "If present and set to 'true', it will create one drop file, if set to any other value or not present it will create multiple drop files (defined by the sysdef file). This is to save traffic to the server."
+ ":hlm-p:`ats.singledrop.enabled`", "[allowed]", "If present and set to 'true', it will create one drop file, if set to any other value or not present it will create multiple drop files (defined by the sysdef file). This is to save traffic to the server."
+ ":hlm-p:`ats.java.importer.enabled`", "[allowed]", "If set to 'true', for older uploader is used for ats3 which shows improved error message."
+ ":hlm-p:`ats.test.filterset`", "[allowed]", "(new Helium v.10.79)Contains a name of test filterset (see example below). A filterset is used to select/unselect test components. The filter(s) is/are effective when the same filters are defined in the package definition file for component(s)."
An example of setting up properties:
@@ -159,12 +205,19 @@
<property name="ats.target.platform" value="armv5 urel" />
<property name="ats.test.timeout" value="60" />
<property name="ats.testrun.name" value="${r'$'}{build.id}_${r'$'}{ats.product.name}_${r'$'}{major.version}.${r'$'}{minor.version}" />
- <property name="ats.trace.enabled" value="False" />
- <property name="ats.ctc.enabled" value="False" />
- <property name="ats.obey.pkgfiles.rule" value="False" />
+ <property name="ats.trace.enabled" value="false" />
+ <property name="ats.ctc.enabled" value="false" />
+ <property name="ats.obey.pkgfiles.rule" value="false" />
<property name="ats.report.location" value="${r'$'}{publish.dir}/${r'$'}{publish.subdir}" />
<property name="eunit.test.package" value="" />
<property name="eunitexerunner.flags" value="/E S60AppEnv /R Off" />
+ <property name="ats.test.filterset" value="sysdef.filters.tests" />
+
+ <hlm:sysdefFilterSet id="sysdef.filters.tests">
+ <filter filter="test, " type="has" />
+ <config file="bldvariant.hrh" includes="" />
+ </hlm:sysdefFilterSet>
+
...
<import file="${r'$'}{helium.dir}/helium.ant.xml" />
@@ -183,7 +236,7 @@
**STEP 3: Call target ats-test**
-To execute the target, a property should be set(``<property name="enabled.ats" value="true" />``).
+To execute the target, a property should be set(``<property name="ats.enabled" value="true" />``).
Then call :hlm-t:`ats-test`, which will create the ATSDrop.zip (test package).
@@ -221,7 +274,7 @@
Skip Sending AtsDrop to ATS
----------------------------
-By setting property of ``skip.ats.sending``, ``ats-test`` target only creates a drop file, and does not send the drop (or package) to ATS server.
+By setting property of ``ats.upload.enabled`` to ``false``, ``ats-test`` target only creates a drop file, and does not send the drop (or package) to ATS server.
Customizing the test.xml in ATS
--------------------------------
@@ -239,11 +292,11 @@
* **prepostaction.xml** goes before first postaction
* **postpostaction.xml** goes after last postaction
-The files must be in the directory custom under the tsrc folder processed.
+The files must be in the directory 'custom' under the 'tsrc' or 'group' folder to be processed.
The files need to be proper XML snippets that fit to their place. In case of an error an error is logged and a comment inserted to the generated XML file.
-A postaction section customization file ( prepostaction.xml or postpostaction.xml) could look like this
+A postaction section customization file (prepostaction.xml or postpostaction.xml) could look like this
.. code-block:: xml
@@ -376,7 +429,7 @@
.. csv-table:: ATS Ant properties
:header: "Property name", "Edit status", "Description"
- ":hlm-p:`ats.server`", "[must]", "For example: ``4fio00105`` or ``catstresrv001.cats.noklab.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as ``80``. The host can be different depending on site and/or product."
+ ":hlm-p:`ats.server`", "[must]", "For example: ``4fio00105`` or ``catstresrv001.company.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as ``80``. The host can be different depending on site and/or product."
":hlm-p:`ats.drop.location`", "[must]", "Server location (UNC path) to save the ATSDrop file, before sending to the ATS. For example: ``\\\\trwsem00\\some_folder\\``. In case, ``ats.script.type`` is set to ``import``, ATS doesn't need to have access to :hlm-p:`ats.drop.location`, its value can be any local folder on build machine, for example ``c:/temp`` (no network share needed)."
":hlm-p:`ats.product.name`", "[must]", "Name of the product to be tested."
":hlm-p:`ats.aste.testasset.location`", "[must]", "Location of SW Test Assets, if the TestAsset is not packaged then it is first compressed to a ``.zip`` file. It should be a UNC path."