--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/buildframework/helium/doc/src/manual/stage_ats.rst.inc.ftl Tue Apr 27 08:33:08 2010 +0300
@@ -0,0 +1,441 @@
+<#--
+============================================================================
+Name : stage_ats.rst.inc.ftl
+Part of : Helium
+
+Copyright (c) 2009 Nokia Corporation and/or its subsidiary(-ies).
+All rights reserved.
+This component and the accompanying materials are made available
+under the terms of the License "Eclipse Public License v1.0"
+which accompanies this distribution, and is available
+at the URL "http://www.eclipse.org/legal/epl-v10.html".
+
+Initial Contributors:
+Nokia Corporation - initial contribution.
+
+Contributors:
+
+Description:
+
+============================================================================
+-->
+
+.. index::
+ single: ATS - STIF, TEF, RTEST, MTF and EUnit
+
+.. _`Stage-ATS-label`:
+
+Stage: ATS - STIF, TEF, RTEST, MTF and EUnit (also Qt)
+=======================================================
+
+ATS testing is the automatic testing of the phone code once it has been compiled and linked to create a ROM image.
+
+Explanation of the process for getting ATS (`STIF`_ and `EUnit`_) tests compiled and executed by Helium, through the use of the ``ats-test`` target.
+
+http://developer.symbian.org/wiki/index.php/Symbian_Test_Tools
+
+<#if !(ant?keys?seq_contains("sf"))>
+.. _`STIF`: http://s60wiki.nokia.com/S60Wiki/STIF
+.. _`EUnit`: http://s60wiki.nokia.com/S60Wiki/EUnit
+</#if>
+
+.. image:: ats.dot.png
+
+Prerequisites
+----------------
+
+* `Harmonized Test Interface (HTI)`_ needs to be compiled and into the image.
+* The reader is expected to already have a working ATS setup in which test cases can be executed. ATS server names,
+ access rights and authentication etc. is supposed to be already taken care of.
+
+<#if !(ant?keys?seq_contains("sf"))>
+.. _`Harmonized Test Interface (HTI)`: http://s60wiki.nokia.com/S60Wiki/HTI
+<#else>
+.. _`Harmonized Test Interface (HTI)`: http://developer.symbian.org/wiki/index.php/HTI_Tool
+</#if>
+
+Test source components
+-------------------------
+
+Test source usually lives in a component's ``tsrc`` directory. Test source components are created like any other Symbian SW component;
+there is a ``group`` directory with a ``bld.inf`` file for building, ``.mmp`` files for defining the targets, and so on.
+
+The test generation code expects ``.pkg`` file in the ``group`` directory of test component to be compiled, to get the paths of the files
+(can be data, configuration, initialization, etc files) to be installed and where to install on the phone.
+
+Three STEPS to setup ATS with Helium
+--------------------------------------
+
+**Step 1: Configure System Definition Files**
+ If the tsrc directory structure meets the criteria defined in the `new API test automation guidelines`_, then test components
+ should be included in the System Definition files; **layers** in ``layers.sysdef.xml`` file and **configuration** in ``build.sysdef.xml``
+ file (`Structure of System Definition files`_).
+
+ <#if !(ant?keys?seq_contains("sf"))>
+.. _`new API test automation guidelines`: http://s60wiki.nokia.com/S60Wiki/Test_Asset_Guidelines
+.. _`Structure of System Definition files`: http://delivery.nmp.nokia.com/trac/helium/wiki/SystemDefinitionFiles
+</#if>
+
+A template of layer in layers.sysdef.xml
+
+.. code-block:: xml
+
+ <layer name="name_test_layer">
+ <module name="module_name_one">
+ <unit unitID="unit_id1" name="unit_name1" bldFile="path_of_tsrc_folder_to_be_built" mrp="" />
+ </module>
+
+ <module name="module_name_two">
+ <unit unitID="unit_id2" name="unit_name2" bldFile="path_of_tsrc_folder_to_be_built" mrp="" />
+ </module>
+ </layer>
+
+* Layer name should end with **_test_layer**
+* Two standard names for ATS test layers are being used; ``unit_test_layer`` and ``api_test_layer``. Test components (the``unit`` tags)
+ should be specified under these layers and grouped by ``module`` tag(s).
+* In the above, two modules means two drop files will be created; ``module`` may have one or more ``unit``
+* By using property ``exclude.test.layers``, complete layers can be excluded and the components inside that layer will not be included in the AtsDrop. This property is a comma (,) separated list
+
+
+**Step 2: Configure ATS properties in build.xml**
+
+**(A)** Username and Password for the ATS should be set in the `.netrc file`_::
+
+ machine ats login ats_user_name password ats_password
+
+Add the above line in the ``.netrc`` file and replace ``ats_user_name`` with your real ATS username and ``ats_password`` with ATS password.
+
+**(B)** The following properties are ATS dependent with their edit status
+
+* [must] - must be set by user
+* [recommended] - should be set by user but not mandatory
+* [allowed] - should **not** be set by user however, it is possible.
+
+.. csv-table:: ATS Ant properties
+ :header: "Property name", "Edit status", "Description"
+
+ ":hlm-p:`ats.server`", "[must]", "For example: ``4fix012345`` or ``catstresrv001.cats.noklab.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as 80. The host can be different depending on site and/or product."
+ ":hlm-p:`ats.drop.location`", "[must]", "Server location (UNC path) to save the ATSDrop file, before sending to the ATS Server. For example: ``\\\\trwsem00\\some_folder\\``. In case, :hlm-p:`ats.script.type` is set to ``import``, ATS doesn't need to have access to :hlm-p:`ats.drop.location`, its value can be any local folder on build machine, for example ``c:/temp`` (no network share needed)."
+ ":hlm-p:`ats.product.name`", "[must]", "Name of the product to be tested."
+ ":hlm-p:`eunit.test.package`", "[recommended]", "The EUnit package name to be unzipped on the environment, for executing EUnit tests."
+ ":hlm-p:`eunitexerunner.flags`", "[recommended]", "Flags for EUnit exerunner can be set by setting the value of this variable. The default flags are set to ``/E S60AppEnv /R Off``."
+ ":hlm-p:`ats.email.list`", "[recommended]", "The property is needed if you want to get an email from ATS server after the tests are executed. There can be one to many semicolon-separated email addresses."
+ ":hlm-p:`ats.flashfiles.minlimit`", "[recommended]", "Limit of minimum number of flash files to execute :hlm-t:`ats-test` target, otherwise ``ATSDrop.zip`` will not be generated. Default value is 2 files."
+ ":hlm-p:`ats.plan.name`", "[recommended]", "Modify the plan name if you have understanding of ``test.xml`` file or leave it as it is. Default value is ``plan``."
+ ":hlm-p:`ats.product.hwid`", "[recommended]", "Product HardWare ID (HWID) attached to ATS. By default the value of HWID is not set."
+ ":hlm-p:`ats.script.type`", "[recommended]", "There are two types of ats script files to send drop to ATS server, ``runx`` and ``import``; only difference is that with ``import`` ATS doesn't have to have access rights to ``testdrop.zip`` file, as it is sent to the system over http and import doesn't need network shares. If that is not needed ``import`` should not be used. Default value is ``runx`` as ``import`` involves heavy processing on ATS server."
+ ":hlm-p:`ats.target.platform`", "[recommended]", "Sets target platform for compiling test components. Default value is ``armv5 urel``."
+ ":hlm-p:`ats.test.timeout`", "[recommended]", "To set test commands execution time limit on ATS server, in seconds. Default value is ``60``."
+ ":hlm-p:`ats.testrun.name`", "[recommended]", "Modify the test-run name if you have understanding of ``test.xml`` file or leave it as it is. Default value is a string consist of build id, product name, major and minor versions."
+ ":hlm-p:`ats.trace.enabled`", "[recommended]", "Should be ``True`` if tracing is needed during the tests running on ATS. Default value is ``False``, the values are case-sensitive. See http://s60wiki.nokia.com/S60Wiki/CATS/TraceTools."
+ ":hlm-p:`ats.ctc.enabled`", "[recommended]", "Should be ``True`` if coverage measurement and dynamic analysis (CTC) tool support is to be used by ATS. Default value is ``False``. The values are case-sensitive."
+ ":hlm-p:`ats.ctc.host`", "[recommended]", "CTC host, provided by CATS used to create coverage measurement reports. MON.sym files are copied to this location, for example ``10.0.0.1``. If not given, code coverage reports are not created"
+ ":hlm-p:`ats.obey.pkgfiles.rule`", "[recommended]", "If the property is set to ``True``, then the only test components which will have PKG files, will be included into the ``test.xml`` as a test-set. Which means, even if there's a test component (executable) but there's no PKG file, it should not be considered as a test component and hence not included into the test.xml as a separate test. By default the property value is ``False``."
+ "``reference.ats.flash.images``", "[recommended]", "Fileset for list of flash images (can be .fpsx, .C00, .V01 etc) It is recommended to set the fileset, default filset is given below which can be overwritten. set *dir=""* attribute of the filset to ``${r'$'}{build.output.dir}/variant_images`` if hlm-t:`variant-image-creation` target is being used."
+ ":hlm-p:`tsrc.data.dir`", "[allowed]", "The default value is ``data`` and refers to the 'data' directory under 'tsrc' directory."
+ ":hlm-p:`tsrc.path.list`", "[allowed]", "Contains list of the tsrc directories. Gets the list from system definition layer files. Assuming that the test components are defined already in te ``layers.sysdef.xml`` files to get compiled. Not recommended, but the property value can be set if there are no System Definition file(s), and tsrc directories paths to set manually."
+ ":hlm-p:`ats.report.location`", "[allowed]", "Sets ATS reports store location. Default location is ``${r'$'}{publish.dir}/${r'$'}{publish.subdir}``."
+ ":hlm-p:`ats.multiset.enabled`", "[allowed]", "Should be ``True`` so a set is used for each pkg file in a component, this allows tests to run in parallel on several devices."
+ ":hlm-p:`ats.diamonds.signal`", "[allowed]", "Should be ``true`` so at end of the build diamonds is checked for test results and Helium fails if tests failed."
+ ":hlm-p:`ats.delta.enabled`", "[allowed]", "Should be ``true`` so only ADOs changed during :hlm-t:`do-prep-work-area` are tested by ATS."
+ ":hlm-p:`ats4.enabled`", "[allowed]", "Should be ``true`` if ATS4 is to be used."
+ ":hlm-p:`ats.emulator.enable`", "[allowed]", "Should be ``true`` if ``WINSCW`` emulator is to be used."
+ ":hlm-p:`ats.specific.pkg`", "[allowed]", "Text in name of PKG files to use eg. 'sanity' would only use xxxsanity.pkg files from components."
+
+An example of setting up properties:
+
+.. code-block:: xml
+
+ <property name="ats.server" value="4fio00105" />
+ <property name="ats.drop.location" location="\\trwsimXX\ATS_TEST_SHARE\" />
+ <property name="ats.email.list" value="temp.user@company.com; another.email@company.com" />
+ <property name="ats.flashfiles.minlimit" value="2" />
+ <property name="ats.product.name" value="PRODUCT" />
+ <property name="ats.plan.name" value="plan" />
+ <property name="ats.product.hwid" value="" />
+ <property name="ats.script.type" value="runx" />
+ <property name="ats.target.platform" value="armv5 urel" />
+ <property name="ats.test.timeout" value="60" />
+ <property name="ats.testrun.name" value="${r'$'}{build.id}_${r'$'}{ats.product.name}_${r'$'}{major.version}.${r'$'}{minor.version}" />
+ <property name="ats.trace.enabled" value="False" />
+ <property name="ats.ctc.enabled" value="False" />
+ <property name="ats.obey.pkgfiles.rule" value="False" />
+ <property name="ats.report.location" value="${r'$'}{publish.dir}/${r'$'}{publish.subdir}" />
+ <property name="eunit.test.package" value="" />
+ <property name="eunitexerunner.flags" value="/E S60AppEnv /R Off" />
+
+ ...
+ <import file="${r'$'}{helium.dir}/helium.ant.xml" />
+ ...
+
+ <fileset id="reference.ats.flash.images" dir="${r'$'}{release.images.dir}">
+ <include name="**/${r'$'}{build.id}*.core.fpsx"/>
+ <include name="**/${r'$'}{build.id}*.rofs2.fpsx"/>
+ <include name="**/${r'$'}{build.id}*.rofs3.fpsx"/>
+ </fileset>
+
+
+.. Note::
+
+ Always declare *Properties* before and *filesets* after importing helium.ant.xml.
+
+**STEP 3: Call target ats-test**
+
+To execute the target, a property should be set(``<property name="enabled.ats" value="true" />``).
+
+Then call ``ats-test``, which will create the ATSDrop.zip (test package).
+
+If property *ats.email.list* is set, an email (test report) will be sent when the tests are ready on ATS.
+
+CTC:
+----
+
+CTC code coverage measurements reports can be created as part of Test Automation process.
+
+1. Build the src using ``build_ctc`` configuration, which is in ``build.sysdef.xml`` file, to create ``MON.sym`` files. It means that a property ``sysdef.configurations.list`` should be modified either add or replace current build configuration with ``build_ctc``
+
+2. Set the property, ``ats.ctc.host``, as described above, for sending the ``MON.sym`` files to the network drive. *(Please contact ATS server administrator and ask for the value to set this property)*
+
+3. Enable CTC process by setting up property ``ats.ctc.enabled`` to "true"
+
+4. Test drops are sent to the ATS server, where, after executing tests ``ctcdata.txt`` files are created. ``ctcdata.txt`` and ``MON.sym`` files are then further processed to create code coverage reports.
+
+5. View or download the Code coverage reports by following the link provided in the ATS report email (sent after the tests are executed on ATS)
+
+*NOTE: After receiving the email notification, it may take a few minutes before the code coverage reports are available.*
+
+
+Qt Tests:
+---------
+
+QtTest.lib is supported and the default harness is set to EUnit. If ``QtTest.lib`` is there in ``.mmp`` file, Helium sets the Harness to Eunit and ATS supported Qt steps are added to test.xml file
+
+In ``layers.sysdef.xml`` file, the layer name should end with "_test_layer" e.g. "qt_unit_test_layer".
+
+There are several ``.PKG`` files created after executing ``qmake``, but only one is selected based on which target platform is set. Please read the property (``ats.target.platform``) description above.
+
+.. _`Skip-Sending-AtsDrop-label`:
+
+Skip Sending AtsDrop to ATS
+----------------------------
+
+By setting property of ``skip.ats.sending``, ``ats-test`` target only creates a drop file, and does not send the drop (or package) to ATS server.
+
+Customizing the test.xml in ATS
+--------------------------------
+
+The user can customize the generated test.xml with files:
+
+* **preset_custom.xml** goes before first set
+* **postset_custom.xml** goes after last set
+* **precase_custom.xml** goes before first case
+* **postcase_custom.xml** goes after last case
+* **prestep_custom.xml** goes before first step
+* **poststep_custom.xml** goes after last step
+* **prerun_custom.xml** goes before first run or execute step
+* **postrun_custom.xml** goes after last run or execute step
+* **prepostaction.xml** goes before first postaction
+* **postpostaction.xml** goes after last postaction
+
+The files must be in the directory custom under the tsrc folder processed.
+
+The files need to be proper XML snippets that fit to their place. In case of an error an error is logged and a comment inserted to the generated XML file.
+
+A postaction section customization file ( prepostaction.xml or postpostaction.xml) could look like this
+
+.. code-block:: xml
+
+ <postAction>
+ <type>Pre PostAction from custom file</type>
+ <params>
+ <param name="foo2" value="bar2" />
+ </params>
+ </postAction>
+
+
+
+The ``prestep_custom.xml`` can be used to flash and unstall something custom.
+
+.. code-block:: xml
+
+ <step name="Install measurement tools" harness="STIF" significant="false">
+ <!-- Copy SIS-packages to DUT -->
+ <command>install</command>
+ <params>
+ <param src="Nokia_Energy_Profiler_1_1.sisx"/>
+ <param dst="c:\data\Nokia_Energy_Profiler_1_1.sisx"/>
+ </params>
+ ...
+ </step>
+
+
+And then the ``prerun_custom.xml`` can be used to start measuring.
+
+.. code-block:: xml
+
+ <step name="Start measurement" harness="STIF" significant="false">
+ <!-- Start measurement -->
+ <command>execute</command>
+ <params>
+ <param file="neplauncher.exe"/>
+ <param parameters="start c:\data\nep.csv"/>
+ <param timeout="30"/>
+ </params>
+ </step>
+
+
+
+**Note:** The users is expected to check the generated test.xml manually, as there is no validation. Invalid XML input files will be disregarded and a comment will be inserted to the generated XML file.
+
+Overriding Test xml values
+--------------------------
+
+Set the property ``ats.config.file`` to the location of the config file.
+
+Example configuration:
+
+.. code-block:: xml
+
+ <ATSConfigData>
+ <config name="common" abstract="true">
+
+ <!-- Properties to add/ modify -->
+ <config type="properties">
+ <set name="HARNESS" value="STIF" />
+ <set name="2" value="3" />
+ </config>
+
+ <!-- Settings to add/ modify -->
+ <config type="settings">
+ <set name="HARNESS" value="STIF" />
+ <set name="2" value="3" />
+ </config>
+
+ <!-- Attributes to modify -->
+ <config type="attributes">
+ <set name="xyz" value="2" />
+ <set name="significant" value="true" />
+ </config>
+ </config>
+ </ATSConfigData>
+
+
+.. index::
+ single: ATS - ASTE
+
+Stage: ATS - ASTE
+===================
+
+Explanation of the process for getting ATS `ASTE`_ tests compiled and executed by Helium, through the use of the ``ats-aste`` target.
+
+<#if !(ant?keys?seq_contains("sf"))>
+.. _`ASTE`: http://s60wiki.nokia.com/S60Wiki/ASTE
+</#if>
+
+Prerequisites
+--------------
+
+* `Harmonized Test Interface (HTI)`_ needs to be compiled and into the image.
+* The reader is expected to already have a working ATS setup in which test cases can be executed. ATS server names, access rights and authentication etc. is supposed to be already taken care of.
+* `SW Test Asset`_ location and type of test should be known.
+
+<#if !(ant?keys?seq_contains("sf"))>
+.. _`Harmonized Test Interface (HTI)`: http://s60wiki.nokia.com/S60Wiki/HTI
+.. _`SW Test Asset`: http://s60wiki.nokia.com/S60Wiki/MC_SW_Test_Asset_documentation
+</#if>
+
+Test source components
+--------------------------
+
+Unlike STIF, EUnit etc tests, test source components (or ``tsrc`` structure) is not needed for `ASTE`_ tests.
+
+Two STEPS to setup ASTE with Helium
+------------------------------------
+
+**STEP 1: Configure ASTE properties in build.xml**
+
+**(A)** Username and Password for the ATS should be set in the `.netrc file`_
+
+.. code-block:: text
+
+ machine ats login ats_user_name password ats_password
+
+Add the above line in the .netrc file and replace *ats_user_name* with your real ats username and "ats_password" with ats password.
+
+.. _`.netrc file`: configuring.html?highlight=netrc#passwords
+
+
+**(B)** The following properties are ASTE dependent with their edit status
+
+* [must] - must be set by user
+* [recommended] - should be set by user but not mandatory
+* [allowed] - should **not** be set by user however, it is possible.
+
+.. csv-table:: ATS Ant properties
+ :header: "Property name", "Edit status", "Description"
+
+ ":hlm-p:`ats.server`", "[must]", "For example: ``4fio00105`` or ``catstresrv001.cats.noklab.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as ``80``. The host can be different depending on site and/or product."
+ ":hlm-p:`ats.drop.location`", "[must]", "Server location (UNC path) to save the ATSDrop file, before sending to the ATS. For example: ``\\\\trwsem00\\some_folder\\``. In case, ``ats.script.type`` is set to ``import``, ATS doesn't need to have access to :hlm-p:`ats.drop.location`, its value can be any local folder on build machine, for example ``c:/temp`` (no network share needed)."
+ ":hlm-p:`ats.product.name`", "[must]", "Name of the product to be tested."
+ ":hlm-p:`ats.aste.testasset.location`", "[must]", "Location of SW Test Assets, if the TestAsset is not packaged then it is first compressed to a ``.zip`` file. It should be a UNC path."
+ ":hlm-p:`ats.aste.software.release`", "[must]", "Flash images releases, for example 'SPP 51.32'."
+ ":hlm-p:`ats.aste.software.version`", "[must]", "Version of the software to be tested. For example: 'W810'"
+ ":hlm-p:`ats.aste.email.list`", "[recommended]", "The property is needed if you want to get an email from ATS server after the tests are executed. There can be one to many semicolon(s) ";" separated email addresses."
+ ":hlm-p:`ats.flashfiles.minlimit`", "[recommended]", "Limit of minimum number of flash files to execute ats-test target, otherwise ATSDrop.zip will not be generated. Default value is "2" files."
+ ":hlm-p:`ats.aste.plan.name`", "[recommended]", "Modify the plan name if you have understanding of test.xml file or leave it as it is. Default value is "plan"."
+ ":hlm-p:`ats.product.hwid`", "[recommended]", "Product HardWare ID (HWID) attached to ATS. By default the value of HWID is not set."
+ ":hlm-p:`ats.test.timeout`", "[recommended]", "To set test commands execution time limit on ATS server, in seconds. Default value is '60'."
+ ":hlm-p:`ats.aste.testrun.name`", "[recommended]", "Modify the test-run name if you have understanding of ``test.xml`` file or leave it as it is. Default value is a string consists of build id, product name, major and minor versions."
+ ":hlm-p:`ats.aste.test.type`", "[recommended]", "Type of test to run. Default is 'smoke'."
+ ":hlm-p:`ats.aste.testasset.caseids`", "[recommended]", "These are the cases that which tests should be run from the TestAsset. For example, value can be set as ``100,101,102,103,105,106,``. A comma is needed to separate case IDs"
+ ":hlm-p:`ats.aste.language`", "[recommended]", "Variant Language to be tested. Default is 'English'"
+ "``reference.ats.flash.images``", "[recommended]", "Fileset for list of flash images (can be .fpsx, .C00, .V01 etc) It is recommended to set the fileset, default filset is given below which can be overwritten. set *dir=\"\"* attribute of the filset to ``${r'$'}{build.output.dir}/variant_images`` if :hlm-t:`variant-image-creation` target is being used."
+
+
+An example of setting up properties:
+
+.. code-block:: xml
+
+ <property name="ats.server" value="4fio00105" />
+ <property name="ats.drop.location" value="\\trwsimXX\ATS_TEST_SHARE\" />
+ <property name="ats.aste.email.list" value="temp.user@company.com; another.email@company.com" />
+ <property name="ats.flashfiles.minlimit" value="2" />
+ <property name="ats.product.name" value="PRODUCT" />
+ <property name="ats.aste.plan.name" value="plan" />
+ <property name="ats.product.hwid" value="" />
+ <property name="ats.test.timeout" value="60" />
+ <property name="ats.aste.testrun.name" value="${r'$'}{build.id}_${r'$'}{ats.product.name}_${r'$'}{major.version}.${r'$'}{minor.version}" />
+ <property name="ats.aste.testasset.location" value="" />
+ <property name="ats.aste.software.release" value="SPP 51.32" />
+ <property name="ats.aste.test.type" value="smoke" />
+ <property name="ats.aste.testasset.caseids" value="100,101,102,104,106," />
+ <property name="ats.aste.software.version" value="W810" />
+ <property name="ats.aste.language" value="English" />
+
+ ...
+ <import file="${r'$'}{helium.dir}/helium.ant.xml" />
+ ...
+
+ <fileset id="reference.ats.flash.images" dir="${r'$'}{release.images.dir}">
+ <include name="**/${r'$'}{build.id}*.core.fpsx"/>
+ <include name="**/${r'$'}{build.id}*.rofs2.fpsx"/>
+ <include name="**/${r'$'}{build.id}*.rofs3.fpsx"/>
+ </fileset>
+
+
+*PLEASE NOTE:* Always declare *Properties* before and *filesets* after importing helium.ant.xml.
+
+**STEP 2: Call target ats-aste**
+
+To execute the target, a property should be set(``<property name="enabled.aste" value="true" />``).
+
+Then call ``ats-aste``, which will create the ATSDrop.zip (test package).
+
+If property ``ats.aste.email.list`` is set, an email (test report) will be sent when the tests are ready on ATS/ASTE.
+
+
+Skip Sending AtsDrop to ATS
+------------------------------
+
+click :ref:`Skip-Sending-AtsDrop-label`:
+