diff -r 0f5e3a7fb6af -r 85df38eb4012 buildframework/helium/doc/src/manual/stage_ats.rst.inc.ftl --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/buildframework/helium/doc/src/manual/stage_ats.rst.inc.ftl Tue Apr 27 08:33:08 2010 +0300 @@ -0,0 +1,441 @@ +<#-- +============================================================================ +Name : stage_ats.rst.inc.ftl +Part of : Helium + +Copyright (c) 2009 Nokia Corporation and/or its subsidiary(-ies). +All rights reserved. +This component and the accompanying materials are made available +under the terms of the License "Eclipse Public License v1.0" +which accompanies this distribution, and is available +at the URL "http://www.eclipse.org/legal/epl-v10.html". + +Initial Contributors: +Nokia Corporation - initial contribution. + +Contributors: + +Description: + +============================================================================ +--> + +.. index:: + single: ATS - STIF, TEF, RTEST, MTF and EUnit + +.. _`Stage-ATS-label`: + +Stage: ATS - STIF, TEF, RTEST, MTF and EUnit (also Qt) +======================================================= + +ATS testing is the automatic testing of the phone code once it has been compiled and linked to create a ROM image. + +Explanation of the process for getting ATS (`STIF`_ and `EUnit`_) tests compiled and executed by Helium, through the use of the ``ats-test`` target. + +http://developer.symbian.org/wiki/index.php/Symbian_Test_Tools + +<#if !(ant?keys?seq_contains("sf"))> +.. _`STIF`: http://s60wiki.nokia.com/S60Wiki/STIF +.. _`EUnit`: http://s60wiki.nokia.com/S60Wiki/EUnit + + +.. image:: ats.dot.png + +Prerequisites +---------------- + +* `Harmonized Test Interface (HTI)`_ needs to be compiled and into the image. +* The reader is expected to already have a working ATS setup in which test cases can be executed. ATS server names, + access rights and authentication etc. is supposed to be already taken care of. + +<#if !(ant?keys?seq_contains("sf"))> +.. _`Harmonized Test Interface (HTI)`: http://s60wiki.nokia.com/S60Wiki/HTI +<#else> +.. _`Harmonized Test Interface (HTI)`: http://developer.symbian.org/wiki/index.php/HTI_Tool + + +Test source components +------------------------- + +Test source usually lives in a component's ``tsrc`` directory. Test source components are created like any other Symbian SW component; +there is a ``group`` directory with a ``bld.inf`` file for building, ``.mmp`` files for defining the targets, and so on. + +The test generation code expects ``.pkg`` file in the ``group`` directory of test component to be compiled, to get the paths of the files +(can be data, configuration, initialization, etc files) to be installed and where to install on the phone. + +Three STEPS to setup ATS with Helium +-------------------------------------- + +**Step 1: Configure System Definition Files** + If the tsrc directory structure meets the criteria defined in the `new API test automation guidelines`_, then test components + should be included in the System Definition files; **layers** in ``layers.sysdef.xml`` file and **configuration** in ``build.sysdef.xml`` + file (`Structure of System Definition files`_). + + <#if !(ant?keys?seq_contains("sf"))> +.. _`new API test automation guidelines`: http://s60wiki.nokia.com/S60Wiki/Test_Asset_Guidelines +.. _`Structure of System Definition files`: http://delivery.nmp.nokia.com/trac/helium/wiki/SystemDefinitionFiles + + +A template of layer in layers.sysdef.xml + +.. code-block:: xml + + + + + + + + + + + +* Layer name should end with **_test_layer** +* Two standard names for ATS test layers are being used; ``unit_test_layer`` and ``api_test_layer``. Test components (the``unit`` tags) + should be specified under these layers and grouped by ``module`` tag(s). +* In the above, two modules means two drop files will be created; ``module`` may have one or more ``unit`` +* By using property ``exclude.test.layers``, complete layers can be excluded and the components inside that layer will not be included in the AtsDrop. This property is a comma (,) separated list + + +**Step 2: Configure ATS properties in build.xml** + +**(A)** Username and Password for the ATS should be set in the `.netrc file`_:: + + machine ats login ats_user_name password ats_password + +Add the above line in the ``.netrc`` file and replace ``ats_user_name`` with your real ATS username and ``ats_password`` with ATS password. + +**(B)** The following properties are ATS dependent with their edit status + +* [must] - must be set by user +* [recommended] - should be set by user but not mandatory +* [allowed] - should **not** be set by user however, it is possible. + +.. csv-table:: ATS Ant properties + :header: "Property name", "Edit status", "Description" + + ":hlm-p:`ats.server`", "[must]", "For example: ``4fix012345`` or ``catstresrv001.cats.noklab.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as 80. The host can be different depending on site and/or product." + ":hlm-p:`ats.drop.location`", "[must]", "Server location (UNC path) to save the ATSDrop file, before sending to the ATS Server. For example: ``\\\\trwsem00\\some_folder\\``. In case, :hlm-p:`ats.script.type` is set to ``import``, ATS doesn't need to have access to :hlm-p:`ats.drop.location`, its value can be any local folder on build machine, for example ``c:/temp`` (no network share needed)." + ":hlm-p:`ats.product.name`", "[must]", "Name of the product to be tested." + ":hlm-p:`eunit.test.package`", "[recommended]", "The EUnit package name to be unzipped on the environment, for executing EUnit tests." + ":hlm-p:`eunitexerunner.flags`", "[recommended]", "Flags for EUnit exerunner can be set by setting the value of this variable. The default flags are set to ``/E S60AppEnv /R Off``." + ":hlm-p:`ats.email.list`", "[recommended]", "The property is needed if you want to get an email from ATS server after the tests are executed. There can be one to many semicolon-separated email addresses." + ":hlm-p:`ats.flashfiles.minlimit`", "[recommended]", "Limit of minimum number of flash files to execute :hlm-t:`ats-test` target, otherwise ``ATSDrop.zip`` will not be generated. Default value is 2 files." + ":hlm-p:`ats.plan.name`", "[recommended]", "Modify the plan name if you have understanding of ``test.xml`` file or leave it as it is. Default value is ``plan``." + ":hlm-p:`ats.product.hwid`", "[recommended]", "Product HardWare ID (HWID) attached to ATS. By default the value of HWID is not set." + ":hlm-p:`ats.script.type`", "[recommended]", "There are two types of ats script files to send drop to ATS server, ``runx`` and ``import``; only difference is that with ``import`` ATS doesn't have to have access rights to ``testdrop.zip`` file, as it is sent to the system over http and import doesn't need network shares. If that is not needed ``import`` should not be used. Default value is ``runx`` as ``import`` involves heavy processing on ATS server." + ":hlm-p:`ats.target.platform`", "[recommended]", "Sets target platform for compiling test components. Default value is ``armv5 urel``." + ":hlm-p:`ats.test.timeout`", "[recommended]", "To set test commands execution time limit on ATS server, in seconds. Default value is ``60``." + ":hlm-p:`ats.testrun.name`", "[recommended]", "Modify the test-run name if you have understanding of ``test.xml`` file or leave it as it is. Default value is a string consist of build id, product name, major and minor versions." + ":hlm-p:`ats.trace.enabled`", "[recommended]", "Should be ``True`` if tracing is needed during the tests running on ATS. Default value is ``False``, the values are case-sensitive. See http://s60wiki.nokia.com/S60Wiki/CATS/TraceTools." + ":hlm-p:`ats.ctc.enabled`", "[recommended]", "Should be ``True`` if coverage measurement and dynamic analysis (CTC) tool support is to be used by ATS. Default value is ``False``. The values are case-sensitive." + ":hlm-p:`ats.ctc.host`", "[recommended]", "CTC host, provided by CATS used to create coverage measurement reports. MON.sym files are copied to this location, for example ``10.0.0.1``. If not given, code coverage reports are not created" + ":hlm-p:`ats.obey.pkgfiles.rule`", "[recommended]", "If the property is set to ``True``, then the only test components which will have PKG files, will be included into the ``test.xml`` as a test-set. Which means, even if there's a test component (executable) but there's no PKG file, it should not be considered as a test component and hence not included into the test.xml as a separate test. By default the property value is ``False``." + "``reference.ats.flash.images``", "[recommended]", "Fileset for list of flash images (can be .fpsx, .C00, .V01 etc) It is recommended to set the fileset, default filset is given below which can be overwritten. set *dir=""* attribute of the filset to ``${r'$'}{build.output.dir}/variant_images`` if hlm-t:`variant-image-creation` target is being used." + ":hlm-p:`tsrc.data.dir`", "[allowed]", "The default value is ``data`` and refers to the 'data' directory under 'tsrc' directory." + ":hlm-p:`tsrc.path.list`", "[allowed]", "Contains list of the tsrc directories. Gets the list from system definition layer files. Assuming that the test components are defined already in te ``layers.sysdef.xml`` files to get compiled. Not recommended, but the property value can be set if there are no System Definition file(s), and tsrc directories paths to set manually." + ":hlm-p:`ats.report.location`", "[allowed]", "Sets ATS reports store location. Default location is ``${r'$'}{publish.dir}/${r'$'}{publish.subdir}``." + ":hlm-p:`ats.multiset.enabled`", "[allowed]", "Should be ``True`` so a set is used for each pkg file in a component, this allows tests to run in parallel on several devices." + ":hlm-p:`ats.diamonds.signal`", "[allowed]", "Should be ``true`` so at end of the build diamonds is checked for test results and Helium fails if tests failed." + ":hlm-p:`ats.delta.enabled`", "[allowed]", "Should be ``true`` so only ADOs changed during :hlm-t:`do-prep-work-area` are tested by ATS." + ":hlm-p:`ats4.enabled`", "[allowed]", "Should be ``true`` if ATS4 is to be used." + ":hlm-p:`ats.emulator.enable`", "[allowed]", "Should be ``true`` if ``WINSCW`` emulator is to be used." + ":hlm-p:`ats.specific.pkg`", "[allowed]", "Text in name of PKG files to use eg. 'sanity' would only use xxxsanity.pkg files from components." + +An example of setting up properties: + +.. code-block:: xml + + + + + + + + + + + + + + + + + + + + ... + + ... + + + + + + + + +.. Note:: + + Always declare *Properties* before and *filesets* after importing helium.ant.xml. + +**STEP 3: Call target ats-test** + +To execute the target, a property should be set(````). + +Then call ``ats-test``, which will create the ATSDrop.zip (test package). + +If property *ats.email.list* is set, an email (test report) will be sent when the tests are ready on ATS. + +CTC: +---- + +CTC code coverage measurements reports can be created as part of Test Automation process. + +1. Build the src using ``build_ctc`` configuration, which is in ``build.sysdef.xml`` file, to create ``MON.sym`` files. It means that a property ``sysdef.configurations.list`` should be modified either add or replace current build configuration with ``build_ctc`` + +2. Set the property, ``ats.ctc.host``, as described above, for sending the ``MON.sym`` files to the network drive. *(Please contact ATS server administrator and ask for the value to set this property)* + +3. Enable CTC process by setting up property ``ats.ctc.enabled`` to "true" + +4. Test drops are sent to the ATS server, where, after executing tests ``ctcdata.txt`` files are created. ``ctcdata.txt`` and ``MON.sym`` files are then further processed to create code coverage reports. + +5. View or download the Code coverage reports by following the link provided in the ATS report email (sent after the tests are executed on ATS) + +*NOTE: After receiving the email notification, it may take a few minutes before the code coverage reports are available.* + + +Qt Tests: +--------- + +QtTest.lib is supported and the default harness is set to EUnit. If ``QtTest.lib`` is there in ``.mmp`` file, Helium sets the Harness to Eunit and ATS supported Qt steps are added to test.xml file + +In ``layers.sysdef.xml`` file, the layer name should end with "_test_layer" e.g. "qt_unit_test_layer". + +There are several ``.PKG`` files created after executing ``qmake``, but only one is selected based on which target platform is set. Please read the property (``ats.target.platform``) description above. + +.. _`Skip-Sending-AtsDrop-label`: + +Skip Sending AtsDrop to ATS +---------------------------- + +By setting property of ``skip.ats.sending``, ``ats-test`` target only creates a drop file, and does not send the drop (or package) to ATS server. + +Customizing the test.xml in ATS +-------------------------------- + +The user can customize the generated test.xml with files: + +* **preset_custom.xml** goes before first set +* **postset_custom.xml** goes after last set +* **precase_custom.xml** goes before first case +* **postcase_custom.xml** goes after last case +* **prestep_custom.xml** goes before first step +* **poststep_custom.xml** goes after last step +* **prerun_custom.xml** goes before first run or execute step +* **postrun_custom.xml** goes after last run or execute step +* **prepostaction.xml** goes before first postaction +* **postpostaction.xml** goes after last postaction + +The files must be in the directory custom under the tsrc folder processed. + +The files need to be proper XML snippets that fit to their place. In case of an error an error is logged and a comment inserted to the generated XML file. + +A postaction section customization file ( prepostaction.xml or postpostaction.xml) could look like this + +.. code-block:: xml + + + Pre PostAction from custom file + + + + + + + +The ``prestep_custom.xml`` can be used to flash and unstall something custom. + +.. code-block:: xml + + + + install + + + + + ... + + + +And then the ``prerun_custom.xml`` can be used to start measuring. + +.. code-block:: xml + + + + execute + + + + + + + + + +**Note:** The users is expected to check the generated test.xml manually, as there is no validation. Invalid XML input files will be disregarded and a comment will be inserted to the generated XML file. + +Overriding Test xml values +-------------------------- + +Set the property ``ats.config.file`` to the location of the config file. + +Example configuration: + +.. code-block:: xml + + + + + + + + + + + + + + + + + + + + + + + + + +.. index:: + single: ATS - ASTE + +Stage: ATS - ASTE +=================== + +Explanation of the process for getting ATS `ASTE`_ tests compiled and executed by Helium, through the use of the ``ats-aste`` target. + +<#if !(ant?keys?seq_contains("sf"))> +.. _`ASTE`: http://s60wiki.nokia.com/S60Wiki/ASTE + + +Prerequisites +-------------- + +* `Harmonized Test Interface (HTI)`_ needs to be compiled and into the image. +* The reader is expected to already have a working ATS setup in which test cases can be executed. ATS server names, access rights and authentication etc. is supposed to be already taken care of. +* `SW Test Asset`_ location and type of test should be known. + +<#if !(ant?keys?seq_contains("sf"))> +.. _`Harmonized Test Interface (HTI)`: http://s60wiki.nokia.com/S60Wiki/HTI +.. _`SW Test Asset`: http://s60wiki.nokia.com/S60Wiki/MC_SW_Test_Asset_documentation + + +Test source components +-------------------------- + +Unlike STIF, EUnit etc tests, test source components (or ``tsrc`` structure) is not needed for `ASTE`_ tests. + +Two STEPS to setup ASTE with Helium +------------------------------------ + +**STEP 1: Configure ASTE properties in build.xml** + +**(A)** Username and Password for the ATS should be set in the `.netrc file`_ + +.. code-block:: text + + machine ats login ats_user_name password ats_password + +Add the above line in the .netrc file and replace *ats_user_name* with your real ats username and "ats_password" with ats password. + +.. _`.netrc file`: configuring.html?highlight=netrc#passwords + + +**(B)** The following properties are ASTE dependent with their edit status + +* [must] - must be set by user +* [recommended] - should be set by user but not mandatory +* [allowed] - should **not** be set by user however, it is possible. + +.. csv-table:: ATS Ant properties + :header: "Property name", "Edit status", "Description" + + ":hlm-p:`ats.server`", "[must]", "For example: ``4fio00105`` or ``catstresrv001.cats.noklab.net:80``. Default server port is ``8080``, but it is not allowed between intra and Noklab. Because of this we need to define server port as ``80``. The host can be different depending on site and/or product." + ":hlm-p:`ats.drop.location`", "[must]", "Server location (UNC path) to save the ATSDrop file, before sending to the ATS. For example: ``\\\\trwsem00\\some_folder\\``. In case, ``ats.script.type`` is set to ``import``, ATS doesn't need to have access to :hlm-p:`ats.drop.location`, its value can be any local folder on build machine, for example ``c:/temp`` (no network share needed)." + ":hlm-p:`ats.product.name`", "[must]", "Name of the product to be tested." + ":hlm-p:`ats.aste.testasset.location`", "[must]", "Location of SW Test Assets, if the TestAsset is not packaged then it is first compressed to a ``.zip`` file. It should be a UNC path." + ":hlm-p:`ats.aste.software.release`", "[must]", "Flash images releases, for example 'SPP 51.32'." + ":hlm-p:`ats.aste.software.version`", "[must]", "Version of the software to be tested. For example: 'W810'" + ":hlm-p:`ats.aste.email.list`", "[recommended]", "The property is needed if you want to get an email from ATS server after the tests are executed. There can be one to many semicolon(s) ";" separated email addresses." + ":hlm-p:`ats.flashfiles.minlimit`", "[recommended]", "Limit of minimum number of flash files to execute ats-test target, otherwise ATSDrop.zip will not be generated. Default value is "2" files." + ":hlm-p:`ats.aste.plan.name`", "[recommended]", "Modify the plan name if you have understanding of test.xml file or leave it as it is. Default value is "plan"." + ":hlm-p:`ats.product.hwid`", "[recommended]", "Product HardWare ID (HWID) attached to ATS. By default the value of HWID is not set." + ":hlm-p:`ats.test.timeout`", "[recommended]", "To set test commands execution time limit on ATS server, in seconds. Default value is '60'." + ":hlm-p:`ats.aste.testrun.name`", "[recommended]", "Modify the test-run name if you have understanding of ``test.xml`` file or leave it as it is. Default value is a string consists of build id, product name, major and minor versions." + ":hlm-p:`ats.aste.test.type`", "[recommended]", "Type of test to run. Default is 'smoke'." + ":hlm-p:`ats.aste.testasset.caseids`", "[recommended]", "These are the cases that which tests should be run from the TestAsset. For example, value can be set as ``100,101,102,103,105,106,``. A comma is needed to separate case IDs" + ":hlm-p:`ats.aste.language`", "[recommended]", "Variant Language to be tested. Default is 'English'" + "``reference.ats.flash.images``", "[recommended]", "Fileset for list of flash images (can be .fpsx, .C00, .V01 etc) It is recommended to set the fileset, default filset is given below which can be overwritten. set *dir=\"\"* attribute of the filset to ``${r'$'}{build.output.dir}/variant_images`` if :hlm-t:`variant-image-creation` target is being used." + + +An example of setting up properties: + +.. code-block:: xml + + + + + + + + + + + + + + + + + + ... + + ... + + + + + + + + +*PLEASE NOTE:* Always declare *Properties* before and *filesets* after importing helium.ant.xml. + +**STEP 2: Call target ats-aste** + +To execute the target, a property should be set(````). + +Then call ``ats-aste``, which will create the ATSDrop.zip (test package). + +If property ``ats.aste.email.list`` is set, an email (test report) will be sent when the tests are ready on ATS/ASTE. + + +Skip Sending AtsDrop to ATS +------------------------------ + +click :ref:`Skip-Sending-AtsDrop-label`: +