After tests complete execution, result logs can be found on the system drive of the device in the \logs\testexecute\ directory. On the PC, if tests are executed using testdriver, results can be found in the following location:
%EPOCROOT%\epoc32\TestDriver\results\ or
%EPOCROOT%\epoc32\winscw\c\logs\testexecute
if executing the tests manually.
When using testdriver, a new test result folder is created for each test run that is, results are not overwritten each time a test run is executed. A .htm log file can be found for each script contained in the suite, in its own subfolder.
Each test case in the script will consists of test blocks (usually just one), which will pass or fail. Each test block will execute a number of commands. Each of the commands performs an action on the tested APIs (calling a certain function of tested classes). A test case passes if all its test blocks pass. A test block passes if all its commands and other instructions pass.
The log will display the percentage of test cases that have passed successfully. The script should ideally have a 100% pass rate.
Each test suite produces its own HTML log file containing the results of the test run. When examining the log file, the results at the bottom of the log should be checked first. This gives a summary of the number of tests in the suite that have passed or failed. In the body of the log, passing tests are highlighted in green and failing tests in red.
If a failing test has been identified, its reason for failing can be seen by examining each test block in the test, listed between the “START_TESTCASE ” and “END_TESTCASE ” tags. Starting and ending tags that is, “START_TEST_BLOCK ” and “END_TEST_BLOCK ” of a test block which passes will be reflected in green with text “Result = PASS ” after the “END_TEST_BLOCK ”. A failing test block will instead be in red or blue (depending on the reason of failure), its “END_TEST_BLOCK ” will be followed by “Result = FAIL ” and will refer to the reason of the failure (“ABORT ” in case of abort by timeout, “PANIC ” followed by a panic string and a code in case of an unexpected panic, etc.). If the reason of a failure is not displayed after the “END_TEST_BLOCK ” then it can be seen within instructions contained by the test block and/or in logs printed in black within the test block. Failed instruction will be displayed in red and will end with “Result = FAIL ”.
When this test suite is run on a real device, some of the tests may fail due to limitations of the device. This will be because of the device not having the correct hardware (for example, no modem exists for a test which needs one), or lacking the right software plug-ins to run the test. This can be spotted if the log displays that a test has failed with a -5 error (which means KErrNotSupported). This -5 error basically means that the test has failed because the functionality in this test is not supported on the device used. Therefore, any tests failing with -5 are not due to defects, but to the limitations of the device used.
Copyright ©2010 Nokia Corporation and/or its subsidiary(-ies).
All rights
reserved. Unless otherwise stated, these materials are provided under the terms of the Eclipse Public License
v1.0.