1 <?xml version="1.0" encoding="utf-8"?>

     2 <!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. -->

     3 <!-- This component and the accompanying materials are made available under the terms of the License 

     4 "Eclipse Public License v1.0" which accompanies this distribution, 

     5 and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". -->

     6 <!-- Initial Contributors:

     7     Nokia Corporation - initial contribution.

     8 Contributors: 

     9 -->

    10 <!DOCTYPE concept

    11   PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">

    12 <concept id="GUID-C824D5A1-3330-4293-9975-496D4D6F7D45" xml:lang="en"><title>Analyzing

    13 Test Results</title><abstract><p>After tests complete execution, result logs can be found on the

    14 system drive of the device in the <filepath>\logs\testexecute\</filepath> directory.

    15 On the PC, if tests are executed using testdriver, results can be found in

    16 the following location: </p> <p> <filepath>%EPOCROOT%\epoc32\TestDriver\results\</filepath> or </p> <p> <filepath>%EPOCROOT%\epoc32\winscw\c\logs\testexecute </filepath>  </p> <p>if executing the tests manually. </p> <p>When using testdriver,

    17 a new test result folder is created for each test run that is, results are

    18 not overwritten each time a test run is executed. A .htm log file can be found

    19 for each script contained in the suite, in its own subfolder. </p> <p>Each

    20 test case in the script will consists of test blocks (usually just one), which

    21 will pass or fail. Each test block will execute a number of commands. Each

    22 of the commands performs an action on the tested APIs (calling a certain function

    23 of tested classes). A test case passes if all its test blocks pass. A test

    24 block passes if all its commands and other instructions pass. </p> <p>The

    25 log will display the percentage of test cases that have passed successfully.

    26 The script should ideally have a 100% pass rate. </p></abstract><prolog><metadata><keywords/></metadata></prolog><conbody>

    27 <section><title>How to interpret test results</title>       <p>Each

    28 test suite produces its own HTML log file containing the results of the test

    29 run. When examining the log file, the results at the bottom of the log should

    30 be checked first. This gives a summary of the number of tests in the suite

    31 that have passed or failed. In the body of the log, passing tests are highlighted

    32 in green and failing tests in red. </p> <p>If a failing test has been identified,

    33 its reason for failing can be seen by examining each test block in the test,

    34 listed between the “<codeph>START_TESTCASE</codeph> ” and “<codeph>END_TESTCASE</codeph> ”

    35 tags. Starting and ending tags that is, “<codeph>START_TEST_BLOCK</codeph> ”

    36 and “<codeph>END_TEST_BLOCK</codeph> ” of a test block which passes will be

    37 reflected in green with text “<codeph>Result = PASS</codeph> ” after the “<codeph>END_TEST_BLOCK</codeph> ”.

    38 A failing test block will instead be in red or blue (depending on the reason

    39 of failure), its “<codeph>END_TEST_BLOCK</codeph> ” will be followed by “<codeph>Result

    40 = FAIL</codeph> ” and will refer to the reason of the failure (“<codeph>ABORT</codeph> ”

    41 in case of abort by timeout, “<codeph>PANIC</codeph> ” followed by a panic

    42 string and a code in case of an unexpected panic, etc.). If the reason of

    43 a failure is not displayed after the “<codeph>END_TEST_BLOCK</codeph> ” then

    44 it can be seen within instructions contained by the test block and/or in logs

    45 printed in black within the test block. Failed instruction will be displayed

    46 in red and will end with “<codeph>Result = FAIL</codeph> ”. </p> <p>When this

    47 test suite is run on a real device, some of the tests may fail due to limitations

    48 of the device. This will be because of the device not having the correct hardware

    49 (for example, no modem exists for a test which needs one), or lacking the

    50 right software plug-ins to run the test. This can be spotted if the log displays

    51 that a test has failed with a -5 error (which means <codeph>KErrNotSupported</codeph>).

    52 This -5 error basically means that the test has failed because the functionality

    53 in this test is not supported on the device used. Therefore, any tests failing

    54 with -5 are not due to defects, but to the limitations of the device used. </p> 

    55    </section>

    56 </conbody></concept>