<?xml version="1.0" encoding="utf-8"?>

<!-- Copyright (c) 2007-2010 Nokia Corporation and/or its subsidiary(-ies) All rights reserved. -->

<!-- This component and the accompanying materials are made available under the terms of the License 

"Eclipse Public License v1.0" which accompanies this distribution, 

and is available at the URL "http://www.eclipse.org/legal/epl-v10.html". -->

<!-- Initial Contributors:

    Nokia Corporation - initial contribution.

Contributors: 

-->

<!DOCTYPE concept

  PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">

<concept id="GUID-C824D5A1-3330-4293-9975-496D4D6F7D45" xml:lang="en"><title>Analyzing

Test Results</title><abstract><p>After tests complete execution, result logs can be found on the

system drive of the device in the <filepath>\logs\testexecute\</filepath> directory.

On the PC, if tests are executed using testdriver, results can be found in

the following location: </p> <p> <filepath>%EPOCROOT%\epoc32\TestDriver\results\</filepath> or </p> <p> <filepath>%EPOCROOT%\epoc32\winscw\c\logs\testexecute </filepath>  </p> <p>if executing the tests manually. </p> <p>When using testdriver,

a new test result folder is created for each test run that is, results are

not overwritten each time a test run is executed. A .htm log file can be found

for each script contained in the suite, in its own subfolder. </p> <p>Each

test case in the script will consists of test blocks (usually just one), which

will pass or fail. Each test block will execute a number of commands. Each

of the commands performs an action on the tested APIs (calling a certain function

of tested classes). A test case passes if all its test blocks pass. A test

block passes if all its commands and other instructions pass. </p> <p>The

log will display the percentage of test cases that have passed successfully.

The script should ideally have a 100% pass rate. </p></abstract><prolog><metadata><keywords/></metadata></prolog><conbody>

<section><title>How to interpret test results</title>       <p>Each

test suite produces its own HTML log file containing the results of the test

run. When examining the log file, the results at the bottom of the log should

be checked first. This gives a summary of the number of tests in the suite

that have passed or failed. In the body of the log, passing tests are highlighted

in green and failing tests in red. </p> <p>If a failing test has been identified,

its reason for failing can be seen by examining each test block in the test,

listed between the “<codeph>START_TESTCASE</codeph> ” and “<codeph>END_TESTCASE</codeph> ”

tags. Starting and ending tags that is, “<codeph>START_TEST_BLOCK</codeph> ”

and “<codeph>END_TEST_BLOCK</codeph> ” of a test block which passes will be

reflected in green with text “<codeph>Result = PASS</codeph> ” after the “<codeph>END_TEST_BLOCK</codeph> ”.

A failing test block will instead be in red or blue (depending on the reason

of failure), its “<codeph>END_TEST_BLOCK</codeph> ” will be followed by “<codeph>Result

= FAIL</codeph> ” and will refer to the reason of the failure (“<codeph>ABORT</codeph> ”

in case of abort by timeout, “<codeph>PANIC</codeph> ” followed by a panic

string and a code in case of an unexpected panic, etc.). If the reason of

a failure is not displayed after the “<codeph>END_TEST_BLOCK</codeph> ” then

it can be seen within instructions contained by the test block and/or in logs

printed in black within the test block. Failed instruction will be displayed

in red and will end with “<codeph>Result = FAIL</codeph> ”. </p> <p>When this

test suite is run on a real device, some of the tests may fail due to limitations

of the device. This will be because of the device not having the correct hardware

(for example, no modem exists for a test which needs one), or lacking the

right software plug-ins to run the test. This can be spotted if the log displays

that a test has failed with a -5 error (which means <codeph>KErrNotSupported</codeph>).

This -5 error basically means that the test has failed because the functionality

in this test is not supported on the device used. Therefore, any tests failing

with -5 are not due to defects, but to the limitations of the device used. </p> 

   </section>