F32 Test Technology Description

Scope

Components under test

The Base subsystem is the lowest layer in the Symbian platform software stack. It contains the core components, such as the kernel, user library, file server, and device drivers, for every supported hardware platform.

This suite tests the F32_EKA2 (file server) and FAT32 components.

Test types

The suite has three sub-suites: validation, performance and conformance.

Validation test suite

The Validation Test Suite provides a series of tests to verify that published APIs perform as described in the Symbian platform library. The tests are intended for regression and compatibility verification.

The validation suite uses Dual Driver tests to test the dual driver feature on the phone. DUAL Drive tests are configured to run on two drives of the same type having common interface.

Performance Test Suite

The Performance Test Suite is intended for assessing and benchmarking file server performance.

Conformance Test Suite

The FAT32 Conformance Test Suite ensures the file server's compliance with FAT32.

Test Environment

Device Setup

The device under test (a hardware platform, a phone or an emulator) must contain an appropriate Symbian platform build or licensee UI build.

PC Setup

The PC must have a Symbian Developer Kit (or licensee equivalent), an PDT and appropriate compilers to build the test code (for example Code Warrior, RVCT or GCCE).

Test Dependencies

The base test suite has no dependencies other than those common to all test suites.

Building and Execution

For general information about test suite structure refer to Symbian Verification Suite » TestDriver.

The following illustrates the F32 test suite structure:

<Component> - base
  <Validation suite> - validation
    <Sub-component> - f32
      <Manual tests> - f32-manual
        <RAM-drive specific tests> - ram-manual
          <Suite> - pbase-f32-sfsrv-publicapi-rem-manual-suite
       <Dual Drive Tests>- f32-dualdrive>
          <Suite> - pbase-f32-sfsrv-Publicapi-ram-DualDrive-suite
        <Automated tests> - f32-automated
          <Suite> - pbase-f32-sfsrv-publicapi-any-suite
          <System drive specific tests> - os
            <Suite> - pbase-f32-sfsrv-publicapi-os-suite
          <ROM-drive specific tests> - rom
            <Suite> - pbase-f32-publicapi-sfsrv-rom-suite
          <NAND-drive specific tests> - nand
            <Suite> - pbase-f32-sfsrv-publicapi-nand-suite
          <RAM-drive specific tests> - ram
            <Non-removable RAM-drive specific tests> - fixed
              <Suite> - pbase-f32-publicapi-sfsrv-ram-suite
            <Suite> - pbase-f32-publicapi-sfsrv-rem-suite
  <Performance suite> - performance
    <Sub-component> - f32
      <Scenario variant> - small
        <Suite> - pbase-f32-rfile-performance-small
        <Suite> - pbase-f32-rfs-performance-small
      <Scenario variant> - medium
        <Suite> - pbase-f32-rfile-performance-medium
        <Suite> - pbase-f32-rfs-performance-medium
      <Scenario variant> - large
        <Suite> - pbase-f32-rfile-performance-large
        <Suite> - pbase-f32-rfs-performance-large
  <Conformance suite> - conformance
    <Sub-component> - f32
      <Sub-component> - FAT32
        <Suite> - f32-fat32-conformance

This structure can be seen in the following location after building the test suite: %EPOCROOT%\epoc32\testdriver\testproduct.

The hierarchical test suite structure facilitates easy build and execution of the complete test suite or part of it. For example, if the path specified is:

• base: The entire suite is built/run.

• base.validation.f32: All the suites F32 are built/run.

• base.validation.f32.f32-automated: Only F32 automated test is built/run.

• base.validation.f32.f32-manual: Only F32 manual test is built/run.

• base.validation.f32.f32-dualdrive: Only f32 dual drive tests is built/run.

• base.validation.f32.f32-automated.ram: Only tests on RAM are built/run.

• base.validation.f32.f32-automated.ram.fixed: Only tests on non-removable RAM are built/run.

• base.validation.f32.f32-automated.rom: Only tests on ROM are built/run.

• base.validation.f32.f32-automated.os: Only OS suite is built/run.

• base.validation.f32.f32-automated.ram.fixed: Only tests on non-removable RAM are built/run.

General environment setup

1. Refer to specific sub-component test suite description in Base Verification Suite for details about configuration and building additional components before building the test suite.

2. The entire Base Test product can be built from ...\baseapitest\basesvs\group\.

3. Individual test components can be built from the respective group directories. For example, ...\baseapitest\basesvs\validation\f32\group\.

   bldmake bldfiles

   abld test build <WINSCW/ARMV5> <udeb/urel>

Steps to build and execute tests using TestDriver

Communication between the PC and the device under test (Symbian platform device) is enabled using STAT. So, the device under test must have STAT installed.

For WINSCW platform:

1. Configure TestDriver using the following script:

   ...\baseapitest\basesvs\group\testdriversetup.bat

2. Build the test code:

   testdriver build –p <winscw/armv5> –b <urel/udeb> –s base[.<specific test suite path>]

   Testdriver commands can be run from any location.

3. To execute the Tests :

   testdriver run

For ARMV5 platform:

Communication between PC and hardware:

The hardware reference board (for example H4) must be connected to the PC through USB or serial cable. The ROM image is built on the PC and transferred to the hardware board through MMC card.

1. Configure TestDriver using the following script:

   ...\baseapitest\basesvs\group\testdriversetup.bat

2. Build the test code:

   testdriver build –p <winscw/armv5> –b <urel/udeb> –s base[.<specific test suite path>]

   Testdriver commands can be run from any location.

3. Build the ROM image:

   buildrom -D_STARTUPMODE2 -D_EABI=ARMV5 -DRVCT -DUSE_STRONG_CRYPTOGRAPHY <h2/h4hrp> techview_statapi td_base

   Note The ROM building commands are different for NAND images and enabling Dual Drive.

4. Transfer the ROM to the device and start-up.

5. Execute the Tests:

   testdriver run -t <transport service>

   Transport service may be “serial<COM port number that is connected to the board>”, “bt<COM port number for BT dongle>” or “usb<mapped port number>”

Steps to build and execute tests manually (Tests built into ROM)

For WINSCW platform:

1. Select your Setup Script:

   For example setup-PBASE-F32-Sfsrv-PublicApi.script is used to run only the F32_EKA2 validation test suite.

2. Change directory to:

   %EPOCROOT%\epoc32\release\winscw\<urel/udeb>

3. Execute the Tests:

   testexecute z:\base\setup_<test suite name>.script

   setup_t_base.script is used to copy the test data files on to c:\drive.

   testexecute c:\base\<test suite name>.script

   t_base.script is used to run the entire base test suite.

For ARMV5 platform:

1. Build ROM image:

   buildrom -D_EABI=ARMV5 –DRVCT -DUSE_STRONG_CRYPTOGRAPHY <h2/h4hrp> techview t_base

   The ROM image is built on the PC and transferred to the hardware board through MMC card. Note: The ROM building commands is different for NAND images. Refer to Base Verification Suite for more details.

2. Select your Setup Script:

   The setup-PBASE-F32-Sfsrv-PublicApi.script is used to run only F32_EKA2 validation test suite.

3. Running Scripts:

   Open eshell

4. Execute the Tests:

   testexecute z:\base\setup_<test suite name>.script

   setup_t_base.script is used to copy the test data files on to c:\drive.

   testexecute c:\base\<test suite name>.script

   t_base.script is used to run the entire base test suite.

Steps to build and execute tests on phone directly using .pkg file

For ARMV5 platform:

1. EShell and TestExecute:

   The phone must have eshell and testexecute installed.

2. Certificate files:

   Ensure .cer and .key files are placed into /epoc32/pkg/ directory on the PC and rename aszeus.cer and SymbianACS.key correspondingly.

3. Create and install SIS-file:

   Run makesisfiles.bat from /epoc32/pkg/ directory.

   makesisfiles.bat

   This creates t_sfsrv.sis file. The file must be copied to an MMC card, transferred to the phone and executed.

   When the t_sfsrv.sis file is run, the base test suite is installed on the device.

4. Running Scripts:

   Run the test scripts. For more information refer to Steps to build and execute tests manually (Tests built into ROM).

Selective Execution of Test Cases using TCS File

Test cases can be excluded from a test run by using a .tcs file. The .tcs file contains IDs of test cases to be excluded.

The test case IDs can be specified as a range, a list, or a combination of both. Each range or individual test case ID is delimited by a new-line character.

For example, range of test case IDs to be excluded can be specified as:

pbase-f32-rfile-performance-large-0001: pbase-f32-rfile-performance-large-0005

This excludes all test cases from 1 to 5 (1 and 5 included) during execution.

A list of test cases IDs to be excluded can be:

COMINF-ESOCK-RSubConnection-PublicAPI-0001

COMINF-ESOCK-RSubConnection-PublicAPI-0005

COMINF-ESOCK-RSubConnection-PublicAPI-0007

A combination of range and list can be:

pbase-f32-rfile-performance-large-0001: pbase-f32-rfile-performance-large-0005

pbase-f32-rfile-performance-large-0010: pbase-f32-rfile-performance-large-0015

pbase-f32-rfile-performance-large-0020

pbase-f32-rfile-performance-large-0021

There is only one .tcs file located at …\baseapitest\basesvs\config\t_base.tcs

For more information on location of .tcs files, see TCS File Source Location in F32 Performance Test Suite, TCS File Source Location in FAT32 Conformance Test Suite, and TCS File Source Location in F32_EKA2 Validation Test Suite.

The .tcs file must be modified when using the TestDriver for test execution. When executing the tests manually, the following testexecute command illustrates the usage of the .tcs file:

testexecute %EPOCROOT%\epoc32\release\winscw\<urel/udeb>\<test suite name>.script –[tcx|tci] %EPOCROOT%\epoc32\release\winscw\<urel/udeb>\t_base.tcs

The -tcx switch is used to execute all the tests in the specified script excluding the test cases specified in the .tcs file. The –tci switch is used to execute only the tests specified in the .tcs file from the specified script.

Test Results

Test Results log file location

After tests complete execution, result logs can be found on the system drive of the device in the \logs\testexecute\ directory.

On the PC, if tests are run using testdriver, results can be found in the following location: %EPOCROOT%\epoc32\TestDriver\results\. If executing the tests manually the results can be found in the following location: %EPOCROOT%\epoc32\winscw\c\logs\testexecute

When using testdriver, a new test result folder is created for each test run that is, results are not overwritten each time a test run is run. A .htm log file for each script contained in the suite can be found in its own subfolder.

Each test case in the script consists of test blocks (usually just one), which either pass or fail. Each test block runs some of the commands. Each of the commands performs an action on the tested APIs. A test block passes if all its commands and other instructions pass. A test case passes if all its test blocks pass.

The log displays the percentage of test cases that have passed successfully.

How to interpret test results

Each test suite produces its own HTML log file containing the results of the test run. When examining the log file, the results at the bottom of the log should be checked first. This gives a summary of the tests in the suite that have passed, failed or panicked.

In the body of the log tests which passed are highlighted in green, tests which failed in red and tests which caused a panic in blue. The reason for any failure or panic is also given.

When this test suite is run on a real device some tests may fail due to limitations of the device. Either the device does not have the correct hardware (for example, no modem exists for a test which needs one) or it lacks the right software plug-ins to run the test. In such cases the log displays that a test has failed with a -5 error (which means KErrNotSupported). Tests appearing to fail with -5 do not indicate defects, but limitations of the device.

Refer to Symbian Verification Suite » TestDriver » Using TestDriver » Results » Viewing the test results for more information on test driver results.

OS Focussed Components

The following tables illustrate the components and classes tested on each Symbian platform baseline:

Table 1: Validation testing of Base, F32_EKA2 component on each Symbian platform baseline

Table 2: Performance testing of Base, F32_EKA2 on each Symbian platform baseline

The FAT32 conformance test suite is not API specific and is available on Symbian platform versions 9.4 and 9.5.