Multimedia test technology

Multimedia test technology area scope

Components under test

The Multimedia test suites test the following components under the Multimedia technology area:

• Camera functionality:

  ECam is an API which allows an application to access and control any camera hardware which is attached to the device.

• Image conversion library (ICL):

  ICL provides a lightweight Image Conversion Library which supports the decoding and encoding of image files.

• Multimedia Framework (MMF):

  MMF including audio and video recording and playback, tone playback and MIDI playback.

  • Controller framework:

    Some of the client-side functionality is implemented as part of the framework. Base classes are provided to help device creators implement plugins.

  • Audio:

    • Audio playback utility

    • Audio recording utility

    • Audio conversion utility

    • Tone playback utility

  • Video:

    Video player utility - APIs for video playback and recording.

  • MIDI:

    MIDI Client Utility - APIs for MIDI playback and recording.

Test scope

Published API testing

This technology area provides a number of test suites to test published APIs. These test suites can be used as a regression test suite and also as a part of compatibility testing program.

For more details about the individual classes, see API test suites for Multimedia.

Test omissions

The Multimedia test suites test only the common audio, video, and image codecs that the Symbian platform implements.

For more details about the individual classes, see API test suites for Multimedia.

Multimedia test technology environment

Device under test setup

The device under test (hardware platform/phone/emulator) should have an appropriate Symbian platform or device creator UI build (S60, UIQ, MOAP for example).

PC setup

The PC should have a Symbian or equivalent device creator developer kit, PDT and relevant compilers (Code Warrior, RVCT, GCCE) installed to build and execute the test code.

Network environment setup

None

Test dependencies

Note: Any mention of “epoc32” is offset by %EPOCROOT%. That is, the full path where the EPOC tree is installed.

Test data

The test data for each test suite is in the following location:

• …\os\mm\mmapitest\mmsvs\suite\ecam\T_Camera\testdata

• …\os\mm\mmapitest\mmsvs\suite\icl\T_ImageEncoder\testdata

• …\os\mm\mmapitest\mmsvs\suite\icl\T_ImageDecoder\testdata

• …\os\mm\mmapitest\mmsvs\suite\mmf\T_MdaAudioConvertUtility\testdata

• …\os\mm\mmapitest\mmsvs\suite\mmf\T_MdaAudioInputStream\testdata

• …\os\mm\mmapitest\mmsvs\suite\mmf\T_MdaAudioOutputStream\testdata

• …\os\mm\mmapitest\mmsvs\suite\mmf\T_MdaAudioPlayerUtility\testdata

• …\os\mm\mmapitest\mmsvs\suite\mmf\T_MdaAudioRecorderUtility\testdata

• …\os\mm\mmapitest\mmsvs\suite\mmf\T_MdaAudioToneUtility\testdata

• …\os\mm\mmapitest\mmsvs\suite\mmf\T_MidiClientUtility\testdata

• …\os\mm\mmapitest\mmsvs\suite\mmf\T_VideoPlayerUtility\testdata

• …\os\mm\mmapitest\mmsvs\suite\mmf\T_VideoRecorderUtility\testdata

For more details about the individual classes, see API test suites for Multimedia.

Also, there are two global environment files, t_multimedia.ini and t_multimedia_location.ini, which contain configurable parameters and their default values. Test that are device setup specific have these details defined in t_multimedia.ini. Test that are dependent on location have these details defined in t_multimedia_location.ini.

Your can configure these parameters by modifying their default values in these files. Details regarding each parameter can be found in the file comments. These files are in the following location:

…\os\mm\mmapitest\mmsvs\suite\testdata

For more details about the individual classes, see API test suites for Multimedia.

Some of the test scripts use an audio file as a seed. As each test case runs the test data is converted into the test data format required for input to the subsequent test case. The approach is taken to help reduce the size on disk that the test data occupies and speed-up transfer of data to device under test.

Testing with corrupt files is difficult, as they are reported by most virus checker applications as viruses, when saved on the PC. To get around this the files are stored in the source tree as complete files, modified during the test cases on the device under test and cleaned up afterwards. Such files can still potentially cause problems while executing tests on the emulator if the test run is terminated before the test data has been cleaned up.

Building and Execution

The following structure illustrates the Multimedia test suite hierarchical structure after the build process. This structure can be seen in the following location after building the test suite:

%EPOCROOT%\epoc32\testdriver\testproduct

For more information on for general information about testdriver test suite structure, see Symbian Verification Suite » TestDriver V2 » TestDriver Output » Viewing the test results.

<Component> - Multimedia
            <Sub-component> - ICL
                    <suite> - MM-ICL-DECDE-PublicApi-suite
                    <suite> - MM-ICL-ENCDE-PublicApi-suite
            <Sub-component> - MMF
                    <suite> - MM-MMF-ACLNT-CNVRT-PublicAPI-suite
                    <suite> - MM-MMF-ACLNT-INPT-PublicAPI-suite
                    <suite> - MM-MMF-ACLNT-OUTPT-PublicAPI-suite
                    <suite> - MM-MMF-ACLNT-PLYR-PublicAPI-suite
                    <suite> - MM-MMF-ACLNT-RCRDR-PublicAPI-suite
                    <suite> - MM-MMF-ACLNT-TONE-PublicAPI-suite
                    <suite> - MM-MMF-MIDI-PublicAPI-suite
                    <suite> - MM-MMF-VCLNT-PLYR-PublicAPI-suite
                    <suite> - MM-MMF-VCLNT-RCRDR-PublicAPI-suite
            <Sub-component> - ECAM
                    <suite> - MM-ECM-PublicApi-suite
                    <suite> - MM-MMF-VCLNT-PLYR-PublicAPI-suite
                    <suite> - MM-MMF-VCLNT-RCRDR-PublicAPI-suite

The hierarchical test suite structure facilitates easy build and execution of the complete test suite or part of it. For example, if the path specified is:

• <multimedia> - The entire suite is built/executed.

• <multimedia.audio> - All the six suites under audio are built/executed.

• <multimedia.audio.MM-MMF-ACLNT-CNVRT-PublicAPI-suite> - MM-MMF-ACLNT-CNVRT-PublicAPI-suite is built/executed.

The following list show the sequence of commands to be executed on emulator and hardware:

• General environment setup for WINSCW and ARMV5

• Steps to build and execute tests using test driver

• Steps to build and execute tests manually (tests built into ROM)

General environment setup for WINSCW and ARMV5

1. Ensure that all the parameters in the files t_multimedia.ini and t_multimedia_location.ini are set to the correct values. For more details, see Test data.

2. Build additional components. For details regarding downloading and building additional components before building the test suite, see the specific sub-component test suite in API test suites for Multimedia.

3. The entire Multimedia test component can be built from …\os\mm\mmapitest\mmsvs\suite\group\….

   Individual test components can be built from the respective group directory:

   • …\os\mm\mmapitest\mmsvs\suite\ecam\t_<sub-component>\group

   • …\os\mm\mmapitest\mmsvs\suite\icl\t_<sub-component>\group

   • …\os\mm\mmapitest\mmsvs\suite\mmf\t_<sub-component>\group

   Execute the following commands:

   1. bldmake bldfiles

   2. abld test build [winscw|armv5] [urel|udeb]

Steps to build and execute tests using test driver

On WINSCW platform

1. Communication between the PC and the device under test (Symbian platform device) is enabled using STAT. Hence, the device under test should have STAT installed.

2. TestDriver can be configured using the automated scripts located at, …\os\mm\mmapitest\mmsvs\suite\group\… using the following command:

   testdriversetup

   testdriversetup.bat calls testdriversetup.pl, which applies appropriate set of configuration parameters.

3. Build test code using the following command:

   testdriver build [–p [winscw|armv5]] [–b [udeb|urel]] –s multimedia[.audio|image|video[.<test suite name >]]

   Note: You can execute TestDriver commands from any location.

   For the multimedia test suite hierarchical structure, refer to Building and Execution section.

4. Execute the tests on the emulator using the following command:

   testdriver run

Note: All scripts and test data are automatically removed from the system drive of the device under test after the execution.

On ARMV5 platform

1. Communication between the PC and the device under test (Symbian platform device) is enabled using STAT. Hence, the device under test should have STAT installed.

2. Communication between PC and hardware. For more details, refer to Symbian Verification Suite » TestDriver » Using TestDriver » Communicating with the device. The ROM image is built on the PC and transferred to the hardware board via MMC card.

3. TestDriver can be configured using the automated scripts located at, …\os\mm\mmapitest\mmsvs\suite\group\… using the following command:

   testdriversetup

   testdriversetup.bat calls testdriversetup.pl, which applies appropriate set of configuration parameters.

4. Build test code using the following command:

   testdriver build [–p [winscw|armv5]] [–b [udeb|urel]] –s multimedia[.mmf|icl|ecam[.<test suite name >]]

   Note: You can execute TestDriver commands from any location.

   For the multimedia test suite hierarchical structure, refer to Building and Execution section.

5. Build the ROM image using the following command:

   buildrom -D_STARTUPMODE2 -D_EABI=ARMV5 -DRVCT -DUSE_STRONG_CRYPTOGRAPHY -D_CUSTOM_COMMSDAT <h2/h4hrp> techview_statapi td_multimedia

6. Execute the tests on the device using the following command:

   testdriver run -t <transport service used>

   The transport service used can be one of the following:

   • serial<COM port number that is connected to the board>

   • bt<COM port number for BT dongle>

   • usb<mapped port number>

   For more details, refer to Symbian Verification Suite » TestDriver » Using TestDriver » Communicating with the device.

Note: All scripts and test data are automatically removed from the system drive of the device under test after the execution.

Steps to build and execute tests manually (tests built into ROM)

On WINSCW platform

1. The following setup scripts are used to execute tests:

   • Execute the entire multimedia test suite using the setup script, setup_t_multimedia.script

   • Execute the test suite for a specific class using one of the following setup scripts:

     • setup-MM-ECM-PublicApi.script

     • setup-MM-ICL-DECDE-PublicApi.script

     • setup-MM-ICL-ENCDE-PublicApi.script

     • setup-MM-MMF-ACLNT-CNVRT-PublicAPI.script

     • setup-MM-MMF-ACLNT-INPT-PublicAPI.script

     • setup-MM-MMF-ACLNT-OUTPT-PublicAPI.script

     • setup-MM-MMF-ACLNT-PLYR-PublicAPI.script

     • setup-MM-MMF-ACLNT-RCRDR-PublicAPI.script

     • setup-MM-MMF-ACLNT-TONE-PublicAPI.script

     • setup-MM-MMF-MIDI-PublicAPI.script

     • setup-MM-MMF-VCLNT-PLYR-PublicAPI.script

     • setup-MM-MMF-VCLNT-RCRDR-PublicAPI.script

2. Run scripts on the emulator by changing the directory to, %EPOCROOT%\epoc32\release\winscw\<urel/udeb>.

3. Execute tests using the following commands:

   testexecute z:\multimedia\setup_<test suite name>.script

   setup_t_multimedia.script is used to execute the entire multimedia setup script.

   testexecute c:\multimedia\<test suite name>.script

   t_multimedia.script can be used to execute the entire multimedia test suite. The scripts mentioned in the t_multimedia.script can be run individually.

On ARMV5 platform

1. Build the ROM image using the following command:

   buildrom -D_EABI=ARMV5 –DRVCT -DUSE_STRONG_CRYPTOGRAPHY -D_CUSTOM_COMMSDAT <h2/h4hrp> techview t_multimedia

2. The following setup scripts are used to execute tests:

   • Execute the entire multimedia test suite using the setup script, setup_t_multimedia.script

   • Execute the test suite for a specific class using one of the following setup scripts:

     • setup-MM-ECM-PublicApi.script

     • setup-MM-ICL-DECDE-PublicApi.script

     • setup-MM-ICL-ENCDE-PublicApi.script

     • setup-MM-MMF-ACLNT-CNVRT-PublicAPI.script

     • setup-MM-MMF-ACLNT-INPT-PublicAPI.script

     • setup-MM-MMF-ACLNT-OUTPT-PublicAPI.script

     • setup-MM-MMF-ACLNT-PLYR-PublicAPI.script

     • setup-MM-MMF-ACLNT-RCRDR-PublicAPI.script

     • setup-MM-MMF-ACLNT-TONE-PublicAPI.script

     • setup-MM-MMF-MIDI-PublicAPI.script

     • setup-MM-MMF-VCLNT-PLYR-PublicAPI.script

     • setup-MM-MMF-VCLNT-RCRDR-PublicAPI.script

3. Run scripts on the device using eshell.

4. Execute tests using the following commands:

   testexecute z:\multimedia\setup_<test suite name>.script

   setup_t_multimedia.script is used to execute the entire multimedia setup script.

   testexecute c:\multimedia\<test suite name>.script

   t_multimedia.script can be used to execute the entire multimedia test suite. The scripts mentioned in the t_multimedia.script can be run individually.

Steps to build and execute tests on phone directly using .pkg file

For ARMV5 platform:

1. EShell and TestExecute:

   The phone must have eshell and testexecute installed.

2. Certificate files:

   Ensure .cer and .key files are placed into /epoc32/pkg/ directory on the PC and renamed aszeus.cer and SymbianACS.key correspondingly.

3. Create and install SIS-file:

   Execute makesisfiles.bat from /epoc32/pkg/ directory.

   makesisfiles.bat

   This creates files.

   • t_camera.sis

   • t_imagedecoder.sis

   • t_imageencoder.sis

   • t_mdaaudioconvertutility.sis

   • t_mdaaudioinputstream.sis

   • t_mdaaudiooutputstream.sis

   • t_mdaaudioplayerutility.sis

   • t_mdaaudiorecorderutility.sis

   • t_mdaaudiotoneutility.sis

   • t_midiclientutility.sis

   • t_videoplayerutility.sis

   • t_videorecorderutility.sis

   The files must be copied to an MMC card, transferred to the phone and executed.

   When the above .sis files are executed, all multimedia test suites are installed on the device.

4. Running Scripts:

   Run the test scripts. For more information refer to Steps to build and execute tests manually (tests built into ROM) .

Excluding Test Cases from Test Run using TCS File

Test cases can be executed selectively using a .tcs file. The .tcs file contains the IDs of test cases:

• to be excluded from a test run

• to be executed on their own. Therefore, excluding all other test cases in the test script from a test run.

The test case IDs can be specified as a range, a list, or a combination of both, where each range or individual test case ID is delimited by a new-line character.

For example, range of test case IDs to be excluded can be specified as:

MM-ICL-DECDE-PublicApi-0003: MM-ICL-DECDE-PublicApi-0012

Test cases from 3 to 12 are (3 and 12 included) either excluded or are the only ones to be executed during a test run.

A list of test cases IDs to be excluded can be:

MM-ICL-ENCDE-PublicApi-0007

MM-ICL-ENCDE-PublicApi-0010

MM-ICL-ENCDE-PublicApi-0012

A combination of range and list can be:

MM-MMF-ACLNT-CNVRT-PublicAPI-0005

MM-MMF-ACLNT-CNVRT-PublicAPI-0008

MM-MMF-ACLNT-CNVRT-PublicAPI-0010: MM-MMF-ACLNT-CNVRT-PublicAPI-0015

There is only one .tcs file, which is located at the following location:

…\os\mm\mmapitest\mmsvs\suite\config\t_multimedia.tcs

For more information on location of .tcs files, see t_camera TCS files source location, , image conversion TCS files source location, and multimedia framework TCS files source location.

Only the .tcs files has to be modified when using the test driver for test execution. When executing the tests manually, the following testexecute command illustrates the usage of the .tcs file:

testexecute %EPOCROOT%\epoc32\release\winscw\<urel/udeb>\<test suite name>.script –[tcx|tci] %EPOCROOT%\epoc32\release\winscw\<urel/udeb>\t_multimedia.tcs

The -tcx switch is used to execute all the tests in the specified script excluding the test cases specified in the .tcs file. While the –tci switch is used to execute only the tests specified in the .tcs file from the specified script.

Test Results

Test Results log file location

After tests complete execution, result logs can be found on the system drive of the device in the \logs\testexecute\directory. On the PC, if tests are executed using test driver, results can be in any of the following locations:

• %EPOCROOT%\epoc32\TestDriver\results\

• %EPOCROOT%\epoc32\winscw\c\logs\testexecute

If the tests are manually run, when using test driver, a new test result folder is created for each test run. That is, results are not overwritten each time a test run is executed. In its own subfolder, an .html log file can be found for each script contained in the suite.

Each test case in the script contains one test block, which performs a number of actions on an API. The log will display the percentage of test cases that have passed successfully. The script should ideally have a 100% pass rate.

How to interpret test results

Each test suite produces its own HTML log file containing the results of the test run. When examining the log file, the results at the bottom of the log should be checked first. This gives a summary of the number of tests in the suite that have passed, failed or panicked. In the body of the log, passing tests are highlighted in green, and failing tests in red, while tests which cause a panic are shown in blue.

If a failing or panicking test has been identified, its reason for failing or panicking can be seen by examining each step in the test, listed between the “START_TESTCASE” and “END_TESTCASE” tags. The steps which passes have the word “INFO” followed by some information regarding the step. A failing step will instead have the word “ERROR”, followed by an explanation of the error (usually with an error code). If a panic occurs there may be no textual explanation of the problem, but a panic code will be given (within the blue highlighted text).

When this test suite is run on a real device, some of the tests may fail due to limitations of the device. This happens because of the device not having the correct hardware (for example, no modem exists for a test which needs one), or lacking the right software plugins to run the test. This can be spotted if the log displays that a test has failed with a -5 error (which means KErrNotSupported). The -5 error basically means that the test has failed because the functionality in this test is not supported on the device used. Therefore, any tests failing with -5 are not due to defects, but to the limitations of the device used.

For more information on test driver results, see R4Error! Reference source not found in Symbian Verification Suite » TestDriver V2 » TestDriver Output » Viewing the test results.

OS Focussed Components

The following table illustrates the sub-components and classes available on each Symbian platform.

Table 1: Component Multimedia sub-components and classes tested on each Symbian platform baseline