Graphics Test Technology

Graphics test technology area scope

The test suites test the following components in the graphics technology area:

• The Font and Bitmap Server (FBSERV ) API, which allows an application to access and control fonts and bitmaps on the device.

• The Window Server (WSERV) API, which allows an application to run on more than one screen and specifies on which screen of the device it is to be viewed.

• BitGDI. BitGDI provides a concrete implementation of the graphics context class and some graphics devices classes.

• The Font Store (FntStore), which contains the typeface store.

The scope of testing is as follows:

• Published API testing

  The test suites test published APIs. The test suites can be used as a regression test suite and also as part of a compatibility testing program.

• Test omissions

  CClickMaker, CAnim, CAnimDll, CAnimGc, CFreeTimerWindowAnim, CSpriteAnim and CWindowAnim are omitted because they are implemented by the device creator.

Graphics test technology environment

Device setup

The device under test (for example, a hardware platform, phone or emulator) must have an appropriate build of the Symbian platform or a Symbian UI (S60, UIQ or MOAP).

PC setup

To build and run the test code, the PC must have installed a Symbian platform or application developer kit or equivalent, the Symbian Product Developer Toolkit (PDT) and relevant compilers (such as RVCT or GCCE).

Network environment setup

None.

Test dependencies

Test data for each test suite can be found in the following location:

• ...\graphics\graphicsapitest\graphicssvs\fbserv\testdata

• ...\graphics\graphicsapitest\graphicssvs\wserv\testdata

• ...\graphics\graphicsapitest\graphicssvs\bitgdi\testdata

• ...\graphics\graphicsapitest\graphicssvs\fntstore\testdata

For links to information about each test suite, see Graphics Verification Suite Component.

There is a global environment file, t_graphics.ini, which contains configurable parameters and their default values. You can configure these parameters by modifying the default values. Details about each parameter can be found in the file comments. These files can be found in the following location:

...\graphics\graphicsapitest\graphicssvs\testdata\<armv5|winscw>

When using Test Driver, communication between the PC and the Symbian platform device under test is enabled using STAT. Therefore, the device under test must have STAT installed. For information on the STAT tool, see Symbian Verification Suite > TestDriver > Using TestDriver > Communicating with the device > STAT plug-in.

Building and execution

The following illustrates the Graphics Test Suite hierarchical structure after the build process. This structure can be seen in %EPOCROOT%\epoc32\testdriver\testproduct after building the test suite.

Note: For information about the TestDriver output, see Symbian Verification Suite > TestDriver > Using TestDriver > Results > Viewing the test results.

<Component> - Graphics 
  <Sub-component> - Fbserv
    <suite>-graphics-fbserv-bitmaputil-publicapi-suite
    <suite>-graphics-fbserv-fbsbitmap-publicapi-suite
    <suite>-graphics-fbserv-fbscolor256bitmaputil-publicapi-suite
    <suite>-graphics-fbserv-fbsfont-publicapi-suite
    <suite>-graphics-fbserv-fbssession-publicapi-suite
    <suite>-graphics-fbserv-fbstypefacestore-publicapi-suite
    <Sub-component>-Wserv
    <suite>-graphics-wserv-wsscreendevice-publicapi-suite
    <suite>-graphics-wserv-wssprite-publicapi-suite
    <suite>-graphics-wserv-wspointercursor-publicapi-suite
    <suite>-graphics-wserv-frame-publicapi-suite
    <suite>-graphics-wserv-anim-publicapi-suite
    <suite>-graphics-wserv-animdll-publicapi-suite
    <suite>-graphics-wserv-animgfwe-publicapi-suite
    <suite>-graphics-wserv-animwindowfunctions-publicapi-suite
    <suite>-graphics-wserv-animgeneralfunctions-publicapi-suite
    <suite>-graphics-wserv-animspritefunctions-publicapi-suite
    <suite>-graphics-wserv-events-publicapi-suite
    <suite>-graphics-wserv-backedupwindow-publicapi-suite
    <suite>-graphics-wserv-blankwindow-publicapi-suite
    <suite>-graphics-wserv-windowgc-publicapi-suite
    <suite>-graphics-wserv-windowgroup-publicapi-suite
    <suite>-graphics-wserv-wsbitmap-publicapi-suite
    <suite>-graphics-wserv-wsgraphicmsgbuf-publicapi-suite
    <suite>-graphics-wserv-wsgraphicbitmap-publicapi-suite
    <suite>-graphics-wserv-session-publicapi-suite
    <suite>-graphics-wserv-soundplugin-publicapi-suite
    <suite>-graphics-wserv-directscreenaccess-publicapi-suite
    <suite>-graphics-wserv-dsasession-publicapi-suite
    <suite>-graphics-wserv-wsgraphicbitmapanimation-publicapi-suite
    <suite>-graphics-wserv-window-publicapi-suite
  <Sub-component>-Bitgdi
    <suite>-graphics-bitgdi-fbsbitmapdevice-publicapi-suite
    <suite>-graphics-bitgdi-fbsscreendevice-publicapi-suite
    <suite>-graphics-bitgdi-fbsbitgc-publicapi-suite
    <suite>-graphics-bitgdi-polygonfiller-publicapi-suite
    <suite>-graphics-bitgdi-FbsBitGcFont-publicapi-suite
    <suite>-graphics-bitgdi-FbsBitGcBitmap-publicapi-suite
    <suite>-graphics-bitgdi-FbsDevice-publicapi-suite
  <Sub-component>-FntStore
    <suite>-graphics-fntstore-fontstore-publicapi-suite
    <suite>-graphics-fntstore-bitmapfont-publicapi-suite
    <suite>-graphics-fntstore-OpenFontRasterizerContext-publicapi-suite

The hierarchical test suite structure facilitates easy build and execution of the complete test suite or part of it. For example, if the path specified is:

• <graphics.panicless> the entire graphics test suite is built with the exception of panicking test cases. For more information, see Test environment and execution in T_FBServAPI Regression Test Suite.

• <graphics.panic> only the panicking test cases are built in the entire Graphics test suite. For more information, see Test environment and execution in T_FBServAPI Regression Test Suite.

• <graphics.[panic|panicless].fbserv> all the six FBSERV suites are built and run.

• <graphics.[panic|panicless].fbserv.graphics-fbserv-bitmaputil-publicapi-suite > the graphics-fbserv-bitmaputil-publicapi-suite is built and run.

General environment setup

The following shows the general environment setup for both WINSCW and ARMV5.

• Ensure that all the parameters in this t_graphics.ini file are set to the correct values. For more information on test data, refer to the Graphics test technology environment section above.

• The entire Graphics test component can be built from the following location:

  ...\graphics\graphicsapitest\graphicssvs\group\...

• Individual test components can be built from the respective group directory:

  • ...\graphics\graphicsapitest\graphicssvs\fbserv\group

  • ...\graphics\graphicsapitest\graphicssvs\wserv\group

  • ...\graphics\graphicsapitest\graphicssvs\bitgdi\group

  • ...\graphics\graphicsapitest\graphicssvs\fntstore\group

• Run the following commands:

  bldmake bldfiles

  abld test build [winscw|armv5] [urel|udeb]

Steps to build and run tests using TestDriver

Communication between the PC and the Symbian platform device under test is enabled using STAT. Therefore, the device under test must have STAT installed. See Symbian Verification Suite > TestDriver > Using TestDriver > Communicating with the device > STAT plug-in for more information.

Use the following steps to build and run the tests:

• For ARMV5:

  The ROM image is built on the PC and transferred to the hardware board through the MMC card. See the STAT plug-in documentation for details on the various methods of communication between the PC and hardware.

• Configure the test driver using the automated scripts of the testdriversetup.bat, which can be found in \graphicsapitest\graphicssvs\group\....

  testdriversetup.bat calls testdriversetup.pl, which applies an appropriate set of configuration parameters.

• Build the test code:

  To build the test code, perform the following steps:

  1. Navigate to the \graphicsapitest\graphicssvs\group directory and run the following commands:

     bldmake bldfiles

     abld test export

  2. Run the following testdriver command, which can be executed from any location:

     testdriver build [–p [winscw|armv5]] [–b [udeb|urel]] –s graphics.[panic|panicless].[fbserv|wserv[.<test suite name >]]

• Build the ROM image:

  • For ARMV5:

    buildrom -D_STARTUPMODE2 -D_EABI=ARMV5 -DRVCT -DUSE_STRONG_CRYPTOGRAPHY <h2|h4hrp> techview_statapi td_graphics

• Run the test: To run the test on an emulator (WINSCW), run the following command:

  testdriver run

  To run the test on hardware (ARMV5), run the following command:

  testdriver run -t <transport service used>

  The transport service can be one of the following:

  • serial<COM port number that is connected to the board>.

  • bt<COM port number for BT dongle>.

  • usb<mapped port number>.

Refer to Symbian Verification Suite > TestDriver > Using TestDriver > Communicating with the device > STAT plug-in for more information.

Note: All scripts and test data are automatically removed from the system drive of the device under test after execution.

Steps to build and run tests manually (tests built into ROM)

The steps to build and run tests manually are as follows:

• Build the ROM:

  • For ARMV5:

    buildrom -D_EABI=ARMV5 –DRVCT -DUSE_STRONG_CRYPTOGRAPHY <h2|h4hrp> techview t_graphics

• Setup Scripts:

  setup_t_graphics.script is used to execute the entire graphics test suite.

  The following setup scripts are used to run the class-specific test suites:

  • setup-GRAPHICS-FBSERV-BitmapUtil-PublicApi.script

  • setup-GRAPHICS-FBSERV-FbsBitmap-PublicApi.script

  • setup-GRAPHICS-FBSERV-FbsColor256BitmapUtil-PublicApi.script

  • setup-GRAPHICS-FBSERV-FbsFont-PublicApi.script

  • setup-GRAPHICS-FBSERV-FbsSession-PublicApi.script

  • setup-GRAPHICS-FBSERV-FbsTypefaceStore-PublicApi.script

  • setup-GRAPHICS-FBSERV-PublicApi.script

  • setup-T-GRAPHICS-WSERV-PublicApi.script

  • setup-T-GRAPHICS-BITGDI-PublicApi.script

  • setup-T-GRAPHICS-FNTSTORE-PublicApi.script

• Running Scripts on an emulator (WINSCW): Change the directory to %EPOCROOT%\epoc32\release\winscw\<urel|udeb>

• Running Scripts on hardware (ARMV5): Open eshell.

• Execute the tests:

  testexecute ...\graphics\setup_<test suite name>.script

  testexecute ...\graphics\<test suite name>.script

  Refer to Graphics Verification Suite Component for links to information about the various test suites.

Buildrom overflow error

The default ROM size for an OMAP (H2, H4) board is currently up to 32 MB.

If the ROM is greater than 32MB and less than 64MB in size, the h2.oby or h4hrp.oby file can be modified, depending on the device. The value of "ROMMEGS" must be set to the desired ROM size. This value must be specified in hexadecimal format.

Images executing in RAM consume part of the 64MB of available system memory. Therefore it is important to ensure that there is enough memory remaining to run the operating system and the required application. The availability of memory can vary dynamically depending on usage, so making ROMs that are close to the limit is risky and can slow down the execution rate dramatically. Alternatively, a custom ROM can be built with only a subset of the tests included.

Note: The above options must also be available for other, non-OMAP, platforms.

Build and execute tests on phone directly using a .pkg file

For ARMV5:

1. EShell and TestExecute:

   The phone must have eshell and testexecute installed.

2. Certificate files:

   Ensure the .cer and .key files are placed into the /epoc32/pkg/ directory on the PC and rename them as the ones used in the makesisfiles.bat file.

3. Create and install SIS-file:

   Execute makesisfiles.bat from /epoc32/pkg/ directory.

   makesisfiles.bat

   This creates t_fbservapi.sis file. The file must be copied to an MMC card, transferred to the phone and executed.

   When the t_fbservapi.sis file is executed, the graphics test suite is installed on the device.

   Note: If there is no SIS installer available, you can use a different tool (such as STAT desktop, which can be found in %EPOCROOT%\epoc32\tools\stat\statdesktop.exe) to install the .sis file.

4. Running Scripts:

   Run the test scripts. For more information refer to Steps to build and run tests manually (tests built into ROM).

Selective execution of test cases using TCS files

Test cases can be executed selectively using a .tcs file. The .tcs file contains the IDs of test cases:

• to be excluded from a test run

• to be executed on their own. Therefore, excluding all other test cases in the test script from a test run.

The test case IDs can be specified as a range, a list, or a combination of both, where each range or individual test case ID is delimited by a new-line character.

For example, a range of test case IDs can be specified as:

GRAPHICS-FBSERV-FbsBitmap-PublicApi-0021: GRAPHICS-FBSERV-FbsBitmap-PublicApi-0052

Test cases from 21 to 52 (3 and 12 included) are either excluded or are the only ones to be executed during a test run.

A list of test cases IDs can be:

GRAPHICS-FBSERV-FbsBitmap-PublicApi-0062

GRAPHICS-FBSERV-FbsBitmap-PublicApi-0070

GRAPHICS-FBSERV-FbsBitmap-PublicApi-0079

A combination of range and list can be:

GRAPHICS-FBSERV-FbsBitmap-PublicApi-0052:GRAPHICS-FBSERV-FbsBitmap-PublicApi-0055

GRAPHICS-FBSERV-FbsBitmap-PublicApi-0062

GRAPHICS-FBSERV-FbsBitmap-PublicApi-0070

GRAPHICS-FBSERV-FbsBitmap-PublicApi-0079

GRAPHICS-FBSERV-FbsBitmap-PublicApi-0097:GRAPHICS-FBSERV-FbsBitmap-PublicApi-0098

There are two .tcs files, which are located in:

• ...\graphics\graphicsapitest\graphicssvs\config\t_graphics.tcs

• ...\graphics\graphicsapitest\graphicssvs\config\t_graphics_panic.tcs

The t_graphics.tcs file is used to exclude test cases that do not cause panic, and the t_graphics_panic.tcs file is used for test cases that cause a panic.

For more information about the exclusion of panicking test cases, see Excluding panicking test cases in T_FBServAPI Regression Test Suite.

For more information regarding location, see TCS files source tree location in T_FBServAPI Regression Test Suite and TCS files source tree location in T_WServAPI Regression Test Suite.

Only the .tcs files are modified when using the test driver for test execution. When executing the tests manually, the following test execute command illustrates the usage of the .tcs file:

testexecute %EPOCROOT%\epoc32\release\winscw\<urel/udeb>\<test suite name>.script –[tcx|tci] %EPOCROOT%\epoc32\release\winscw\<urel/udeb>\< t_graphics/ t_graphics_panic>.tcs

The -tcx switch is used to execute all the tests in the specified script excluding the test cases specified in the .tcs file. While the –tci switch is used to execute only the tests specified in the .tcs file from the specified script.

Test results

Test results log file location

After running the test, the result logs can be found on the system drive of the device in the \logs\testexecute\ directory. If the tests are run manually using TestDriver on the PC, results can be found in the following location:

%EPOCROOT%\epoc32\TestDriver\results\ or

%EPOCROOT%\epoc32\winscw\c\logs\testexecute

If the test are run manually, when you use TestDriver a new test result folder is created for each test run. This means that old results are not overwritten when you run a new test. An HTML log file can be found for each script contained in the suite, in its own subfolder.

Each test case in the script contains one test block, which performs a number of actions on an API. The log displays the percentage of test cases that have passed successfully. Ideally the script should have a 100% pass rate.

How to interpret the test results

Each test suite produces its own HTML log file containing the results of the test run (see Platform Focused Components section). When examining the log file, check the results at the bottom of the file first. This gives a summary of the number of tests in the suite that have passed, failed or caused panic. In the body of the log, passed tests are highlighted in green, failed tests in red and tests causing panic are highlighted in blue.

If a test fails or results in a panic, you can find out why it failed or panicked by examining each step in the test. The steps are listed between START_TESTCASE and END_TESTCASE tags. The steps that pass have the word INFO followed by some information regarding the step. A failing step instead has the word ERROR, followed by an explanation of the error (usually with an error code). If a panic occurs, there may be no textual explanation for the problem except the panic code (within the blue highlighted text).

When this test suite is run on a real device, some tests may fail due to device limitations. For example, because the device does not have the correct hardware (such as no modem exists for a test that needs one) or lacks the right software plugins to run the test. This can be spotted if the log displays that a test has failed with -5 error (KErrNotSupported). This means that the functionality in this test is not supported on the device used.

Please refer to Symbian Verification Suite > TestDriver > Using TestDriver > Results > Viewing the test results for more information on TestDriver results.

Platform focused components

The following tables show which classes are tested on the various Symbian releases.

FBSERV classes tested on Symbian versions

WSERV classes tested on Symbian versions

BitGDI classes tested on Symbian versions

Font Store classes tested on Symbian versions