Comms-infras Test Technology

Scope

Components under test

The Comms-Infras (Communications Architecture Infrastructure) subsystem provides a framework for the operation of all communication protocols on a Symbian device.

The Comms-Infras test suites test the esock sub-component.

Test Scope

Published API testing

This technology area provides a number of test suites to test APIs that Symbian identify as PublishedAll and are documented in the Symbian platform Library. These test suites can be used as a regression test suite and also as part of a compatibility testing program.

For more details, refer to the individual classes in Comms-Infras API .

Test Omissions

The behaviour of the Comms-Infras APIs is dependent on the protocol being used. The protocols available are dependent on the device used and so cannot be exhaustively tested. Consequently the test suites are only tested using the most common protocols (that is, TCP and UDP).

For more details about omissions in the individual classes, refer to Comms-Infras API .

Environment

Device under Test Setup

The device under test (hardware platform/phone/emulator) must have an appropriate Symbian platform or licensee-UI build.

PC Setup

The PC must have a Symbian or licensee developer kit, PDT and the relevant compilers (Code Warrior, RVCT, GCCE) installed to build and execute the test code.

Network Environment Setup

The PC and the device under test must be on the same network.

The Comms-Infras test suites use sockets to send and receive data. To ensure that the full networking stack is tested correctly, the sockets on the PC and the device under test must be able send and receive data between each other. Two PC ports, port 10007 for TCP data and port 10008 for UDP data, listen for and receive TCP and UDP data respectively sent by the device under test.

Follow the instructions in Appendix A - Echo Server and Appendix B - NT RAS to set up TCP/IP communications between the PC and device under test.

However, it is also possible to use other TCP/IP connections such as Ethernet and Wi-fi instead of NT RAS. To do this the appropriate table needs to be referenced from the Comms database (COMMSDAT). You must change the section [iap_settings] in t_comms-infras.env with the table used to establish the connection.

       
        
       
       [iap_settings]
NetworkTable=1 <Network table entry number in CommDbCommsInfras.xml>
IAPTable=1 <The IAP entry number in CommDbCommsInfras.xml>

The default preference for commdb in CommDbCommsInfras.xml is given below:

       
        
       
       <ConnectionPreferencesTable>
   <ConnectionPreferences operation="add">
     <Name>DefaultRecordName-1</Name>
     <Ranking>1</Ranking>          
     <Direction>OUTGOING</Direction>
     <BearerSet>CSD</BearerSet>    
     <DialogPref>DONOTPROMPT</DialogPref>
     <IAPRef><my NTAS/ppp connection IAP></IAPRef>
   </ConnectionPreferences>
</ConnectionPreferencesTable>

You need your own CommDbCommsInfras.xml to do this.

Note : When executing the Comms-Infras test suites on the emulator or the hardware reference board(H4), the preferred connection channel should be the NT RAS. The preferred connection cannot be NT RAS when you execute the Comms-Infras test suites on the real phone WIFI-enabled.

The CommDbCommsInfras.xml file is in the following location:

...\commsinfrastructureapitest\commsinfrastructuresvs\suite\group

For command details refer Building and Execution section.

When execute the Comms-Infras test suites on the emulator or the hardware reference board(H4), you must first configure the NT RAS using the instructions in Appendix B - NT RAS , and a serial port cable to connect the COM port. After configuring NT RAS, you must build and execute the tests.

For information about how to execute the Comms-Infras test suites on the real phone with WIFI enable, see the Appendix C - Testing on real phone through WIFI connection .

Test dependencies

Note: Any mention of “epoc32” is offset by %EPOCROOT% that is, the full path where the epoc tree is installed.

Test data

Test data for each test suite is in the following location:

…\commsinfrastructureapitest\commsinfrastructuresvs\suite\esock\t_rconnection\testdata
…\commsinfrastructureapitest\commsinfrastructuresvs\suite\esock\t_rsocket\testdata
…\commsinfrastructureapitest\commsinfrastructuresvs\suite\esock\t_rsocketserv\testdata
…\commsinfrastructureapitest\commsinfrastructuresvs\suite\esock\t_rsubconnection\testdata

For more details, refer to Comms-Infras API .

Building and Execution

The Comms-infras Test Suite Hierarchical Structure shown below illustrates the Test Suite structure after building the test suite. General information about the testdriver test suite structure may be found here » Symbian Verification Suite » TestDriver V2 » TestDriver Output » Storing tests and results.

Note: This structure can be seen in the following location after building the test suite %EPOCROOT%\epoc32\testdriver\testproduct

The Comms-infras Test Suite Hierarchical Structure is as shown below:

       
        
       
       <Component> - Comms-Infras
  <Sub-component>-Esock
    csd (Circuit Switched Data)
      <Class>-RConnection
        <suite>-COMINF-ESOCK-RConnection-PublicAPI-Other-Suite
        <suite>-COMINF-ESOCK-RConnection-PublicAPI-TCP-Suite
        <suite>-COMINF-ESOCK-RConnection-PublicAPI-UDP-Suite
      <Class>-RSocket
        <suite>-COMINF-ESOCK-RSocket-PublicAPI-Other-Suite
        <suite>-COMINF-ESOCK-RSocket-PublicAPI-TCP-Suite
        <suite>-COMINF-ESOCK-RSocket-PublicAPI-UDP-Suite
      <Class>-RSocketServ
        <suite>-COMINF-ESOCK-RSocketServ-PublicAPI-Other-Suite
        <suite>-COMINF-ESOCK-RSocketServ-PublicAPI-TCP-Suite
        <suite>-COMINF-ESOCK-RSocketServ-PublicAPI-UDP-Suite
    psd (Packet Switched Data)
      <Class>-RSubConnection
        <suite>-COMINF-ESOCK-RSubConnection-PublicAPI-Other-Suite
        <suite>-COMINF-ESOCK-RSubConnection-PublicAPI-TCP-Suite
        <suite>-COMINF-ESOCK-RSubConnection-PublicAPI-UDP-Suite

The hierarchical test suite structure facilitates easy build and execution of the complete test suite or part of it. For example, if the path specified is:

<comms-infras> - The entire suite is built or executed
<comms-infras.esock.csd> - All the suites below csd that is, RConnection, RSocket, RSocketServ are built/executed.
<comms-infras.esock.csd.RConnection> - All the three RConnection suites are built/executed.
<comms-infras.esock.csd.RConnection.RConnection-PublicAPI-Other-Suite> - RConnection-PublicAPI-Other-Suite is built or executed.

The commands to be executed on emulator and hardware.

Note : Refer to Comms-infras Test Suite hierarchical structure for sub-component and test suite names.

General environment setup

IAPSettings in the t_comms-infras.ini file:
- For WINSCW platform:
  
  Ensure that under the [IAPSettings] section, IAPTable = 2 to use WinTap.
- For ARMV5 platform:
  
  Ensure that under the [IAPSettings] section, IAPTable = 1 to use NT RAS.
The t_comms-infras.ini file is generated by executing the InsertIPAddress batch file, which uses the t_comms-infras.env as a template, in the following location:

…\commsinfrastructureapitest\commsinfrastructuresvs\suite\esock\testdata

See Appendix A - Echo Server for more details on environment settings
The entire Comms-Infras Test component can be built from …\commsinfrastructureapitest\commsinfrastructuresvs\suite\group\…

Individual test components can be built from the respective group directory. For example, …\commsinfrastructureapitest\commsinfrastructuresvs\suite\esock\t_<sub-component>\group

Execute the following commands:

InsertIpAddr

bldmake bldfiles

abld test build [winscw|armv5] [urel|udeb]

%EPOCROOT%\epoc32\release\winscw\<udeb/urel>ced z:\comms-infras\CommDbCommsInfras.xml

Note: This command accomplishes the following:
- Updates CommDB to use the appropriate transport mechanism (NT RAS by default). See Appendix B - NT RAS for more details.
- Suppresses any WinTap prompts.
- Updates CommDb to use the right IAP table.
startechoserver

Steps to build and execute automatic tests using Test Driver

For WINSCW platform:

Communication between the PC and the device under test (Symbian platform device) is enabled using STAT. So, the device under test must have STAT installed.

Note: Refer to Symbian Verification Suite » TestDriver » Using TestDriver » Communicating with the device » STAT plug-in for more information.
Test Driver can be configured using the automated scripts located at:

…\commsinfrastructureapitest\commsinfrastructuresvs\suite\group\…

testdriversetup

testdriversetup.bat calls testdriversetup.pl which applies an appropriate set of configuration parameters.

Note : If you execute the tests on the real phone using Test Driver, the eshell and testexecute must be installed on the phone. See details in Appendix C – Testing on real phone through WIFI connection .
Build the test code:

testdriver build [–p [winscw|armv5]] [–b [udeb|urel]] –s comms-infras[.esock[.<csd|psd>[.<class name>[.<test suite name >]]]]

Testdriver commands can be executed from any location.

Note: For tree structure, refer to Comms-infras Test Suite Hierarchical Structure in Building and Execution section.
Execute the Tests:

testdriver run

For ARMV5 platform:

Communication between the PC and the device under test (Symbian platform device) is enabled using STAT. So, the device under test must have STAT installed.
Communication between PC and hardware:

The hardware reference board (for example, H4) must be connected to the PC through USB or serial cable. The ROM image is built on the PC and transferred to the hardware board through MMC card.

Note: Refer to Symbian Verification Suite » TestDriver » Using TestDriver » Communicating with the device » STAT plug-in for more information.
Test Driver can be configured using the automated scripts located at:

…\commsinfrastructureapitest\commsinfrastructuresvs\suite\group\…

testdriversetup

testdriversetup.bat calls testdriversetup.pl which applies an appropriate set of configuration parameters.
Build the test code:

testdriver build [–p [winscw|armv5]] [–b [udeb|urel]] –s comms-infras[.esock[.<csd|psd>[.<class name>[.<test suite name >]]]]

Testdriver commands can be executed from any location.

Note: For tree structure, refer to Comms-infras Test Suite Hierarchical Structure in Building and Execution section.
To build the ROM image:

buildrom -D_STARTUPMODE2 -D_EABI=ARMV5 -DRVCT -DUSE_STRONG_CRYPTOGRAPHY -D_CUSTOM_COMMSDAT <h2/h4hrp> techview_statapi td_comms-infras
Execute the Tests:

testdriver run -t <transport service used>

Transport service used may be “serial<COM port number that is connected to the board>”, “bt<COM port number for BT dongle>” or “usb<mapped port number>”

Note:

All scripts and test data are automatically removed from the system drive of the device under test after the execution.
The RSubconnection tests and COMINF-ESOCK-RSocket-PublicAPI-0024 test case are excluded from execution. For more information, see Exclusion of Packet Switched Data (PSD) Tests .

Steps to build and execute tests manually (Tests built into ROM)

For WINSCW platform:

Setup Scripts:

setup_t_comms-infras.script is used to execute the entire comms-infras test suite.

setup_COMINF-ESOCK-<class name>-PublicAPI.script is used to execute class specific test suites.
Running Scripts:

Change directory to

%EPOCROOT%\epoc32\release\winscw\<urel/udeb>
Execute the Tests:

testexecute z:\comms-infras\esock\setup_<test suite name>.script

setup_t_comms-infras.script is used to copy the test data files on to c:\drive.

testexecute c:\comms-infras\esock\<test suite name>.script

t_comms-infras.script is used to execute the entire comms-infras test suite.

For ARMV5 platform:

Build ROM image:

buildrom -D_EABI=ARMV5 –DRVCT -DUSE_STRONG_CRYPTOGRAPHY -D_CUSTOM_COMMSDAT <h2/h4hrp> techview td_comms-infras
Setup Scripts:

setup_t_comms-infras.script is used to execute the entire comms-infras test suite.

setup_COMINF-ESOCK-<class name>-PublicAPI.script is used to execute class specific test suites.
Running Scripts:

Open eshell
Execute the Tests:

testexecute z:\comms-infras\esock\setup_<test suite name>.script

setup_t_comms-infras.script is used to copy the test data files on to c:\drive.

testexecute c:\comms-infras\esock\<test suite name>.script

t_comms-infras.script is used to execute the entire comms-infras test suite.

Steps to build and execute tests on phone directly using .pkg file

For ARMV5 platform:

EShell and TestExecute:

The phone must have eshell and testexecute installed.
Certificate files:

Ensure .cer and .key files are placed into …/epoc32/pkg/ directory on the PC and renamed as zeus.cer and SymbianACS.key correspondingly.
Create and install SIS-file:

Execute makesisfiles.bat from …/epoc32/pkg/ directory.

makesisfiles.bat

This creates t_rconnection.sis, t_rsocket.sis, t_rsocketserv.sis, and t_rsubconnection.sis files. The files must be copied to an MMC card, transferred to the phone and executed.

When the above files are executed, the comms-infras test suite is installed on the device.
Running Scripts:

Run the test scripts. For more information refer to Steps to build and execute tests manually (Tests built into ROM) .

Selective Execution of Test Cases using TCS File

Test cases can be excluded from a test run by using a .tcs file. The .tcs file contains the IDs of test cases to be excluded.

The test case IDs can be specified as a range, a list, or a combination of both, where each range or individual test case ID is delimited by a new-line character.

For example, range of test case IDs to be excluded can be specified as:

COMINF-ESOCK-RSubConnection-PublicAPI-0001: COMINF-ESOCK-RSubConnection-PublicAPI-0039

A list of test cases IDs to be excluded can be:

COMINF-ESOCK-RSubConnection-PublicAPI-0001

COMINF-ESOCK-RSubConnection-PublicAPI-0005

COMINF-ESOCK-RSubConnection-PublicAPI-0007

A combination of range and list can be:

COMINF-ESOCK-RSubConnection-PublicAPI-0005: COMINF-ESOCK-RSubConnection-PublicAPI-0010

COMINF-ESOCK-RSubConnection-PublicAPI-0015: COMINF-ESOCK-RSubConnection-PublicAPI-0020

COMINF-ESOCK-RSubConnection-PublicAPI-0030

There is only one .tcs file, which is located at the following location:

…\commsinfrastructureapitest\commsinfrastructuresvs\suite\config\t_comms-infras.tcs

For more information on location of .tcs files , see TCS File Source Location in Comms-Infras ESOCK Test Suite.

Only .tcs file is modified during test execution using the test driver.

When executing the tests manually, the following testexecute command illustrates the usage of the .tcs file:

testexecute %EPOCROOT%\epoc32\release\winscw\<urel/udeb>\<test suite name>.script –[tcx|tci] %EPOCROOT%\epoc32\release\winscw\<urel/udeb>\ t_comms-infras.tcs

This executes the specified script excluding the test cases specified in the .tcs file.

Test Results

Test Results log file location

After the tests complete execution, result logs can be found on the system drive of the device in the \logs\testexecute\ directory. On the PC the location of the results depends on whether the tests were run using TestDriver or Testexecute. TestDriver places the results in, \epoc32\TestDriver\results\ and creates a new folder for each test run. Testexecute places the results in \epoc32\winscw\c\logs\testexecute . A .html log file can be found for each script contained in the suite, in its own subfolder

Each test case in the script contains one test block, which performs a number of actions on an API. The log will display the percentage of test cases that have passed successfully. The script should ideally have a 100% pass rate.

How to interpret test results

Each test suite produces its own HTML log file containing the results of the test run . When examining the log file, the results at the bottom of the log should be checked first. This gives a summary of the number of tests in the suite that have passed, failed or panicked. In the body of the log tests which pass are highlighted in green, tests which fail in red and tests which cause a panic in blue.

If a failing/panicking test has been identified, its reason for failing/panicking can be seen by examining each step in the test, listed between the “START_TESTCASE” and “END_TESTCASE” tags. The steps which pass will have the word “INFO” followed by some information regarding the step. A failing step will instead have the word “ERROR”, followed by an explanation of the error (usually with an error code). If a panic occurs there may be no textual explanation of the problem, but a panic code will be given (within the blue highlighted text).

When this test suite is run on a real device, some of the tests may fail due to limitations of the device. This will be because of the device not having the correct hardware (for example, no modem exists for a test which needs one), or lacking the right software plugins to run the test. This can be spotted if the log displays that a test has failed with a -5 error (which means KErrNotSupported). This -5 error basically means that the test has failed because the functionality in this test is not supported on the device used. Therefore, any tests failing with -5 are not due to defects, but to the limitations of the device used.

Refer to Symbian Verification Suite » TestDriver » Using TestDriver » Results » Viewing the test results for more information on test driver results.

OS Focussed Components

Table 1 illustrates the sub-components and classes available on each Symbian platform

Table 1 – Component ESOCK class tested on each Symbian platform baseline

Symbian platform Baseline	9.1	9.2	9.3	9.4
Class1 - RConnection
Class2 – RSocketX
Class3 – RSocketServ
Class4 - RSubConnection
X	X	X	X	X
X	X	X	X	X
X	X	X	X	X
X	X	X	X	X

Appendix A - Echo Server

To ensure that the full networking stack is correctly tested, an external IP Echo Server is used to receive data on the PC from the device under test and echo it back to a port on the device under test.

In order to send data to the EchoServer, the tests need its IP address (that is, the real IP address of the PC running the server). This is set in the global Comms-Infras environment file, t_comms-infras.env, found in the following location

…\commsinfrastructureapitest\commsinfrastructuresvs\suite\testdata\

t_comms-infras.env file has the following lines:

       
        
       
       [Network]
LocalIpAddress    = <insert_local_ip_address_here>
TcpPort        =10007
UdpPort    =10008
LocalPort    =2024
BindPort    =10001

The <insert_local_ip_address_here> tag needs to be replaced with IP address of the PC running the EchoServer. For example:

       
        
       
       [Network]
LocalIpAddress    = 10.10.10.10

The bind port is the port on the device under test that sends TCP and UDP data to the PC. The local port is the port on the device that receives the data echoed by the echo server on the PC. These ports may be configured in the t_comms-infras.env file

To generate the t_comms-infras.ini referred to by the test data files and replace the IP address tag with the IP address of the PC, execute the following batch file:

…\commsinfrastructureapitest\commsinfrastructuresvs\suite\group\InsertIpAddr.bat

The batch file executes the command, ipconfig, to retrieve the required IP address.

Note: By default InsertIpAddr gets the IP address from the ipconfig section ending “Local Area Connection”. If your IP address is in a different section, you must edit InsertIpAddr.bat, changing “Local Area Connection” to your section name.

Run Echo Server on the test PC using the following batch file:

…\commsinfrastructureapitest\commsinfrastructuresvs\suite\group\StartEchoServer.bat

This will launch two command prompt windows. One each for the TCP and UDP connections made to the server. After running all of the tests the EchoServer can be stopped by closing the two Echo Server windows.

Appendix B - NT RAS

If the tests are to be executed on the emulator, the PC must have two COM ports which are connected through a null-modem cable. The Symbian emulator talks to one of the COM ports and the NT RAS service talks to the other.

If the tests are to be executed on hardware one of the serial (COM) ports on the PC has to be connected to the COM3 port on the hardware board.

Here are the full instructions for setting up NT RAS:

B.1 Install a Null modem

Open the Control Panel
Open Phone and Modem Settings (In Windows XP the option can be found in printers and other Hardware)
Goto the Modems tab
Select Add...
Check the "Do not detect my modem; I will select it from a list." check box
Click Next
Select the (Standard Modem Types) - Communications cable between two computers
Select the appropriate com port
Click Finish
Open the Properties of the Modem you have just installed
Set the maximum port speed to be 115200 (on the General tab)
Goto the Advanced tab
Click Change Default Preferences...
Make sure the Port speed is 115200 and the Flow control is None (on the General tab)
Goto the Advanced tab
Make sure the Data bits = 8, Parity = None and Stop bits = 1
Keep clicking OK to close all the open dialogs

B.2 Configuring NT RAS (Incoming Networking Connections)

Open the Control Panel
Open Networking and Dial-up Connections (*)
Open Make New Connection
Click Next
Select the Accept incoming connections and click Next
Check the modem you have just installed and click Next
Select Do no allow virtual private connections and click Next
Add a new user
Click Add
1. Enter the following details (noting the case):
  1. User name: RasUser
  2. Password: pass
  Note : Networks differ and may enforce strong passwords. In such a scenario, the password must be modified in the commdb. This can be done by modifying the following lines in the CommDbCommsInfras.xml file
  - <IfAuthName>RasUser</IfAuthName>
  - <IfAuthPass>Pass</IfAuthPass>
2. Click OK
3. Make sure the new user is checked
4. Click Next
Setup the Networking Components
1. If TCP/IP is not set up, uncheck "Allow callers to access my area Networks".
2. Check "Specify TCP/IP addresess" and add "192.68.0.1" to "From:"-mask and "192.68.0.2" to "To:" mask.
  
  Note: This step may be machine or network dependent.
3. Click Next
Enter whatever name you like for the connection name.
Click Finish
Open the Properties of the connection you have just created
Go to the Users tab
Make sure the "Always allow directly......... a password" check box is unchecked
Click OK

* Steps 1-6 on XP

1. Select Network and Internet Connections
2. Select network connections
3. Click on “create a new connection -> new connection wizard
4. Click next
5. Select radio button “set up an advanced connection
6. Accept incoming connections

Continue as instructed from step 6 onwards.

Appendix C - Testing on real phone through WIFI connection

If the tests are to be executed on the real phone through WIFI connection, a wireless router must be connected to the PC to provide a wireless connection channel between the PC and the real phone.

Before you start testing, configure the WIFI-enabled phone’s IP address within the wireless IP domain. The phone can be configured using two types of IP addresses, static IP address and DHCP IP address.

Due to the convenience consideration of automated running the test suites, the phone’s WIFI card IP address are configured as the static IP address . The following are the instructions to set the static IP address:

Connect the wireless router to the PC
Switch on the phone’s WIFI card and search for the WIFI hot spot.
Find a router to be connected to the WIFI hotspot.
Open the web browser from the PC and logon to the administration web page, and move to the static IP address configuration web page.

The web page of static IP address configuration shows the MAC address of WIFI card of phone.
Click the Add new record button to bind new static IP address to the MAC address of the phone’s WIFI card. Once it is done, the wireless router assigns an IP address to this phone’s WIFI card. Each time when the phone connected, the router assigns this specified IP address to the phone.

When the phone’s WIFI card IP is specified, you also must let the PC run within the same network domain. This enables you to run InsertIpAddr.bat during the wireless connection between the PC and router. You can also allocate a static IP address to the PC through the configuration web page of the wireless router.

The following are the steps to execute the test on the real phone through the WIFI connection:

Turn on the WIFI on the real phone and connect it to the PC-connected wireless router.
Keep the router connecting to the PC, and execute the InsertIpAddr.bat to generate the correct IP allocated with in the wireless router’s domain in the t_comms-infras.ini.
The entire Comms-Infras test component can be built from …\commsinfrastructureapitest\commsinfrastructuresvs\suite\group\...

Individual test components can be built from the respective group directory. For example, …\commsinfrastructureapitest\commsinfrastructuresvs\suite\esock\t_<sub-component>\group .

Run the following commands :

bldmake bldfiles

abld test build armv5 urel

Note : To connect to the licensee server , change back the network cable connection to the original network from PC side might is needed. When it finished , change back the connection to the wireless router.
Run the following command:

startechoserver

If you want to build and execute the tests using the testdriver, make sure the statapi application exists in the phone and it must be run on the real phone before you proceed for further steps.
Configure the TestDriver using the automated scripts located at …\commsinfrastructureapitest\commsinfrastructuresvs\suite\group\…

Execute the following commands:

testdriversetup

testdriversetup.bat calls testdriversetup.pl which applies an appropriate set of configuration parameters.

testdriver config --testexec OFF

This command notifies the testdriver that it must not transfer the TEF component sis file to the real phone if the real phone has already built-in TEF module.

testdriver config --cert phone_applied_cert_file

testdriver config --key phone_applied_key_file
Connect the PC and the real phone with USB or some other cable, and execute the following commands:

testdriver build -p armv5 -b urel -s comms-infras.esock.csd.RConnection

testdriver run -p armv5 -b urel -s comms-infras.esock.csd.RConnection –t usb1

Notes :

During the execution, if the phone’s CommsDB has not cancel the dialog prompt in the ConnectionPreferences table, you must manually select the WIFI hot spot (IAP) to be connected to the Echoserver.
Ensure that the wireless connection of the phone remains unaffected because phone has different wireless signal strength under different situations. For example, wireless strength may vary with distance.
While testing under the wireless environment, the phone package can be blocked by the PC’s firewall. To avoid this problem, modify the firewall configuration to turn off this blocking or close the firewall.