|
1 <html> |
|
2 <head> |
|
3 <title>Eclipse Testing Instructions</title> |
|
4 </head> |
|
5 |
|
6 <body> |
|
7 <h1>The Eclipse Test Framework</h1><p>Last Modified: June 13, 2007</p><p><font size="+1"><br> |
|
8 Introduction</font></p><p>The testing framework is comprised of the org.eclipse.test plugin and the org.eclipse.ant.optional.junit |
|
9 fragment. </p><p>These two projects are available from the dev.eclipse.org repository and are |
|
10 included in the eclipse-test-framework-<buildId>.zip from the eclipse.org |
|
11 downloads page.</p><p><font size="+1">Building and Installation</font></p><p>Since the org.eclipse.test plugin is stored in the repository in source-code |
|
12 form, it needs to be compiled before it can be used. The org.eclipse.ant.optional.junit |
|
13 fragment does not contain any source and can be used as is from the repository.</p><ol> |
|
14 <li>Turn of automatic builds. Window->Preferences->Workbench and uncheck "Perform |
|
15 build automatically on resource modification"</li><li>Load org.eclipse.test into your workspace.</li><li>Right-click on the org.eclipse.test project in either the Navigator or Packages |
|
16 view. Select 'Rebuild Project' from the context menu. This will compile the |
|
17 entire org.eclipse.test plugin.</li><li>Finally, copy the org.eclipse.test plugin into your target Eclipse.</li> |
|
18 <li>The org.eclipse.ant.optional.junit fragment only needs to be present in |
|
19 the environment of the Eclipse that is overseeing the test process. If you |
|
20 are running the test script from within the Workbench, this means that the |
|
21 fragments need to be present withinn your development Eclipse. If you are |
|
22 running the tests from the command line, then the fragments will need to be |
|
23 present in your target Eclipse.</li></ol> |
|
24 <p><font size="+1">Setup</font></p> |
|
25 <p>Follow the steps given above to build and install all of the neccessary plugins |
|
26 and fragments. Please note that the current version of the test framework is |
|
27 not compatible with the PDE notion of self-hosting. If you want to run the tests, |
|
28 you will need to setup a full target Eclipse so that the testing framework can |
|
29 detect everything that is needed.</p> |
|
30 <p>If you are writing tests for one or more Eclipse plugins, then you should |
|
31 create a separate plugin for tests. The test plugin will contain all of the |
|
32 tests that will be run on your development plugins, as well as defining how |
|
33 those tests get run.</p> |
|
34 |
|
35 <p>If you are not writing tests for an Eclipse plugin, then you should look into |
|
36 using JUnit on its own to test your work. JUnit is designed for testing Java |
|
37 code, and by default has no knowledge of Eclipse. There are separate mechanisms |
|
38 for using JUnit on Java code in Eclipse. See the documentation provided here:</p> |
|
39 <p> <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/jdt-ui-home/plugins/org.eclipse.jdt.junit/index.html"> |
|
40 JUnit Support in Eclipse</a><br> |
|
41 <a href="http://dev.eclipse.org:8080/help/content/help:/org.eclipse.jdt.doc.user/tasks/tasks-207.htm?resultof=%6a%75%6e%69%74%20&toc=%2Forg.eclipse.jdt.doc.user%2Ftoc.xml"> |
|
42 Using JUnit</a></p> |
|
43 |
|
44 <p>Once a test plugin has been created, you must then create an Ant script that |
|
45 will run all of the tests. Create a file called 'test.xml' in the root of your plug-in or bundle. |
|
46 This is an Ant file that defines how each test is going to be run. The 'test.xml' |
|
47 file is like a DOS batch file that scripts the entire testing process. Typically, |
|
48 the 'test.xml' file will contain targets for setting up the test run, executing |
|
49 the tests, cleaning up afterwards, and running the entire process.</p> |
|
50 |
|
51 <p><font size="+1">Converting existing Test Suites</font></p> |
|
52 |
|
53 <p>If you are converting an existing set of tests to use the new framework, the |
|
54 actual tests that have been written should not need much change.</p> |
|
55 <p>If you have tests in multiple plugins, move these to a single test plug-in for your component.</p> |
|
56 |
|
57 <p>Make sure that the tests are defined in a plug-in. This is probably the most |
|
58 common cause of confusion in the entire test process. Your tests need to be |
|
59 in a plug-in so that Eclipse can find them when it tries to load them.</p> |
|
60 |
|
61 <p><font size="+1">Creating new Test Suites</font></p> |
|
62 |
|
63 <p>Creating new JUnit tests for an Eclipse plugin should be no more difficult |
|
64 than writing standard JUnit tests. Since the framework allows tests to be run |
|
65 inside of a working Eclipse, any tests that you write have available to them |
|
66 any of the methods supplied by the Eclipse platform, provided that you add the |
|
67 appropriate dependencies to your tests' manifest.</p> |
|
68 |
|
69 <p><font size="+1">Performance Issues</font></p> |
|
70 <p>You should keep in mind the number of times that Eclipse needs to be |
|
71 started. Launching Eclipse has a substantial cost in terms of runtime. To |
|
72 minimize the number of times the platform is started, you should consider |
|
73 writing a TestSuite called AllTests for each of your test plugins. AllTests |
|
74 should invoke each of the tests that you want to run for a particular plugin. |
|
75 The 'test.xml' file can then run the AllTests class, which will run all of your |
|
76 tests, but the platform will only ever be started once for each of your test |
|
77 plugins.</p><p>Note: Sometimes tests involve shutting down, restarting, and testing the state |
|
78 of metadata that was written to disk. These session tests will require Eclipse |
|
79 to be launched several times in sequence, which will increase the runtime of |
|
80 your tests. This cannot be avoided.</p><p> </p> |
|
81 |
|
82 <p><font size="+1">Running the Test Suite from the UI</font></p><p>Right click on the test.xml file and select 'Run Ant...' from the pull-down |
|
83 menu. The Running Ant dialog box pops up. From the list of targets, select the |
|
84 one that runs all of your tests. If you are using the example file provided |
|
85 below, this target is called 'Run', and will be selected by default. Hit the |
|
86 'Finish' button to start the test process.</p> |
|
87 |
|
88 <p><font size="+1">Running the Test Suite from the command line</font></p> |
|
89 <p>When the test suites are invoked automatically, they are run from command |
|
90 line. From the ${eclipse-home} directory, the command to use is:</p> |
|
91 <code>java -jar plugins\org.eclipse.equinox.launcher_<version>.jar -application org.eclipse.ant.core.antRunner -buildfile ${test-plugin-path}\test.xml |
|
92 -Declipse-home=${eclipse-home} -Dos=<operating system> -Dws=<windowing system> -Darch=<architecture></code> |
|
93 |
|
94 <p>Individual tests can also be invoked directly. From the ${eclipse-home} |
|
95 directory, use the command:</p><code>java -jar plugins\org.eclipse.equinox.launcher_<version>.jar |
|
96 -application ${launcher} -os <operating system> -ws <windowing system> -arch <architecture> -dev bin -testpluginname ${plugin-name} |
|
97 -classname ${test-classname} formattter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,<path to output file with .xml extension></code> |
|
98 <p>where ${launcher} is one of: <i>org.eclipse.test.uitestapplication</i> or <i>org.eclipse.test.coretestapplication</i> |
|
99 depending on whether or not you want to run your tests within an active Workbench.</p><p><font size="+1">Output</font></p><p>By default, output from each test run is logged as XML. For each test that |
|
100 is run, the output is logged into the file called ${classname}.xml. The individual |
|
101 XML files are located in the ${eclipse-home} directory. When the test run is |
|
102 finished, you can call the <b>"collect"</b> target in the library |
|
103 file, which will collect the XML into a single file. See below for an example |
|
104 of how to use this target correctly.</p> |
|
105 |
|
106 <h3>Other issues</h3><h4>Headless Testing vs. UI testing</h4><p>Many plugin tests will not need the Workbench active in order to run. Indeed, |
|
107 only the minimum number of plugins needed to run the plugin being tested need |
|
108 to be present when testing in a target Eclipse. There are two different Ant |
|
109 targets provided for running Eclipse plugin tests. One target starts the entire |
|
110 Workbench. The other starts Eclipse with the minimum number of plugins needed. |
|
111 It is up to you to decide which target is most appropriate. For examples, look |
|
112 at the <b>"ui-test"</b> and <b>"core-test"</b> targets below.</p> |
|
113 |
|
114 <h4>Disposing the Display</h4><p>Some low-level tests for the Eclipse platform take actions that are not |
|
115 normally possible inside of Eclipse. An example of this behaviour would be |
|
116 disposing the display. While this action can be performed while running the UI, |
|
117 it will also kill the UI for the copy of Eclipse that is running, and cause |
|
118 errors when the Workbench tries to shutdown. If you need to test disposing the |
|
119 display, or other similar actions, your tests should be running without a UI. </p> |
|
120 |
|
121 <h4>Tests that are not plugins</h4><p>It is very easy to forget to define your tests inside of a plugin. If your |
|
122 tests will not load properly, make sure that a plug-in manifest exists in your |
|
123 test project, and also that the plugin is being loaded by the platform. Make |
|
124 sure that all of the dependencies are satisfied for your test plugin.</p> |
|
125 |
|
126 <p><font size="+1">Interface</font></p><p>The org.eclipse.test plugin defines many useful Ant tasks/targets to aid developers |
|
127 in writing their test.xml scripts. Currently, there is only Ant targets defined, |
|
128 which can be called using Ant's built-in <ant> task. To use these targets, |
|
129 add the following line to the top of your script, and reference the ${library-file} |
|
130 property when calling <ant>:</p><code><property name="library-file" |
|
131 value="${eclipse-home}/fragments/org.eclipse.test/library.xml"/> </code> |
|
132 |
|
133 <p>The targets that are defined are: </p><ul> |
|
134 <li><b>ui-test</b> - This target runs a JUnit test suite inside of an Eclipse |
|
135 Workbench. This target is mainly for testing plugins that use the Eclipse |
|
136 UI and JFace. The output of the test pass is automatically logged in an XML |
|
137 file called ${classname}.xml. It takes four arguments: </li><ul> |
|
138 <li><i>data-dir</i> - The |
|
139 data directory of the Eclipse that gets run</li><li><i>plugin-name</i> - |
|
140 The name of the plugin that the test suite is defined in</li><li><i>classname</i> - |
|
141 The name of the class that the test suite is defined in</li><li><i>vmargs</i> - An |
|
142 optional argument string to pass to the VM running the tests</li></ul><p>For example, the following code will run the test |
|
143 org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a new Eclipse |
|
144 workbench. It passes the string "-Dbaz=true" to the VM. The Eclipse |
|
145 stores its metadata in the directory "data-folder". |
|
146 |
|
147 <code><pre> |
|
148 <ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}"> |
|
149 <property name="data-dir" value="data-folder"/> |
|
150 <property name="plugin-name" value="org.eclipse.foo"/> |
|
151 <property name="classname" value="org.eclipse.foo.bar.MyTest"/> |
|
152 <property name="vmargs" value="-Dbaz=true"/> |
|
153 </ant> |
|
154 </pre></code> |
|
155 |
|
156 <li><b>core-test</b> - This target runs a JUnit test suite inside of an IPlatformRunnable. |
|
157 This target is for testing plugins that use the Eclipse platform, but do not |
|
158 require a UI to be running. The output of the test pass is automatically logged |
|
159 in an XML file called ${classname}.xml. It takes four arguments: </li><ul> |
|
160 <li><i>data-dir</i> - The |
|
161 data directory of the Eclipse that gets run</li><li><i>plugin-name</i> - |
|
162 The name of the plugin that the test suite is defined in</li><li><i>classname</i> - |
|
163 The name of the class that the test suite is defined in</li><li><i>vmargs</i> - An |
|
164 optional argument string to pass to the VM running the tests</li></ul><p>For example, the following code will run the test |
|
165 org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a headless Eclipse. |
|
166 It passes the string "-Dbaz=true" to the VM. The Eclipse stores its |
|
167 metadata in the directory "data-folder". |
|
168 |
|
169 <code><pre> |
|
170 <ant target="core-test" antfile="${library-file}" dir="${eclipse-home}"> |
|
171 <property name="data-dir" value="data-folder"/> |
|
172 <property name="plugin-name" value="org.eclipse.foo"/> |
|
173 <property name="classname" value="org.eclipse.foo.bar.MyTest"/> |
|
174 <property name="vmargs" value="-Dbaz=true"/> |
|
175 </ant> |
|
176 </pre></code> |
|
177 |
|
178 <li><b>collect</b> - This |
|
179 target collects the XML files that are produced over the course of the |
|
180 test script. It takes two arguments: </li><ul> |
|
181 <li><i>includes</i> - A |
|
182 pattern matching all XML files to be included in the test report. This |
|
183 argument is typically "org*.xml"</li><li><i>output-file</i> - |
|
184 The filename where the output of the test gets stored. For the automated |
|
185 build process, this file should be ${pluginname}.xml, and be located in |
|
186 the ECLIPSE_HOME directory.</li></ul><p>For example, the following code collects all of the files |
|
187 matching the pattern "org*.xml" in the directory ${eclipse-home}, |
|
188 into the file named "logfile.xml". |
|
189 |
|
190 <code><pre> |
|
191 <ant target="collect" antfile="${library-file}" dir="${eclipse-home}"> |
|
192 <property name="includes" value="org*.xml"/> |
|
193 <property name="output-file" value="logfile.xml"/> |
|
194 </ant> |
|
195 </pre></code> |
|
196 |
|
197 |
|
198 </ul><h2>Examples:</h2><p>Included is the 'test.xml' file from the org.eclipse.jdt.ui.tests.refactoring plugin. This |
|
199 file controls all of the automated testing that is done for the org.eclipse.jdt.ui.tests.refactoring |
|
200 plugin. It can be run from inside of Eclipse or from the command line. It is |
|
201 intended to serve as a template file for testing any other plugin.</p><p>Notice that the structure of the file roughly mirrors that of a JUnit test. |
|
202 Targets are defined for setting up the tests, defining what needs to be done, |
|
203 cleaning up after the tests, and running everything in the right order.</p> |
|
204 <code><pre> |
|
205 <?xml version="1.0" encoding="UTF-8"?> |
|
206 |
|
207 <project name="testsuite" default="run" basedir="."> |
|
208 <!-- The property ${eclipse-home} should be passed into this script --> |
|
209 <!-- Set a meaningful default value for when it is not. --> |
|
210 <property name="eclipse-home" value="${basedir}\..\.."/> |
|
211 |
|
212 <!-- sets the properties eclipse-home, and library-file --> |
|
213 <property name="plugin-name" value="org.eclipse.jdt.ui.tests.refactoring"/> |
|
214 <property name="library-file" |
|
215 value="${eclipse-home}/plugins/org.eclipse.test/library.xml"/> |
|
216 |
|
217 <!-- This target holds all initialization code that needs to be done for --> |
|
218 <!-- all tests that are to be run. Initialization for individual tests --> |
|
219 <!-- should be done within the body of the suite target. --> |
|
220 <target name="init"> |
|
221 <tstamp/> |
|
222 <delete> |
|
223 <fileset dir="${eclipse-home}" includes="org*.xml"/> |
|
224 </delete> |
|
225 </target> |
|
226 |
|
227 <!-- This target defines the tests that need to be run. --> |
|
228 <target name="suite"> |
|
229 <property name="refactoring-folder" |
|
230 value="${eclipse-home}/refactoring_folder"/> |
|
231 <delete dir="${refactoring-folder}" quiet="true"/> |
|
232 <ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}"> |
|
233 <property name="data-dir" value="${refactoring-folder}"/> |
|
234 <property name="plugin-name" value="${plugin-name}"/> |
|
235 <property name="classname" |
|
236 value="org.eclipse.jdt.ui.tests.refactoring.all.AllAllRefactoringTests"/> |
|
237 </ant> |
|
238 </target> |
|
239 |
|
240 <!-- This target holds code to cleanup the testing environment after --> |
|
241 <!-- after all of the tests have been run. You can use this target to --> |
|
242 <!-- delete temporary files that have been created. --> |
|
243 <target name="cleanup"> |
|
244 </target> |
|
245 |
|
246 <!-- This target runs the test suite. Any actions that need to happen --> |
|
247 <!-- after all the tests have been run should go here. --> |
|
248 <target name="run" depends="init,suite,cleanup"> |
|
249 <ant target="collect" antfile="${library-file}" dir="${eclipse-home}"> |
|
250 <property name="includes" value="org*.xml"/> |
|
251 <property name="output-file" value="${plugin-name}.xml"/> |
|
252 </ant> |
|
253 </target> |
|
254 </project> |
|
255 </pre></code> |
|
256 |
|
257 <h2>Known Issues:</h2><h4>Problem 1: Issues with Ant 1.3</h4><p>Ant 1.3 has some issues when used with the <style> tag and absolute |
|
258 paths. Also, any tests that use the <code>System.exit()</code> call will not |
|
259 log their output properly when using Ant 1.3.</p><h4>Problem 2: ECLIPSE_HOME</h4><p>The test suites need to know where the root of the eclipse install is on the |
|
260 file system (the ECLIPSE_HOME variable). However, this variable is only defined |
|
261 in JDT. The ${eclipse-home} property can be set to a reasonable default inside |
|
262 of the test.xml script. Then tests can be run from the standard Ant window, |
|
263 without having to specify -Declipse-home=%ECLIPSE_HOME%. If a value for |
|
264 ${eclipse-home} does get passed in, the default (specified in test.xml) gets |
|
265 overridden. The parameter is passed in by the build mechanism. For most cases, |
|
266 the value "${basedir}/../.." is a reasonable default.</p><h4>Problem 3: Ugly reference to library.xml</h4><p>org.eclipse.test should provide Ant tasks, not template scripts.</p><h4>Problem 4: No console output</h4><p>When you run a TestSuite using the standard JUnit, it normally outputs a |
|
267 series of dots to the console so that you can track the TestSuite's progress. |
|
268 It is not possible to add this feature to the automated testing process at this |
|
269 point in time.</p><h4>Problem 5: Ant java task on Linux</h4><p>Ant expects there to be a java executable on the system path. Furthermore, the executable |
|
270 must be a real file, not a symbolic link. If the test framework is throwing an exception |
|
271 <code>java.io.IOException: java: not found</code>, ensure that the java executable is on |
|
272 your system path.</p><h4>Problem 6: PDE</h4><p>The testing framework currently has no knowledge of PDE. In order to run the automated |
|
273 you must be running a self hosting environment with a full development and target Eclipse.</p> |
|
274 |
|
275 </body> |
|
276 |
|
277 </html> |