|
637
|
1 |
<html>
|
|
|
2 |
<head>
|
|
|
3 |
<title>Eclipse Testing Instructions</title>
|
|
|
4 |
</head>
|
|
|
5 |
|
|
|
6 |
<body>
|
|
|
7 |
<h1>The Eclipse Test Framework</h1><p>Last Modified: June 13, 2007</p><p><font size="+1"><br>
|
|
|
8 |
Introduction</font></p><p>The testing framework is comprised of the org.eclipse.test plugin and the org.eclipse.ant.optional.junit
|
|
|
9 |
fragment. </p><p>These two projects are available from the dev.eclipse.org repository and are
|
|
|
10 |
included in the eclipse-test-framework-<buildId>.zip from the eclipse.org
|
|
|
11 |
downloads page.</p><p><font size="+1">Building and Installation</font></p><p>Since the org.eclipse.test plugin is stored in the repository in source-code
|
|
|
12 |
form, it needs to be compiled before it can be used. The org.eclipse.ant.optional.junit
|
|
|
13 |
fragment does not contain any source and can be used as is from the repository.</p><ol>
|
|
|
14 |
<li>Turn of automatic builds. Window->Preferences->Workbench and uncheck "Perform
|
|
|
15 |
build automatically on resource modification"</li><li>Load org.eclipse.test into your workspace.</li><li>Right-click on the org.eclipse.test project in either the Navigator or Packages
|
|
|
16 |
view. Select 'Rebuild Project' from the context menu. This will compile the
|
|
|
17 |
entire org.eclipse.test plugin.</li><li>Finally, copy the org.eclipse.test plugin into your target Eclipse.</li>
|
|
|
18 |
<li>The org.eclipse.ant.optional.junit fragment only needs to be present in
|
|
|
19 |
the environment of the Eclipse that is overseeing the test process. If you
|
|
|
20 |
are running the test script from within the Workbench, this means that the
|
|
|
21 |
fragments need to be present withinn your development Eclipse. If you are
|
|
|
22 |
running the tests from the command line, then the fragments will need to be
|
|
|
23 |
present in your target Eclipse.</li></ol>
|
|
|
24 |
<p><font size="+1">Setup</font></p>
|
|
|
25 |
<p>Follow the steps given above to build and install all of the neccessary plugins
|
|
|
26 |
and fragments. Please note that the current version of the test framework is
|
|
|
27 |
not compatible with the PDE notion of self-hosting. If you want to run the tests,
|
|
|
28 |
you will need to setup a full target Eclipse so that the testing framework can
|
|
|
29 |
detect everything that is needed.</p>
|
|
|
30 |
<p>If you are writing tests for one or more Eclipse plugins, then you should
|
|
|
31 |
create a separate plugin for tests. The test plugin will contain all of the
|
|
|
32 |
tests that will be run on your development plugins, as well as defining how
|
|
|
33 |
those tests get run.</p>
|
|
|
34 |
|
|
|
35 |
<p>If you are not writing tests for an Eclipse plugin, then you should look into
|
|
|
36 |
using JUnit on its own to test your work. JUnit is designed for testing Java
|
|
|
37 |
code, and by default has no knowledge of Eclipse. There are separate mechanisms
|
|
|
38 |
for using JUnit on Java code in Eclipse. See the documentation provided here:</p>
|
|
|
39 |
<p> <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/jdt-ui-home/plugins/org.eclipse.jdt.junit/index.html">
|
|
|
40 |
JUnit Support in Eclipse</a><br>
|
|
|
41 |
<a href="http://dev.eclipse.org:8080/help/content/help:/org.eclipse.jdt.doc.user/tasks/tasks-207.htm?resultof=%6a%75%6e%69%74%20&toc=%2Forg.eclipse.jdt.doc.user%2Ftoc.xml">
|
|
|
42 |
Using JUnit</a></p>
|
|
|
43 |
|
|
|
44 |
<p>Once a test plugin has been created, you must then create an Ant script that
|
|
|
45 |
will run all of the tests. Create a file called 'test.xml' in the root of your plug-in or bundle.
|
|
|
46 |
This is an Ant file that defines how each test is going to be run. The 'test.xml'
|
|
|
47 |
file is like a DOS batch file that scripts the entire testing process. Typically,
|
|
|
48 |
the 'test.xml' file will contain targets for setting up the test run, executing
|
|
|
49 |
the tests, cleaning up afterwards, and running the entire process.</p>
|
|
|
50 |
|
|
|
51 |
<p><font size="+1">Converting existing Test Suites</font></p>
|
|
|
52 |
|
|
|
53 |
<p>If you are converting an existing set of tests to use the new framework, the
|
|
|
54 |
actual tests that have been written should not need much change.</p>
|
|
|
55 |
<p>If you have tests in multiple plugins, move these to a single test plug-in for your component.</p>
|
|
|
56 |
|
|
|
57 |
<p>Make sure that the tests are defined in a plug-in. This is probably the most
|
|
|
58 |
common cause of confusion in the entire test process. Your tests need to be
|
|
|
59 |
in a plug-in so that Eclipse can find them when it tries to load them.</p>
|
|
|
60 |
|
|
|
61 |
<p><font size="+1">Creating new Test Suites</font></p>
|
|
|
62 |
|
|
|
63 |
<p>Creating new JUnit tests for an Eclipse plugin should be no more difficult
|
|
|
64 |
than writing standard JUnit tests. Since the framework allows tests to be run
|
|
|
65 |
inside of a working Eclipse, any tests that you write have available to them
|
|
|
66 |
any of the methods supplied by the Eclipse platform, provided that you add the
|
|
|
67 |
appropriate dependencies to your tests' manifest.</p>
|
|
|
68 |
|
|
|
69 |
<p><font size="+1">Performance Issues</font></p>
|
|
|
70 |
<p>You should keep in mind the number of times that Eclipse needs to be
|
|
|
71 |
started. Launching Eclipse has a substantial cost in terms of runtime. To
|
|
|
72 |
minimize the number of times the platform is started, you should consider
|
|
|
73 |
writing a TestSuite called AllTests for each of your test plugins. AllTests
|
|
|
74 |
should invoke each of the tests that you want to run for a particular plugin.
|
|
|
75 |
The 'test.xml' file can then run the AllTests class, which will run all of your
|
|
|
76 |
tests, but the platform will only ever be started once for each of your test
|
|
|
77 |
plugins.</p><p>Note: Sometimes tests involve shutting down, restarting, and testing the state
|
|
|
78 |
of metadata that was written to disk. These session tests will require Eclipse
|
|
|
79 |
to be launched several times in sequence, which will increase the runtime of
|
|
|
80 |
your tests. This cannot be avoided.</p><p> </p>
|
|
|
81 |
|
|
|
82 |
<p><font size="+1">Running the Test Suite from the UI</font></p><p>Right click on the test.xml file and select 'Run Ant...' from the pull-down
|
|
|
83 |
menu. The Running Ant dialog box pops up. From the list of targets, select the
|
|
|
84 |
one that runs all of your tests. If you are using the example file provided
|
|
|
85 |
below, this target is called 'Run', and will be selected by default. Hit the
|
|
|
86 |
'Finish' button to start the test process.</p>
|
|
|
87 |
|
|
|
88 |
<p><font size="+1">Running the Test Suite from the command line</font></p>
|
|
|
89 |
<p>When the test suites are invoked automatically, they are run from command
|
|
|
90 |
line. From the ${eclipse-home} directory, the command to use is:</p>
|
|
|
91 |
<code>java -jar plugins\org.eclipse.equinox.launcher_<version>.jar -application org.eclipse.ant.core.antRunner -buildfile ${test-plugin-path}\test.xml
|
|
|
92 |
-Declipse-home=${eclipse-home} -Dos=<operating system> -Dws=<windowing system> -Darch=<architecture></code>
|
|
|
93 |
|
|
|
94 |
<p>Individual tests can also be invoked directly. From the ${eclipse-home}
|
|
|
95 |
directory, use the command:</p><code>java -jar plugins\org.eclipse.equinox.launcher_<version>.jar
|
|
|
96 |
-application ${launcher} -os <operating system> -ws <windowing system> -arch <architecture> -dev bin -testpluginname ${plugin-name}
|
|
|
97 |
-classname ${test-classname} formattter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,<path to output file with .xml extension></code>
|
|
|
98 |
<p>where ${launcher} is one of: <i>org.eclipse.test.uitestapplication</i> or <i>org.eclipse.test.coretestapplication</i>
|
|
|
99 |
depending on whether or not you want to run your tests within an active Workbench.</p><p><font size="+1">Output</font></p><p>By default, output from each test run is logged as XML. For each test that
|
|
|
100 |
is run, the output is logged into the file called ${classname}.xml. The individual
|
|
|
101 |
XML files are located in the ${eclipse-home} directory. When the test run is
|
|
|
102 |
finished, you can call the <b>"collect"</b> target in the library
|
|
|
103 |
file, which will collect the XML into a single file. See below for an example
|
|
|
104 |
of how to use this target correctly.</p>
|
|
|
105 |
|
|
|
106 |
<h3>Other issues</h3><h4>Headless Testing vs. UI testing</h4><p>Many plugin tests will not need the Workbench active in order to run. Indeed,
|
|
|
107 |
only the minimum number of plugins needed to run the plugin being tested need
|
|
|
108 |
to be present when testing in a target Eclipse. There are two different Ant
|
|
|
109 |
targets provided for running Eclipse plugin tests. One target starts the entire
|
|
|
110 |
Workbench. The other starts Eclipse with the minimum number of plugins needed.
|
|
|
111 |
It is up to you to decide which target is most appropriate. For examples, look
|
|
|
112 |
at the <b>"ui-test"</b> and <b>"core-test"</b> targets below.</p>
|
|
|
113 |
|
|
|
114 |
<h4>Disposing the Display</h4><p>Some low-level tests for the Eclipse platform take actions that are not
|
|
|
115 |
normally possible inside of Eclipse. An example of this behaviour would be
|
|
|
116 |
disposing the display. While this action can be performed while running the UI,
|
|
|
117 |
it will also kill the UI for the copy of Eclipse that is running, and cause
|
|
|
118 |
errors when the Workbench tries to shutdown. If you need to test disposing the
|
|
|
119 |
display, or other similar actions, your tests should be running without a UI. </p>
|
|
|
120 |
|
|
|
121 |
<h4>Tests that are not plugins</h4><p>It is very easy to forget to define your tests inside of a plugin. If your
|
|
|
122 |
tests will not load properly, make sure that a plug-in manifest exists in your
|
|
|
123 |
test project, and also that the plugin is being loaded by the platform. Make
|
|
|
124 |
sure that all of the dependencies are satisfied for your test plugin.</p>
|
|
|
125 |
|
|
|
126 |
<p><font size="+1">Interface</font></p><p>The org.eclipse.test plugin defines many useful Ant tasks/targets to aid developers
|
|
|
127 |
in writing their test.xml scripts. Currently, there is only Ant targets defined,
|
|
|
128 |
which can be called using Ant's built-in <ant> task. To use these targets,
|
|
|
129 |
add the following line to the top of your script, and reference the ${library-file}
|
|
|
130 |
property when calling <ant>:</p><code><property name="library-file"
|
|
|
131 |
value="${eclipse-home}/fragments/org.eclipse.test/library.xml"/> </code>
|
|
|
132 |
|
|
|
133 |
<p>The targets that are defined are: </p><ul>
|
|
|
134 |
<li><b>ui-test</b> - This target runs a JUnit test suite inside of an Eclipse
|
|
|
135 |
Workbench. This target is mainly for testing plugins that use the Eclipse
|
|
|
136 |
UI and JFace. The output of the test pass is automatically logged in an XML
|
|
|
137 |
file called ${classname}.xml. It takes four arguments: </li><ul>
|
|
|
138 |
<li><i>data-dir</i> - The
|
|
|
139 |
data directory of the Eclipse that gets run</li><li><i>plugin-name</i> -
|
|
|
140 |
The name of the plugin that the test suite is defined in</li><li><i>classname</i> -
|
|
|
141 |
The name of the class that the test suite is defined in</li><li><i>vmargs</i> - An
|
|
|
142 |
optional argument string to pass to the VM running the tests</li></ul><p>For example, the following code will run the test
|
|
|
143 |
org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a new Eclipse
|
|
|
144 |
workbench. It passes the string "-Dbaz=true" to the VM. The Eclipse
|
|
|
145 |
stores its metadata in the directory "data-folder".
|
|
|
146 |
|
|
|
147 |
<code><pre>
|
|
|
148 |
<ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}">
|
|
|
149 |
<property name="data-dir" value="data-folder"/>
|
|
|
150 |
<property name="plugin-name" value="org.eclipse.foo"/>
|
|
|
151 |
<property name="classname" value="org.eclipse.foo.bar.MyTest"/>
|
|
|
152 |
<property name="vmargs" value="-Dbaz=true"/>
|
|
|
153 |
</ant>
|
|
|
154 |
</pre></code>
|
|
|
155 |
|
|
|
156 |
<li><b>core-test</b> - This target runs a JUnit test suite inside of an IPlatformRunnable.
|
|
|
157 |
This target is for testing plugins that use the Eclipse platform, but do not
|
|
|
158 |
require a UI to be running. The output of the test pass is automatically logged
|
|
|
159 |
in an XML file called ${classname}.xml. It takes four arguments: </li><ul>
|
|
|
160 |
<li><i>data-dir</i> - The
|
|
|
161 |
data directory of the Eclipse that gets run</li><li><i>plugin-name</i> -
|
|
|
162 |
The name of the plugin that the test suite is defined in</li><li><i>classname</i> -
|
|
|
163 |
The name of the class that the test suite is defined in</li><li><i>vmargs</i> - An
|
|
|
164 |
optional argument string to pass to the VM running the tests</li></ul><p>For example, the following code will run the test
|
|
|
165 |
org.eclipse.foo.bar.MyTest in the plugin org.eclipse.foo in a headless Eclipse.
|
|
|
166 |
It passes the string "-Dbaz=true" to the VM. The Eclipse stores its
|
|
|
167 |
metadata in the directory "data-folder".
|
|
|
168 |
|
|
|
169 |
<code><pre>
|
|
|
170 |
<ant target="core-test" antfile="${library-file}" dir="${eclipse-home}">
|
|
|
171 |
<property name="data-dir" value="data-folder"/>
|
|
|
172 |
<property name="plugin-name" value="org.eclipse.foo"/>
|
|
|
173 |
<property name="classname" value="org.eclipse.foo.bar.MyTest"/>
|
|
|
174 |
<property name="vmargs" value="-Dbaz=true"/>
|
|
|
175 |
</ant>
|
|
|
176 |
</pre></code>
|
|
|
177 |
|
|
|
178 |
<li><b>collect</b> - This
|
|
|
179 |
target collects the XML files that are produced over the course of the
|
|
|
180 |
test script. It takes two arguments: </li><ul>
|
|
|
181 |
<li><i>includes</i> - A
|
|
|
182 |
pattern matching all XML files to be included in the test report. This
|
|
|
183 |
argument is typically "org*.xml"</li><li><i>output-file</i> -
|
|
|
184 |
The filename where the output of the test gets stored. For the automated
|
|
|
185 |
build process, this file should be ${pluginname}.xml, and be located in
|
|
|
186 |
the ECLIPSE_HOME directory.</li></ul><p>For example, the following code collects all of the files
|
|
|
187 |
matching the pattern "org*.xml" in the directory ${eclipse-home},
|
|
|
188 |
into the file named "logfile.xml".
|
|
|
189 |
|
|
|
190 |
<code><pre>
|
|
|
191 |
<ant target="collect" antfile="${library-file}" dir="${eclipse-home}">
|
|
|
192 |
<property name="includes" value="org*.xml"/>
|
|
|
193 |
<property name="output-file" value="logfile.xml"/>
|
|
|
194 |
</ant>
|
|
|
195 |
</pre></code>
|
|
|
196 |
|
|
|
197 |
|
|
|
198 |
</ul><h2>Examples:</h2><p>Included is the 'test.xml' file from the org.eclipse.jdt.ui.tests.refactoring plugin. This
|
|
|
199 |
file controls all of the automated testing that is done for the org.eclipse.jdt.ui.tests.refactoring
|
|
|
200 |
plugin. It can be run from inside of Eclipse or from the command line. It is
|
|
|
201 |
intended to serve as a template file for testing any other plugin.</p><p>Notice that the structure of the file roughly mirrors that of a JUnit test.
|
|
|
202 |
Targets are defined for setting up the tests, defining what needs to be done,
|
|
|
203 |
cleaning up after the tests, and running everything in the right order.</p>
|
|
|
204 |
<code><pre>
|
|
|
205 |
<?xml version="1.0" encoding="UTF-8"?>
|
|
|
206 |
|
|
|
207 |
<project name="testsuite" default="run" basedir=".">
|
|
|
208 |
<!-- The property ${eclipse-home} should be passed into this script -->
|
|
|
209 |
<!-- Set a meaningful default value for when it is not. -->
|
|
|
210 |
<property name="eclipse-home" value="${basedir}\..\.."/>
|
|
|
211 |
|
|
|
212 |
<!-- sets the properties eclipse-home, and library-file -->
|
|
|
213 |
<property name="plugin-name" value="org.eclipse.jdt.ui.tests.refactoring"/>
|
|
|
214 |
<property name="library-file"
|
|
|
215 |
value="${eclipse-home}/plugins/org.eclipse.test/library.xml"/>
|
|
|
216 |
|
|
|
217 |
<!-- This target holds all initialization code that needs to be done for -->
|
|
|
218 |
<!-- all tests that are to be run. Initialization for individual tests -->
|
|
|
219 |
<!-- should be done within the body of the suite target. -->
|
|
|
220 |
<target name="init">
|
|
|
221 |
<tstamp/>
|
|
|
222 |
<delete>
|
|
|
223 |
<fileset dir="${eclipse-home}" includes="org*.xml"/>
|
|
|
224 |
</delete>
|
|
|
225 |
</target>
|
|
|
226 |
|
|
|
227 |
<!-- This target defines the tests that need to be run. -->
|
|
|
228 |
<target name="suite">
|
|
|
229 |
<property name="refactoring-folder"
|
|
|
230 |
value="${eclipse-home}/refactoring_folder"/>
|
|
|
231 |
<delete dir="${refactoring-folder}" quiet="true"/>
|
|
|
232 |
<ant target="ui-test" antfile="${library-file}" dir="${eclipse-home}">
|
|
|
233 |
<property name="data-dir" value="${refactoring-folder}"/>
|
|
|
234 |
<property name="plugin-name" value="${plugin-name}"/>
|
|
|
235 |
<property name="classname"
|
|
|
236 |
value="org.eclipse.jdt.ui.tests.refactoring.all.AllAllRefactoringTests"/>
|
|
|
237 |
</ant>
|
|
|
238 |
</target>
|
|
|
239 |
|
|
|
240 |
<!-- This target holds code to cleanup the testing environment after -->
|
|
|
241 |
<!-- after all of the tests have been run. You can use this target to -->
|
|
|
242 |
<!-- delete temporary files that have been created. -->
|
|
|
243 |
<target name="cleanup">
|
|
|
244 |
</target>
|
|
|
245 |
|
|
|
246 |
<!-- This target runs the test suite. Any actions that need to happen -->
|
|
|
247 |
<!-- after all the tests have been run should go here. -->
|
|
|
248 |
<target name="run" depends="init,suite,cleanup">
|
|
|
249 |
<ant target="collect" antfile="${library-file}" dir="${eclipse-home}">
|
|
|
250 |
<property name="includes" value="org*.xml"/>
|
|
|
251 |
<property name="output-file" value="${plugin-name}.xml"/>
|
|
|
252 |
</ant>
|
|
|
253 |
</target>
|
|
|
254 |
</project>
|
|
|
255 |
</pre></code>
|
|
|
256 |
|
|
|
257 |
<h2>Known Issues:</h2><h4>Problem 1: Issues with Ant 1.3</h4><p>Ant 1.3 has some issues when used with the <style> tag and absolute
|
|
|
258 |
paths. Also, any tests that use the <code>System.exit()</code> call will not
|
|
|
259 |
log their output properly when using Ant 1.3.</p><h4>Problem 2: ECLIPSE_HOME</h4><p>The test suites need to know where the root of the eclipse install is on the
|
|
|
260 |
file system (the ECLIPSE_HOME variable). However, this variable is only defined
|
|
|
261 |
in JDT. The ${eclipse-home} property can be set to a reasonable default inside
|
|
|
262 |
of the test.xml script. Then tests can be run from the standard Ant window,
|
|
|
263 |
without having to specify -Declipse-home=%ECLIPSE_HOME%. If a value for
|
|
|
264 |
${eclipse-home} does get passed in, the default (specified in test.xml) gets
|
|
|
265 |
overridden. The parameter is passed in by the build mechanism. For most cases,
|
|
|
266 |
the value "${basedir}/../.." is a reasonable default.</p><h4>Problem 3: Ugly reference to library.xml</h4><p>org.eclipse.test should provide Ant tasks, not template scripts.</p><h4>Problem 4: No console output</h4><p>When you run a TestSuite using the standard JUnit, it normally outputs a
|
|
|
267 |
series of dots to the console so that you can track the TestSuite's progress.
|
|
|
268 |
It is not possible to add this feature to the automated testing process at this
|
|
|
269 |
point in time.</p><h4>Problem 5: Ant java task on Linux</h4><p>Ant expects there to be a java executable on the system path. Furthermore, the executable
|
|
|
270 |
must be a real file, not a symbolic link. If the test framework is throwing an exception
|
|
|
271 |
<code>java.io.IOException: java: not found</code>, ensure that the java executable is on
|
|
|
272 |
your system path.</p><h4>Problem 6: PDE</h4><p>The testing framework currently has no knowledge of PDE. In order to run the automated
|
|
|
273 |
you must be running a self hosting environment with a full development and target Eclipse.</p>
|
|
|
274 |
|
|
|
275 |
</body>
|
|
|
276 |
|
|
|
277 |
</html>
|