author | Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com> |
Fri, 17 Sep 2010 09:07:27 +0300 | |
branch | RCL_3 |
changeset 1 | 9d347b658349 |
parent 0 | 4f2f89ce4247 |
permissions | -rw-r--r-- |
0
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
1 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
2 |
* Add more test cases. Categories we'd like to cover (with reasonably |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
3 |
real-world tests, preferably not microbenchmarks) include: |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
4 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
5 |
(X marks the ones that are fairly well covered now). |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
6 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
7 |
X math (general) |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
8 |
X bitops |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
9 |
X 3-d (the math bits) |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
10 |
- crypto / encoding |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
11 |
X string processing |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
12 |
- regexps |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
13 |
- date processing |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
14 |
- array processing |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
15 |
- control flow |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
16 |
- function calls / recursion |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
17 |
- object access (unclear if it is possible to make a realistic |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
18 |
benchmark that isolates this) |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
19 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
20 |
I'd specifically like to add all the computer language shootout |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
21 |
tests that Mozilla is using. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
22 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
23 |
* Normalize tests. Most of the test cases available have a repeat |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
24 |
count of some sort, so the time they take can be tuned. The tests |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
25 |
should be tuned so that each category contributes about the same |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
26 |
total, and so each test in each category contributes about the same |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
27 |
amount. The question is, what implementation should be the baseline? |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
28 |
My current thought is to either pick some specific browser on a |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
29 |
specific platform (IE 7 or Firefox 2 perhaps), or try to target the |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
30 |
average that some set of same-generation release browsers get on |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
31 |
each test. The latter is more work. IE7 is probably a reasonable |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
32 |
normalization target since it is the latest version of the most |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
33 |
popular browser, so results on this benchmark will tell you how much |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
34 |
you have to gain or lose by using a different browser. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
35 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
36 |
* Instead of using the standard error, the correct way to calculate |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
37 |
a 95% confidence interval for a small sample is the t-test. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
38 |
<http://en.wikipedia.org/wiki/Student%27s_t-test>. Basically this involves |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
39 |
using values from a 2-tailed t-distribution table instead of 1.96 to |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
40 |
multiply by the error function, a table is available at |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
41 |
<http://www.medcalc.be/manual/t-distribution.php> |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
42 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
43 |
* Add support to compare two different engines (or two builds of the |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
44 |
same engine) interleaved. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
45 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
46 |
* Add support to compare two existing sets of saved results. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
47 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
48 |
* Allow repeat count to be controlled from the browser-hosted version |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
49 |
and the WebKitTools wrapper script. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
50 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
51 |
* Add support to run only a subset of the tests (both command-line and |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
52 |
web versions). |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
53 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
54 |
* Add a profile mode for the command-line version that runs the tests |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
55 |
repeatedly in the same command-line interpreter instance, for ease |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
56 |
of profiling. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
57 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
58 |
* Make the browser-hosted version prettier, both in general design and |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
59 |
maybe using bar graphs for the output. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
60 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
61 |
* Make it possible to track change over time and generate a graph per |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
62 |
result showing result and error bar for each version. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
63 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
64 |
* Hook up to automated testing / buildbot infrastructure. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
65 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
66 |
* Possibly... add the ability to download iBench from its original |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
67 |
server, pull out the JS test content, preprocess it, and add it as a |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
68 |
category to the benchmark. |
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
69 |
|
4f2f89ce4247
Revision: 201037
Dremov Kirill (Nokia-D-MSW/Tampere) <kirill.dremov@nokia.com>
parents:
diff
changeset
|
70 |
* Profit. |