This is a simple regression testing framework for Arthur.+
−
+
−
There are three parts to it.+
−
+
−
1) datagenerator . which is the main part. it's used to +
−
   run all the tests, create output png files,+
−
   measure rendering performance and output it in+
−
   data.xml file. datagenerator reads in+
−
   framework.ini to figure out where to look for tests.+
−
   if the framework.ini file isn't in the current directory+
−
   you have to specify its location with the -framework+
−
   parameter. by default datagenerator generates output in the+
−
   current directory - to specify output directory pass in+
−
   -output parameter followed by a directory name. if the given+
−
   directory doesn't exist it will be created. +
−
+
−
   it's recommended to always specify -iterations argument (followed+
−
   by the number). if the argument has been specified datagenerator+
−
   will try to render all testcases the given amount of times yielding +
−
   a lot more precise results.+
−
+
−
   if one wants to test just a specified engine -engine argument+
−
   should be specified (followed by the desired engine name).+
−
+
−
   individual testsuites can be run with -suite paramenter.+
−
   individual testcases from testcases can also be reran with+
−
   -testcase paramenter.+
−
+
−
   given files can also be tested individually. if the option -file+
−
   is specified with a SVG file engines will be tested against just+
−
   that file and output will be written directly to stdout (not files).+
−
   it's useful when optimizing one engine with particular case in+
−
   mind.+
−
+
−
   Example command line:+
−
   ./bin/datagenerator -framework data/framework.ini -output sampleout -iterations 5 -engine [ENGINE]+
−
+
−
2) htmlgenerator. which is used to generate html pages out of+
−
   generated output (generated by datagenerator). htmlgenerator also+
−
   reads in the framework.ini file. so the option -framework also+
−
   applies. but htmlgenerator takes a mandatory directory name as the+
−
   second argument. the mandatory directory name is the directory to+
−
   which you output the files generated by datagenerator (either+
−
   whatever followed the -output or the current directory)+
−
+
−
   htmlgenerator will generate html pages that can be viewed to+
−
   manually inspect the results.+
−
+
−
   Example command line:+
−
   ./bin/htmlgenerator -framework data/framework.ini sampleout+
−
+
−
3) performancediff. diffs the performance data between two output+
−
   directories. so assuming you already have data for all the engines,+
−
   you change something in an engine. to see the performance+
−
   difference run+
−
   ./bin/datagenerator -framework data/framework.ini -output newoutput -iterations 5+
−
   and then+
−
   ./bin/performancediff sampleout newoutput+
−
   +
−
   The performance results will be printed out.+
−
+
−
   The two first columns are in ms. The last one is a +
−
   percentage of difference. +
−
+
−
   After the percentage a '+' or a '-' sign followed by a +
−
   number will be sometimes printed.+
−
+
−
   '+' signifies the new results were mathematically better. '-' is of+
−
   course just the opposite.+
−
   the number following '+' is the minimum rendering time the engine+
−
   spent rendering the given file up till this moment.+
−
   the number following '-' is the maximum rendering time the engine+
−
   spent rendering the given file up till this moment.+
−
+
−
   (if the current results are out of the scope from within maximum and+
−
   minimum number then the respective sign is printed out.)+
−
-------------------------------------------------------------------+
−
+
−
Note that the generated output directories can be copied from one+
−
machine to the other. htmlgenerator will generate webpages for all+
−
engines with valid data.xml files. So if you want add NativeWin32 and+
−
NativeMac engines you'll have to have someone with a windows and mac+
−
run datagenerator with "-engine NativeWin32" or "-engine NativeMac"+
−
options and then sending you the respective output directories. +
−