|
18
|
1 |
Resources were copied using make rules. This is sensible in local
|
|
|
2 |
machine builds but non-optimal in cluster builds. It is entirely IO
|
|
|
3 |
bound so that instead of benefitting from running on the cluster it
|
|
|
4 |
simply creates more IO as files need to be transferred over the network,
|
|
|
5 |
possibly multiple times.
|
|
|
6 |
|
|
|
7 |
This change introduces the <finalcopy> tag to the log which the frontend
|
|
|
8 |
reads in a new "default" filter called FilterCopyFile. Thus the python
|
|
|
9 |
frontend does the copying rather than the cluster build engine.
|
|
|
10 |
|
|
|
11 |
This happens at the end of each invocation of the build engine or "stage".
|
|
|
12 |
Since resources are built in their own stage, the copying is completed
|
|
|
13 |
before build tasks in the other stages require them. The copied files
|
|
|
14 |
are not needed in the resource stage itself.
|
|
|
15 |
|
|
|
16 |
The filter uses <progress:end ...> tags to determine when a stage
|
|
|
17 |
is finished and this requires that the timing feature be switched on
|
|
|
18 |
permanently.
|
|
|
19 |
|
|
|
20 |
The format of the tag is:
|
|
|
21 |
<finalcopy source="sourcefilename">dest_filename1 dest_filename2 ...</finalcopy>
|
|
|
22 |
|
|
|
23 |
Spaces may not be used in filenames. The sequence "%20" may be used
|
|
|
24 |
instead.
|