1 Resources were copied using make rules. This is sensible in local |
1 Resources were copied using make rules. This is sensible in local |
2 machine builds but non-optimal in cluster builds. It is entirely IO |
2 machine builds but non-optimal in cluster builds. It is entirely IO |
3 bound and the cluster aspect simply creates more IO as files need to |
3 bound so that instead of benefitting from running on the cluster it |
4 be transferred over the network, possibly multiple times. This change |
4 simply creates more IO as files need to be transferred over the network, |
5 introduces the <copy> tag to the log which the frontend reads in a new |
5 possibly multiple times. |
6 "default" filter called FilterCopyFile. Thus the python frontend does |
|
7 the copying rather than the cluster build engine. |
|
8 |
6 |
9 This happens at the end of the build and since resources are built in |
7 This change introduces the <finalcopy> tag to the log which the frontend |
10 their own "stage" it's is all completed before any other build tasks |
8 reads in a new "default" filter called FilterCopyFile. Thus the python |
11 are invoked. The copied files are not needed in the resource stage |
9 frontend does the copying rather than the cluster build engine. |
12 itself so it is ok that it happens at the end of that. |
|
13 |
10 |
|
11 This happens at the end of each invocation of the build engine or "stage". |
|
12 Since resources are built in their own stage, the copying is completed |
|
13 before build tasks in the other stages require them. The copied files |
|
14 are not needed in the resource stage itself. |
|
15 |
|
16 The filter uses <progress:end ...> tags to determine when a stage |
|
17 is finished and this requires that the timing feature be switched on |
|
18 permanently. |
14 |
19 |
15 The format of the tag is: |
20 The format of the tag is: |
16 <copy source="sourcefilename">dest_filename1 dest_filename2 ...</copy> |
21 <finalcopy source="sourcefilename">dest_filename1 dest_filename2 ...</finalcopy> |
17 |
22 |
18 Spaces may not be used in filenames. The sequence "%20" may be used instead. |
23 Spaces may not be used in filenames. The sequence "%20" may be used |
|
24 instead. |