sbsv2/raptor/notes/localresourcecopying.txt
changeset 18 de5b887c98f7
equal deleted inserted replaced
14:eb060913c963 18:de5b887c98f7
       
     1 Resources were copied using make rules.  This is sensible in local
       
     2 machine builds but non-optimal in cluster builds.  It is entirely IO
       
     3 bound so that instead of benefitting from running on the cluster it
       
     4 simply creates more IO as files need to be transferred over the network,
       
     5 possibly multiple times.
       
     6 
       
     7 This change introduces the <finalcopy> tag to the log which the frontend
       
     8 reads in a new "default" filter called FilterCopyFile.  Thus the python
       
     9 frontend does the copying rather than the cluster build engine.
       
    10 
       
    11 This happens at the end of each invocation of the build engine or "stage".
       
    12 Since resources are built in their own stage, the copying is completed
       
    13 before build tasks in the other stages require them.  The copied files
       
    14 are not needed in the resource stage itself.
       
    15 
       
    16 The filter uses <progress:end ...> tags to determine when a stage
       
    17 is finished and this requires that the timing feature be switched on
       
    18 permanently.
       
    19 
       
    20 The format of the tag is:
       
    21 <finalcopy source="sourcefilename">dest_filename1 dest_filename2 ...</finalcopy>
       
    22 
       
    23 Spaces may not be used in filenames. The sequence "%20" may be used
       
    24 instead.