diff -r 238f4cb8391f -r ad8ffc8e1982 sbsv2/raptor/notes/localresourcecopying.txt --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/sbsv2/raptor/notes/localresourcecopying.txt Mon Jul 26 11:04:29 2010 +0800 @@ -0,0 +1,24 @@ +Resources were copied using make rules. This is sensible in local +machine builds but non-optimal in cluster builds. It is entirely IO +bound so that instead of benefitting from running on the cluster it +simply creates more IO as files need to be transferred over the network, +possibly multiple times. + +This change introduces the tag to the log which the frontend +reads in a new "default" filter called FilterCopyFile. Thus the python +frontend does the copying rather than the cluster build engine. + +This happens at the end of each invocation of the build engine or "stage". +Since resources are built in their own stage, the copying is completed +before build tasks in the other stages require them. The copied files +are not needed in the resource stage itself. + +The filter uses tags to determine when a stage +is finished and this requires that the timing feature be switched on +permanently. + +The format of the tag is: +dest_filename1 dest_filename2 ... + +Spaces may not be used in filenames. The sequence "%20" may be used +instead.