sbsv2/raptor/notes/localresourcecopying.txt
author jjkang
Wed, 23 Jun 2010 17:27:59 +0800
changeset 591 22486c9c7b15
permissions -rw-r--r--
raptor v2.14.0
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
591
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
     1
Resources were copied using make rules.  This is sensible in local
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
     2
machine builds but non-optimal in cluster builds.  It is entirely IO
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
     3
bound so that instead of benefitting from running on the cluster it
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
     4
simply creates more IO as files need to be transferred over the network,
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
     5
possibly multiple times.
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
     6
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
     7
This change introduces the <finalcopy> tag to the log which the frontend
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
     8
reads in a new "default" filter called FilterCopyFile.  Thus the python
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
     9
frontend does the copying rather than the cluster build engine.
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    10
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    11
This happens at the end of each invocation of the build engine or "stage".
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    12
Since resources are built in their own stage, the copying is completed
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    13
before build tasks in the other stages require them.  The copied files
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    14
are not needed in the resource stage itself.
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    15
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    16
The filter uses <progress:end ...> tags to determine when a stage
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    17
is finished and this requires that the timing feature be switched on
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    18
permanently.
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    19
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    20
The format of the tag is:
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    21
<finalcopy source="sourcefilename">dest_filename1 dest_filename2 ...</finalcopy>
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    22
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    23
Spaces may not be used in filenames. The sequence "%20" may be used
22486c9c7b15 raptor v2.14.0
jjkang
parents:
diff changeset
    24
instead.