sbsv2/raptor/notes/localresourcecopying.txt
author timothy.murphy@nokia.com
Tue, 11 May 2010 21:10:52 +0100
branchfix
changeset 551 b41ce675e7b2
parent 547 9fe7d0ab0f8f
permissions -rw-r--r--
fix: pull less modules into depcrunch. fix timing test. Don't report RPP unnecessarily in resource FLM.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     1
Resources were copied using make rules.  This is sensible in local
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     2
machine builds but non-optimal in cluster builds.  It is entirely IO
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
     3
bound so that instead of benefitting from running on the cluster it
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
     4
simply creates more IO as files need to be transferred over the network,
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
     5
possibly multiple times.
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
     6
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
     7
This change introduces the <finalcopy> tag to the log which the frontend
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
     8
reads in a new "default" filter called FilterCopyFile.  Thus the python
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
     9
frontend does the copying rather than the cluster build engine.
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    10
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    11
This happens at the end of each invocation of the build engine or "stage".
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    12
Since resources are built in their own stage, the copying is completed
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    13
before build tasks in the other stages require them.  The copied files
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    14
are not needed in the resource stage itself.
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    15
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    16
The filter uses <progress:end ...> tags to determine when a stage
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    17
is finished and this requires that the timing feature be switched on
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    18
permanently.
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    19
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    20
The format of the tag is:
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    21
<finalcopy source="sourcefilename">dest_filename1 dest_filename2 ...</finalcopy>
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    22
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    23
Spaces may not be used in filenames. The sequence "%20" may be used
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 533
diff changeset
    24
instead.