author | yiluzhu |
Tue, 25 May 2010 16:26:21 +0100 | |
branch | fix |
changeset 571 | 25e377052d9f |
parent 549 | d633be326c9f |
permissions | -rw-r--r-- |
533
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
1 |
# |
547
9fe7d0ab0f8f
fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents:
537
diff
changeset
|
2 |
# Copyright (c) 2008-2010 Nokia Corporation and/or its subsidiary(-ies). |
533
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
3 |
# All rights reserved. |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
4 |
# This component and the accompanying materials are made available |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
5 |
# under the terms of the License "Eclipse Public License v1.0" |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
6 |
# which accompanies this distribution, and is available |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
7 |
# at the URL "http://www.eclipse.org/legal/epl-v10.html". |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
8 |
# |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
9 |
# Initial Contributors: |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
10 |
# Nokia Corporation - initial contribution. |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
11 |
# |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
12 |
# Contributors: |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
13 |
# |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
14 |
# Description: |
537
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
15 |
# Filter class for copying files in serial in python. This |
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
16 |
# is important in cluster builds where file copying is |
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
17 |
# very inefficient. |
547
9fe7d0ab0f8f
fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents:
537
diff
changeset
|
18 |
# The one-to-many <finalcopy> tag is searched for and copy |
537
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
19 |
# instructions are built up in a hash table. |
547
9fe7d0ab0f8f
fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents:
537
diff
changeset
|
20 |
# <finalcopy source='sourcefilename'>destfilename1 destfilename2 . . . .destfilenameN</copy> |
537
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
21 |
# destinations must be full filenames not directories. |
533
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
22 |
# |
537
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
23 |
# This filter monitors build progress |
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
24 |
# via the <progress> tags and flushes copies as build |
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
25 |
# stages end (e.g. after resource so resources are ready for the next stage) |
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
26 |
# |
533
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
27 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
28 |
import os |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
29 |
import sys |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
30 |
import tempfile |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
31 |
import filter_interface |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
32 |
import shutil |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
33 |
import generic_path |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
34 |
import stat |
547
9fe7d0ab0f8f
fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents:
537
diff
changeset
|
35 |
from raptor_utilities import copyfile |
533
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
36 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
37 |
class FilterCopyFile(filter_interface.Filter): |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
38 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
39 |
def open(self, params): |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
40 |
"initialise" |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
41 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
42 |
self.ok = True |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
43 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
44 |
self.files = {} |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
45 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
46 |
return self.ok |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
47 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
48 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
49 |
def write(self, text): |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
50 |
"process some log text" |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
51 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
52 |
for line in text.splitlines(): |
547
9fe7d0ab0f8f
fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents:
537
diff
changeset
|
53 |
if line.startswith("<finalcopy"): |
9fe7d0ab0f8f
fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents:
537
diff
changeset
|
54 |
source_start = line.find("source='") |
9fe7d0ab0f8f
fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents:
537
diff
changeset
|
55 |
source = line[source_start+8:line.find("'", source_start+8)] |
9fe7d0ab0f8f
fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents:
537
diff
changeset
|
56 |
destinations = line[line.find(">",source_start)+1:line.find("</finalcopy>")].split(" ") |
533
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
57 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
58 |
if source in self.files: |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
59 |
self.files[source].update(destinations) |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
60 |
else: |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
61 |
self.files[source] = set(destinations) |
537
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
62 |
elif line.startswith("<progress:end object_type='makefile' task='build'"): |
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
63 |
self.flushcopies() # perform copies at end of each invocation of the make engine |
547
9fe7d0ab0f8f
fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents:
537
diff
changeset
|
64 |
# to ensure dependencies are in place for the next one. |
533
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
65 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
66 |
return self.ok |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
67 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
68 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
69 |
def summary(self): |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
70 |
"finish off" |
537
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
71 |
self.flushcopies() |
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
72 |
return self.ok |
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
73 |
|
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
74 |
def flushcopies(self): |
533
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
75 |
for source in self.files.keys(): |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
76 |
for dest in self.files[source]: |
549
d633be326c9f
fix: depcrunch was using re.match where re.findall was what was wanted. Add a test.
timothy.murphy@nokia.com
parents:
547
diff
changeset
|
77 |
try: |
d633be326c9f
fix: depcrunch was using re.match where re.findall was what was wanted. Add a test.
timothy.murphy@nokia.com
parents:
547
diff
changeset
|
78 |
copyfile(source, dest) |
d633be326c9f
fix: depcrunch was using re.match where re.findall was what was wanted. Add a test.
timothy.murphy@nokia.com
parents:
547
diff
changeset
|
79 |
except IOError, e: |
d633be326c9f
fix: depcrunch was using re.match where re.findall was what was wanted. Add a test.
timothy.murphy@nokia.com
parents:
547
diff
changeset
|
80 |
print "<error>%s</error>" % str(e) |
537
164e587fef9f
fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents:
534
diff
changeset
|
81 |
self.files = {} |
533
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
82 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
83 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
84 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
85 |
def close(self): |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
86 |
"nop" |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
87 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
88 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
89 |
return self.ok |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
90 |
|
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
91 |
# the end |
408bfff46ad7
fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff
changeset
|
92 |