sbsv2/raptor/python/plugins/filter_copyfile.py
author yiluzhu
Tue, 25 May 2010 16:26:21 +0100
branchfix
changeset 571 25e377052d9f
parent 549 d633be326c9f
permissions -rw-r--r--
edit comment
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     1
#
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 537
diff changeset
     2
# Copyright (c) 2008-2010 Nokia Corporation and/or its subsidiary(-ies).
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     3
# All rights reserved.
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     4
# This component and the accompanying materials are made available
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     5
# under the terms of the License "Eclipse Public License v1.0"
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     6
# which accompanies this distribution, and is available
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     7
# at the URL "http://www.eclipse.org/legal/epl-v10.html".
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     8
#
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
     9
# Initial Contributors:
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    10
# Nokia Corporation - initial contribution.
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    11
#
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    12
# Contributors:
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    13
#
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    14
# Description: 
537
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    15
# Filter class for copying files in serial in python. This
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    16
# is important in cluster builds where file copying is 
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    17
# very inefficient.  
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 537
diff changeset
    18
# The one-to-many <finalcopy> tag is searched for and copy
537
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    19
# instructions are built up in a hash table.
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 537
diff changeset
    20
# <finalcopy source='sourcefilename'>destfilename1 destfilename2 . . . .destfilenameN</copy>
537
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    21
# destinations must be full filenames not directories.
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    22
#
537
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    23
# This filter monitors build progress
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    24
# via the <progress> tags and flushes copies as build 
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    25
# stages end (e.g. after resource so resources are ready for the next stage)
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    26
# 
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    27
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    28
import os
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    29
import sys
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    30
import tempfile
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    31
import filter_interface
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    32
import shutil
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    33
import generic_path
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    34
import stat
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 537
diff changeset
    35
from raptor_utilities import copyfile
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    36
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    37
class FilterCopyFile(filter_interface.Filter):
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    38
	
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    39
	def open(self, params):
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    40
		"initialise"
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    41
		
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    42
		self.ok = True
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    43
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    44
		self.files = {}
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    45
		
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    46
		return self.ok
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    47
	
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    48
	
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    49
	def write(self, text):
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    50
		"process some log text"
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    51
		
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    52
		for line in text.splitlines():
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 537
diff changeset
    53
			if line.startswith("<finalcopy"):
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 537
diff changeset
    54
				source_start = line.find("source='")
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 537
diff changeset
    55
				source = line[source_start+8:line.find("'", source_start+8)]
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 537
diff changeset
    56
				destinations = line[line.find(">",source_start)+1:line.find("</finalcopy>")].split(" ")
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    57
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    58
				if source in self.files:
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    59
					self.files[source].update(destinations)
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    60
				else:
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    61
					self.files[source] = set(destinations)
537
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    62
			elif line.startswith("<progress:end object_type='makefile' task='build'"):
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    63
				self.flushcopies() # perform copies at end of each invocation of the make engine
547
9fe7d0ab0f8f fixes for review comments. better docs. copyrights. make copy filter more modular, change <copy> tag to <filtercopy>
timothy.murphy@nokia.com
parents: 537
diff changeset
    64
						   # to ensure dependencies are in place for the next one.
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    65
				
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    66
		return self.ok
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    67
	
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    68
	
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    69
	def summary(self):
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    70
		"finish off"
537
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    71
		self.flushcopies()
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    72
		return self.ok
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    73
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    74
	def flushcopies(self):
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    75
		for source in self.files.keys():
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    76
			for dest in self.files[source]:
549
d633be326c9f fix: depcrunch was using re.match where re.findall was what was wanted. Add a test.
timothy.murphy@nokia.com
parents: 547
diff changeset
    77
				try:
d633be326c9f fix: depcrunch was using re.match where re.findall was what was wanted. Add a test.
timothy.murphy@nokia.com
parents: 547
diff changeset
    78
					copyfile(source, dest)
d633be326c9f fix: depcrunch was using re.match where re.findall was what was wanted. Add a test.
timothy.murphy@nokia.com
parents: 547
diff changeset
    79
				except IOError, e:
d633be326c9f fix: depcrunch was using re.match where re.findall was what was wanted. Add a test.
timothy.murphy@nokia.com
parents: 547
diff changeset
    80
					print "<error>%s</error>" % str(e)
537
164e587fef9f fix: take action on copy tags at the end of build phases.
timothy.murphy@nokia.com
parents: 534
diff changeset
    81
		self.files = {}
533
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    82
		
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    83
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    84
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    85
	def close(self):
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    86
		"nop"
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    87
		
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    88
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    89
		return self.ok
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    90
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    91
# the end				
408bfff46ad7 fix performance: copy resources in the frontend. Helps cluster builds since remote copying is inefficient.
timothy.murphy@nokia.com
parents:
diff changeset
    92