# HG changeset patch # User Daniel Jacobs # Date 1260531954 0 # Node ID 55ef265012ad55984c69bb7eb5e1e4765f9d7967 # Parent 000ba5e4ba7d2b8cc20a33221197ad72da02e09b# Parent 5808d36c3722a251c8f40561e0ccbc11374c8c8b Merging changes from pull. diff -r 000ba5e4ba7d -r 55ef265012ad sbsv2/raptor/bin/sbs_check_exports.py diff -r 000ba5e4ba7d -r 55ef265012ad sbsv2/raptor/lib/config/variants.xml --- a/sbsv2/raptor/lib/config/variants.xml Thu Dec 10 16:31:46 2009 +0000 +++ b/sbsv2/raptor/lib/config/variants.xml Fri Dec 11 11:45:54 2009 +0000 @@ -187,7 +187,6 @@ - diff -r 000ba5e4ba7d -r 55ef265012ad sbsv2/raptor/python/raptor.py --- a/sbsv2/raptor/python/raptor.py Thu Dec 10 16:31:46 2009 +0000 +++ b/sbsv2/raptor/python/raptor.py Fri Dec 11 11:45:54 2009 +0000 @@ -11,7 +11,7 @@ # # Contributors: # -# Description: +# Description: # raptor module # This module represents the running Raptor program. Raptor is started # either by calling the Main() function, which creates an instance of @@ -119,7 +119,7 @@ self.type = type self.specs = [] self.deps = [] - self.children = set() + self.children = set() self.unfurled = False self.parent = parent @@ -163,15 +163,15 @@ for c in self.children: c.unfurl_all(build) - + def realise_exports(self, build): - """Do the things that are needed such that we can fully unfurl all + """Do the things that are needed such that we can fully unfurl all sibling nodes. i.e. this step is here to "take care" of the dependencies - between siblings. + between siblings. """ pass - + def realise_makefile(self, build, specs): makefilename_base = build.topMakefile if self.name is not None: @@ -190,16 +190,16 @@ key = str(makefile.path)) return makefileset - + def realise(self, build): - """Give the spec trees to the make engine and actually - "build" the product represented by this model node""" + """Give the spec trees to the make engine and actually + "build" the product represented by this model node""" # Must ensure that all children are unfurled at this point self.unfurl_all(build) - sp = self.specs + sp = self.specs build.AssertBuildOK() @@ -210,8 +210,8 @@ result = build.Make(m) build.InfoEndTime(object_type = "layer", task = "build", key = (str(m.directory) + "/" + str(m.filenamebase))) - - + + return result @@ -229,7 +229,7 @@ def makefile(self, makefilename_base, engine, named = False): """Makefiles for individual mmps not feasible at the moment""" - pass # Cannot, currently, "unfurl an mmp" directly but do want + pass # Cannot, currently, "unfurl an mmp" directly but do want # to be able to simulate the overall recursive unfurling of a build. class Component(ModelNode): @@ -252,9 +252,9 @@ class Layer(ModelNode): - """ Some components that should be built togther - e.g. a Layer in the system definition. - """ + """ Some components that should be built togther + e.g. a Layer in the system definition. + """ def __init__(self, name, componentlist=[]): super(Layer,self).__init__(name) self.name = name @@ -263,11 +263,11 @@ self.children.add(Component(c)) def unfurl(self, build): - """Discover the children of this layer. This involves parsing the component MetaData (bld.infs, mmps). + """Discover the children of this layer. This involves parsing the component MetaData (bld.infs, mmps). Takes a raptor object as a parameter (build), together with a list of Configurations. We currently have parsers that work on collections of components/bld.infs and that cannot - parse at a "finer" level. So one can't 'unfurl' an mmp at the moment. + parse at a "finer" level. So one can't 'unfurl' an mmp at the moment. Returns True if the object was successfully unfurled. """ @@ -300,7 +300,7 @@ def meta_realise(self, build): """Generate specs that can be used to "take care of" finding out more about this metaunit - i.e. one doesn't want to parse it immediately - but to create a makefile that will parse it. + but to create a makefile that will parse it. In this case it allows bld.infs to be parsed in parallel by make.""" # insert the start time into the Makefile name? @@ -310,44 +310,44 @@ # Pass certain CLI flags through to the makefile-generating sbs calls cli_options = "" - + if build.debugOutput == True: cli_options += " -d" - + if build.ignoreOsDetection == True: cli_options += " -i" - + if build.keepGoing == True: cli_options += " -k" - + if build.quiet == True: cli_options += " -q" - + if build.timing == True: cli_options += " --timing" - + nc = len(self.children) number_blocks = build.jobs block_size = (nc / number_blocks) + 1 component_blocks = [] spec_nodes = [] - + b = 0 childlist = list(self.children) while b < nc: component_blocks.append(childlist[b:b+block_size]) b += block_size - + while len(component_blocks[-1]) <= 0: component_blocks.pop() number_blocks -= 1 - + build.Info("Parallel Parsing: bld.infs split into %d blocks\n", number_blocks) - # Cause the binding makefiles to have the toplevel makefile's - # name. The bindee's have __pp appended. + # Cause the binding makefiles to have the toplevel makefile's + # name. The bindee's have __pp appended. tm = build.topMakefile.Absolute() - binding_makefiles = raptor_makefile.MakefileSet(str(tm.Dir()), build.maker.selectors, makefiles=None, filenamebase=str(tm.File())) + binding_makefiles = raptor_makefile.MakefileSet(str(tm.Dir()), build.maker.selectors, makefiles=None, filenamebase=str(tm.File())) build.topMakefile = generic_path.Path(str(build.topMakefile) + "_pp") loop_number = 0 @@ -357,16 +357,16 @@ componentList = " ".join([str(c.bldinf_filename) for c in block]) - + configList = " ".join([c.name for c in self.configs if c.name != "build" ]) - + makefile_path = str(build.topMakefile) + "_" + str(loop_number) try: os.unlink(makefile_path) # until we have dependencies working properly except Exception,e: # print "couldn't unlink %s: %s" %(componentMakefileName, str(e)) pass - + # add some basic data in a component-wide variant var = raptor_data.Variant() var.AddOperation(raptor_data.Set("COMPONENT_PATHS", componentList)) @@ -385,13 +385,13 @@ if build.noBuild: var.AddOperation(raptor_data.Set("NO_BUILD", "1")) specNode.AddVariant(var) - + try: interface = build.cache.FindNamedInterface("build.makefiles") specNode.SetInterface(interface) except KeyError: build.Error("Can't find flm interface 'build.makefiles' ") - + spec_nodes.append(specNode) binding_makefiles.addInclude(str(makefile_path)+"_all") @@ -430,7 +430,7 @@ M_BUILD = 1 - M_VERSION = 2 + M_VERSION = 2 def __init__(self, home = None): @@ -554,7 +554,7 @@ self.Warn("ignoring target %s because --what or --check is specified.\n", target) else: self.targets.append(target) - + def AddSourceTarget(self, filename): # source targets are sanitised and then added as if they were a "normal" makefile target # in addition they have a default, empty, top-level target assigned in order that they can @@ -615,7 +615,7 @@ def SetNoDependInclude(self, TrueOrFalse): self.noDependInclude = TrueOrFalse return True - + def SetKeepGoing(self, TrueOrFalse): self.keepGoing = TrueOrFalse return True @@ -669,7 +669,7 @@ return False return True - + def SetTiming(self, TrueOrFalse): self.timing = TrueOrFalse return True @@ -717,9 +717,9 @@ self.Info("Set-up %s", str(self.raptorXML)) self.Info("Command-line-arguments %s", " ".join(self.args)) self.Info("Current working directory %s", os.getcwd()) - + # the inherited environment - for e, value in os.environ.items(): + for e, value in sorted( os.environ.items() ): self.Info("Environment %s=%s", e, value.replace("]]>", "]]>")) # and some general debug stuff @@ -822,7 +822,7 @@ return self.home.Append(aGenericPath) else: return aGenericPath - + # make generic paths absolute (if required) self.configPath = map(mkAbsolute, self.configPath) self.cache.Load(self.configPath) @@ -859,12 +859,12 @@ return x def GetBuildUnitsToBuild(self, configNames): - """Return a list of the configuration objects that correspond to the + """Return a list of the configuration objects that correspond to the list of configuration names in the configNames parameter. raptor.GetBuildUnitsToBuild(["armv5", "winscw"]) >>> [ config1, config2, ... , configN ] - """ + """ if len(configNames) == 0: # use default config @@ -878,9 +878,9 @@ for c in set(configNames): self.Debug("BuildUnit: %s", c) - try: + try: x = self.GetConfig(c) - gb = x.GenerateBuildUnits(self.cache) + gb = x.GenerateBuildUnits(self.cache) buildUnitsToBuild.update( gb ) except Exception, e: self.FatalError(str(e)) @@ -894,7 +894,7 @@ return buildUnitsToBuild def CheckToolset(self, evaluator, configname): - """Check the toolset for a particular config, allow other objects access + """Check the toolset for a particular config, allow other objects access to the toolset for this build (e.g. the raptor_make class).""" if self.toolset is None: if self.toolcheck == 'on': @@ -968,7 +968,7 @@ def FindComponentIn(self, aDir = None): - # look for a bld.inf + # look for a bld.inf if aDir is None: dir = generic_path.CurrentDir() @@ -1109,14 +1109,14 @@ for a,v in dictionary.items(): atts += " " + a + "='" + v + "'" return atts - + def Info(self, format, *extras, **attributes): """Send an information message to the configured channel (XML control characters will be escaped) """ self.out.write("" + escape(format % extras) + "\n") - + def InfoDiscovery(self, object_type, count): if self.timing: try: @@ -1124,7 +1124,7 @@ count = count)) except Exception, exception: Error(exception.Text, function = "InfoDiscoveryTime") - + def InfoStartTime(self, object_type, task, key): if self.timing: try: @@ -1132,7 +1132,7 @@ task = task, key = key)) except Exception, exception: Error(exception.Text, function = "InfoStartTime") - + def InfoEndTime(self, object_type, task, key): if self.timing: try: @@ -1154,7 +1154,7 @@ """Send a warning message to the configured channel (XML control characters will be escaped) """ - self.out.write("" + + self.out.write("" + escape(format % extras) + "\n") def FatalError(self, format, *extras, **attributes): @@ -1165,7 +1165,7 @@ further errors are probably triggered by the first. """ if not self.fatalErrorState: - self.out.write("" + + self.out.write("" + (format % extras) + "\n") self.errorCode = 1 self.fatalErrorState = True @@ -1174,7 +1174,7 @@ """Send an error message to the configured channel (XML control characters will be escaped) """ - self.out.write("" + + self.out.write("" + escape(format % extras) + "\n") self.errorCode = 1 @@ -1212,7 +1212,7 @@ if self.systemDefinitionFile != None: systemModel = raptor_xml.SystemModel(self, self.systemDefinitionFile, self.systemDefinitionBase) layers = self.GatherSysModelLayers(systemModel, self.systemDefinitionRequestedLayers) - + # Now get components specified on a commandline - build them after any # layers in the system definition. if len(self.commandlineComponents) > 0: @@ -1246,7 +1246,7 @@ self.Introduction() # establish an object cache self.AssertBuildOK() - + self.LoadCache() # find out what configurations to build @@ -1299,7 +1299,7 @@ for l in layers: # create specs for a specific group of components l.realise(self) - + except BuildCannotProgressException,b: if str(b) != "": self.Info(str(b)) @@ -1326,7 +1326,7 @@ build.ProcessConfig() build.CommandLine(argv) - return build + return build diff -r 000ba5e4ba7d -r 55ef265012ad sbsv2/raptor/python/raptor_data.py --- a/sbsv2/raptor/python/raptor_data.py Thu Dec 10 16:31:46 2009 +0000 +++ b/sbsv2/raptor/python/raptor_data.py Fri Dec 11 11:45:54 2009 +0000 @@ -722,15 +722,23 @@ def Apply(self, oldValue): try: value = os.environ[self.name] - - # if this value is a "path" or a "tool" then we need to make sure - # it is a proper absolute path in our preferred format. - if value and (self.type == "path" or self.type == "tool"): - try: - path = generic_path.Path(value) - value = str(path.Absolute()) - except ValueError,e: - raise BadToolValue("the environment variable %s is incorrect: %s" % (self.name, str(e))) + + if value: + # if this value is a "path" or a "tool" then we need to make sure + # it is a proper absolute path in our preferred format. + if self.type == "path" or self.type == "tool": + try: + path = generic_path.Path(value) + value = str(path.Absolute()) + except ValueError,e: + raise BadToolValue("the environment variable %s is incorrect: %s" % (self.name, str(e))) + # if this value ends in an un-escaped backslash, then it will be treated as a line continuation character + # in makefile parsing - un-escaped backslashes at the end of values are therefore escaped + elif value.endswith('\\'): + # an odd number of backslashes means there's one to escape + count = len(value) - len(value.rstrip('\\')) + if count % 2: + value += '\\' except KeyError: if self.default != None: value = self.default diff -r 000ba5e4ba7d -r 55ef265012ad sbsv2/raptor/test/timing_tests/parse_time_exports_mmps.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/sbsv2/raptor/test/timing_tests/parse_time_exports_mmps.py Fri Dec 11 11:45:54 2009 +0000 @@ -0,0 +1,77 @@ + +from raptor_tests import SmokeTest, ReplaceEnvs +import os + +def generate_files(): + try: + os.makedirs(ReplaceEnvs("$(SBS_HOME)/test/timing_tests/test_resources/parse_time")) + except: + pass + bldinf_path = ReplaceEnvs("$(SBS_HOME)/test/timing_tests/test_resources/parse_time/bld.inf") + bldinf = open(bldinf_path, "w") + bldinf_content = """prj_mmpfiles +""" + test_dir = ReplaceEnvs("$(SBS_HOME)/test/timing_tests/test_resources/parse_time") + for number in range(0, 250): + mmp_path = ("parse_timing_" + str(number).zfill(3) + ".mmp") + mmp_file = open((test_dir + "/" + mmp_path), "w") + mmp_file.write("""targettype none +""") + mmp_file.close() + bldinf_content += (mmp_path + "\n") + + bldinf_content += "\nprj_exports\n" + + for number1 in range(0, 10): + source_dir = ("export_source_" + str(number1)) + try: + os.mkdir(test_dir + "/" + source_dir) + except: + pass + + for number2 in range (0, 10): + source_file = ("/file_" + str(number2) + ".txt ") + export_file = open((test_dir + "/" + source_dir + source_file), "w") + export_file.write(str(number2)) + export_file.close() + + for number3 in range (0, 10): + dest_dir = ("epoc32/include/export_destination_" + \ + str(number1) + str(number2) + str(number3)) + + for number4 in range(0, 10): + bldinf_content += source_dir + source_file + dest_dir + \ + "/export_destination_" + str(number4) + "\n" + bldinf.write(bldinf_content) + bldinf.close() + + +def delete_files(): + import shutil + + test_dir = ReplaceEnvs("$(SBS_HOME)/test/timing_tests/test_resources/parse_time") + objects = os.listdir(test_dir) + for object in objects: + object_path = (test_dir + "/" + object) + if os.path.isfile(object_path): + os.remove(object_path) + else: + shutil.rmtree(object_path) + + +def run(): + + generate_files() + + t = SmokeTest() + + t.id = "1" + t.name = "parse_time_exports_mmps" + t.description = """Test to measure time taken to parse a large number of + exports and mmps""" + t.command = "sbs -b timing_tests/test_resources/parse_time/bld.inf -n " + \ + "-c armv5_urel --toolcheck=off --timing" + t.run() + + delete_files() + return t diff -r 000ba5e4ba7d -r 55ef265012ad sbsv2/raptor/test/unit_suite/raptor_data_unit.py --- a/sbsv2/raptor/test/unit_suite/raptor_data_unit.py Thu Dec 10 16:31:46 2009 +0000 +++ b/sbsv2/raptor/test/unit_suite/raptor_data_unit.py Fri Dec 11 11:45:54 2009 +0000 @@ -30,24 +30,18 @@ def SetEnv(self, name, value): - # set environment variable and remember the old value - - try: - old = os.environ[name] - self.envStack[name] = old - os.environ[name] = value - except KeyError: - self.envStack[name] = None # was not defined + # set environment variable and remember the old value (if there is one) + if os.environ.has_key(name): + self.envStack[name] = os.environ[name] + os.environ[name] = value def RestoreEnv(self, name): # put environment back to its state before SetEnv - saved = self.envStack[name] - - if saved == None: + if self.envStack.has_key(name): + os.environ[name] = self.envStack[name] + else: del os.environ[name] # was not defined - else: - os.environ[name] = saved def testSimpleSpecification(self): @@ -359,6 +353,32 @@ # test the Resolve wrt EPOCROOT varcfg = eval.Resolve("VARIANT_CFG") self.assertEqual(varcfg, "/C/variant/variant.cfg") + + def testProblematicEnvironment(self): + # ask for environment variable values that will break makefile parsing due to + # backslashes forming line continuation characters + self.SetEnv("ENVVAR_BSLASH_END1", "C:\\test1a\\;C:\\test1b\\") + self.SetEnv("ENVVAR_BSLASH_END2", "C:\\test2a\\;C:\\test2b\\\\") + self.SetEnv("ENVVAR_BSLASH_END3", "C:\\test3a\\;C:\\test3b\\\\\\") + var = raptor_data.Variant("my.var") + var.AddOperation(raptor_data.Env("ENVVAR_BSLASH_END1")) + var.AddOperation(raptor_data.Env("ENVVAR_BSLASH_END2")) + var.AddOperation(raptor_data.Env("ENVVAR_BSLASH_END3")) + + aRaptor = raptor.Raptor() + eval = aRaptor.GetEvaluator(None, var.GenerateBuildUnits(aRaptor.cache)[0]) + self.RestoreEnv("ENVVAR_BSLASH_END1") + self.RestoreEnv("ENVVAR_BSLASH_END2") + self.RestoreEnv("ENVVAR_BSLASH_END3") + + value = eval.Get("ENVVAR_BSLASH_END1") + self.assertEqual(value, "C:\\test1a\\;C:\\test1b\\\\") + + value = eval.Get("ENVVAR_BSLASH_END2") + self.assertEqual(value, "C:\\test2a\\;C:\\test2b\\\\") + + value = eval.Get("ENVVAR_BSLASH_END3") + self.assertEqual(value, "C:\\test3a\\;C:\\test3b\\\\\\\\") def testMissingEnvironment(self): # ask for an environment variable that is not set