Arnaudl adding work done by Darios on raptor summarise.

Merge.

Brag script 1st draft by Arnaudl and DarioS

Add raptor build html summary computation

Changed regular expressions and temp files to be more agnostic about build ID conventions.

Bug 289 Issue in populatezip.pl when doing Platform build in step preprocess-zip-config

Adjusted matching for processing TSV files, so it will not fail even if the dir has not been created.

Truclean should not be performed if the sources.csv line doesn't have a sysdef field

Merge

Set Raptor -j option to 2 for package builds

Added disk space check for all builds. Made the platform build configurable via props and refs in the same way that the package build is.

First drop of script to generate build summary

=Merge

Enable analysis of Raptor warnings. Publish to Diamonds

Merge

Moved all targets specific to the postbuild phase of the build into the postbuild ant file.

Created framework for separating postbuild related activities into a separate file.

Moved comments around to be adjacent to the thing that they're describing.

Added Build environment check feature

platform build temporary patch - overidden sf-build-noprep to not generate the model from the fragments. Needed for split model builds and furhter work on simplifying model.

catchup merge

BOM fix - Changed to reading project location ${sf.project.location} instead of ${sf.config.dir}/../build/config, which isn't always correct.

catchup merge

Smoketest change - Added 'sf-find-ATS-worker-root' to build.xml so that the smoketest goes where the worker is expecting it on a individual basis, rather than it being hardcoded to D:\ATS\...

ATS testing on emulator - Renamed properties for lon-engbuild87, lon-engbuild89, and sym-build02 so they are the same as the machine name. Modified the TEMPDIR for c100726 so that it runs smoketests

sf-list-dir - added new tool 'listdir.py' which can exclude a directory and changed the behaviour of sf-list-dir to stop scanning epoc32/build to speed things up.

sf-postbuild - running analysis in parallel with evalid and post-build zipping.

Fix packaging source optimization does not work for package builds

Merge

change made in smoketest.xml

change made in ats3_specialise_test_drop.pl script.

Simplify the create-canonical-sysdef-file target as we can depend on there only being a single sysdef in the supplied reference.

Simplified create-canonical-sysdef-file so that it's just a simple copy.

Merge

Add my computer following ATS installation instructions found at: https://sites.google.com/a/symbian.org/build_and_test/Home/ats-automated-test-system/ats-howtos,

changing smoketest.xml

catchup merge

Changed 'publish' to 'sf-publish' to only publish the zips, not all logs. Changed behaviour so that 'sf.spec.package.src.enable'=false stops the source repos from getting packaged, but not the RnD ones (which are extracted again in platform builds)

source syncing ftl - Added 'sf.spec.sourcesync.archive' preference to do 'hg archive' instead of 'hg sync'. If set and the repo sync is s'posed to be synced by tag it still does the slower clone+update behaviour.

platform build - added 'sf.os.compile.iterate' variable to iterate though the main targets rather than all at once to work around raptor problems with single model.

Split sf-run-analysis into bit-sized targets, which can run in parallel to reduce the post-build time on a platform build.

catchup merge

minor fix - creation of 'generated' dir for caching files.

raptor parser.pl - adding XML::SAX perl modules (public domain code - http://www.saxproject.org/faq.html) so we don't have to patch up every single build machine.

merge

Add syslibssvs ats test drop creation script

ats change

Add multimediasvs ats test drop creation script

updating smoketest.xml

updating to add build-id for diamond in build.xml

Zip BOM and releasables changing the top level folder from "output" to "build_info"

updating ATS

updating device reg. file

Making device property file

Making device property file

updating security file for ATS

Caught up

Created Graphics Perl script to create package and finished adding to test XML

Reference to the Font Store API Server

Added graphics test.xml

changing "sf.spec.test.host.name" to "v800005" as ats server.

catchup merge

Yarp.pl - Added manual XML parsing. Memory leaks using XML::Simple cause perl to run out of memory when parsing a large raptor log.

Updating ATS security file.

catchup merge

catchup merge

Yarp - Modified to output failed recipes to STDOUT instead of having to use dump_recipe_at_line.pl, and modified build.xml to save these logs.

Added a new "sf.os.compile.fast" parameter to skip redundant steps when executing the builds in separate config targets.

Make ATS drop path by UNC not mapped drive.

Fixed smoketest perl target.

Move to classicui folder

Fixed usage of 7z in ats_specialise_test_drop.pl

smoketest creation - .pl files not always executable on build machines.

merge

Add keyevent test at end of apploader test suite to remove prompt for location/time settings Remove redundant batch file. Use perl script instead.

Caught up from mercurial_development

Added step to submit Dynamic BC tests to ATS server

reverting change

reverting the change

Fixed FBF management of single build.

Merge to tip

SYM-BUILD02 had wrong tempdir, should have been D

Added my machine to STAF_Security

merge

Passing diamond build-id as parameter for ats_specialise_test_drop.pl to insert build-id tag in test.xml

adding tags to enable diamonds post.

create tag for PDK2.0.d

Make BC testing more robust. Now requires PDT_HOME env var to be set rather than fail silently. Plus Logs zip tidy up.

Add zipfile validation to packaging

Fix packaging issues. Released zips now delivered in zips\release folder.

Un-hardwired an interim location for the system definition. Removed a target that is over-ridden in both the platform and package specific files.

Added output of the package definition incorporated for each package.

Added some diagnostic output when the system definition is built from the sources.csv file.

Commented out clean and tools targets. Changed tool2 target into tools2_rel

Removed commented & obsolete code.

Changed preference for selecting package definition files, so that it takes the one from our tree in preference to the one in the package itself. This means that we can just delete items from our tree when we're happy to switch to using the one in the package (as this will vary from package to package). Added sorting of the attributes when outputting the system definition, to match the de facto standard in the system definition we started with.

Add new raptor error category to raptor parser (tool_didnt_return_version)

Set project name and codeline from subproject only if these properties are not found in project's job_props.ant.xml

Merge

Fix again releaseables dir not existing as erroronmissing dir is not supported

If a package does not have a definition in the sources.csv then it won't get built, so don't try to incorporate it into the system definition.

Avoid build failure if releaseable dir is not present

Made error messages more useful.

Should fail the build if we can't generate the system model.

Fixed paths and drives for executing script.

Added invokation to build.xml and adjusted path to backup locations to match my development environment.

Added tool for collating a build definition from the set of package defintions indicated by the sources.csv Location of tree of package defintions will need to be adjusted...

Change package baseline to PDK 2.0.d

Added 3 raptor error categories, adjusted severity of all categories

Fix regexp to take account of new codeline naming convention

Allow subprojects within a project repo and model file relative to the subproject dir

Generate error and summary file even if build reports no error

Restructured package build targets and sbs calls

Reverted category-property to build.family

Enable build tagging on Diamonds (default: 'test')

Fix to account of log file names for test builds

Added transformation of model file from version 3.0.0 to a version usable by Raptor

Merge

Add sdk version file for 5.0 run

Move runtarget outside if conditional

Fix cenrep to run for both single and split model builds

Use GENERIC interface on ATS server and add machine name to test package to aid origin

merge

Remove redundant files Add missing files for bctestlist suite

Add all tests

Remove extra column

Add branch and changeset support for getting source. BOM sources.csv can now be reused directly.

Find_collisions.pl - modified tool to use 'FC' instead of 'diff' for finding origins. (gcc 'diff' not currently in builds)

catchup merge

Build HG Caching - added new variable 'sf.spec.sourcesync.cachelocation' and changed caching invocation to use it. Defaults to previous location.

cachefiles.pl - stopped cache from being excessively flattened, which could cause problems.

Add delay to give expected processes a chance to start

Add tools baseline to BOM

Added creation of zips of zips of OSS packages (though these are not yet published in a PDK release). Updated names of zips of zips of SFL packages to avoid clashing with the ideal long term name of the zips of zips of OSS packages.

Added tweak for bldmefirst zip filename

Small speed improvement (fingers crossed) for zipping up binaries.

Small tweak to get the right content with the right filename for doing a PDK release. Corrected some indentation.

Stopped it putting useless information about zips of zips into the release_metadata.xml file.

Create zips of zips for SFL src.

Merge

Raptor error analysis to be published to Diamonds

Added analysis of Raptor errors

BOM and TSVs now zipped up in an alternative zip file. Zip files renamed to fit in with other published build files.

Corrected filenames used for logging output of zip commands.

Zips binaries into three zips rather than one, to make them easier to download.

Simplified the locations of some generated files, so they are created where they need to be rather than created on one place and moved to another. Zipping of binaries now assumes that there's always an rnd exclude file, as it is always created by populateziptemplate.pl, even if there's nothing put into it.

Added sorting and removal of duplicates from TSV files.

Merge

Raptor parser to provide summary csv file so that faults can be readily put into db

Updated preprocess_log to deal with mal-formed <archive> tags - workaround for Bug 170. Also re-jigged to use STDIN and STDOUT rather than filename options on the CLI.

Modularized Raptor parser scripts

generate-layer overridden target from platform/build.xml to common/build.xml

Adding truclean step to the package build

Merge

Fix log zip structure, fix udeb/urel issue in MD5s and also package MD5s into zip in correct epoc32\relinfo layout

Integrate extraction of whatlog info and other minor changes from pkgbuild config

Enable build machine based testing via ATS worker

Tag source matching PDK_2.0.c

Fixed single model genxml pre-processing workaround to enable single system definition builds. Added "tools" and "adaptation" to the list of packages to be prepended by sf where needed.

Tidy up generate-layers and add missing quote

merge diamondize and dynamic bc testing

merge using wildcard on destination removes all of destination not just src files

Add option for dynamic bc testing

Export build-info.xml to BOM folder

Export system model files to BOM folder

Merge commit

Rollback the removal of overriden generate-layers target - required platform build

Adding ATS device registration files

Catchup from platform build

merge in changes

Packaging change - Tools now zipped into 'tools_epoc.zip', which isn't 'emu' but 'tools' for package and platform builds. Should work when we update the metadata.

Added Hg local cache management feature.

Tidy up smoke test ats file Add BC ats files

Remove the "abld import" stuff, add "No rule to make" and "Can't find mmp file" processing

Limit source parallelisation to NUMBER_OF_PROCESSORS

Clear list as we dont want to exclude anything from RnD unpack

Build smoketest as udeb plus ensure that sf-postbuild is run for sf-build-noprep.

Add the missing column heading in PkgComponentAnalysisSummary.csv

Added new helium target: 'sf-build-and-pack' for use if you need to babysit 'sf-prebuild'

Smoketest fix - changed ip address to point to symbuild01, caused by machines moving.

rebuildok.txt - removed 'epoc32/tools/*' for build machines with no vc6 compiler, so all the tools get unzipped in 'sf-unpack-rnd'.

parsewhatlog.pl - put back in the regex 1-liner what got taken out (multiple 'master' repos out of sync)

'WHAT' build removed. parselistdirs.pl fixes for raptor logs.

'sf-unpack-rnd' target: changed to overwrite mode, due to TEF being present earlier in build.

Results Publish - changed network location from \\lon-engbuild87\SF_Build\ to \\v800008\SF_build\

Applied the /accepteula option to all calls to junction so that it will never stall a build waiting for the user to accept the EULA,

Amend STAF machine trust levels to waive intranet component

Committing merges

Adding global STAF security settings file

merged in parsewhatlog.pl defect fix for Raptor 2.7.2 and find_collisions.pl minor arg cleanup.

find_collisions.pl - removed unused command-line argument from script to tidy things up.

parsewhatlog.pl - regex bug fix for raptor 2.7.2

Reorder the PkgComponentAnalysisSummary.csv columns, and change "KO" to "Bad".

Remove files that are not in test package

Merge changes

Add <sf-getenv-tools> target to unpack tools baseline and also <sf-build-smoketestpkg> to create smoketest package to be sent to ATS3 server

Add MD5 creation

Adding extraction of whatlog information into "deliverables" files

Changed PDK to 2.0.c

Fix for Error doesn't correctly report the number of failing raptor recipes

Added output stream to failures line in yarp. Made the failure string more robust

Yarp file parser to get number of failures from the correct field

Yarp to print failures

Commented out print line

Moved location of build results to \\v800008\g$

Add missing file pare_yarp_files.pl

Add missing files for yarp publishing to diamonds

Added yarp to diamonds publishing

Initial implementation of publishing to Diamonds

Catchup from platform build config

Added yarp to the post-build analysis

Changed location of PDK

Changed location of source code to point to v800008.ad-sfpd.intra

Catchup from platform build config

Fixed typo. $-sign should have been &

Added yarp.pl - (Yet Another Raptor Parser) and dump_recipe_at_line.pl Usage for yarp.pl: perl yarp.pl <compile_log> <output_csv> Usage for dump_recipe_at_line.pl: perl dump_recipe_at_line.pl <compile_log> <line|line_list_file> The idea of dump_recipe_at_line.pl is so you can take a column of line numbers from the CSV created by yarp and see what happened (without having to open the 300MB file) with a single column: ... 226242 ... you get: ... 226242 <recipe name blah blah blah... ... With two columns in the file, you can have some more readable output: ... 226242 m:/sf/mw/classicui/group/bld.inf ... you get: ... m:/sf/mw/classicui/group/bld.inf(226242): <recipe name blah blah blah... ...

Catch up

Add perl script to build test zip Remove unwanted files for smoke test

merge

Comments and code tidying for ats3_testdrop.pl Adding new ATS support script ats_specialise_test_drop.pl

.xml missing from file names Add option to exclude header/library BC static checking

File was rendered obsolete a long time ago

Added analyse_components.pl and related call to build.xml Modified merge_csv.pl to use "7za" instead of "7z"

Remove duplicates from rebuiltok list

catchup merge

adding merge_csv_orphans.pl Untrusted at the moment

Added merge_csv_orphans.pl - Would just change merge_csv.pl but I've been hacking and don't trust it yet.

Merge to default branch

Exclude list of things from RnD unpack that we know we can build

merge - catch up

Add optiona BC checking to build process

Merge of change to merge_csv.pl change. Caused by moving of the Repo: No files changed.

merge_csv.pl - Added an optional post-build rnd zip argument which adds a timestamp field to the CSV. Requires '7z' not '7za'.

Add ats local winscw device Add tool to build ats amoketest package

Catchup from default config

Remove abld.bat - generated during build

merge - catch up

Add first draft of smoketests

Use 7za not 7z as this is what is available on build machines

Merge to default branch

Move analysis after packaging

Adding license boilerplate to perl scripts.

Add zip tool to support postbuild RnD packaging.

Add postbuild RnD packaging. Add .whatlog to os tools build.

Merge changes to default branch

Unpack actual RnD zips rather than fastcopy. Add build collisions tool to <sf-run-analysis> target.

Added license header

Merge

Added filter to exclude Hg control files from package zips.

Fixed minor bug in ats3_testdrop.pl: Quote marks within quotation needed to be escaped.

Disable hg tagging by default

Fix test property typos

Fix populateziptemplate.pl to work with new repo structure

Added find_collisions.pl Updated the readme.txt for instructions.

parselistdirs.pl - fixed export scanning and added output path for logs. - 'uptodate' export scanning looks in all logs, not just 'clean' ones. - Exported is a new group, and those exports have been removed form other groups. - Logs output path flexiblility improved - The following are now the same: I:\logs\generated>perl parselistdirs.pl ..\ >list_results.log I:\>perl logs\generated\parselistdirs.pl logs\ logs\generated\ >logs\generated\list_results.log

Add ATS test client target <sf-send-testpkg> to default configuration. Default is off.

Merge updates from master

Add build analysis tools to <sf-run-analysis> target. All output generated in <logs>\analysis dir.

Fixed "zip.cfg.xml.ftl.template" to use "oss" and not "epl" in the configuration name.

Merge with previous modifications.

Modified populateziptemplate.pl to use "oss" instead of "epl" keyword.

Exclude .hg repositories from epoc32 merge

Exclude *.sym files from binary packaging

1) Update binary packaging solution to invoke 7za directly. 2) Add partially complete ATS sf-send-testpkg target and associated tools

Removed obsolete variable (@includes). Moved remaining code round so that an XML entry is created even if there are no local files detected. (Ie same approach as src repositories.)

Copy Text::CSV module into source and change to use it, as we can't rely on it appearing in the Perl library when Perl 5.6.1 is installed.

Package RnD binaries from rnd root location to reduce size of include list

Make use of <hlm:fastcopy> taskdef to speed up copying of RnD binary tree.

Make RnD merge conditional, only on discovery of rnd binary tree

Add merge of RnD binaries into epoc32 tree as HLM target. Also create empty sources.csv and baseline.txt files if source sync/get baseline is disabled.

Improved zip template generation script, to take a third argument for the required location of the output file. Updated build script to supply third argument. Standardised indentation in build.xml - 4 columns; no tabs.

Fix the generation of the binary zip, so epoc32/include is zipped, but epoc32/build is not.

Exclude the .hg repository directories from the packaged src.

Added RnD binaries support in PRE-BUILD stage.

Fixed ant syntax from previous commit.

Implemented the "sf.spec.baseline.clean" switch.

Strip drive and dir root from listing

Exclude epoc32/build to stop intermediate obj files from being packaged

Include epl/sfl category in zipfile name

Add missing </if>

1. Run Packaging src at start and bin at end of build. 2. Make src/bin packaging configurable.

Add first cut of packaging solution

If found then export cli_build_error.log to logs directory

Remove old s60 bldmelast step and replace with call to generate cenrep files.

Include --check on CLEAN and also remove S60_bldmelast configuration. It didnt work anyway as s60cli.exe is not present.

Fix generated sysdef filename and also remove erroneous comment

Make BOM identify thread safe

Fix missing <if> typo

Remove GenXML filtering and merge as platform System Model xml files should now be complete

Add intersection target to FBF

Add intersection option for difflist.pl

Correct source and target diff files

Disabled tagging after build by changing default value to false

Changed PDK and filter options to get PDK_candidate_1 binaries

Made smoketest optional

Changing default value of sf.spec.smoketest.enable to false

New version of make_junction.pl

First drop of smoketest support

Setting sf.spec.logs.raptorfilter.enable by default to true, as this will enable the full log and also the scanlog html report in the build results

Reimplemented --check as the previous version didn't work (also most of the arguments are parameterized now)

Doing WHAT with .whatlog. Added check with an exec task (some arguments are not parameterized).

added whatlog and check

Changed default location for logs/results to \\sym-build01\f$\SF_builds

Changed common default for sf.spec.systemdefinition.assemble to false

Changed location of PDK to \\lon-engbuild87\d$\PDK\PDK_candidate_0

Changed --check to CHECK to see if it works

Given a dummy value to the configuration list

Added check stage

Merge

Changed package-common method of baseline retrieval

Changed package-common location of baseline to \\lon-engbuild98\d$\GRACE\PDK_20090401

Overridden gernate-layers, so that sysdef file is not filtered

The prefix "hlm" for element "hlm:argSet" was not bound.

Changes to support clean check and avoid filtering the logs

Download of "emu" and "patch" baseline zips common to all packages

Catchup from main repo

1. Update list and delta targets 2. Add WHAT target 3. Make raptor filter configurable

Add tools to create file tree deltas and integrate them into build f/w

Update to allow foundation model build in single step

Add sf-zip-logs target to be run before sf-postbuild. Packages logs in zipfile for easy transfer.

Remove platform.properties as properties are no longer loaded this way.

Apply sbs clean before build. TODO add generic clean target next.

Update Platform Config to build tools2 before tools - which must be run in single thread. Also update sync template to sync in parallel.

Commit

Fixed include option to -i (was -l)

Add config changes in order to support s60 and os platform builds

Diamondize BOM info

Fixed error

Added project and config info to BOM files

Added the revision number, moved BOM folder under /logs/

Fixed misspelling of variable

Added recording of baseline info in a BOM file

Removed leftover notes at end of file

Added try/catch around the tagging, so that the build doesn't fail if tag gives an error

Targets must be separated by commas

Adding removal of sources.csv before populating it

Fixed path to sources.csv file

Changed comment

Removing old code

Fixing nested vars issue

New implementation of generation of BOM sources.csv

Added revision identification

Changed filter option to -i

Changed publish share to lon-engbuild87

Build drive G: gives issues on Foundation machines

Tag level should be "normal" otherwise the tags are not copied to the original repo

Added tagging of source code used in the build

name, codeline, number

Job drive defaults to G: and job number defaults to 0

Changed default rootdir and made it be used

Changed name of product and release spec vars

Added publishing of logs and reports

Added publishing stage

Enabled sf.spec.systemdefinition.assemble flag to control sysdef merge stage

Moved back the override

Moved override of create-canonical-sysdef-file

Changes to enable system definition file to be provided and used as is

Finalize catchup from Shabe's changes

Catchup from Shabe's changes

Reorganized property/reference importing structure

Fix FMPP template to allow multiple repositories

Correct new project filename

Correct location

Restore properties.default.ant.xml and allow projec spec to be configurable

Progress in moving old vars to new formats

merged with shabe's changes

Enabled sf properties: sf.baseline.location and sf.baseline.getenv_options

Added tag fbf_configs_0.0.a for changeset 29394d95b3e1

Merge all changes

Remove duplicate sync and rename sf-build-run to correct sf-build-noprep target

Rename and move of generated source sync xml and also tidy up

Various changes

Reorder target for check-tool-dependencies

Fix target name

Fix error in template counter

Update and add build.sysdef.xml

test commital

Initial drop of FBF Helium configuration