Use prebuilt baseline rather than emu for package and platform builds

Remove redundant toolsbaseline functionality

Merge

Ensure "prebuilt" binaries are classified as such. Also populate details about the release, such as build.version.

Fix toolsbaseline build logic

Major rework to enable rebuilds of the platform from single sf-build call - Change the way that sources and system model are selected: now property based - Split sources.csv to have publicly available http repos only; internal repos are in sources_rnd.csv - System model output in BOM renamed to system_model.xml rather than canonical_system_definition_{config}.xml to make reuse predictable - Public property configuration public_override_props.ant.xml now used. - Fix all exports to run based upon configuration specified (no export configuration makes SBS assume RVCT is being used, which is not available to all) - sf.production.build MUST be defined and TRUE to enable production build properties, public config is DEFAULT behaviour

Smoketest source path wrong in build.test.xml

Try different approach to qmake

Added call to run-qmake to build.test.xml

Added auto-startup of ATS4 Engine

Modified ATS4 Agent startup method

Added exporting links to Diamonds for ATS4 Smoketest report

Modified ATS4 Engine startup command

Changed method of starting ATS4 components

Modified ats_specialise_test_drop to update FlashTask with the ROM image location

Modified update_brag_status to handle ATS4 reports

Increased 'sleep' after starting ATS4 components

Added automatic start-up of the ATS4 Agent(s) and Engine components.

Replaced ats3_testdrop with ats_testdrop to handle sending test drops to ATS4 server

Catchup merge

Modified ats_specialise_test_drop to handle ats4 test plans

Correct handling of tag name and changeset author in sf-tag-hg-code Addition of sf-push-hg-tags and associated template file, for pushing the tags created in sf-tag-hg-code back out to the MCL. Minor whitespace rationalisation.

Added specification of changeset author when committing. Removed execution of hg out - that's taken care of in the push tags target/template.

Fix: package build fail because there is no package_definitions dir

Introduced new Helium property (sf.spec.ats.version) to specify ATS version

Changed locations of package_definitions to make the files a little bit more usable

Added call to produce package definition peices from system definition(s).

Enabled tool to create arbitrarily deep paths for output

Added a few strategically placed spaces, to improve the indentation of the output.

Merge in tools for manipulating sytem defintions from their own repo.

Move files into a directory appropriate for merging into main config repo.

updated commit message as no longer triggered by change in packages repo

Separate the generation of the system model and the generation of the derived package definition into 2 script so can be run separately

change destination directory for auto generated model to sydefs\auto

Fix all symbian3 package_definition have disappeared from packages

Fix: update back to HighFidelityModel is not possible because there are changed files and it crosses branches

Exec hg add/addremove commands before hg diff

Auto generate system model into codeline/single/sysdef instead of auto directory. Minor fixes

Fix some issues

Add auto update of package models also for CompilerCompatibility builds

Add automatic update of package models to the packages repository (default branch)

Add README.txt file with instructions

Use MCL for packages repo

Use MCL for the platforms repo

Uncomment pushing to web

Add generate_system_models.pl

Fix: wrong name for diamonds_envinfo file

Add tool envinfo2diamonds.pl

Use new tools to publish and compare environment information

Remove hard-coded dependency to lib.path

Take envinfo.pl from the utilities repo

Pick correct truclean.pl from the myriad of available ones

Fix location of truclean script

Update run-qmake template to add make install step to configure Orbit

Reverse order of Diamonds summary info

Fix: Diamonds summary build info should be more related to the raptor build breakdown from the UH parser

Add sgimagelite_support to Orbit

Run qmake after exports too, to allow -configure options to be applied

Change Qt qmake order to allow -config qmake args to be passed correctly

Updated template for tagging source code so it uses the TAGS branch.

Tweaked so that all revs are cloned from the utilities web repo, but it's still synced to the head of the default branch.

Fix: build fails if any source URL doesn't contain '/MCL/'

Fix: when sf.spec.sources.revision is defined sources are not updated to it

Add new property sf.spec.sources.revision to override revisions of source repos

Include explicit export log in tsv file creation

Added script for updating BRAG status with test results

Added template for manual smoketest results

Updated brag stylesheet to handle smoketest criteria

Resolve packaging oddity with missing build.drive fwdslash

Fix property typo and add sf-package-tools dependency on sf-merge-manifest

Add re-packaging for prebuilt binaries into the PDK, driven from a manifest_* file delivered with the prebuilt zip file

Merge

Make manifest generation more accurate, based upon filtered WHAT output

Change PDK baseline for symbian3 and CompilerCompatibility builds to PDK 3.0.i

Bug 2670: binaries_epoc.zip contains megabytes of useless Emulator files Exclude files in \epoc32\data\media from release zip.

Added tag PDK_3.0.i for changeset 8ee0b12c2384

Merge

Merge s^4 and ats changes.

Merge ats_specialise_test_drop modification to automatically set the test-target param.

Merge ats_specialise_drop to insert ROM image parameters.

Add explicit qtextensions configuration step

Correct location of Qt deviceconfiguration

Build qmake in .ftl, prior to running on all .pro files

Run qmake on Qt package before Orbit

Add sf-do-compile depends on create-canonical-sysdef-file

Simplify compilation by requiring a prebuilt qmake, which will generate bld.infs first.

Merge

Update run-qmake template to run Orbit configure.py first, followed by mkspec export. This should allow hb SYSTEMINCLUDEs for subsequently processed .pro files.

Use config list in run-qmake

Improve guard by checking for qmake.exe rather than qmake.bat

Ensure inheritRefs is set to true

1. Run exports post-qmake to ensure exports from generated bld.infs 2. Explicitly run qmake to save duplicate calls for iterative builds

Include prebuilt features.dat in tools_epoc.zip, as its built via tools2

Correct path separators

Repackage a generated manifest with binary zip output

list contents of zip file

buildinfo.txt now zipped up in tools_epoc.zip to make it more available for those wishing to build the platform.

Rationalised indentation

Fixed launching of ATS3, for all servers.

Modified ats_specialise_test_drop to automatically set test-target param based on device alias

Merge

Moving call to sf-package-tools outside the /parallel tag

Adding property sf.spec.prebuilt.excludelist in platform_props.ant.xml

Modifying sf-package-binary target to exclude the overlay contents from packaging

Merge

Increased Boot Time for Syborg target to 500 secs

Told perl where to look for libraries, so that Text::CSV can be located even when the script is in utilities, and the module is in an fbf repo.

Changed value of test-target property to Syborg_tef

Merge

Improve exclusion list generation for tools build

Fixed: exec doesn't support the nested "if" element

Introduced new property for passing ROM image path to ats_specialise_test_drop

Fixed ats_specialise_drop to insert ROM image parameters

Whoops - no closing tag for if in sf-start-ats.

Fixed ats_specialise_drop to insert ROM image parameters

Merge

Added ROM image parameters to ats_specialise_drop

Merge

Run start.cmd only if it exists

Added deletion of existing syborg.dtb file from ATS temp dir

Added commands to invoke new release note generator.

Added conditional statement for copying syborg.dtb file

Added copying syborg.dtb to ATS Worker Root directory

Updated comms settings for Syborg on C100782

Added new device registration files for Syborg on C100782

Introduced new property (sf.spec.test.target) for running tests on different targets

Update qmake template to run Orbit's configure.py tool

Enable qmake for all packages that have proFile attributes

Retry "module" for ant scripts, initially used by source syncing operations.

Update csvToSysDef.pl to understand proFile and qmakeArgs attributes

Prevent qmake from being run before it is built

Adding back in the spawn attribute, because sf-start-ats is hanging

Simplified the sf-start-ats target to see if it works better

Prevent qmake from failing build by suppressing 'signalRunQMakeInput' signal

Use qt.qmake.ant.template property for custom qmake template

Merged changes from MCL

Added property files for devices on c100782 test machine.

Shift down BRAG colour attribution also for Diamonds

Backout start-ats logging as its not supported when spawn=true

Update packaging to deliver output into epoc32\ location

Add logging for sf-start-ats target

Add new sf-tools build type for use with tools projects

Shift down BRAG colour attribution (i.e. critical now make the build only red, major or unknown only amber)

Add number of missing files to major failures count

Fix canonical_system_definition.xml doesn't have XML special characters escaped

Changed packaging so that /epoc/gcc_mingw/** is put into tools_epoc.zip

Adding sf-merge-brag target

FIxed package config script to be able to handle utilities repo being at old URL, or new URL.

Merge

Add new sf-build-roms target to invoke iMaker rom building

Bug 2274 - Detection of physical drives doesn't work on Vista/Win7

Updated to deal with utilities repo moving to a new URL.

Removed Helium Version Check from BuildEnv.xml to avoid CheckBuildEnv.pl problems

Re-apply changes to allow sources.csv to specify a revision by a *local* tag in the web repository. Uses "hg id" instead of "hg in", so not affected by the aborts we had previously. And no need for an empty repo this time either.

uh2brag.pl to take account of new 'missing' column in index.html

Change URL to utilities repo

Back out emptyRepo changes - due to issues

Merge

Remove call to releaseables.pl script as now UH parser handles releaseables as well

Wrote a script to extract data for e32tests & f32tests based on input from: richardcoles / Symbian Foundation Community Member and Krishna Achugatla from Symbian.

Added -n flag (newest first) to invocation of hg in, so that it gives us the most tip-ward revision that would be needed, not the oldest (revision 0 in almost all cases).

Extended source syncing slightly, to be able to manage use of local tags on (web) server.

Update pckg_test_status_patterns.csv with more info to look for tests patterns.

Remove call to releaseables.pl script as now UH parser handles releaseables as well

Wrote a script to extract data for e32tests & f32tests based on input from: richardcoles / Symbian Foundation Community Member and Krishna Achugatla from Symbian.

Include -j option for sf-do-full-export target

1) Add support for parallel makefile generation by spliting compile into two-step operation. export first, followed by compile. 2) Remove smoketest from sources.csv and into sources_test.csv. This enables a fully resolved csv file to be fed directly back into a platform build.

Merge

Include armv5 bootloader binaries into lib zipfile.

Updated version checker to permit all versions of hg from v1.2 to v1.5

Moved bc_filter tool to utilities repo.

Changes from v800008

Adjusted invocation of ConE so it doesn't completely fail if /epoc32/rom/config/ does not exist. Should get the CI build running again.

Regularised indentation

Fix java version is not available

Update baseline for CompilerCompatibility builds

Update baseline for symbian3 builds

Added ATS property files for v800004

Turn off zipContentMacro failure for empty zip targets in Helium 7

Relocated Helium 7 *.log.dir property settings from root build.xml to common\build.xml so that 'sf.spec.job.drive' works again.

build_md5.zip now in release_metadata.xml

findphysicaldrive.pl - Fixed so it works with empty (freshly formatted) drives.

Catchup merge

Catchup merge

catchup merge.

Packaging tools with the binaries.

Catchup of Shabe's Helium7 work.

Added tag PDK_3.0.h for changeset ebd555518364

Move copying of System Model to after compile-main

Export system model to BOM directory. This used to be done by generate-layers, however this isnt called by compile-main anymore.

Merge

Fix scan_antlogs.pl execution plus fix ConE calls to use PDT_HOME env variable

Add initial support for Helium7 for the platform. * Use sf.spec.sbs.options to pass args direct to Raptor. * Backwards compatibility with Helium5 maintained. * Listdir function to be provided by sbs check filter, which will be added later.

quick fix for zipping winscw lib files. Added property 'sf.spec.package.zip.format' for the post-build zip format (default is '7z').

Changes to zipping during platform build; binary files are divided into 4 more zips, reducing individual sizes.

Added tool for filtering BC header reports

sourcesync template - Updated generation of 'changes.txt' so that it does a 'follow' rather than being restricted to a 'branch'. This enables us to see the history if the previous tag was an ancestor but on a different branch.

Corrected property name for activating the hg cache, so it should take effect!

Remove Raptor log preprocessing as no longer needed

Take the UH parser from the utilities repo

Add Helium and java to the enironment info

Update baseline for symbian3 and CC. Remove symbian2 property file as we are no loger building for symbian2

s/bishare/v800020/

Reviewed rules defining the BRAG status

Merge

Merge

Add Helium 7 support for package builds. - During operation, if last.major.helium.version is detected as 7.0 then it will use new sbsinput mechanism. - Restore sf-postbuild dependency on sf-build-noprep target

Added error correction, so if a cache repository is corrupted in some way, then it will by-pass it, and try to recover it, or delete it.

Moved definition of dollar, so it's available for use when needed.

Stopped cache being located/created in a stupid place, relative to the CWD on the cache drive. Work around limitation of ant-contrib regular expressions, so they're not thwarted by \s in the cache location property.

Add missing / Avoids potential problem of it not seeing an existing cache, depending on the CWD on the cache drive at the point the target is running (which could be anything really).

Turned on use of cache for all platform builds (unless over-ridden by the particular job properties of course).

Bug 1661: Simplified the sequence of pulls/clones via the cache, so the changes simply go from source -> cache -> destination. Made use of --uncompressed flag to speed things up. Including setting the flag when we create a cache entry.

Bug 1661: Improve build speed with better caching strategy Changed the way that the cache is located, so the same code can run on any build machine without any configuration. Made work around for Bug 419 the only option, as we're not using the alternative anyway, and I don't want to commit completely untested code! Changed caching algorithm, to improve build times.

Updated capabilities of findPhysicalDrive to support imminent work on locating the hg cache.

Indentation and other minor changes to make the file easier to read and understand.

Added profiling timer to sf-get-source target.

Fix version of new mwccsym2 is reported as N.A.

Added tag PDK_3.0.g for changeset fc600bbaa15c

Fix BRAG status is different as viewed from Diamonds and from the _BRAG.xml page

Update baseline for symbian3 and CompilerCompatibility builds. Removed unused property file for NewGraphicsArchitecture

Dump environment info to Diamonds

Fix when accessing uh index from brag summary, pages linked from uh index are not found

Make BRAG point to the results of UH parser instead of intermediate artifacts

Remove redundant computation of some BRAG intermediate xml files

Commented out log-to-brag call on build environment checker as its results shouldn't be brag affecting

Make index.html report on all packages, including the ones built successfully. Make the list of packages be alphabetically sorted

Enabled filtering results after Static BC analysis

Added tag PDK_3.0.f for changeset ef25bf39d67c

Creation of TAGS branch. The TAGS branch should be used for *all* tags, including PDK release tags and package release tags. Do not merge this branch with any "code" branches.

Merge

Remove check-tool-dependencies to allow > Perl 5.6.1

Disabled filtering results after Static BC analysis

adding EPL headers to smoke test

sf-run-analysis-missing-files - changed to depend on a new undefined parameter 'sf.spec.checkbuild' so that it can be turned on and off

sf-s60-cone-cenrep - S60 config with epoc32/data/z combintation was missing.

Add ConE or ConfigurationTool support. ConfigurationTool is used for Symbian2 only.

Updated to ignore messages of the form "Nothing to be done for ..." as these do not represent a problem of any kind at all.

Removed copying smoketest source to EPOCROOT from sf-build-smoketestpkg target

Changed the way that the sf.spec.sourcesync.enable property is used to control the syncing, so that it can sync test repos even if it's not syncing any source repos.

Move targets round so that those related to syncing the source are all together.

Intelligent way of getting timings for ALL uses of sf-zip-content

Removed changes in build.test.xml to copy smoketest source to EPOCROOT

Removed copying smoketest source to EPOCROOT

Added stopwatch tasks through most of the code hit during a platform build.

Added sf-scrub-build, for completely freeing up the resources used by a build (ie deleting it from the HD). Removed import of generated ant script as it had no effect, and was causing strange failures with sf-scrub-build.

Removed check for value of PDT_HOME, as it doesn't matter where it's installed, and it's turning the CI builds red.

Code to automatically find a physical drive for running a build, triggered as necessary.

Added new logic to deal with drive letter allocation when builds are launched from Hudson.

Added framework for specifying a "sources.csv" for test repos (separate from both the FBF and the sources repos).

Catchup merge

Fixed websrv bc test suite. Increased time-outs for selected bc tests.

updated version of sysdefdowngrade.xsl from BobR

add xsl for merging / downgrading system model

Added smoke test for Media management

Fixed ats_generate_index.pl to not include the link to Smoke Test Report

Disabled BC check for Platform builds

Fixed creation of the ats_reports folder

Fixed copying temp index

Implemented publishing Dynamic BC Test Reports

Added new property for publishing Smoke Test Report

Added tool for generating the top-level index for BC test reports

Add escape left angle bracket to preprocessing of raptor log (see bug #1519)

PDK baseline for symbian3 builds back to PDK_3.0.d as PDK_3.0.e is not stable yet

ArnaudL: corrected last entry that had "????" left behind.

Arnaudl: Added scripts and files related to the development of "task 286 - Package test status prep"

Added tag PDK_3.0.e for changeset a1ebf7f8e66f

Move gcc into tools_epoc.zip so that syborg/beagle rom building works during build from source only.

Added new param for calling ats_specialise_test_drop.pl from build.test.xml

Added smoke test for making a connection to a web server through ethernet

Update the PDK baseline for symbian3 builds

Fixed smoke test for Localisation

Added smoke test for Graphics

Added tag PDK_3.0.d for changeset b9c988316957

Added tag PDK_2.0.2 for changeset 2a9e6403efaf

Platform Build - Added 'sf-postbuild' onto deps for 'sf-build-noprep' as although it isn't the best place to have it, it's the same as the deps for the version in 'common' and is needed for the external Hudson jobs.

Ensured build environment is always compared with expected environment, whether or not it would abort the build because of it.

Changed building of tools targets so that it builds the targets listed in ${sf.spec.sbs.tools.config}, in sequence, rather than just buildng tools2_rel.whatlog followed by tools_rel.whatlog, regardless of the property.

Moved hlm:argSet definitions in sf-os-compile to be adjacent to the call that uses them. (And deleted three unused ones in the process.)

Update the PDK baseline for symbian2 builds

Updates to make the build environment check more reasonable: Mercurial v1.3 permitted The Java compiler is not a showstopping issue 7-zip can be installed in any location Update to Helium 5 Helium can be installed in PDT 1.*, not necessarily 1.0 Raptor installation path not significant Update to Raptor 2.9.* The Raptor patch to update the bundled version of python is no longer relevant BRAG calculations updated to ignore items not being in the system path, as this just doesn't matter. Overall effect is that the build environment check should pass on a machine that is able to do a build!

Regularised indentation

Fixed postAction in ats_specialise_test_drop.pl

Implemented FileStoreAction for BC tests

Implemented exporting Smoke Test Report to Diamonds

Updated postbuild to publish smoketest results

Updated ats_specialise_test_drop.pl to insert FileStoreAction param

Added new postAction to smoketest.xml

Restored signifcance of the MS tools not being available, as this only occurs when running a configuration that would try to use them.

Re-indented target "sf-os-compile" (following best guess as to the standard) to make it easier to read.

Re-assessed some of the raptor error messages to reflect the inconsequential nature of missing the MS dev tools when not trying to build the tools target.

Changed mergeXML.pl to use a better version of the XML printing algorithm, so it can generate well-formed content if the input includes some rogue &s or <>s.

If any files are in both the list of SDK headers, and the list of RnD headers, then they're only zipped in the binaries_epoc_additional zip, and not the binaries_epoc_sdk zip.

Fixed format checking of summary.csv to deal with new field.

Test: Add MediaMgmt smoke tests (not being built or run ATM)

Test: Remove the media management smoke test for now

Re-instated call to process raptor logs for BRAG purposes.

Remove s60 postfix from sf-s60-create-cenrep for Symbian^3

Fixed format checking of summary file so that it actually checks something!

Updated framework for packaging SDK headers so the contents to zip can be set by a project property (or it can zip everything together if it's not set).

add media mgmt to smoketest build

Fix cenrep export location

SDK file list - Fixed duplicates

Fixed and enabled smoke test for Set Language

Defined new devices for v800017

Disabled smoke test for Set language

Small changes to make the output file more regular, and therefor easier to read and understand.

Added processing of error tags.

Fixed perl scripts for building BC test drops

Fixed Set language smoke test

Changed to package the SDK header files in a different zip to the rest of the epoc32 tree.

Fixed smoketest.xml

Added new smoke test for phone languages

Removed code for building ROMs (now done by Raptor) but added code for publishing ROM building logs.

Fix releaseables are missing. Fix stray call to summarize.pl

Catchup merge

Disabled BC check from running as default on builds. Should be enabled specifically at the command line.

Added new smoke test for Setting wallpaper

Disabled bccheck for normal builds

Catchup merge

Platform Build cenrep generatation - modified 'sf-s60-create-cenrep' to hopefully cope with the different CRML export paths coming from the different platform_paths.hrh. This isn't pretty.

Fix Diamonds links are no longer appearing

Added conditions for BC check results to be published

Changed delays between key presses

Added smoke test to close UI setup dialogs

Increased delay before presskey

Enabled bccheck for normal builds

Update send_xml_to_diamonds.py and update invoking code to match new i/f

Updated IF statements for BC reports

Add new recipe-error categories armcc_modifier_not_allowed, mwccsym2_file_cannot_be_opened

Changed current baseline version for BC check.

Fix freemarker.core.ParseException: Encountered { at line 25, column 15 in linksForDiamonds.xml.ftl

Added Static BC reports for Platform Builds

Add NewGraphicsArchitecture_props.ant.xml

Update PDK baselines for package builds

Merge Tags

Add Raptor Build Summary to the links for Diamonds

Publish HTML files summary of Raptor build

Now creates src-by-layer entries on the fly, so it doesn't create release zips with no content.

Added interpretation of recipe "win32def2lib" failing.

Fix XML-SAX bug #43174 (see http://rt.cpan.org/Public/Bug/Display.html?id=43174)

Improved Raptor parser, now "uh"

Changed versions of the baselines for Static BC test

Disabled Dynamic BC tests

Merge

Added tag PDK_2.0.1 for changeset e13acd883fbe

merged heads - non trivial merge in smoketest. Manual merge of those changes.

Test: Input methods smoke tests added.

Test: Fixed up the app install smoke tests so that both now pass

Added new smoketest

Add call to parse (preprocessed) raptor logs directly for BRAGG.

Updated XSL to include counts of failures at each severity, and also the new effect/causes distinction, for failures that are not package specific.

New script which parses raptor logs directly to produce a BRAG summary. New version of XML structure accommodates causes of failure as well as the effect of the failure, and deals with excessive error messages. New structure also avoids a potential problem (bug?, questionable feature?) that both IE and firefox exhibit. Some simplifications of brag utilities module.

Moved common code to shared module.

Updated the options to pass to getenv, to include -k. This avoids aborts if things are not exactly as the tool expects.

Added tag PDK_3.0.c for changeset 12815724a053

Added new smoketest

Added new smoketest

Bug 881: Now also provides the locations of the two split-build system definitions to the release note generator, so it will work with both single and split builds.

source sync template - enabled getting the mercurial changes if you have sf.spec.sourcesync.archive set

Transplant fix in ats BC Test - 'bld_sis' change that was knocked out with the rest of the transplant (bcdrivers/utilities)

Added call to munge license headers at the top of the postbuild phase, so it should be done within the platorm build.

Switched to using revised Package/FCL analysis script.

Remove meta data from package definition and replace with helium script

Test: Forgot to execute Profile smoke tests

Test: Changes to test drop files to run AppInstall

Test: Tweaked Profile tests and added AppInstall

Add new raptor-analysis recipe-failure category: elf2e32_symbol_missing_from_elf_file

Override Helium defaults for RVCT checks to allow winscw builds without RVCT

Update PDK baselines for package builds

Add CompilerCompatibility properties file for package build

Changed device name for lon-engbuild89

Changed device name for lon-engbuild87

Changed device name for C100725

Changed device name

host name is added for any number of devices

Added smoketest for Notepad

Add recipe-failure category armcc_internal_fault

fix bug with multiple harnesses

export to 'smoketest' instead of 'testframework'

capitals

updated smoketest plan to support both harnesses

Add recipe-failure categories armcc_controlling_expression_is_constant and armcc_generic_warnings_errors

Adding profiles smoke tests