clone_packages/clone_all_packages.pl
author William Roberts <williamr@symbian.org>
Wed, 26 Aug 2009 17:34:08 +0100
changeset 39 674034f6180c
parent 38 759027e1c6b3
child 40 0c8b7325c98e
permissions -rw-r--r--
Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script Added better return code handling, to allow for "hg diff" which returns 0 = same, 1 = different Refactor the reading of the supplied text files to use -packagelist processing internally Added %REV% for -exec usage, which gives access to the revision identifies from build-info.xml
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
     1
#! perl
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
     2
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
     3
# Copyright (c) 2009 Symbian Foundation Ltd
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
     4
# This component and the accompanying materials are made available
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
     5
# under the terms of the License "Eclipse Public License v1.0"
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
     6
# which accompanies this distribution, and is available
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
     7
# at the URL "http://www.eclipse.org/legal/epl-v10.html".
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
     8
#
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
     9
# Initial Contributors:
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    10
# Symbian Foundation Ltd - initial contribution.
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    11
# 
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    12
# Contributors:
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    13
#
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    14
# Description:
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    15
# Perl script to clone or update all of the Foundation MCL repositories
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    16
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    17
use strict;
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    18
use Getopt::Long;
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    19
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    20
sub Usage($)
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    21
  {
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    22
  my ($msg) = @_;
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    23
  
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    24
  print "$msg\n\n" if ($msg ne "");
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    25
  
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    26
	print <<'EOF';
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    27
clone_all_repositories - simple script for cloning Symbian repository tree
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    28
	
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    29
This script will clone repositories, or pull changes into a previously
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    30
cloned repository. The script will prompt for your username and
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    31
password, which will be needed to access the SFL repositories, or you can
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    32
supply them with command line arguments.
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    33
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
    34
The list of packages can be supplied in a text file using the -packagelist
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
    35
option, which is capable of reading the build-info.xml files supplied with 
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
    36
Symbian PDKs. Supplying a build-info.xml file will cause the clone or update
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
    37
operation to use the exact revision for each of the relevant repositories.
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
    38
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    39
Important: 
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    40
  This script uses https access to the repositories, so the username and
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    41
  password will be stored as cleartext in the .hg/hgrc file for each repository.
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    42
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    43
Used with the "-mirror" option, the script will copy both MCL and FCL
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    44
repositories into the same directory layout as the Symbian website, and will
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    45
use the Mercurial "--noupdate" option when cloning.
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    46
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    47
Options:
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    48
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    49
-username      username at the Symbian website
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    50
-password      password to go with username
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    51
-mirror        create a "mirror" of the Symbian repository tree
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
    52
-packagelist   file containing the URLs for the packages to be processed
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    53
-retries       number of times to retry a failed operation (default 1)
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    54
-verbose       print the underlying "hg" commands before executing them
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    55
-n             do nothing - don't actually execute the commands
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    56
-help          print this help information
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    57
-exec          execute command on each repository
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    58
-filter <RE>   only process repository paths matching regular expression <RE>
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    59
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    60
The -exec option processes the rest of the command line, treating it as
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    61
a command to apply to each repository in turn. Some keywords are expanded
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    62
to repository-specific values, and "hg" is always expanded to "hg -R %REPO%"
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    63
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    64
%REPO%         relative path to the repository
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    65
%WREPO%        relative path to repository, with Windows path separators
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    66
%URL%          URL of the master repository
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    67
%PUSHURL%      URL suitable for pushing (always includes username & password)
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
    68
%REV%          revision associated with the repository (defaults to "tip")
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    69
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    70
It's often useful to use "--" to separate the exec command from the options
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    71
to this script, e.g. "-exec -- hg update -C tip"
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    72
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    73
EOF
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    74
  exit (1);  
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    75
  }
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    76
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
    77
my @clone_options = (); # use ("--noupdate") to clone without extracting the source
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
    78
my @pull_options  = (); # use ("--rebase") to rebase your changes when pulling
7
2184cc44590a Clone from developer.symbian.org (Bug 79) and use https for SFL packages
William Roberts <williamr@symbian.org>
parents: 5
diff changeset
    79
my $hostname = "developer.symbian.org";
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    80
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    81
my $username = "";
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    82
my $password = "";
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
    83
my $mirror = 0; # set to 1 if you want to mirror the repository structure
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
    84
my $retries = 1;  # number of times to retry problem repos
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    85
my $verbose = 0;  # turn on more tracing
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    86
my $do_nothing = 0; # print the hg commands, don't actually do them
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    87
my $help = 0;
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    88
my $exec = 0;
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
    89
my $filter = "";
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
    90
my @packagelist_files = ();
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    91
38
759027e1c6b3 Improved robustness of package list location searching.
thomase
parents: 37
diff changeset
    92
my $program_path = $0;
37
a1155a089aba Updated script to use external lists of repositories managed dynamically by SCM people.
thomase
parents: 36
diff changeset
    93
38
759027e1c6b3 Improved robustness of package list location searching.
thomase
parents: 37
diff changeset
    94
# Analyse the rest of command-line parameters
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    95
if (!GetOptions(
12
dda0176e838b Fix mistake in -password and -username options, tweak the (doomed) sfl repository screen scraping
William Roberts <williamr@symbian.org>
parents: 11
diff changeset
    96
    "u|username=s" => \$username,
dda0176e838b Fix mistake in -password and -username options, tweak the (doomed) sfl repository screen scraping
William Roberts <williamr@symbian.org>
parents: 11
diff changeset
    97
    "p|password=s" => \$password,
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    98
    "m|mirror" => \$mirror, 
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
    99
    "r|retries=i" => \$retries,
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   100
    "v|verbose" => \$verbose,
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   101
    "n" => \$do_nothing,
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   102
    "h|help" => \$help,
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   103
    "e|exec" => \$exec,
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   104
    "f|filter=s" => \$filter,
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   105
    "l|packagelist=s" => \@packagelist_files,
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   106
    ))
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   107
  {
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   108
  Usage("Invalid argument");
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   109
  }
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   110
  
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   111
Usage("Too many arguments") if (scalar @ARGV > 0 && !$exec);
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   112
Usage("Too few arguments for -exec") if (scalar @ARGV == 0 && $exec);
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   113
Usage("") if ($help);
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   114
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   115
# Important: This script uses http access to the repositories, so
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   116
# the username and password will be stored as cleartext in the
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   117
# .hg/hgrc file in each repository.
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   118
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   119
my $needs_id = 1; # assumed necessary for clone/pull
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   120
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   121
my @exec_cmd = @ARGV;
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   122
if ($exec)
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   123
  {
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   124
  if ($exec_cmd[0] eq "hg")
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   125
    {
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   126
    shift @exec_cmd;
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   127
    unshift @exec_cmd, "hg", "-R", "%REPO%";
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   128
    }
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   129
  if ($verbose)
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   130
    {
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   131
    print "* Exec template = >", join("<,>", @exec_cmd), "<\n";
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   132
    }
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   133
  $needs_id = grep /URL%/,@exec_cmd; # only need id if using %URL% or %PUSHURL%
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   134
  }
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   135
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   136
if ($needs_id && $username eq "" )
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   137
  {
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   138
  print "Username: ";
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   139
  $username = <STDIN>;
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   140
  chomp $username;
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   141
  }
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   142
if ($needs_id && $password eq "" )
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   143
  {
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   144
  print "Password: ";
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   145
  $password = <STDIN>;
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   146
  chomp $password;
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   147
  }
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   148
30
cf4bd27a8280 Add special case handling for export controlled OSS repositories (currently sf/os/security)
William Roberts <williamr@symbian.org>
parents: 29
diff changeset
   149
my %export_control_special_case = (
cf4bd27a8280 Add special case handling for export controlled OSS repositories (currently sf/os/security)
William Roberts <williamr@symbian.org>
parents: 29
diff changeset
   150
  "oss/MCL/sf/os/security" => 1,
cf4bd27a8280 Add special case handling for export controlled OSS repositories (currently sf/os/security)
William Roberts <williamr@symbian.org>
parents: 29
diff changeset
   151
  "oss/FCL/sf/os/security" => 1,
cf4bd27a8280 Add special case handling for export controlled OSS repositories (currently sf/os/security)
William Roberts <williamr@symbian.org>
parents: 29
diff changeset
   152
  );
cf4bd27a8280 Add special case handling for export controlled OSS repositories (currently sf/os/security)
William Roberts <williamr@symbian.org>
parents: 29
diff changeset
   153
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   154
sub do_system(@)
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   155
  {
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   156
  my (@cmd) = @_;
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   157
  
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   158
  if ($verbose)
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   159
    {
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   160
    print "* ", join(" ", @cmd), "\n";
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   161
    }
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   162
  return 0 if ($do_nothing);
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   163
  
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   164
  return system(@cmd);
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   165
  }
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   166
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   167
my %revisions;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   168
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   169
sub process_one_repo($)
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   170
  {
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   171
  my ($package) = @_;
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   172
  my @dirs = split /\//, $package;
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   173
  my $license = shift @dirs;
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   174
  my $repotree = shift @dirs; # remove the MCL or FCL repo tree information
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   175
  my $destdir = pop @dirs;  # ignore the package name, because Mercurial will create that
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   176
  
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   177
  if ($mirror)
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   178
    {
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   179
    # Mirror the full directory structure, so put back the license & repotree dirs
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   180
    unshift @dirs, $repotree;
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   181
    unshift @dirs, $license;
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   182
    }
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   183
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   184
  # Ensure the directories already exist as far as the parent of the repository
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   185
  my $path = "";
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   186
  foreach my $dir (@dirs)
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   187
    {
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   188
    $path = ($path eq "") ? $dir : "$path/$dir";
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   189
    if (!-d $path)
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   190
      {
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   191
      mkdir $path;
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   192
      }
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   193
    }
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   194
  
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   195
  $path .= "/$destdir";   # this is where the repository will go
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   196
7
2184cc44590a Clone from developer.symbian.org (Bug 79) and use https for SFL packages
William Roberts <williamr@symbian.org>
parents: 5
diff changeset
   197
  my $repo_url = "https://$username:$password\@$hostname/$package/";
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   198
  my $repo_push_url =$repo_url;
30
cf4bd27a8280 Add special case handling for export controlled OSS repositories (currently sf/os/security)
William Roberts <williamr@symbian.org>
parents: 29
diff changeset
   199
  if ($license ne "sfl" && !$export_control_special_case{$package})
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   200
    {
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   201
    # user registration is not required for reading public package repositories
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   202
    $repo_url = "http://developer.symbian.org/$package/";
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   203
    }
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   204
  
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   205
  my @rev_options = ();
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   206
  my $revision = $revisions{$package};
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   207
  if (defined($revision))
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   208
    {
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   209
    @rev_options = ("--rev", $revision);
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   210
    }
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   211
  else
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   212
    {
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   213
    $revision = "tip";
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   214
    # and leave the rev_options list empty
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   215
    }
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   216
  
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   217
  my $ret;
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   218
  if ($exec)
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   219
    {
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   220
    # iteration functionality - process the keywords
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   221
    my $wpath = $path;
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   222
    $wpath =~ s/\//\\/g;  # win32 path separator
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   223
    my @repo_cmd = ();
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   224
    foreach my $origcmd (@exec_cmd)
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   225
      {
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   226
      my $cmd = $origcmd; # avoid altering the original
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   227
      $cmd =~ s/%REPO%/$path/;
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   228
      $cmd =~ s/%WREPO%/$wpath/;
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   229
      $cmd =~ s/%URL%/$repo_url/;
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   230
      $cmd =~ s/%PUSHURL%/$repo_push_url/;
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   231
      $cmd =~ s/%REV%/$revision/;
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   232
      push @repo_cmd, $cmd;
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   233
      }
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   234
    print "Processing $path...\n";
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   235
    $ret = do_system(@repo_cmd);
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   236
    }
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   237
  elsif (-d "$path/.hg")
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   238
    {
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   239
    # The repository already exists, so just do an update
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   240
    
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   241
    print "Updating $destdir from $package...\n";
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   242
    $ret = do_system("hg", "pull", @pull_options, "-R", $path, $repo_url, @rev_options);
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   243
    }
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   244
  else
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   245
    {
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   246
    # Clone the repository
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   247
    
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   248
    print "Cloning $destdir from $package...\n";
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   249
    $ret = do_system("hg", "clone", @clone_options, $repo_url, $path, @rev_options);
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   250
    }
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   251
  
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   252
  $ret = $ret >> 8;   # extract the exit status
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   253
  print "* Exit status $ret for $path\n\n" if ($verbose);
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   254
  return $ret;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   255
  }
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   256
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   257
if (scalar @packagelist_files == 0)
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   258
  {
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   259
  # Read the package list files alongside the script itself
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   260
  
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   261
  # Extract the path location of the program and locate package list files
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   262
  $program_path =~ s/\\/\//g;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   263
  $program_path =~ s/(^.*\/)[^\/]+$/$1/;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   264
  foreach my $file ("sf_mcl_packages.txt", "sftools_mcl_packages.txt", "other_packages.txt")
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   265
    {
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   266
    push @packagelist_files, $program_path.$file;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   267
    }
5
c34a018f3291 Re-introduce the clone_all_packages.pl script
William Roberts <williamr@symbian.org>
parents:
diff changeset
   268
  }
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   269
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   270
my @all_packages = ();
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   271
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   272
foreach my $file (@packagelist_files)
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   273
  {
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   274
  print "* reading package information from $file...\n" if ($verbose);
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   275
  open PKG_LIST, "<$file" or die "Can't open $file: $!\n";
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   276
  foreach my $line (<PKG_LIST>)
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   277
    {
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   278
    chomp($line);
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   279
    
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   280
    my $revision; # set when processing build-info listings
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   281
    
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   282
    # build-info.xml format
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   283
    # <baseline>//v800008/Builds01/mercurial_master_prod/sfl/MCL/sf/adaptation/stubs/#:e086c7f635d5</baseline>
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   284
    if ($line =~ /<baseline>(.*)#:([0-9a-z]+)<\/baseline>/i)
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   285
      {
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   286
      $line = $1;   # discard the wrapping
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   287
      $revision = $2;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   288
      }
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   289
 
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   290
    # Look for the oss/MCL/ prefix to a path e.g.
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   291
    # https://developer.symbian.org/oss/FCL/interim/contrib/WidgetExamples
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   292
    if ($line =~ /((oss|sfl)\/(FCL|MCL)\/.*)\s*$/)
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   293
      {
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   294
      my $repo_path = $1;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   295
      $repo_path =~ s/\/$//;  # remove trailing slash, if any
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   296
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   297
      push @all_packages, $repo_path;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   298
      $revisions{$repo_path} = $revision if (defined $revision);
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   299
      next;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   300
      }
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   301
    }
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   302
  close PKG_LIST;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   303
  }
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   304
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   305
if ($mirror)
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   306
  {
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   307
  push @clone_options, "--noupdate";
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   308
  
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   309
  # Assume that every MCL has a matching FCL
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   310
  my @list_with_fcls = ();
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   311
  foreach my $package (@all_packages)
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   312
    {
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   313
    push @list_with_fcls, $package;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   314
    if ($package =~ /MCL/)
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   315
      {
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   316
      # If mirroring, get the matching FCLs as well as MCLs
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   317
      $package =~ s/MCL/FCL/;
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   318
      push @list_with_fcls, $package;
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   319
      }
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   320
    }
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   321
  @all_packages = @list_with_fcls;
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   322
  
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   323
  }
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   324
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   325
my @problem_packages = ();
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   326
my $total_packages = 0;
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   327
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   328
foreach my $package (@all_packages)
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   329
  {
15
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   330
  if ($filter && $package !~ /$filter/)
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   331
    {
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   332
    next; # skip repos which don't match the filter
880bb4a0a72d Add the -exec and -filter options to clone_all_repositories, giving simple "iterator" functionality
William Roberts <williamr@symbian.org>
parents: 12
diff changeset
   333
    }
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   334
  my $err = process_one_repo($package);
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   335
  $total_packages++;
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   336
  push @problem_packages, $package if ($err < 0 || $err > 127); 
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   337
  }
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   338
  
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   339
# retry problem packages
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   340
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   341
my $attempt = 0;
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   342
while ($attempt < $retries && scalar @problem_packages) 
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   343
  {
11
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   344
  $attempt++;
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   345
  printf "\n\n------------\nRetry attempt %d on %d packages\n",
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   346
    $attempt, scalar @problem_packages;
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   347
  print join("\n", @problem_packages, ""), "\n";
319764718a57 Add proper usage and more options (Bug 80). Add prototype screen-scraping to read repository list.
William Roberts <williamr@symbian.org>
parents: 10
diff changeset
   348
    
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   349
  my @list = @problem_packages;
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   350
  @problem_packages = ();
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   351
  foreach my $package (@list)
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   352
    {
39
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   353
    my $err = process_one_repo($package);
674034f6180c Add -packagelist option, which also supports build-info.xml to provide the equivalent of the clone_BOM.pl script
William Roberts <williamr@symbian.org>
parents: 38
diff changeset
   354
    push @problem_packages, $package if ($err < 0 || $err > 127); 
10
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   355
   }
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   356
  }
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   357
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   358
printf "\n------------\nProcessed %d packages, of which %d reported errors\n", 
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   359
  $total_packages, scalar @problem_packages;
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   360
if (scalar @problem_packages)
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   361
  {
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   362
  print join("\n", @problem_packages, "");
ccca32510405 Prompt for username and password (Bug 80), support retries and repo mirroring
William Roberts <williamr@symbian.org>
parents: 7
diff changeset
   363
  }