downloadkit/downloadkit.py
author William Roberts <williamr@symbian.org>
Tue, 29 Jun 2010 17:58:44 +0100
changeset 274 3b8bce67b574
parent 199 a99c74c53f62
child 276 c1b745b16f58
permissions -rw-r--r--
version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures Give a clear statement of success or failure after all the unzipping (if any) is completed This version will also resume any interrupted transfers (unless you say --noresume) based on the file sizes.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
     1
#!/usr/bin/python
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
     2
# Copyright (c) 2009 Symbian Foundation.
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
     3
# All rights reserved.
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
     4
# This component and the accompanying materials are made available
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
     5
# under the terms of the License "Eclipse Public License v1.0"
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
     6
# which accompanies this distribution, and is available
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
     7
# at the URL "http://www.eclipse.org/legal/epl-v10.html".
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
     8
#
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
     9
# Initial Contributors:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    10
# Symbian Foundation - Initial contribution
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    11
# 
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    12
# Description:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    13
# Script to download and unpack a Symbian PDK - assumes "7z" installed to unzip the files
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    14
174
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    15
import socket
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    16
import urllib2
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    17
import urllib
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    18
import os.path
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    19
import cookielib
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    20
import sys
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    21
import getpass
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    22
import re
140
55dd69d60bbc version 0.5 - fix unzipping bug, add "--progress" option for info on long downloads
William Roberts <williamr@symbian.org>
parents: 139
diff changeset
    23
import time
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    24
from BeautifulSoup import BeautifulSoup
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
    25
from optparse import OptionParser
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
    26
import hashlib
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
    27
import xml.etree.ElementTree as ET 
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    28
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
    29
version = '0.17'
171
5f1dcce7f6d4 version 0.10 - add --nowinscw option to avoid winscw binaries, add version to the user-agent string
William Roberts <williamr@symbian.org>
parents: 170
diff changeset
    30
user_agent = 'downloadkit.py script v' + version
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    31
headers = { 'User-Agent' : user_agent }
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    32
top_level_url = "https://developer.symbian.org"
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    33
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()	# not relevant for live Symbian website
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
    34
download_list = []
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
    35
failure_list = []
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
    36
unzip_list = []
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    37
170
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
    38
def build_opener(debug=False):
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    39
	# Create a HTTP and HTTPS handler with the appropriate debug
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    40
	# level.  We intentionally create a new one because the
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    41
	# OpenerDirector class in urllib2 is smart enough to replace
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    42
	# its internal versions with ours if we pass them into the
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    43
	# urllib2.build_opener method.  This is much easier than trying
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    44
	# to introspect into the OpenerDirector to find the existing
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    45
	# handlers.
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    46
	http_handler = urllib2.HTTPHandler(debuglevel=debug)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    47
	https_handler = urllib2.HTTPSHandler(debuglevel=debug)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    48
	
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    49
	# We want to process cookies, but only in memory so just use
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    50
	# a basic memory-only cookie jar instance
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    51
	cookie_jar = cookielib.LWPCookieJar()
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    52
	cookie_handler = urllib2.HTTPCookieProcessor(cookie_jar)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    53
	
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    54
	# add HTTP authentication password handler (only relevant for Symbian staging server)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    55
	authhandler = urllib2.HTTPBasicAuthHandler(passman)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    56
	
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    57
	handlers = [authhandler, http_handler, https_handler, cookie_handler]
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    58
	opener = urllib2.build_opener(*handlers)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    59
	
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    60
	# Save the cookie jar with the opener just in case it's needed
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    61
	# later on
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    62
	opener.cookie_jar = cookie_jar
170
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
    63
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    64
	return opener
170
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
    65
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    66
urlopen = urllib2.urlopen
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    67
Request = urllib2.Request
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
    68
174
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    69
def quick_networking_check():
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    70
	global options
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    71
	defaulttimeout = socket.getdefaulttimeout()
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    72
	socket.setdefaulttimeout(15)
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    73
	probesite = top_level_url
174
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    74
	probeurl = probesite + '/main/user_profile/login.php'
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    75
	headers = { 'User-Agent' : user_agent }
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    76
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    77
	req = urllib2.Request(probeurl, None, headers)
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    78
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    79
	try:
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    80
		response = urllib2.urlopen(req)
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    81
		doc=response.read()
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    82
	except urllib2.URLError, e:
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    83
		if hasattr(e, 'code') and e.code == 401:#
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    84
			# Needs HTTP basic authentication
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    85
			print >> sys.stderr, 'HTTP username: ',
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    86
			http_username=sys.stdin.readline().strip()
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    87
			http_password=getpass.getpass('HTTP password: ')
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    88
			passman.add_password(None, top_level_url, http_username, http_password)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    89
			# now try again...
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    90
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    91
	try:
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    92
		response = urllib2.urlopen(req)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    93
		doc=response.read()
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
    94
	except urllib2.URLError, e:
174
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    95
		print '*** Problem accessing ' + probesite
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    96
		if hasattr(e, 'reason'):
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    97
			print '*** Reason: ', e.reason
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    98
		elif hasattr(e, 'code'):
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
    99
			print '*** Error code: ', e.code
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   100
		print "Do you need to use a proxy server to access the %s website?" % probesite
174
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
   101
		sys.exit(1)
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
   102
	socket.setdefaulttimeout(defaulttimeout)	# restore the default timeout
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
   103
	if options.progress:
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
   104
		print "Confirmed that we can access " + probesite
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
   105
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   106
def login(prompt):
160
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   107
	global options
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   108
	loginurl =  top_level_url + '/main/user_profile/login.php'
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   109
	
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   110
	if prompt:
160
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   111
		if options.username == '':
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   112
			print >> sys.stderr, 'username: ',
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   113
			options.username=sys.stdin.readline().strip()
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   114
		if options.password == '':
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   115
			options.password=getpass.getpass()
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   116
	
160
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   117
	values = {'username' : options.username,
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   118
	          'password' : options.password,
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   119
	          'submit': 'Login'}
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   120
	          
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   121
	headers = { 'User-Agent' : user_agent }
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   122
	
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   123
	
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   124
	data = urllib.urlencode(values)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   125
	req = urllib2.Request(loginurl, data, headers)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   126
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   127
	response = urllib2.urlopen(req)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   128
	doc=response.read()      
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   129
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   130
	if doc.find('Please try again') != -1:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   131
		print >> sys.stderr, 'Login failed'
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   132
		return False
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   133
	return True
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   134
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   135
from threading import Thread
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   136
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   137
class unzipfile(Thread):
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   138
	def __init__ (self,filename,levels=1,deletelevels=0):
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   139
		Thread.__init__(self)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   140
		self.filename = filename
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   141
		self.levels = levels
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   142
		self.deletelevels = deletelevels
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   143
		self.status = -1
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   144
		
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   145
	def unzip(self,filename,unziplevel,deletelevel):
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   146
		if unziplevel < 1:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   147
			return 0   # do nothing
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   148
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   149
		print "  Unzipping " + filename
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   150
		filelist = os.popen("7z x -y "+self.filename)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   151
		subzips = []
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   152
		for line in filelist.readlines():
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   153
			# Extracting  src_oss_app_webuis.zip
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   154
			match = re.match(r"^Extracting\s+(\S+.zip)$", line)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   155
			if match is None: continue
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   156
			subzips.append(match.group(1))
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   157
		topstatus = filelist.close()
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   158
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   159
		if deletelevel > 0:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   160
			print "  Deleting " + filename
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   161
			os.remove(filename)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   162
		if unziplevel > 1 and len(subzips) > 0:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   163
			print "  Expanding %d zip files from %s" % (len(subzips), filename)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   164
			for subzip in subzips:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   165
				self.unzip(subzip, unziplevel-1, deletelevel-1)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   166
		return topstatus
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   167
	def run(self):
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   168
		self.status = self.unzip(self.filename, self.levels, self.deletelevels)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   169
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   170
threadlist = []
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   171
def schedule_unzip(filename, unziplevel, deletelevel):
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   172
	global options
139
9baccbcc5509 version 0.4 - added --nounzip option to suppress the unzipping and deletion
William Roberts <williamr@symbian.org>
parents: 138
diff changeset
   173
	if options.nounzip :
9baccbcc5509 version 0.4 - added --nounzip option to suppress the unzipping and deletion
William Roberts <williamr@symbian.org>
parents: 138
diff changeset
   174
		return
153
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   175
	if options.nodelete :
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   176
		deletelevel = 0
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   177
	if options.dryrun :
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   178
		global unzip_list
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   179
		if unziplevel > 0:
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   180
			unzip_list.append("7z x -y %s" % filename)
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   181
			if unziplevel > 1:
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   182
				unzip_list.append("# unzip recursively %d more times" % unziplevel-1)
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   183
		if deletelevel > 0:
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   184
			unzip_list.append("# delete %s" % filename)
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   185
			if deletelevel > 1:
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   186
				unzip_list.append("# delete zip files recursively %d more times" % deletelevel-1)
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   187
		return
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   188
		
140
55dd69d60bbc version 0.5 - fix unzipping bug, add "--progress" option for info on long downloads
William Roberts <williamr@symbian.org>
parents: 139
diff changeset
   189
	unzipthread = unzipfile(filename, unziplevel, deletelevel)
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   190
	global threadlist
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   191
	threadlist.append(unzipthread)
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   192
	unzipthread.start()
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   193
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   194
def complete_outstanding_unzips():
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   195
	global options
139
9baccbcc5509 version 0.4 - added --nounzip option to suppress the unzipping and deletion
William Roberts <williamr@symbian.org>
parents: 138
diff changeset
   196
	if options.dryrun or options.nounzip:
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   197
		return
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   198
	print "Waiting for outstanding commands to finish..."
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   199
	for thread in threadlist:
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   200
		thread.join()  
153
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   201
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   202
def check_unzip_environment():
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   203
	global options
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   204
	if options.nounzip:
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   205
		return True		# if we aren't unzipping, no need to have 7z installed
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   206
	help = os.popen("7z -h")
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   207
	for line in help.readlines():
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   208
		if re.match('7-Zip', line) :
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   209
			help.close()
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   210
			return True
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   211
	help.close()
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   212
	return False
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   213
128
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   214
def orderResults(x,y) :
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   215
	def ranking(name) :
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   216
		# 0th = release_metadata
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   217
		if re.match(r"release_metadata", name):
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   218
			return 0000;
128
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   219
		# 1st = release_metadata, build_BOM.zip (both small things!)
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   220
		if re.match(r"build_BOM", name):
128
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   221
			return 1000;
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   222
		# 2nd = tools, binaries (required for execution and compilation)
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   223
		elif re.match(r"(binaries_|tools_)", name):
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   224
			return 2000;
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   225
		# 3rd = rnd binaries, binary patches
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   226
		elif re.match(r"(bin_)", name):
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   227
			return 3000;
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   228
		# 4th = sources
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   229
		elif re.match(r"(src_sfl|src_oss)", name):
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   230
			return 4000;
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   231
		# 5rd = rnd sources, source patches (not sure we'd ever have those)
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   232
		elif re.match(r"(src_)", name):
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   233
			return 5000;
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   234
		# Last, anything else
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   235
		return 10000;
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   236
	xtitle = x['title']
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   237
	ytitle = y['title']
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   238
	return cmp(ranking(xtitle)+cmp(xtitle,ytitle), ranking(ytitle))
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   239
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   240
def md5_checksum(filename):
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   241
	MD5_BLOCK_SIZE = 128 * 1024
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   242
	md5 = hashlib.md5()
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   243
	try:
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   244
		file = open(filename,"rb")
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   245
	except IOError:
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   246
		print "Terminating script: Unable to open %S" % filename
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   247
		sys.exit()
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   248
	while True:
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   249
		data = file.read(MD5_BLOCK_SIZE)
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   250
		if not data:
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   251
			break
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   252
		md5.update(data)
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   253
	file.close()
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   254
	return md5.hexdigest().upper()
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   255
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   256
checksums = {}
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   257
filesizes = {}
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   258
def parse_release_metadata(filename):
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   259
	if os.path.exists(filename):
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   260
		tree = ET.parse(filename)
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   261
		iter = tree.getiterator('package')
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   262
		for element in iter:
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   263
			if element.keys():
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   264
				file = element.get("name")
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   265
				md5 = element.get("md5checksum")
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   266
				checksums[file] = md5.upper()
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   267
				size = element.get("size")
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   268
				filesizes[file] = int(size)
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   269
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   270
def download_file(filename,url):
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   271
	global options
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   272
	global checksums
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   273
	global filesizes
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   274
	resume_start = 0
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   275
	if os.path.exists(filename):
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   276
		if filename in checksums:
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   277
			print 'Checking existing ' + filename
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   278
			file_size = os.stat(filename).st_size
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   279
			if file_size == filesizes[filename]:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   280
				file_checksum = md5_checksum(filename)
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   281
				if file_checksum == checksums[filename]:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   282
					if options.progress:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   283
						print '- OK ' + filename
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   284
					return True
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   285
			elif file_size < filesizes[filename]:
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   286
				if options.progress:
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   287
					print '- %s is too short' % (filename)
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   288
				if options.debug:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   289
					print '- %s is %d bytes, should be %d bytes' % (filename, file_size, filesizes[filename])
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   290
				if options.resume:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   291
					resume_start = file_size
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   292
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   293
	if options.dryrun and not re.match(r"release_metadata", filename):
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   294
		global download_list
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   295
		download_info = "download %s %s" % (filename, url)
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   296
		download_list.append(download_info)
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   297
		return True
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   298
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   299
	print 'Downloading ' + filename
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   300
	global headers
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   301
	request_headers = headers.copy()		# want a fresh copy for each request
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   302
	if resume_start > 0:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   303
		request_headers['Range'] = "bytes=%d-%d" % (resume_start, filesizes[filename])
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   304
	req = urllib2.Request(url, None, request_headers)
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   305
	
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   306
	CHUNK = 128 * 1024
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   307
	size = 0
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   308
	filesize = -1
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   309
	start_time = time.time()
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   310
	last_time = start_time
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   311
	last_size = size
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   312
	try:
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   313
		response = urllib2.urlopen(req)
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   314
		chunk = response.read(CHUNK)
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   315
		if chunk.find('<div id="sign_in_box">') != -1:
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   316
			# our urllib2 cookies have gone awol - login again
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   317
			login(False)
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   318
			req = urllib2.Request(url, None, request_headers)
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   319
			response = urllib2.urlopen(req)
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   320
			chunk = response.read(CHUNK)
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   321
			if chunk.find('<div id="sign_in_box">') != -1:
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   322
				# still broken - give up on this one
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   323
				print "*** ERROR trying to download %s" % (filename)
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   324
				return False
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   325
		info = response.info()
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   326
		if 'Content-Length' in info:
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   327
			filesize = resume_start + int(info['Content-Length'])		# NB. length of the requested content, taking into account the range
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   328
			if resume_start > 0 and 'Content-Range' not in info:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   329
				# server doesn't believe in our range
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   330
				filesize = int(info['Content-Length'])
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   331
				if options.debug:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   332
					print "Server reports filesize as %d, ignoring our range request (%d-%d)" % (filesize, resume_start, filesizes[filename])
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   333
				resume_start = 0;	# will have to download from scratch
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   334
			if filename in filesizes:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   335
				if filesize != filesizes[filename]:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   336
					print "WARNING:  %s size %d does not match release_metadata.xml (%d)" % ( filename, filesize, filesizes[filename])
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   337
		else:
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   338
			match = re.search('>([^>]+Licen[^<]+)<', chunk, re.IGNORECASE)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   339
			if match:
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   340
				license = match.group(1).replace('&amp;','&')
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   341
				print "*** %s is subject to the %s which you have not yet accepted\n" % (filename,license)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   342
				return False
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   343
			print "*** HTTP response did not contain 'Content-Length' when expected"
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   344
			if options.debug:
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   345
				print info
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   346
				print chunk
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   347
			return False
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   348
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   349
	except urllib2.URLError, e:
160
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   350
		print '- ERROR: Failed to start downloading ' + filename
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   351
		if hasattr(e, 'reason'):
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   352
			print 'Reason: ', e.reason
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   353
		elif hasattr(e, 'code'):
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   354
			print 'Error code: ', e.code
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   355
		return False
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   356
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   357
	# we are now up and running, and chunk contains the start of the download
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   358
	if options.debug:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   359
		print "\nReading %s from effective URL %s" % (filename, response.geturl())
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   360
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   361
	try:
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   362
		if resume_start > 0:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   363
			fp = open(filename, 'a+b')	# append to existing content
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   364
			if options.progress:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   365
				print " - Resuming at offset %d" % (resume_start)
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   366
				size = resume_start
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   367
				last_size = size
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   368
		else:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   369
			fp = open(filename, 'wb')		# write new file
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   370
		md5 = hashlib.md5()
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   371
		while True:
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   372
			fp.write(chunk)
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   373
			md5.update(chunk)
140
55dd69d60bbc version 0.5 - fix unzipping bug, add "--progress" option for info on long downloads
William Roberts <williamr@symbian.org>
parents: 139
diff changeset
   374
			size += len(chunk)
55dd69d60bbc version 0.5 - fix unzipping bug, add "--progress" option for info on long downloads
William Roberts <williamr@symbian.org>
parents: 139
diff changeset
   375
			now = time.time()
55dd69d60bbc version 0.5 - fix unzipping bug, add "--progress" option for info on long downloads
William Roberts <williamr@symbian.org>
parents: 139
diff changeset
   376
			if options.progress and now-last_time > 20:
141
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   377
				rate = (size-last_size)/(now-last_time)
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   378
				estimate = ""
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   379
				if filesize > 0 and rate > 0:
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   380
					remaining_seconds = (filesize-size)/rate
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   381
					if remaining_seconds > 110:
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   382
						remaining = "%d minutes" % (remaining_seconds/60)
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   383
					else:
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   384
						remaining = "%d seconds" % remaining_seconds
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   385
					estimate = "- %d%% est. %s" % ((100*size/filesize), remaining)
1ef616833a37 version 0.5.1 - improve the progress reporting, getting the file size from the HTTP message header
William Roberts <williamr@symbian.org>
parents: 140
diff changeset
   386
				print "- %d Kb (%d Kb/s) %s" % (size/1024, (rate/1024)+0.5, estimate)
140
55dd69d60bbc version 0.5 - fix unzipping bug, add "--progress" option for info on long downloads
William Roberts <williamr@symbian.org>
parents: 139
diff changeset
   387
				last_time = now
55dd69d60bbc version 0.5 - fix unzipping bug, add "--progress" option for info on long downloads
William Roberts <williamr@symbian.org>
parents: 139
diff changeset
   388
				last_size = size
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   389
			chunk = response.read(CHUNK)
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   390
			if not chunk: break
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   391
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   392
		fp.close()
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   393
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   394
	#handle errors
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   395
	except urllib2.URLError, e:
160
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   396
		print '- ERROR: Failed while downloading ' + filename
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   397
		if hasattr(e, 'reason'):
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   398
			print 'Reason: ', e.reason
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   399
		elif hasattr(e, 'code'):
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   400
			print 'Error code: ', e.code
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   401
		return False
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   402
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   403
	if options.debug:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   404
		info = response.info()
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   405
		print "Info from final response of transfer:"
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   406
		print response.info()
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   407
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   408
	if filesize > 0 and size != filesize:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   409
		print "Incomplete transfer - only received %d bytes of the expected %d byte file" % (size, filesize)
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   410
		return False
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   411
	
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   412
	if options.progress:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   413
		now = time.time()
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   414
		print "- Completed %s - %d Kb in %d seconds" % (filename, (filesize/1024)+0.5, now-start_time)
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   415
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   416
	if filename in checksums:
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   417
		download_checksum = md5.hexdigest().upper()
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   418
		if resume_start > 0:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   419
			# did a partial download, so need to checksum the whole file
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   420
			download_checksum = md5_checksum(filename)
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   421
		if download_checksum != checksums[filename]:
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   422
			if options.debug:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   423
				print '- Checksum for %s was %s, expected %s' % (filename, download_checksum, checksums[filename])
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   424
			print '- ERROR: %s checksum does not match' % filename
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   425
			return False
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   426
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   427
	return True
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   428
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   429
def downloadkit(version):	
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   430
	global headers
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   431
	global options
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   432
	global failure_list
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   433
	urlbase = top_level_url + '/main/tools_and_kits/downloads/'
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   434
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   435
	viewid = 5   # default to Symbian^3
199
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   436
	if version[0] == '2':
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   437
		viewid= 1  # Symbian^2
199
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   438
	if version[0] == '3':
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   439
		viewid= 5  # Symbian^3
199
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   440
	url = urlbase + ('view.php?id=%d'% viewid)
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   441
	if len(version) > 1:
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   442
		# single character version means "give me the latest"
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   443
		url = url + '&vId=' + version
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   444
125
c107e11c37e8 Mark all requests with the "downloadkit.py script" User Agent string, so that we can spot them in the HTTP logs
William Roberts <williamr@symbian.org>
parents: 124
diff changeset
   445
	req = urllib2.Request(url, None, headers)
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   446
	response = urllib2.urlopen(req)
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   447
	doc=response.read()
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   448
	
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   449
	# BeatifulSoup chokes on some javascript, so we cut away everything before the <body>
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   450
	try:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   451
		bodystart=doc.find('<body>')
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   452
		doc = doc[bodystart:]
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   453
	except:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   454
		pass
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   455
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   456
	if options.debug:
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   457
		f = open("downloadpage.html","w")
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   458
		print >>f, doc 
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   459
		f.close()
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   460
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   461
	soup=BeautifulSoup(doc)
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   462
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   463
	# check that this is the right version
197
04b0678959bf v0.14 - handle the version checking in the "(Public)" download page as well
William Roberts <williamr@symbian.org>
parents: 196
diff changeset
   464
	match = re.search('Platform Release (\(Public\) )?v(\d\.\d\.[0-9a-z]+)', doc, re.IGNORECASE)
199
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   465
	if not match:
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   466
		print "*** ERROR: no version information in the download page"
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   467
		return 0
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   468
		
199
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   469
	if len(version) > 1:
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   470
		if match.group(2) != version:
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   471
			print "*** ERROR: version %s is not available" % version
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   472
			print "*** the website is offering version %s instead" % match.group(2)
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   473
			return 0
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   474
	else:
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   475
		print "The latest version of Symbian^%s is PDK %s" % (version, match.group(2))
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   476
		
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   477
	# let's hope the HTML format never changes...
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   478
	# <a href='download.php?id=27&cid=60&iid=270' title='src_oss_mw.zip'> ...</a> 
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   479
	threadlist = []
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   480
	results=soup.findAll('a', href=re.compile("^download"), title=re.compile("\.(zip|xml)$"))
128
249ca6c587b6 Changes to control the order of downloading, and hopefully the resulting unzipping (for patches in particular)
William Roberts <williamr@symbian.org>
parents: 127
diff changeset
   481
	results.sort(orderResults)
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   482
	for result in results:
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   483
		downloadurl = urlbase + result['href']
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   484
		filename = result['title']
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   485
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   486
		if options.nosrc and re.match(r"(src_sfl|src_oss)", filename) :
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   487
			continue 	# no snapshots of Mercurial source thanks...
171
5f1dcce7f6d4 version 0.10 - add --nowinscw option to avoid winscw binaries, add version to the user-agent string
William Roberts <williamr@symbian.org>
parents: 170
diff changeset
   488
		if options.nowinscw and re.search(r"winscw", filename) :
5f1dcce7f6d4 version 0.10 - add --nowinscw option to avoid winscw binaries, add version to the user-agent string
William Roberts <williamr@symbian.org>
parents: 170
diff changeset
   489
			continue 	# no winscw emulator...
198
90e6dc719131 version 0.15 - add --noarmv5 option to avoid armv5 binaries. using this with --nowinscw also avoids binaries_epoc.zip and binaries_epoc_sdk.zip.
Shabe Razvi <shaber@symbian.org>
parents: 197
diff changeset
   490
		if options.noarmv5 and re.search(r"armv5", filename) :
90e6dc719131 version 0.15 - add --noarmv5 option to avoid armv5 binaries. using this with --nowinscw also avoids binaries_epoc.zip and binaries_epoc_sdk.zip.
Shabe Razvi <shaber@symbian.org>
parents: 197
diff changeset
   491
			continue 	# no armv5 emulator...
90e6dc719131 version 0.15 - add --noarmv5 option to avoid armv5 binaries. using this with --nowinscw also avoids binaries_epoc.zip and binaries_epoc_sdk.zip.
Shabe Razvi <shaber@symbian.org>
parents: 197
diff changeset
   492
		if options.noarmv5 and options.nowinscw and re.search(r"binaries_epoc.zip|binaries_epoc_sdk", filename) :
90e6dc719131 version 0.15 - add --noarmv5 option to avoid armv5 binaries. using this with --nowinscw also avoids binaries_epoc.zip and binaries_epoc_sdk.zip.
Shabe Razvi <shaber@symbian.org>
parents: 197
diff changeset
   493
			continue 	# skip binaries_epoc and binaries_sdk ...
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   494
		if download_file(filename, downloadurl) != True :
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   495
			failure_list.append(filename)
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   496
			continue # download failed
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   497
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   498
		# unzip the file (if desired)
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   499
		if re.match(r"patch", filename):
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   500
			complete_outstanding_unzips()	# ensure that the thing we are patching is completed first
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   501
			
156
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   502
		if re.match(r"release_metadata", filename):
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   503
			parse_release_metadata(filename)	# read the md5 checksums etc
0df3a90af030 version 0.7 - add parsing of release_metadata.xml and checking of the MD5 checksums (based on suggested code from ChrisD)
William Roberts <williamr@symbian.org>
parents: 154
diff changeset
   504
		elif re.match(r"(bin|tools).*\.zip", filename):
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   505
			schedule_unzip(filename, 1, 0)   # unzip once, don't delete
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   506
		elif re.match(r"src_.*\.zip", filename):
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   507
			schedule_unzip(filename, 1, 1)   # zip of zips, delete top level
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   508
		elif re.match(r"build_BOM.zip", filename):
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   509
			schedule_unzip(filename, 1, 1)   # unpack then delete zip as it's not needed again
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   510
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   511
	# wait for the unzipping threads to complete
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   512
	complete_outstanding_unzips()  
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   513
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   514
	if len(failure_list) > 0:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   515
		print "\n"
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   516
		print "Downloading completed, with failures in %d files\n" % (len(failure_list))
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   517
		print "\n\t".join(failure_list)
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   518
		print "\n"
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   519
	elif not options.dryrun:
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   520
		print "\nDownloading completed successfully"
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   521
	return 1
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   522
171
5f1dcce7f6d4 version 0.10 - add --nowinscw option to avoid winscw binaries, add version to the user-agent string
William Roberts <williamr@symbian.org>
parents: 170
diff changeset
   523
parser = OptionParser(version="%prog "+version, usage="Usage: %prog [options] version")
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   524
parser.add_option("-n", "--dryrun", action="store_true", dest="dryrun",
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   525
	help="print the files to be downloaded, the 7z commands, and the recommended deletions")
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   526
parser.add_option("--nosrc", action="store_true", dest="nosrc",
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   527
	help="Don't download any of the source code available directly from Mercurial")
171
5f1dcce7f6d4 version 0.10 - add --nowinscw option to avoid winscw binaries, add version to the user-agent string
William Roberts <williamr@symbian.org>
parents: 170
diff changeset
   528
parser.add_option("--nowinscw", action="store_true", dest="nowinscw",
5f1dcce7f6d4 version 0.10 - add --nowinscw option to avoid winscw binaries, add version to the user-agent string
William Roberts <williamr@symbian.org>
parents: 170
diff changeset
   529
	help="Don't download the winscw emulator")
198
90e6dc719131 version 0.15 - add --noarmv5 option to avoid armv5 binaries. using this with --nowinscw also avoids binaries_epoc.zip and binaries_epoc_sdk.zip.
Shabe Razvi <shaber@symbian.org>
parents: 197
diff changeset
   530
parser.add_option("--noarmv5", action="store_true", dest="noarmv5",
90e6dc719131 version 0.15 - add --noarmv5 option to avoid armv5 binaries. using this with --nowinscw also avoids binaries_epoc.zip and binaries_epoc_sdk.zip.
Shabe Razvi <shaber@symbian.org>
parents: 197
diff changeset
   531
	help="Don't download the armv5 binaries")
139
9baccbcc5509 version 0.4 - added --nounzip option to suppress the unzipping and deletion
William Roberts <williamr@symbian.org>
parents: 138
diff changeset
   532
parser.add_option("--nounzip", action="store_true", dest="nounzip",
9baccbcc5509 version 0.4 - added --nounzip option to suppress the unzipping and deletion
William Roberts <williamr@symbian.org>
parents: 138
diff changeset
   533
	help="Just download, don't unzip or delete any files")
153
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   534
parser.add_option("--nodelete", action="store_true", dest="nodelete",
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   535
	help="Do not delete files after unzipping")
140
55dd69d60bbc version 0.5 - fix unzipping bug, add "--progress" option for info on long downloads
William Roberts <williamr@symbian.org>
parents: 139
diff changeset
   536
parser.add_option("--progress", action="store_true", dest="progress",
55dd69d60bbc version 0.5 - fix unzipping bug, add "--progress" option for info on long downloads
William Roberts <williamr@symbian.org>
parents: 139
diff changeset
   537
	help="Report download progress")
160
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   538
parser.add_option("-u", "--username", dest="username", metavar="USER",
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   539
	help="login to website as USER")
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   540
parser.add_option("-p", "--password", dest="password", metavar="PWD",
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   541
	help="specify the account password")
170
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
   542
parser.add_option("--debug", action="store_true", dest="debug", 
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
   543
	help="debug HTML traffic (not recommended!)")
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   544
parser.add_option("--webhost", dest="webhost", metavar="SITE",
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   545
	help="use alternative website (for testing!)")
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   546
parser.add_option("--noresume", action="store_false", dest="resume",
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   547
	help="Do not attempt to continue a previous failed transfer")
160
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   548
parser.set_defaults(
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   549
	dryrun=False, 
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   550
	nosrc=False, 
171
5f1dcce7f6d4 version 0.10 - add --nowinscw option to avoid winscw binaries, add version to the user-agent string
William Roberts <williamr@symbian.org>
parents: 170
diff changeset
   551
	nowinscw=False, 
198
90e6dc719131 version 0.15 - add --noarmv5 option to avoid armv5 binaries. using this with --nowinscw also avoids binaries_epoc.zip and binaries_epoc_sdk.zip.
Shabe Razvi <shaber@symbian.org>
parents: 197
diff changeset
   552
	noarmv5=False, 
160
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   553
	nounzip=False, 
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   554
	nodelete=False, 
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   555
	progress=False,
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   556
	username='',
170
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
   557
	password='',
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   558
	webhost = 'developer.symbian.org',
274
3b8bce67b574 version 0.17 - use the size information in release_metadata.xml, treat checksum errors as failures
William Roberts <williamr@symbian.org>
parents: 199
diff changeset
   559
	resume=True,
170
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
   560
	debug=False
160
fed3f1d2c557 version 0.8 - added command line options for username and password. Revised the exception handling
William Roberts <williamr@symbian.org>
parents: 156
diff changeset
   561
	)
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   562
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   563
(options, args) = parser.parse_args()
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   564
if len(args) != 1:
199
a99c74c53f62 v0.16 - generate correct URL for downloading a specific version, and enable "Latest Symbian^X" functionality.
William Roberts <williamr@symbian.org>
parents: 198
diff changeset
   565
	parser.error("Must supply a PDK version, e.g. 3 or 3.0.h")
153
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   566
if not check_unzip_environment() :
77c25e4f2c6f version 0.6 - avoid problem if Content-Length not supplied, add --nodelete option, check 7z works before starting
William Roberts <williamr@symbian.org>
parents: 141
diff changeset
   567
	parser.error("Unable to execute 7z command")
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   568
193
f74ba2451a44 version 0.12 - add checking for click-through licenses and make sure we get the version we asked for!
William Roberts <williamr@symbian.org>
parents: 174
diff changeset
   569
top_level_url = "https://" + options.webhost
170
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
   570
opener = build_opener(options.debug)
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
   571
urllib2.install_opener(opener)
dd0383270574 version 0.9 - added --debug option to enable urllib2 HTTP debugging
William Roberts <williamr@symbian.org>
parents: 160
diff changeset
   572
174
5ebd70511e4c v0.11 - added quick_networking_check(), to confirm that we can talk to https://developer.symbian.org
William Roberts <williamr@symbian.org>
parents: 171
diff changeset
   573
quick_networking_check()
124
b60a149520e7 Added downloadkit.py - script to download and unpack a PDK
William Roberts <williamr@symbian.org>
parents:
diff changeset
   574
login(True)
138
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   575
downloadkit(args[0])
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   576
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   577
if options.dryrun:
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   578
	print "# instructions for downloading kit " + args[0]
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   579
	for download in download_list:
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   580
		print download
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   581
	for command in unzip_list:
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   582
		print command
7b2d146ef884 version 0.3 - add command line processing, --dryrun option, --nosrc option
William Roberts <williamr@symbian.org>
parents: 128
diff changeset
   583