1 '''"Executable documentation" for the pickle module.

     2 

     3 Extensive comments about the pickle protocols and pickle-machine opcodes

     4 can be found here.  Some functions meant for external use:

     5 

     6 genops(pickle)

     7    Generate all the opcodes in a pickle, as (opcode, arg, position) triples.

     8 

     9 dis(pickle, out=None, memo=None, indentlevel=4)

    10    Print a symbolic disassembly of a pickle.

    11 '''

    12 

    13 __all__ = ['dis', 'genops', 'optimize']

    14 

    15 # Other ideas:

    16 #

    17 # - A pickle verifier:  read a pickle and check it exhaustively for

    18 #   well-formedness.  dis() does a lot of this already.

    19 #

    20 # - A protocol identifier:  examine a pickle and return its protocol number

    21 #   (== the highest .proto attr value among all the opcodes in the pickle).

    22 #   dis() already prints this info at the end.

    23 #

    24 # - A pickle optimizer:  for example, tuple-building code is sometimes more

    25 #   elaborate than necessary, catering for the possibility that the tuple

    26 #   is recursive.  Or lots of times a PUT is generated that's never accessed

    27 #   by a later GET.

    28 

    29 

    30 """

    31 "A pickle" is a program for a virtual pickle machine (PM, but more accurately

    32 called an unpickling machine).  It's a sequence of opcodes, interpreted by the

    33 PM, building an arbitrarily complex Python object.

    34 

    35 For the most part, the PM is very simple:  there are no looping, testing, or

    36 conditional instructions, no arithmetic and no function calls.  Opcodes are

    37 executed once each, from first to last, until a STOP opcode is reached.

    38 

    39 The PM has two data areas, "the stack" and "the memo".

    40 

    41 Many opcodes push Python objects onto the stack; e.g., INT pushes a Python

    42 integer object on the stack, whose value is gotten from a decimal string

    43 literal immediately following the INT opcode in the pickle bytestream.  Other

    44 opcodes take Python objects off the stack.  The result of unpickling is

    45 whatever object is left on the stack when the final STOP opcode is executed.

    46 

    47 The memo is simply an array of objects, or it can be implemented as a dict

    48 mapping little integers to objects.  The memo serves as the PM's "long term

    49 memory", and the little integers indexing the memo are akin to variable

    50 names.  Some opcodes pop a stack object into the memo at a given index,

    51 and others push a memo object at a given index onto the stack again.

    52 

    53 At heart, that's all the PM has.  Subtleties arise for these reasons:

    54 

    55 + Object identity.  Objects can be arbitrarily complex, and subobjects

    56   may be shared (for example, the list [a, a] refers to the same object a

    57   twice).  It can be vital that unpickling recreate an isomorphic object

    58   graph, faithfully reproducing sharing.

    59 

    60 + Recursive objects.  For example, after "L = []; L.append(L)", L is a

    61   list, and L[0] is the same list.  This is related to the object identity

    62   point, and some sequences of pickle opcodes are subtle in order to

    63   get the right result in all cases.

    64 

    65 + Things pickle doesn't know everything about.  Examples of things pickle

    66   does know everything about are Python's builtin scalar and container

    67   types, like ints and tuples.  They generally have opcodes dedicated to

    68   them.  For things like module references and instances of user-defined

    69   classes, pickle's knowledge is limited.  Historically, many enhancements

    70   have been made to the pickle protocol in order to do a better (faster,

    71   and/or more compact) job on those.

    72 

    73 + Backward compatibility and micro-optimization.  As explained below,

    74   pickle opcodes never go away, not even when better ways to do a thing

    75   get invented.  The repertoire of the PM just keeps growing over time.

    76   For example, protocol 0 had two opcodes for building Python integers (INT

    77   and LONG), protocol 1 added three more for more-efficient pickling of short

    78   integers, and protocol 2 added two more for more-efficient pickling of

    79   long integers (before protocol 2, the only ways to pickle a Python long

    80   took time quadratic in the number of digits, for both pickling and

    81   unpickling).  "Opcode bloat" isn't so much a subtlety as a source of

    82   wearying complication.

    83 

    84 

    85 Pickle protocols:

    86 

    87 For compatibility, the meaning of a pickle opcode never changes.  Instead new

    88 pickle opcodes get added, and each version's unpickler can handle all the

    89 pickle opcodes in all protocol versions to date.  So old pickles continue to

    90 be readable forever.  The pickler can generally be told to restrict itself to

    91 the subset of opcodes available under previous protocol versions too, so that

    92 users can create pickles under the current version readable by older

    93 versions.  However, a pickle does not contain its version number embedded

    94 within it.  If an older unpickler tries to read a pickle using a later

    95 protocol, the result is most likely an exception due to seeing an unknown (in

    96 the older unpickler) opcode.

    97 

    98 The original pickle used what's now called "protocol 0", and what was called

    99 "text mode" before Python 2.3.  The entire pickle bytestream is made up of

   100 printable 7-bit ASCII characters, plus the newline character, in protocol 0.

   101 That's why it was called text mode.  Protocol 0 is small and elegant, but

   102 sometimes painfully inefficient.

   103 

   104 The second major set of additions is now called "protocol 1", and was called

   105 "binary mode" before Python 2.3.  This added many opcodes with arguments

   106 consisting of arbitrary bytes, including NUL bytes and unprintable "high bit"

   107 bytes.  Binary mode pickles can be substantially smaller than equivalent

   108 text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte

   109 int as 4 bytes following the opcode, which is cheaper to unpickle than the

   110 (perhaps) 11-character decimal string attached to INT.  Protocol 1 also added

   111 a number of opcodes that operate on many stack elements at once (like APPENDS

   112 and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE).

   113 

   114 The third major set of additions came in Python 2.3, and is called "protocol

   115 2".  This added:

   116 

   117 - A better way to pickle instances of new-style classes (NEWOBJ).

   118 

   119 - A way for a pickle to identify its protocol (PROTO).

   120 

   121 - Time- and space- efficient pickling of long ints (LONG{1,4}).

   122 

   123 - Shortcuts for small tuples (TUPLE{1,2,3}}.

   124 

   125 - Dedicated opcodes for bools (NEWTRUE, NEWFALSE).

   126 

   127 - The "extension registry", a vector of popular objects that can be pushed

   128   efficiently by index (EXT{1,2,4}).  This is akin to the memo and GET, but

   129   the registry contents are predefined (there's nothing akin to the memo's

   130   PUT).

   131 

   132 Another independent change with Python 2.3 is the abandonment of any

   133 pretense that it might be safe to load pickles received from untrusted

   134 parties -- no sufficient security analysis has been done to guarantee

   135 this and there isn't a use case that warrants the expense of such an

   136 analysis.

   137 

   138 To this end, all tests for __safe_for_unpickling__ or for

   139 copy_reg.safe_constructors are removed from the unpickling code.

   140 References to these variables in the descriptions below are to be seen

   141 as describing unpickling in Python 2.2 and before.

   142 """

   143 

   144 # Meta-rule:  Descriptions are stored in instances of descriptor objects,

   145 # with plain constructors.  No meta-language is defined from which

   146 # descriptors could be constructed.  If you want, e.g., XML, write a little

   147 # program to generate XML from the objects.

   148 

   149 ##############################################################################

   150 # Some pickle opcodes have an argument, following the opcode in the

   151 # bytestream.  An argument is of a specific type, described by an instance

   152 # of ArgumentDescriptor.  These are not to be confused with arguments taken

   153 # off the stack -- ArgumentDescriptor applies only to arguments embedded in

   154 # the opcode stream, immediately following an opcode.

   155 

   156 # Represents the number of bytes consumed by an argument delimited by the

   157 # next newline character.

   158 UP_TO_NEWLINE = -1

   159 

   160 # Represents the number of bytes consumed by a two-argument opcode where

   161 # the first argument gives the number of bytes in the second argument.

   162 TAKEN_FROM_ARGUMENT1 = -2   # num bytes is 1-byte unsigned int

   163 TAKEN_FROM_ARGUMENT4 = -3   # num bytes is 4-byte signed little-endian int

   164 

   165 class ArgumentDescriptor(object):

   166     __slots__ = (

   167         # name of descriptor record, also a module global name; a string

   168         'name',

   169 

   170         # length of argument, in bytes; an int; UP_TO_NEWLINE and

   171         # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length

   172         # cases

   173         'n',

   174 

   175         # a function taking a file-like object, reading this kind of argument

   176         # from the object at the current position, advancing the current

   177         # position by n bytes, and returning the value of the argument

   178         'reader',

   179 

   180         # human-readable docs for this arg descriptor; a string

   181         'doc',

   182     )

   183 

   184     def __init__(self, name, n, reader, doc):

   185         assert isinstance(name, str)

   186         self.name = name

   187 

   188         assert isinstance(n, int) and (n >= 0 or

   189                                        n in (UP_TO_NEWLINE,

   190                                              TAKEN_FROM_ARGUMENT1,

   191                                              TAKEN_FROM_ARGUMENT4))

   192         self.n = n

   193 

   194         self.reader = reader

   195 

   196         assert isinstance(doc, str)

   197         self.doc = doc

   198 

   199 from struct import unpack as _unpack

   200 

   201 def read_uint1(f):

   202     r"""

   203     >>> import StringIO

   204     >>> read_uint1(StringIO.StringIO('\xff'))

   205     255

   206     """

   207 

   208     data = f.read(1)

   209     if data:

   210         return ord(data)

   211     raise ValueError("not enough data in stream to read uint1")

   212 

   213 uint1 = ArgumentDescriptor(

   214             name='uint1',

   215             n=1,

   216             reader=read_uint1,

   217             doc="One-byte unsigned integer.")

   218 

   219 

   220 def read_uint2(f):

   221     r"""

   222     >>> import StringIO

   223     >>> read_uint2(StringIO.StringIO('\xff\x00'))

   224     255

   225     >>> read_uint2(StringIO.StringIO('\xff\xff'))

   226     65535

   227     """

   228 

   229     data = f.read(2)

   230     if len(data) == 2:

   231         return _unpack("<H", data)[0]

   232     raise ValueError("not enough data in stream to read uint2")

   233 

   234 uint2 = ArgumentDescriptor(

   235             name='uint2',

   236             n=2,

   237             reader=read_uint2,

   238             doc="Two-byte unsigned integer, little-endian.")

   239 

   240 

   241 def read_int4(f):

   242     r"""

   243     >>> import StringIO

   244     >>> read_int4(StringIO.StringIO('\xff\x00\x00\x00'))

   245     255

   246     >>> read_int4(StringIO.StringIO('\x00\x00\x00\x80')) == -(2**31)

   247     True

   248     """

   249 

   250     data = f.read(4)

   251     if len(data) == 4:

   252         return _unpack("<i", data)[0]

   253     raise ValueError("not enough data in stream to read int4")

   254 

   255 int4 = ArgumentDescriptor(

   256            name='int4',

   257            n=4,

   258            reader=read_int4,

   259            doc="Four-byte signed integer, little-endian, 2's complement.")

   260 

   261 

   262 def read_stringnl(f, decode=True, stripquotes=True):

   263     r"""

   264     >>> import StringIO

   265     >>> read_stringnl(StringIO.StringIO("'abcd'\nefg\n"))

   266     'abcd'

   267 

   268     >>> read_stringnl(StringIO.StringIO("\n"))

   269     Traceback (most recent call last):

   270     ...

   271     ValueError: no string quotes around ''

   272 

   273     >>> read_stringnl(StringIO.StringIO("\n"), stripquotes=False)

   274     ''

   275 

   276     >>> read_stringnl(StringIO.StringIO("''\n"))

   277     ''

   278 

   279     >>> read_stringnl(StringIO.StringIO('"abcd"'))

   280     Traceback (most recent call last):

   281     ...

   282     ValueError: no newline found when trying to read stringnl

   283 

   284     Embedded escapes are undone in the result.

   285     >>> read_stringnl(StringIO.StringIO(r"'a\n\\b\x00c\td'" + "\n'e'"))

   286     'a\n\\b\x00c\td'

   287     """

   288 

   289     data = f.readline()

   290     if not data.endswith('\n'):

   291         raise ValueError("no newline found when trying to read stringnl")

   292     data = data[:-1]    # lose the newline

   293 

   294     if stripquotes:

   295         for q in "'\"":

   296             if data.startswith(q):

   297                 if not data.endswith(q):

   298                     raise ValueError("strinq quote %r not found at both "

   299                                      "ends of %r" % (q, data))

   300                 data = data[1:-1]

   301                 break

   302         else:

   303             raise ValueError("no string quotes around %r" % data)

   304 

   305     # I'm not sure when 'string_escape' was added to the std codecs; it's

   306     # crazy not to use it if it's there.

   307     if decode:

   308         data = data.decode('string_escape')

   309     return data

   310 

   311 stringnl = ArgumentDescriptor(

   312                name='stringnl',

   313                n=UP_TO_NEWLINE,

   314                reader=read_stringnl,

   315                doc="""A newline-terminated string.

   316 

   317                    This is a repr-style string, with embedded escapes, and

   318                    bracketing quotes.

   319                    """)

   320 

   321 def read_stringnl_noescape(f):

   322     return read_stringnl(f, decode=False, stripquotes=False)

   323 

   324 stringnl_noescape = ArgumentDescriptor(

   325                         name='stringnl_noescape',

   326                         n=UP_TO_NEWLINE,

   327                         reader=read_stringnl_noescape,

   328                         doc="""A newline-terminated string.

   329 

   330                         This is a str-style string, without embedded escapes,

   331                         or bracketing quotes.  It should consist solely of

   332                         printable ASCII characters.

   333                         """)

   334 

   335 def read_stringnl_noescape_pair(f):

   336     r"""

   337     >>> import StringIO

   338     >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\nEmpty\njunk"))

   339     'Queue Empty'

   340     """

   341 

   342     return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f))

   343 

   344 stringnl_noescape_pair = ArgumentDescriptor(

   345                              name='stringnl_noescape_pair',

   346                              n=UP_TO_NEWLINE,

   347                              reader=read_stringnl_noescape_pair,

   348                              doc="""A pair of newline-terminated strings.

   349 

   350                              These are str-style strings, without embedded

   351                              escapes, or bracketing quotes.  They should

   352                              consist solely of printable ASCII characters.

   353                              The pair is returned as a single string, with

   354                              a single blank separating the two strings.

   355                              """)

   356 

   357 def read_string4(f):

   358     r"""

   359     >>> import StringIO

   360     >>> read_string4(StringIO.StringIO("\x00\x00\x00\x00abc"))

   361     ''

   362     >>> read_string4(StringIO.StringIO("\x03\x00\x00\x00abcdef"))

   363     'abc'

   364     >>> read_string4(StringIO.StringIO("\x00\x00\x00\x03abcdef"))

   365     Traceback (most recent call last):

   366     ...

   367     ValueError: expected 50331648 bytes in a string4, but only 6 remain

   368     """

   369 

   370     n = read_int4(f)

   371     if n < 0:

   372         raise ValueError("string4 byte count < 0: %d" % n)

   373     data = f.read(n)

   374     if len(data) == n:

   375         return data

   376     raise ValueError("expected %d bytes in a string4, but only %d remain" %

   377                      (n, len(data)))

   378 

   379 string4 = ArgumentDescriptor(

   380               name="string4",

   381               n=TAKEN_FROM_ARGUMENT4,

   382               reader=read_string4,

   383               doc="""A counted string.

   384 

   385               The first argument is a 4-byte little-endian signed int giving

   386               the number of bytes in the string, and the second argument is

   387               that many bytes.

   388               """)

   389 

   390 

   391 def read_string1(f):

   392     r"""

   393     >>> import StringIO

   394     >>> read_string1(StringIO.StringIO("\x00"))

   395     ''

   396     >>> read_string1(StringIO.StringIO("\x03abcdef"))

   397     'abc'

   398     """

   399 

   400     n = read_uint1(f)

   401     assert n >= 0

   402     data = f.read(n)

   403     if len(data) == n:

   404         return data

   405     raise ValueError("expected %d bytes in a string1, but only %d remain" %

   406                      (n, len(data)))

   407 

   408 string1 = ArgumentDescriptor(

   409               name="string1",

   410               n=TAKEN_FROM_ARGUMENT1,

   411               reader=read_string1,

   412               doc="""A counted string.

   413 

   414               The first argument is a 1-byte unsigned int giving the number

   415               of bytes in the string, and the second argument is that many

   416               bytes.

   417               """)

   418 

   419 

   420 def read_unicodestringnl(f):

   421     r"""

   422     >>> import StringIO

   423     >>> read_unicodestringnl(StringIO.StringIO("abc\uabcd\njunk"))

   424     u'abc\uabcd'

   425     """

   426 

   427     data = f.readline()

   428     if not data.endswith('\n'):

   429         raise ValueError("no newline found when trying to read "

   430                          "unicodestringnl")

   431     data = data[:-1]    # lose the newline

   432     return unicode(data, 'raw-unicode-escape')

   433 

   434 unicodestringnl = ArgumentDescriptor(

   435                       name='unicodestringnl',

   436                       n=UP_TO_NEWLINE,

   437                       reader=read_unicodestringnl,

   438                       doc="""A newline-terminated Unicode string.

   439 

   440                       This is raw-unicode-escape encoded, so consists of

   441                       printable ASCII characters, and may contain embedded

   442                       escape sequences.

   443                       """)

   444 

   445 def read_unicodestring4(f):

   446     r"""

   447     >>> import StringIO

   448     >>> s = u'abcd\uabcd'

   449     >>> enc = s.encode('utf-8')

   450     >>> enc

   451     'abcd\xea\xaf\x8d'

   452     >>> n = chr(len(enc)) + chr(0) * 3  # little-endian 4-byte length

   453     >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk'))

   454     >>> s == t

   455     True

   456 

   457     >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1]))

   458     Traceback (most recent call last):

   459     ...

   460     ValueError: expected 7 bytes in a unicodestring4, but only 6 remain

   461     """

   462 

   463     n = read_int4(f)

   464     if n < 0:

   465         raise ValueError("unicodestring4 byte count < 0: %d" % n)

   466     data = f.read(n)

   467     if len(data) == n:

   468         return unicode(data, 'utf-8')

   469     raise ValueError("expected %d bytes in a unicodestring4, but only %d "

   470                      "remain" % (n, len(data)))

   471 

   472 unicodestring4 = ArgumentDescriptor(

   473                     name="unicodestring4",

   474                     n=TAKEN_FROM_ARGUMENT4,

   475                     reader=read_unicodestring4,

   476                     doc="""A counted Unicode string.

   477 

   478                     The first argument is a 4-byte little-endian signed int

   479                     giving the number of bytes in the string, and the second

   480                     argument-- the UTF-8 encoding of the Unicode string --

   481                     contains that many bytes.

   482                     """)

   483 

   484 

   485 def read_decimalnl_short(f):

   486     r"""

   487     >>> import StringIO

   488     >>> read_decimalnl_short(StringIO.StringIO("1234\n56"))

   489     1234

   490 

   491     >>> read_decimalnl_short(StringIO.StringIO("1234L\n56"))

   492     Traceback (most recent call last):

   493     ...

   494     ValueError: trailing 'L' not allowed in '1234L'

   495     """

   496 

   497     s = read_stringnl(f, decode=False, stripquotes=False)

   498     if s.endswith("L"):

   499         raise ValueError("trailing 'L' not allowed in %r" % s)

   500 

   501     # It's not necessarily true that the result fits in a Python short int:

   502     # the pickle may have been written on a 64-bit box.  There's also a hack

   503     # for True and False here.

   504     if s == "00":

   505         return False

   506     elif s == "01":

   507         return True

   508 

   509     try:

   510         return int(s)

   511     except OverflowError:

   512         return long(s)

   513 

   514 def read_decimalnl_long(f):

   515     r"""

   516     >>> import StringIO

   517 

   518     >>> read_decimalnl_long(StringIO.StringIO("1234\n56"))

   519     Traceback (most recent call last):

   520     ...

   521     ValueError: trailing 'L' required in '1234'

   522 

   523     Someday the trailing 'L' will probably go away from this output.

   524 

   525     >>> read_decimalnl_long(StringIO.StringIO("1234L\n56"))

   526     1234L

   527 

   528     >>> read_decimalnl_long(StringIO.StringIO("123456789012345678901234L\n6"))

   529     123456789012345678901234L

   530     """

   531 

   532     s = read_stringnl(f, decode=False, stripquotes=False)

   533     if not s.endswith("L"):

   534         raise ValueError("trailing 'L' required in %r" % s)

   535     return long(s)

   536 

   537 

   538 decimalnl_short = ArgumentDescriptor(

   539                       name='decimalnl_short',

   540                       n=UP_TO_NEWLINE,

   541                       reader=read_decimalnl_short,

   542                       doc="""A newline-terminated decimal integer literal.

   543 

   544                           This never has a trailing 'L', and the integer fit

   545                           in a short Python int on the box where the pickle

   546                           was written -- but there's no guarantee it will fit

   547                           in a short Python int on the box where the pickle

   548                           is read.

   549                           """)

   550 

   551 decimalnl_long = ArgumentDescriptor(

   552                      name='decimalnl_long',

   553                      n=UP_TO_NEWLINE,

   554                      reader=read_decimalnl_long,

   555                      doc="""A newline-terminated decimal integer literal.

   556 

   557                          This has a trailing 'L', and can represent integers

   558                          of any size.

   559                          """)

   560 

   561 

   562 def read_floatnl(f):

   563     r"""

   564     >>> import StringIO

   565     >>> read_floatnl(StringIO.StringIO("-1.25\n6"))

   566     -1.25

   567     """

   568     s = read_stringnl(f, decode=False, stripquotes=False)

   569     return float(s)

   570 

   571 floatnl = ArgumentDescriptor(

   572               name='floatnl',

   573               n=UP_TO_NEWLINE,

   574               reader=read_floatnl,

   575               doc="""A newline-terminated decimal floating literal.

   576 

   577               In general this requires 17 significant digits for roundtrip

   578               identity, and pickling then unpickling infinities, NaNs, and

   579               minus zero doesn't work across boxes, or on some boxes even

   580               on itself (e.g., Windows can't read the strings it produces

   581               for infinities or NaNs).

   582               """)

   583 

   584 def read_float8(f):

   585     r"""

   586     >>> import StringIO, struct

   587     >>> raw = struct.pack(">d", -1.25)

   588     >>> raw

   589     '\xbf\xf4\x00\x00\x00\x00\x00\x00'

   590     >>> read_float8(StringIO.StringIO(raw + "\n"))

   591     -1.25

   592     """

   593 

   594     data = f.read(8)

   595     if len(data) == 8:

   596         return _unpack(">d", data)[0]

   597     raise ValueError("not enough data in stream to read float8")

   598 

   599 

   600 float8 = ArgumentDescriptor(

   601              name='float8',

   602              n=8,

   603              reader=read_float8,

   604              doc="""An 8-byte binary representation of a float, big-endian.

   605 

   606              The format is unique to Python, and shared with the struct

   607              module (format string '>d') "in theory" (the struct and cPickle

   608              implementations don't share the code -- they should).  It's

   609              strongly related to the IEEE-754 double format, and, in normal

   610              cases, is in fact identical to the big-endian 754 double format.

   611              On other boxes the dynamic range is limited to that of a 754

   612              double, and "add a half and chop" rounding is used to reduce

   613              the precision to 53 bits.  However, even on a 754 box,

   614              infinities, NaNs, and minus zero may not be handled correctly

   615              (may not survive roundtrip pickling intact).

   616              """)

   617 

   618 # Protocol 2 formats

   619 

   620 from pickle import decode_long

   621 

   622 def read_long1(f):

   623     r"""

   624     >>> import StringIO

   625     >>> read_long1(StringIO.StringIO("\x00"))

   626     0L

   627     >>> read_long1(StringIO.StringIO("\x02\xff\x00"))

   628     255L

   629     >>> read_long1(StringIO.StringIO("\x02\xff\x7f"))

   630     32767L

   631     >>> read_long1(StringIO.StringIO("\x02\x00\xff"))

   632     -256L

   633     >>> read_long1(StringIO.StringIO("\x02\x00\x80"))

   634     -32768L

   635     """

   636 

   637     n = read_uint1(f)

   638     data = f.read(n)

   639     if len(data) != n:

   640         raise ValueError("not enough data in stream to read long1")

   641     return decode_long(data)

   642 

   643 long1 = ArgumentDescriptor(

   644     name="long1",

   645     n=TAKEN_FROM_ARGUMENT1,

   646     reader=read_long1,

   647     doc="""A binary long, little-endian, using 1-byte size.

   648 

   649     This first reads one byte as an unsigned size, then reads that

   650     many bytes and interprets them as a little-endian 2's-complement long.

   651     If the size is 0, that's taken as a shortcut for the long 0L.

   652     """)

   653 

   654 def read_long4(f):

   655     r"""

   656     >>> import StringIO

   657     >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x00"))

   658     255L

   659     >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x7f"))

   660     32767L

   661     >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\xff"))

   662     -256L

   663     >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\x80"))

   664     -32768L

   665     >>> read_long1(StringIO.StringIO("\x00\x00\x00\x00"))

   666     0L

   667     """

   668 

   669     n = read_int4(f)

   670     if n < 0:

   671         raise ValueError("long4 byte count < 0: %d" % n)

   672     data = f.read(n)

   673     if len(data) != n:

   674         raise ValueError("not enough data in stream to read long4")

   675     return decode_long(data)

   676 

   677 long4 = ArgumentDescriptor(

   678     name="long4",

   679     n=TAKEN_FROM_ARGUMENT4,

   680     reader=read_long4,

   681     doc="""A binary representation of a long, little-endian.

   682 

   683     This first reads four bytes as a signed size (but requires the

   684     size to be >= 0), then reads that many bytes and interprets them

   685     as a little-endian 2's-complement long.  If the size is 0, that's taken

   686     as a shortcut for the long 0L, although LONG1 should really be used

   687     then instead (and in any case where # of bytes < 256).

   688     """)

   689 

   690 

   691 ##############################################################################

   692 # Object descriptors.  The stack used by the pickle machine holds objects,

   693 # and in the stack_before and stack_after attributes of OpcodeInfo

   694 # descriptors we need names to describe the various types of objects that can

   695 # appear on the stack.

   696 

   697 class StackObject(object):

   698     __slots__ = (

   699         # name of descriptor record, for info only

   700         'name',

   701 

   702         # type of object, or tuple of type objects (meaning the object can

   703         # be of any type in the tuple)

   704         'obtype',

   705 

   706         # human-readable docs for this kind of stack object; a string

   707         'doc',

   708     )

   709 

   710     def __init__(self, name, obtype, doc):

   711         assert isinstance(name, str)

   712         self.name = name

   713 

   714         assert isinstance(obtype, type) or isinstance(obtype, tuple)

   715         if isinstance(obtype, tuple):

   716             for contained in obtype:

   717                 assert isinstance(contained, type)

   718         self.obtype = obtype

   719 

   720         assert isinstance(doc, str)

   721         self.doc = doc

   722 

   723     def __repr__(self):

   724         return self.name

   725 

   726 

   727 pyint = StackObject(

   728             name='int',

   729             obtype=int,

   730             doc="A short (as opposed to long) Python integer object.")

   731 

   732 pylong = StackObject(

   733              name='long',

   734              obtype=long,

   735              doc="A long (as opposed to short) Python integer object.")

   736 

   737 pyinteger_or_bool = StackObject(

   738                         name='int_or_bool',

   739                         obtype=(int, long, bool),

   740                         doc="A Python integer object (short or long), or "

   741                             "a Python bool.")

   742 

   743 pybool = StackObject(

   744              name='bool',

   745              obtype=(bool,),

   746              doc="A Python bool object.")

   747 

   748 pyfloat = StackObject(

   749               name='float',

   750               obtype=float,

   751               doc="A Python float object.")

   752 

   753 pystring = StackObject(

   754                name='str',

   755                obtype=str,

   756                doc="A Python string object.")

   757 

   758 pyunicode = StackObject(

   759                 name='unicode',

   760                 obtype=unicode,

   761                 doc="A Python Unicode string object.")

   762 

   763 pynone = StackObject(

   764              name="None",

   765              obtype=type(None),

   766              doc="The Python None object.")

   767 

   768 pytuple = StackObject(

   769               name="tuple",

   770               obtype=tuple,

   771               doc="A Python tuple object.")

   772 

   773 pylist = StackObject(

   774              name="list",

   775              obtype=list,

   776              doc="A Python list object.")

   777 

   778 pydict = StackObject(

   779              name="dict",

   780              obtype=dict,

   781              doc="A Python dict object.")

   782 

   783 anyobject = StackObject(

   784                 name='any',

   785                 obtype=object,

   786                 doc="Any kind of object whatsoever.")

   787 

   788 markobject = StackObject(

   789                  name="mark",

   790                  obtype=StackObject,

   791                  doc="""'The mark' is a unique object.

   792 

   793                  Opcodes that operate on a variable number of objects

   794                  generally don't embed the count of objects in the opcode,

   795                  or pull it off the stack.  Instead the MARK opcode is used

   796                  to push a special marker object on the stack, and then

   797                  some other opcodes grab all the objects from the top of

   798                  the stack down to (but not including) the topmost marker

   799                  object.

   800                  """)

   801 

   802 stackslice = StackObject(

   803                  name="stackslice",

   804                  obtype=StackObject,

   805                  doc="""An object representing a contiguous slice of the stack.

   806 

   807                  This is used in conjuction with markobject, to represent all

   808                  of the stack following the topmost markobject.  For example,

   809                  the POP_MARK opcode changes the stack from

   810 

   811                      [..., markobject, stackslice]

   812                  to

   813                      [...]

   814 

   815                  No matter how many object are on the stack after the topmost

   816                  markobject, POP_MARK gets rid of all of them (including the

   817                  topmost markobject too).

   818                  """)

   819 

   820 ##############################################################################

   821 # Descriptors for pickle opcodes.

   822 

   823 class OpcodeInfo(object):

   824 

   825     __slots__ = (

   826         # symbolic name of opcode; a string

   827         'name',

   828 

   829         # the code used in a bytestream to represent the opcode; a

   830         # one-character string

   831         'code',

   832 

   833         # If the opcode has an argument embedded in the byte string, an

   834         # instance of ArgumentDescriptor specifying its type.  Note that

   835         # arg.reader(s) can be used to read and decode the argument from

   836         # the bytestream s, and arg.doc documents the format of the raw

   837         # argument bytes.  If the opcode doesn't have an argument embedded

   838         # in the bytestream, arg should be None.

   839         'arg',

   840 

   841         # what the stack looks like before this opcode runs; a list

   842         'stack_before',

   843 

   844         # what the stack looks like after this opcode runs; a list

   845         'stack_after',

   846 

   847         # the protocol number in which this opcode was introduced; an int

   848         'proto',

   849 

   850         # human-readable docs for this opcode; a string

   851         'doc',

   852     )

   853 

   854     def __init__(self, name, code, arg,

   855                  stack_before, stack_after, proto, doc):

   856         assert isinstance(name, str)

   857         self.name = name

   858 

   859         assert isinstance(code, str)

   860         assert len(code) == 1

   861         self.code = code

   862 

   863         assert arg is None or isinstance(arg, ArgumentDescriptor)

   864         self.arg = arg

   865 

   866         assert isinstance(stack_before, list)

   867         for x in stack_before:

   868             assert isinstance(x, StackObject)

   869         self.stack_before = stack_before

   870 

   871         assert isinstance(stack_after, list)

   872         for x in stack_after:

   873             assert isinstance(x, StackObject)

   874         self.stack_after = stack_after

   875 

   876         assert isinstance(proto, int) and 0 <= proto <= 2

   877         self.proto = proto

   878 

   879         assert isinstance(doc, str)

   880         self.doc = doc

   881 

   882 I = OpcodeInfo

   883 opcodes = [

   884 

   885     # Ways to spell integers.

   886 

   887     I(name='INT',

   888       code='I',

   889       arg=decimalnl_short,

   890       stack_before=[],

   891       stack_after=[pyinteger_or_bool],

   892       proto=0,

   893       doc="""Push an integer or bool.

   894 

   895       The argument is a newline-terminated decimal literal string.

   896 

   897       The intent may have been that this always fit in a short Python int,

   898       but INT can be generated in pickles written on a 64-bit box that

   899       require a Python long on a 32-bit box.  The difference between this

   900       and LONG then is that INT skips a trailing 'L', and produces a short

   901       int whenever possible.

   902 

   903       Another difference is due to that, when bool was introduced as a

   904       distinct type in 2.3, builtin names True and False were also added to

   905       2.2.2, mapping to ints 1 and 0.  For compatibility in both directions,

   906       True gets pickled as INT + "I01\\n", and False as INT + "I00\\n".

   907       Leading zeroes are never produced for a genuine integer.  The 2.3

   908       (and later) unpicklers special-case these and return bool instead;

   909       earlier unpicklers ignore the leading "0" and return the int.

   910       """),

   911 

   912     I(name='BININT',

   913       code='J',

   914       arg=int4,

   915       stack_before=[],

   916       stack_after=[pyint],

   917       proto=1,

   918       doc="""Push a four-byte signed integer.

   919 

   920       This handles the full range of Python (short) integers on a 32-bit

   921       box, directly as binary bytes (1 for the opcode and 4 for the integer).

   922       If the integer is non-negative and fits in 1 or 2 bytes, pickling via

   923       BININT1 or BININT2 saves space.

   924       """),

   925 

   926     I(name='BININT1',

   927       code='K',

   928       arg=uint1,

   929       stack_before=[],

   930       stack_after=[pyint],

   931       proto=1,

   932       doc="""Push a one-byte unsigned integer.

   933 

   934       This is a space optimization for pickling very small non-negative ints,

   935       in range(256).

   936       """),

   937 

   938     I(name='BININT2',

   939       code='M',

   940       arg=uint2,

   941       stack_before=[],

   942       stack_after=[pyint],

   943       proto=1,

   944       doc="""Push a two-byte unsigned integer.

   945 

   946       This is a space optimization for pickling small positive ints, in

   947       range(256, 2**16).  Integers in range(256) can also be pickled via

   948       BININT2, but BININT1 instead saves a byte.

   949       """),

   950 

   951     I(name='LONG',

   952       code='L',

   953       arg=decimalnl_long,

   954       stack_before=[],

   955       stack_after=[pylong],

   956       proto=0,

   957       doc="""Push a long integer.

   958 

   959       The same as INT, except that the literal ends with 'L', and always

   960       unpickles to a Python long.  There doesn't seem a real purpose to the

   961       trailing 'L'.

   962 

   963       Note that LONG takes time quadratic in the number of digits when

   964       unpickling (this is simply due to the nature of decimal->binary

   965       conversion).  Proto 2 added linear-time (in C; still quadratic-time

   966       in Python) LONG1 and LONG4 opcodes.

   967       """),

   968 

   969     I(name="LONG1",

   970       code='\x8a',

   971       arg=long1,

   972       stack_before=[],

   973       stack_after=[pylong],

   974       proto=2,

   975       doc="""Long integer using one-byte length.

   976 

   977       A more efficient encoding of a Python long; the long1 encoding

   978       says it all."""),

   979 

   980     I(name="LONG4",

   981       code='\x8b',

   982       arg=long4,

   983       stack_before=[],

   984       stack_after=[pylong],

   985       proto=2,

   986       doc="""Long integer using found-byte length.

   987 

   988       A more efficient encoding of a Python long; the long4 encoding

   989       says it all."""),

   990 

   991     # Ways to spell strings (8-bit, not Unicode).

   992 

   993     I(name='STRING',

   994       code='S',

   995       arg=stringnl,

   996       stack_before=[],

   997       stack_after=[pystring],

   998       proto=0,

   999       doc="""Push a Python string object.

  1000 

  1001       The argument is a repr-style string, with bracketing quote characters,

  1002       and perhaps embedded escapes.  The argument extends until the next

  1003       newline character.

  1004       """),

  1005 

  1006     I(name='BINSTRING',

  1007       code='T',

  1008       arg=string4,

  1009       stack_before=[],

  1010       stack_after=[pystring],

  1011       proto=1,

  1012       doc="""Push a Python string object.

  1013 

  1014       There are two arguments:  the first is a 4-byte little-endian signed int

  1015       giving the number of bytes in the string, and the second is that many

  1016       bytes, which are taken literally as the string content.

  1017       """),

  1018 

  1019     I(name='SHORT_BINSTRING',

  1020       code='U',

  1021       arg=string1,

  1022       stack_before=[],

  1023       stack_after=[pystring],

  1024       proto=1,

  1025       doc="""Push a Python string object.

  1026 

  1027       There are two arguments:  the first is a 1-byte unsigned int giving

  1028       the number of bytes in the string, and the second is that many bytes,

  1029       which are taken literally as the string content.

  1030       """),

  1031 

  1032     # Ways to spell None.

  1033 

  1034     I(name='NONE',

  1035       code='N',

  1036       arg=None,

  1037       stack_before=[],

  1038       stack_after=[pynone],

  1039       proto=0,

  1040       doc="Push None on the stack."),

  1041 

  1042     # Ways to spell bools, starting with proto 2.  See INT for how this was

  1043     # done before proto 2.

  1044 

  1045     I(name='NEWTRUE',

  1046       code='\x88',

  1047       arg=None,

  1048       stack_before=[],

  1049       stack_after=[pybool],

  1050       proto=2,

  1051       doc="""True.

  1052 

  1053       Push True onto the stack."""),

  1054 

  1055     I(name='NEWFALSE',

  1056       code='\x89',

  1057       arg=None,

  1058       stack_before=[],

  1059       stack_after=[pybool],

  1060       proto=2,

  1061       doc="""True.

  1062 

  1063       Push False onto the stack."""),

  1064 

  1065     # Ways to spell Unicode strings.

  1066 

  1067     I(name='UNICODE',

  1068       code='V',

  1069       arg=unicodestringnl,

  1070       stack_before=[],

  1071       stack_after=[pyunicode],

  1072       proto=0,  # this may be pure-text, but it's a later addition

  1073       doc="""Push a Python Unicode string object.

  1074 

  1075       The argument is a raw-unicode-escape encoding of a Unicode string,

  1076       and so may contain embedded escape sequences.  The argument extends

  1077       until the next newline character.

  1078       """),

  1079 

  1080     I(name='BINUNICODE',

  1081       code='X',

  1082       arg=unicodestring4,

  1083       stack_before=[],

  1084       stack_after=[pyunicode],

  1085       proto=1,

  1086       doc="""Push a Python Unicode string object.

  1087 

  1088       There are two arguments:  the first is a 4-byte little-endian signed int

  1089       giving the number of bytes in the string.  The second is that many

  1090       bytes, and is the UTF-8 encoding of the Unicode string.

  1091       """),

  1092 

  1093     # Ways to spell floats.

  1094 

  1095     I(name='FLOAT',

  1096       code='F',

  1097       arg=floatnl,

  1098       stack_before=[],

  1099       stack_after=[pyfloat],

  1100       proto=0,

  1101       doc="""Newline-terminated decimal float literal.

  1102 

  1103       The argument is repr(a_float), and in general requires 17 significant

  1104       digits for roundtrip conversion to be an identity (this is so for

  1105       IEEE-754 double precision values, which is what Python float maps to

  1106       on most boxes).

  1107 

  1108       In general, FLOAT cannot be used to transport infinities, NaNs, or

  1109       minus zero across boxes (or even on a single box, if the platform C

  1110       library can't read the strings it produces for such things -- Windows

  1111       is like that), but may do less damage than BINFLOAT on boxes with

  1112       greater precision or dynamic range than IEEE-754 double.

  1113       """),

  1114 

  1115     I(name='BINFLOAT',

  1116       code='G',

  1117       arg=float8,

  1118       stack_before=[],

  1119       stack_after=[pyfloat],

  1120       proto=1,

  1121       doc="""Float stored in binary form, with 8 bytes of data.

  1122 

  1123       This generally requires less than half the space of FLOAT encoding.

  1124       In general, BINFLOAT cannot be used to transport infinities, NaNs, or

  1125       minus zero, raises an exception if the exponent exceeds the range of

  1126       an IEEE-754 double, and retains no more than 53 bits of precision (if

  1127       there are more than that, "add a half and chop" rounding is used to

  1128       cut it back to 53 significant bits).

  1129       """),

  1130 

  1131     # Ways to build lists.

  1132 

  1133     I(name='EMPTY_LIST',

  1134       code=']',

  1135       arg=None,