1 '''"Executable documentation" for the pickle module.

     2 

     3 Extensive comments about the pickle protocols and pickle-machine opcodes

     4 can be found here.  Some functions meant for external use:

     5 

     6 genops(pickle)

     7    Generate all the opcodes in a pickle, as (opcode, arg, position) triples.

     8 

     9 dis(pickle, out=None, memo=None, indentlevel=4)

    10    Print a symbolic disassembly of a pickle.

    11 '''

    12 

    13 __all__ = ['dis',

    14            'genops',

    15           ]

    16 

    17 # Other ideas:

    18 #

    19 # - A pickle verifier:  read a pickle and check it exhaustively for

    20 #   well-formedness.  dis() does a lot of this already.

    21 #

    22 # - A protocol identifier:  examine a pickle and return its protocol number

    23 #   (== the highest .proto attr value among all the opcodes in the pickle).

    24 #   dis() already prints this info at the end.

    25 #

    26 # - A pickle optimizer:  for example, tuple-building code is sometimes more

    27 #   elaborate than necessary, catering for the possibility that the tuple

    28 #   is recursive.  Or lots of times a PUT is generated that's never accessed

    29 #   by a later GET.

    30 

    31 

    32 """

    33 "A pickle" is a program for a virtual pickle machine (PM, but more accurately

    34 called an unpickling machine).  It's a sequence of opcodes, interpreted by the

    35 PM, building an arbitrarily complex Python object.

    36 

    37 For the most part, the PM is very simple:  there are no looping, testing, or

    38 conditional instructions, no arithmetic and no function calls.  Opcodes are

    39 executed once each, from first to last, until a STOP opcode is reached.

    40 

    41 The PM has two data areas, "the stack" and "the memo".

    42 

    43 Many opcodes push Python objects onto the stack; e.g., INT pushes a Python

    44 integer object on the stack, whose value is gotten from a decimal string

    45 literal immediately following the INT opcode in the pickle bytestream.  Other

    46 opcodes take Python objects off the stack.  The result of unpickling is

    47 whatever object is left on the stack when the final STOP opcode is executed.

    48 

    49 The memo is simply an array of objects, or it can be implemented as a dict

    50 mapping little integers to objects.  The memo serves as the PM's "long term

    51 memory", and the little integers indexing the memo are akin to variable

    52 names.  Some opcodes pop a stack object into the memo at a given index,

    53 and others push a memo object at a given index onto the stack again.

    54 

    55 At heart, that's all the PM has.  Subtleties arise for these reasons:

    56 

    57 + Object identity.  Objects can be arbitrarily complex, and subobjects

    58   may be shared (for example, the list [a, a] refers to the same object a

    59   twice).  It can be vital that unpickling recreate an isomorphic object

    60   graph, faithfully reproducing sharing.

    61 

    62 + Recursive objects.  For example, after "L = []; L.append(L)", L is a

    63   list, and L[0] is the same list.  This is related to the object identity

    64   point, and some sequences of pickle opcodes are subtle in order to

    65   get the right result in all cases.

    66 

    67 + Things pickle doesn't know everything about.  Examples of things pickle

    68   does know everything about are Python's builtin scalar and container

    69   types, like ints and tuples.  They generally have opcodes dedicated to

    70   them.  For things like module references and instances of user-defined

    71   classes, pickle's knowledge is limited.  Historically, many enhancements

    72   have been made to the pickle protocol in order to do a better (faster,

    73   and/or more compact) job on those.

    74 

    75 + Backward compatibility and micro-optimization.  As explained below,

    76   pickle opcodes never go away, not even when better ways to do a thing

    77   get invented.  The repertoire of the PM just keeps growing over time.

    78   For example, protocol 0 had two opcodes for building Python integers (INT

    79   and LONG), protocol 1 added three more for more-efficient pickling of short

    80   integers, and protocol 2 added two more for more-efficient pickling of

    81   long integers (before protocol 2, the only ways to pickle a Python long

    82   took time quadratic in the number of digits, for both pickling and

    83   unpickling).  "Opcode bloat" isn't so much a subtlety as a source of

    84   wearying complication.

    85 

    86 

    87 Pickle protocols:

    88 

    89 For compatibility, the meaning of a pickle opcode never changes.  Instead new

    90 pickle opcodes get added, and each version's unpickler can handle all the

    91 pickle opcodes in all protocol versions to date.  So old pickles continue to

    92 be readable forever.  The pickler can generally be told to restrict itself to

    93 the subset of opcodes available under previous protocol versions too, so that

    94 users can create pickles under the current version readable by older

    95 versions.  However, a pickle does not contain its version number embedded

    96 within it.  If an older unpickler tries to read a pickle using a later

    97 protocol, the result is most likely an exception due to seeing an unknown (in

    98 the older unpickler) opcode.

    99 

   100 The original pickle used what's now called "protocol 0", and what was called

   101 "text mode" before Python 2.3.  The entire pickle bytestream is made up of

   102 printable 7-bit ASCII characters, plus the newline character, in protocol 0.

   103 That's why it was called text mode.  Protocol 0 is small and elegant, but

   104 sometimes painfully inefficient.

   105 

   106 The second major set of additions is now called "protocol 1", and was called

   107 "binary mode" before Python 2.3.  This added many opcodes with arguments

   108 consisting of arbitrary bytes, including NUL bytes and unprintable "high bit"

   109 bytes.  Binary mode pickles can be substantially smaller than equivalent

   110 text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte

   111 int as 4 bytes following the opcode, which is cheaper to unpickle than the

   112 (perhaps) 11-character decimal string attached to INT.  Protocol 1 also added

   113 a number of opcodes that operate on many stack elements at once (like APPENDS

   114 and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE).

   115 

   116 The third major set of additions came in Python 2.3, and is called "protocol

   117 2".  This added:

   118 

   119 - A better way to pickle instances of new-style classes (NEWOBJ).

   120 

   121 - A way for a pickle to identify its protocol (PROTO).

   122 

   123 - Time- and space- efficient pickling of long ints (LONG{1,4}).

   124 

   125 - Shortcuts for small tuples (TUPLE{1,2,3}}.

   126 

   127 - Dedicated opcodes for bools (NEWTRUE, NEWFALSE).

   128 

   129 - The "extension registry", a vector of popular objects that can be pushed

   130   efficiently by index (EXT{1,2,4}).  This is akin to the memo and GET, but

   131   the registry contents are predefined (there's nothing akin to the memo's

   132   PUT).

   133 

   134 Another independent change with Python 2.3 is the abandonment of any

   135 pretense that it might be safe to load pickles received from untrusted

   136 parties -- no sufficient security analysis has been done to guarantee

   137 this and there isn't a use case that warrants the expense of such an

   138 analysis.

   139 

   140 To this end, all tests for __safe_for_unpickling__ or for

   141 copy_reg.safe_constructors are removed from the unpickling code.

   142 References to these variables in the descriptions below are to be seen

   143 as describing unpickling in Python 2.2 and before.

   144 """

   145 

   146 # Meta-rule:  Descriptions are stored in instances of descriptor objects,

   147 # with plain constructors.  No meta-language is defined from which

   148 # descriptors could be constructed.  If you want, e.g., XML, write a little

   149 # program to generate XML from the objects.

   150 

   151 ##############################################################################

   152 # Some pickle opcodes have an argument, following the opcode in the

   153 # bytestream.  An argument is of a specific type, described by an instance

   154 # of ArgumentDescriptor.  These are not to be confused with arguments taken

   155 # off the stack -- ArgumentDescriptor applies only to arguments embedded in

   156 # the opcode stream, immediately following an opcode.

   157 

   158 # Represents the number of bytes consumed by an argument delimited by the

   159 # next newline character.

   160 UP_TO_NEWLINE = -1

   161 

   162 # Represents the number of bytes consumed by a two-argument opcode where

   163 # the first argument gives the number of bytes in the second argument.

   164 TAKEN_FROM_ARGUMENT1 = -2   # num bytes is 1-byte unsigned int

   165 TAKEN_FROM_ARGUMENT4 = -3   # num bytes is 4-byte signed little-endian int

   166 

   167 class ArgumentDescriptor(object):

   168     __slots__ = (

   169         # name of descriptor record, also a module global name; a string

   170         'name',

   171 

   172         # length of argument, in bytes; an int; UP_TO_NEWLINE and

   173         # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length

   174         # cases

   175         'n',

   176 

   177         # a function taking a file-like object, reading this kind of argument

   178         # from the object at the current position, advancing the current

   179         # position by n bytes, and returning the value of the argument

   180         'reader',

   181 

   182         # human-readable docs for this arg descriptor; a string

   183         'doc',

   184     )

   185 

   186     def __init__(self, name, n, reader, doc):

   187         assert isinstance(name, str)

   188         self.name = name

   189 

   190         assert isinstance(n, int) and (n >= 0 or

   191                                        n in (UP_TO_NEWLINE,

   192                                              TAKEN_FROM_ARGUMENT1,

   193                                              TAKEN_FROM_ARGUMENT4))

   194         self.n = n

   195 

   196         self.reader = reader

   197 

   198         assert isinstance(doc, str)

   199         self.doc = doc

   200 

   201 from struct import unpack as _unpack

   202 

   203 def read_uint1(f):

   204     r"""

   205     >>> import StringIO

   206     >>> read_uint1(StringIO.StringIO('\xff'))

   207     255

   208     """

   209 

   210     data = f.read(1)

   211     if data:

   212         return ord(data)

   213     raise ValueError("not enough data in stream to read uint1")

   214 

   215 uint1 = ArgumentDescriptor(

   216             name='uint1',

   217             n=1,

   218             reader=read_uint1,

   219             doc="One-byte unsigned integer.")

   220 

   221 

   222 def read_uint2(f):

   223     r"""

   224     >>> import StringIO

   225     >>> read_uint2(StringIO.StringIO('\xff\x00'))

   226     255

   227     >>> read_uint2(StringIO.StringIO('\xff\xff'))

   228     65535

   229     """

   230 

   231     data = f.read(2)

   232     if len(data) == 2:

   233         return _unpack("<H", data)[0]

   234     raise ValueError("not enough data in stream to read uint2")

   235 

   236 uint2 = ArgumentDescriptor(

   237             name='uint2',

   238             n=2,

   239             reader=read_uint2,

   240             doc="Two-byte unsigned integer, little-endian.")

   241 

   242 

   243 def read_int4(f):

   244     r"""

   245     >>> import StringIO

   246     >>> read_int4(StringIO.StringIO('\xff\x00\x00\x00'))

   247     255

   248     >>> read_int4(StringIO.StringIO('\x00\x00\x00\x80')) == -(2**31)

   249     True

   250     """

   251 

   252     data = f.read(4)

   253     if len(data) == 4:

   254         return _unpack("<i", data)[0]

   255     raise ValueError("not enough data in stream to read int4")

   256 

   257 int4 = ArgumentDescriptor(

   258            name='int4',

   259            n=4,

   260            reader=read_int4,

   261            doc="Four-byte signed integer, little-endian, 2's complement.")

   262 

   263 

   264 def read_stringnl(f, decode=True, stripquotes=True):

   265     r"""

   266     >>> import StringIO

   267     >>> read_stringnl(StringIO.StringIO("'abcd'\nefg\n"))

   268     'abcd'

   269 

   270     >>> read_stringnl(StringIO.StringIO("\n"))

   271     Traceback (most recent call last):

   272     ...

   273     ValueError: no string quotes around ''

   274 

   275     >>> read_stringnl(StringIO.StringIO("\n"), stripquotes=False)

   276     ''

   277 

   278     >>> read_stringnl(StringIO.StringIO("''\n"))

   279     ''

   280 

   281     >>> read_stringnl(StringIO.StringIO('"abcd"'))

   282     Traceback (most recent call last):

   283     ...

   284     ValueError: no newline found when trying to read stringnl

   285 

   286     Embedded escapes are undone in the result.

   287     >>> read_stringnl(StringIO.StringIO(r"'a\n\\b\x00c\td'" + "\n'e'"))

   288     'a\n\\b\x00c\td'

   289     """

   290 

   291     data = f.readline()

   292     if not data.endswith('\n'):

   293         raise ValueError("no newline found when trying to read stringnl")

   294     data = data[:-1]    # lose the newline

   295 

   296     if stripquotes:

   297         for q in "'\"":

   298             if data.startswith(q):

   299                 if not data.endswith(q):

   300                     raise ValueError("strinq quote %r not found at both "

   301                                      "ends of %r" % (q, data))

   302                 data = data[1:-1]

   303                 break

   304         else:

   305             raise ValueError("no string quotes around %r" % data)

   306 

   307     # I'm not sure when 'string_escape' was added to the std codecs; it's

   308     # crazy not to use it if it's there.

   309     if decode:

   310         data = data.decode('string_escape')

   311     return data

   312 

   313 stringnl = ArgumentDescriptor(

   314                name='stringnl',

   315                n=UP_TO_NEWLINE,

   316                reader=read_stringnl,

   317                doc="""A newline-terminated string.

   318 

   319                    This is a repr-style string, with embedded escapes, and

   320                    bracketing quotes.

   321                    """)

   322 

   323 def read_stringnl_noescape(f):

   324     return read_stringnl(f, decode=False, stripquotes=False)

   325 

   326 stringnl_noescape = ArgumentDescriptor(

   327                         name='stringnl_noescape',

   328                         n=UP_TO_NEWLINE,

   329                         reader=read_stringnl_noescape,

   330                         doc="""A newline-terminated string.

   331 

   332                         This is a str-style string, without embedded escapes,

   333                         or bracketing quotes.  It should consist solely of

   334                         printable ASCII characters.

   335                         """)

   336 

   337 def read_stringnl_noescape_pair(f):

   338     r"""

   339     >>> import StringIO

   340     >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\nEmpty\njunk"))

   341     'Queue Empty'

   342     """

   343 

   344     return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f))

   345 

   346 stringnl_noescape_pair = ArgumentDescriptor(

   347                              name='stringnl_noescape_pair',

   348                              n=UP_TO_NEWLINE,

   349                              reader=read_stringnl_noescape_pair,

   350                              doc="""A pair of newline-terminated strings.

   351 

   352                              These are str-style strings, without embedded

   353                              escapes, or bracketing quotes.  They should

   354                              consist solely of printable ASCII characters.

   355                              The pair is returned as a single string, with

   356                              a single blank separating the two strings.

   357                              """)

   358 

   359 def read_string4(f):

   360     r"""

   361     >>> import StringIO

   362     >>> read_string4(StringIO.StringIO("\x00\x00\x00\x00abc"))

   363     ''

   364     >>> read_string4(StringIO.StringIO("\x03\x00\x00\x00abcdef"))

   365     'abc'

   366     >>> read_string4(StringIO.StringIO("\x00\x00\x00\x03abcdef"))

   367     Traceback (most recent call last):

   368     ...

   369     ValueError: expected 50331648 bytes in a string4, but only 6 remain

   370     """

   371 

   372     n = read_int4(f)

   373     if n < 0:

   374         raise ValueError("string4 byte count < 0: %d" % n)

   375     data = f.read(n)

   376     if len(data) == n:

   377         return data

   378     raise ValueError("expected %d bytes in a string4, but only %d remain" %

   379                      (n, len(data)))

   380 

   381 string4 = ArgumentDescriptor(

   382               name="string4",

   383               n=TAKEN_FROM_ARGUMENT4,

   384               reader=read_string4,

   385               doc="""A counted string.

   386 

   387               The first argument is a 4-byte little-endian signed int giving

   388               the number of bytes in the string, and the second argument is

   389               that many bytes.

   390               """)

   391 

   392 

   393 def read_string1(f):

   394     r"""

   395     >>> import StringIO

   396     >>> read_string1(StringIO.StringIO("\x00"))

   397     ''

   398     >>> read_string1(StringIO.StringIO("\x03abcdef"))

   399     'abc'

   400     """

   401 

   402     n = read_uint1(f)

   403     assert n >= 0

   404     data = f.read(n)

   405     if len(data) == n:

   406         return data

   407     raise ValueError("expected %d bytes in a string1, but only %d remain" %

   408                      (n, len(data)))

   409 

   410 string1 = ArgumentDescriptor(

   411               name="string1",

   412               n=TAKEN_FROM_ARGUMENT1,

   413               reader=read_string1,

   414               doc="""A counted string.

   415 

   416               The first argument is a 1-byte unsigned int giving the number

   417               of bytes in the string, and the second argument is that many

   418               bytes.

   419               """)

   420 

   421 

   422 def read_unicodestringnl(f):

   423     r"""

   424     >>> import StringIO

   425     >>> read_unicodestringnl(StringIO.StringIO("abc\uabcd\njunk"))

   426     u'abc\uabcd'

   427     """

   428 

   429     data = f.readline()

   430     if not data.endswith('\n'):

   431         raise ValueError("no newline found when trying to read "

   432                          "unicodestringnl")

   433     data = data[:-1]    # lose the newline

   434     return unicode(data, 'raw-unicode-escape')

   435 

   436 unicodestringnl = ArgumentDescriptor(

   437                       name='unicodestringnl',

   438                       n=UP_TO_NEWLINE,

   439                       reader=read_unicodestringnl,

   440                       doc="""A newline-terminated Unicode string.

   441 

   442                       This is raw-unicode-escape encoded, so consists of

   443                       printable ASCII characters, and may contain embedded

   444                       escape sequences.

   445                       """)

   446 

   447 def read_unicodestring4(f):

   448     r"""

   449     >>> import StringIO

   450     >>> s = u'abcd\uabcd'

   451     >>> enc = s.encode('utf-8')

   452     >>> enc

   453     'abcd\xea\xaf\x8d'

   454     >>> n = chr(len(enc)) + chr(0) * 3  # little-endian 4-byte length

   455     >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk'))

   456     >>> s == t

   457     True

   458 

   459     >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1]))

   460     Traceback (most recent call last):

   461     ...

   462     ValueError: expected 7 bytes in a unicodestring4, but only 6 remain

   463     """

   464 

   465     n = read_int4(f)

   466     if n < 0:

   467         raise ValueError("unicodestring4 byte count < 0: %d" % n)

   468     data = f.read(n)

   469     if len(data) == n:

   470         return unicode(data, 'utf-8')

   471     raise ValueError("expected %d bytes in a unicodestring4, but only %d "

   472                      "remain" % (n, len(data)))

   473 

   474 unicodestring4 = ArgumentDescriptor(

   475                     name="unicodestring4",

   476                     n=TAKEN_FROM_ARGUMENT4,

   477                     reader=read_unicodestring4,

   478                     doc="""A counted Unicode string.

   479 

   480                     The first argument is a 4-byte little-endian signed int

   481                     giving the number of bytes in the string, and the second

   482                     argument-- the UTF-8 encoding of the Unicode string --

   483                     contains that many bytes.

   484                     """)

   485 

   486 

   487 def read_decimalnl_short(f):

   488     r"""

   489     >>> import StringIO

   490     >>> read_decimalnl_short(StringIO.StringIO("1234\n56"))

   491     1234

   492 

   493     >>> read_decimalnl_short(StringIO.StringIO("1234L\n56"))

   494     Traceback (most recent call last):

   495     ...

   496     ValueError: trailing 'L' not allowed in '1234L'

   497     """

   498 

   499     s = read_stringnl(f, decode=False, stripquotes=False)

   500     if s.endswith("L"):

   501         raise ValueError("trailing 'L' not allowed in %r" % s)

   502 

   503     # It's not necessarily true that the result fits in a Python short int:

   504     # the pickle may have been written on a 64-bit box.  There's also a hack

   505     # for True and False here.

   506     if s == "00":

   507         return False

   508     elif s == "01":

   509         return True

   510 

   511     try:

   512         return int(s)

   513     except OverflowError:

   514         return long(s)

   515 

   516 def read_decimalnl_long(f):

   517     r"""

   518     >>> import StringIO

   519 

   520     >>> read_decimalnl_long(StringIO.StringIO("1234\n56"))

   521     Traceback (most recent call last):

   522     ...

   523     ValueError: trailing 'L' required in '1234'

   524 

   525     Someday the trailing 'L' will probably go away from this output.

   526 

   527     >>> read_decimalnl_long(StringIO.StringIO("1234L\n56"))

   528     1234L

   529 

   530     >>> read_decimalnl_long(StringIO.StringIO("123456789012345678901234L\n6"))

   531     123456789012345678901234L

   532     """

   533 

   534     s = read_stringnl(f, decode=False, stripquotes=False)

   535     if not s.endswith("L"):

   536         raise ValueError("trailing 'L' required in %r" % s)

   537     return long(s)

   538 

   539 

   540 decimalnl_short = ArgumentDescriptor(

   541                       name='decimalnl_short',

   542                       n=UP_TO_NEWLINE,

   543                       reader=read_decimalnl_short,

   544                       doc="""A newline-terminated decimal integer literal.

   545 

   546                           This never has a trailing 'L', and the integer fit

   547                           in a short Python int on the box where the pickle

   548                           was written -- but there's no guarantee it will fit

   549                           in a short Python int on the box where the pickle

   550                           is read.

   551                           """)

   552 

   553 decimalnl_long = ArgumentDescriptor(

   554                      name='decimalnl_long',

   555                      n=UP_TO_NEWLINE,

   556                      reader=read_decimalnl_long,

   557                      doc="""A newline-terminated decimal integer literal.

   558 

   559                          This has a trailing 'L', and can represent integers

   560                          of any size.

   561                          """)

   562 

   563 

   564 def read_floatnl(f):

   565     r"""

   566     >>> import StringIO

   567     >>> read_floatnl(StringIO.StringIO("-1.25\n6"))

   568     -1.25

   569     """

   570     s = read_stringnl(f, decode=False, stripquotes=False)

   571     return float(s)

   572 

   573 floatnl = ArgumentDescriptor(

   574               name='floatnl',

   575               n=UP_TO_NEWLINE,

   576               reader=read_floatnl,

   577               doc="""A newline-terminated decimal floating literal.

   578 

   579               In general this requires 17 significant digits for roundtrip

   580               identity, and pickling then unpickling infinities, NaNs, and

   581               minus zero doesn't work across boxes, or on some boxes even

   582               on itself (e.g., Windows can't read the strings it produces

   583               for infinities or NaNs).

   584               """)

   585 

   586 def read_float8(f):

   587     r"""

   588     >>> import StringIO, struct

   589     >>> raw = struct.pack(">d", -1.25)

   590     >>> raw

   591     '\xbf\xf4\x00\x00\x00\x00\x00\x00'

   592     >>> read_float8(StringIO.StringIO(raw + "\n"))

   593     -1.25

   594     """

   595 

   596     data = f.read(8)

   597     if len(data) == 8:

   598         return _unpack(">d", data)[0]

   599     raise ValueError("not enough data in stream to read float8")

   600 

   601 

   602 float8 = ArgumentDescriptor(

   603              name='float8',

   604              n=8,

   605              reader=read_float8,

   606              doc="""An 8-byte binary representation of a float, big-endian.

   607 

   608              The format is unique to Python, and shared with the struct

   609              module (format string '>d') "in theory" (the struct and cPickle

   610              implementations don't share the code -- they should).  It's

   611              strongly related to the IEEE-754 double format, and, in normal

   612              cases, is in fact identical to the big-endian 754 double format.

   613              On other boxes the dynamic range is limited to that of a 754

   614              double, and "add a half and chop" rounding is used to reduce

   615              the precision to 53 bits.  However, even on a 754 box,

   616              infinities, NaNs, and minus zero may not be handled correctly

   617              (may not survive roundtrip pickling intact).

   618              """)

   619 

   620 # Protocol 2 formats

   621 

   622 from pickle import decode_long

   623 

   624 def read_long1(f):

   625     r"""

   626     >>> import StringIO

   627     >>> read_long1(StringIO.StringIO("\x00"))

   628     0L

   629     >>> read_long1(StringIO.StringIO("\x02\xff\x00"))

   630     255L

   631     >>> read_long1(StringIO.StringIO("\x02\xff\x7f"))

   632     32767L

   633     >>> read_long1(StringIO.StringIO("\x02\x00\xff"))

   634     -256L

   635     >>> read_long1(StringIO.StringIO("\x02\x00\x80"))

   636     -32768L

   637     """

   638 

   639     n = read_uint1(f)

   640     data = f.read(n)

   641     if len(data) != n:

   642         raise ValueError("not enough data in stream to read long1")

   643     return decode_long(data)

   644 

   645 long1 = ArgumentDescriptor(

   646     name="long1",

   647     n=TAKEN_FROM_ARGUMENT1,

   648     reader=read_long1,

   649     doc="""A binary long, little-endian, using 1-byte size.

   650 

   651     This first reads one byte as an unsigned size, then reads that

   652     many bytes and interprets them as a little-endian 2's-complement long.

   653     If the size is 0, that's taken as a shortcut for the long 0L.

   654     """)

   655 

   656 def read_long4(f):

   657     r"""

   658     >>> import StringIO

   659     >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x00"))

   660     255L

   661     >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x7f"))

   662     32767L

   663     >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\xff"))

   664     -256L

   665     >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\x80"))

   666     -32768L

   667     >>> read_long1(StringIO.StringIO("\x00\x00\x00\x00"))

   668     0L

   669     """

   670 

   671     n = read_int4(f)

   672     if n < 0:

   673         raise ValueError("long4 byte count < 0: %d" % n)

   674     data = f.read(n)

   675     if len(data) != n:

   676         raise ValueError("not enough data in stream to read long4")

   677     return decode_long(data)

   678 

   679 long4 = ArgumentDescriptor(

   680     name="long4",

   681     n=TAKEN_FROM_ARGUMENT4,

   682     reader=read_long4,

   683     doc="""A binary representation of a long, little-endian.

   684 

   685     This first reads four bytes as a signed size (but requires the

   686     size to be >= 0), then reads that many bytes and interprets them

   687     as a little-endian 2's-complement long.  If the size is 0, that's taken

   688     as a shortcut for the long 0L, although LONG1 should really be used

   689     then instead (and in any case where # of bytes < 256).

   690     """)

   691 

   692 

   693 ##############################################################################

   694 # Object descriptors.  The stack used by the pickle machine holds objects,

   695 # and in the stack_before and stack_after attributes of OpcodeInfo

   696 # descriptors we need names to describe the various types of objects that can

   697 # appear on the stack.

   698 

   699 class StackObject(object):

   700     __slots__ = (

   701         # name of descriptor record, for info only

   702         'name',

   703 

   704         # type of object, or tuple of type objects (meaning the object can

   705         # be of any type in the tuple)

   706         'obtype',

   707 

   708         # human-readable docs for this kind of stack object; a string

   709         'doc',

   710     )

   711 

   712     def __init__(self, name, obtype, doc):

   713         assert isinstance(name, str)

   714         self.name = name

   715 

   716         assert isinstance(obtype, type) or isinstance(obtype, tuple)

   717         if isinstance(obtype, tuple):

   718             for contained in obtype:

   719                 assert isinstance(contained, type)

   720         self.obtype = obtype

   721 

   722         assert isinstance(doc, str)

   723         self.doc = doc

   724 

   725     def __repr__(self):

   726         return self.name

   727 

   728 

   729 pyint = StackObject(

   730             name='int',

   731             obtype=int,

   732             doc="A short (as opposed to long) Python integer object.")

   733 

   734 pylong = StackObject(

   735              name='long',

   736              obtype=long,

   737              doc="A long (as opposed to short) Python integer object.")

   738 

   739 pyinteger_or_bool = StackObject(

   740                         name='int_or_bool',

   741                         obtype=(int, long, bool),

   742                         doc="A Python integer object (short or long), or "

   743                             "a Python bool.")

   744 

   745 pybool = StackObject(

   746              name='bool',

   747              obtype=(bool,),

   748              doc="A Python bool object.")

   749 

   750 pyfloat = StackObject(

   751               name='float',

   752               obtype=float,

   753               doc="A Python float object.")

   754 

   755 pystring = StackObject(

   756                name='str',

   757                obtype=str,

   758                doc="A Python string object.")

   759 

   760 pyunicode = StackObject(

   761                 name='unicode',

   762                 obtype=unicode,

   763                 doc="A Python Unicode string object.")

   764 

   765 pynone = StackObject(

   766              name="None",

   767              obtype=type(None),

   768              doc="The Python None object.")

   769 

   770 pytuple = StackObject(

   771               name="tuple",

   772               obtype=tuple,

   773               doc="A Python tuple object.")

   774 

   775 pylist = StackObject(

   776              name="list",

   777              obtype=list,

   778              doc="A Python list object.")

   779 

   780 pydict = StackObject(

   781              name="dict",

   782              obtype=dict,

   783              doc="A Python dict object.")

   784 

   785 anyobject = StackObject(

   786                 name='any',

   787                 obtype=object,

   788                 doc="Any kind of object whatsoever.")

   789 

   790 markobject = StackObject(

   791                  name="mark",

   792                  obtype=StackObject,

   793                  doc="""'The mark' is a unique object.

   794 

   795                  Opcodes that operate on a variable number of objects

   796                  generally don't embed the count of objects in the opcode,

   797                  or pull it off the stack.  Instead the MARK opcode is used

   798                  to push a special marker object on the stack, and then

   799                  some other opcodes grab all the objects from the top of

   800                  the stack down to (but not including) the topmost marker

   801                  object.

   802                  """)

   803 

   804 stackslice = StackObject(

   805                  name="stackslice",

   806                  obtype=StackObject,

   807                  doc="""An object representing a contiguous slice of the stack.

   808 

   809                  This is used in conjuction with markobject, to represent all

   810                  of the stack following the topmost markobject.  For example,

   811                  the POP_MARK opcode changes the stack from

   812 

   813                      [..., markobject, stackslice]

   814                  to

   815                      [...]

   816 

   817                  No matter how many object are on the stack after the topmost

   818                  markobject, POP_MARK gets rid of all of them (including the

   819                  topmost markobject too).

   820                  """)

   821 

   822 ##############################################################################

   823 # Descriptors for pickle opcodes.

   824 

   825 class OpcodeInfo(object):

   826 

   827     __slots__ = (

   828         # symbolic name of opcode; a string

   829         'name',

   830 

   831         # the code used in a bytestream to represent the opcode; a

   832         # one-character string

   833         'code',

   834 

   835         # If the opcode has an argument embedded in the byte string, an

   836         # instance of ArgumentDescriptor specifying its type.  Note that

   837         # arg.reader(s) can be used to read and decode the argument from

   838         # the bytestream s, and arg.doc documents the format of the raw

   839         # argument bytes.  If the opcode doesn't have an argument embedded

   840         # in the bytestream, arg should be None.

   841         'arg',

   842 

   843         # what the stack looks like before this opcode runs; a list

   844         'stack_before',

   845 

   846         # what the stack looks like after this opcode runs; a list

   847         'stack_after',

   848 

   849         # the protocol number in which this opcode was introduced; an int

   850         'proto',

   851 

   852         # human-readable docs for this opcode; a string

   853         'doc',

   854     )

   855 

   856     def __init__(self, name, code, arg,

   857                  stack_before, stack_after, proto, doc):

   858         assert isinstance(name, str)

   859         self.name = name

   860 

   861         assert isinstance(code, str)

   862         assert len(code) == 1

   863         self.code = code

   864 

   865         assert arg is None or isinstance(arg, ArgumentDescriptor)

   866         self.arg = arg

   867 

   868         assert isinstance(stack_before, list)

   869         for x in stack_before:

   870             assert isinstance(x, StackObject)

   871         self.stack_before = stack_before

   872 

   873         assert isinstance(stack_after, list)

   874         for x in stack_after:

   875             assert isinstance(x, StackObject)

   876         self.stack_after = stack_after

   877 

   878         assert isinstance(proto, int) and 0 <= proto <= 2

   879         self.proto = proto

   880 

   881         assert isinstance(doc, str)

   882         self.doc = doc

   883 

   884 I = OpcodeInfo

   885 opcodes = [

   886 

   887     # Ways to spell integers.

   888 

   889     I(name='INT',

   890       code='I',

   891       arg=decimalnl_short,

   892       stack_before=[],

   893       stack_after=[pyinteger_or_bool],

   894       proto=0,

   895       doc="""Push an integer or bool.

   896 

   897       The argument is a newline-terminated decimal literal string.

   898 

   899       The intent may have been that this always fit in a short Python int,

   900       but INT can be generated in pickles written on a 64-bit box that

   901       require a Python long on a 32-bit box.  The difference between this

   902       and LONG then is that INT skips a trailing 'L', and produces a short

   903       int whenever possible.

   904 

   905       Another difference is due to that, when bool was introduced as a

   906       distinct type in 2.3, builtin names True and False were also added to

   907       2.2.2, mapping to ints 1 and 0.  For compatibility in both directions,

   908       True gets pickled as INT + "I01\\n", and False as INT + "I00\\n".

   909       Leading zeroes are never produced for a genuine integer.  The 2.3

   910       (and later) unpicklers special-case these and return bool instead;

   911       earlier unpicklers ignore the leading "0" and return the int.

   912       """),

   913 

   914     I(name='BININT',

   915       code='J',

   916       arg=int4,

   917       stack_before=[],

   918       stack_after=[pyint],

   919       proto=1,

   920       doc="""Push a four-byte signed integer.

   921 

   922       This handles the full range of Python (short) integers on a 32-bit

   923       box, directly as binary bytes (1 for the opcode and 4 for the integer).

   924       If the integer is non-negative and fits in 1 or 2 bytes, pickling via

   925       BININT1 or BININT2 saves space.

   926       """),

   927 

   928     I(name='BININT1',

   929       code='K',

   930       arg=uint1,

   931       stack_before=[],

   932       stack_after=[pyint],

   933       proto=1,

   934       doc="""Push a one-byte unsigned integer.

   935 

   936       This is a space optimization for pickling very small non-negative ints,

   937       in range(256).

   938       """),

   939 

   940     I(name='BININT2',

   941       code='M',

   942       arg=uint2,

   943       stack_before=[],

   944       stack_after=[pyint],

   945       proto=1,

   946       doc="""Push a two-byte unsigned integer.

   947 

   948       This is a space optimization for pickling small positive ints, in

   949       range(256, 2**16).  Integers in range(256) can also be pickled via

   950       BININT2, but BININT1 instead saves a byte.

   951       """),

   952 

   953     I(name='LONG',

   954       code='L',

   955       arg=decimalnl_long,

   956       stack_before=[],

   957       stack_after=[pylong],

   958       proto=0,

   959       doc="""Push a long integer.

   960 

   961       The same as INT, except that the literal ends with 'L', and always

   962       unpickles to a Python long.  There doesn't seem a real purpose to the

   963       trailing 'L'.

   964 

   965       Note that LONG takes time quadratic in the number of digits when

   966       unpickling (this is simply due to the nature of decimal->binary

   967       conversion).  Proto 2 added linear-time (in C; still quadratic-time

   968       in Python) LONG1 and LONG4 opcodes.

   969       """),

   970 

   971     I(name="LONG1",

   972       code='\x8a',

   973       arg=long1,

   974       stack_before=[],

   975       stack_after=[pylong],

   976       proto=2,

   977       doc="""Long integer using one-byte length.

   978 

   979       A more efficient encoding of a Python long; the long1 encoding

   980       says it all."""),

   981 

   982     I(name="LONG4",

   983       code='\x8b',

   984       arg=long4,

   985       stack_before=[],

   986       stack_after=[pylong],

   987       proto=2,

   988       doc="""Long integer using found-byte length.

   989 

   990       A more efficient encoding of a Python long; the long4 encoding

   991       says it all."""),

   992 

   993     # Ways to spell strings (8-bit, not Unicode).

   994 

   995     I(name='STRING',

   996       code='S',

   997       arg=stringnl,

   998       stack_before=[],

   999       stack_after=[pystring],

  1000       proto=0,

  1001       doc="""Push a Python string object.

  1002 

  1003       The argument is a repr-style string, with bracketing quote characters,

  1004       and perhaps embedded escapes.  The argument extends until the next

  1005       newline character.

  1006       """),

  1007 

  1008     I(name='BINSTRING',

  1009       code='T',

  1010       arg=string4,

  1011       stack_before=[],

  1012       stack_after=[pystring],

  1013       proto=1,

  1014       doc="""Push a Python string object.

  1015 

  1016       There are two arguments:  the first is a 4-byte little-endian signed int

  1017       giving the number of bytes in the string, and the second is that many

  1018       bytes, which are taken literally as the string content.

  1019       """),

  1020 

  1021     I(name='SHORT_BINSTRING',

  1022       code='U',

  1023       arg=string1,

  1024       stack_before=[],

  1025       stack_after=[pystring],

  1026       proto=1,

  1027       doc="""Push a Python string object.

  1028 

  1029       There are two arguments:  the first is a 1-byte unsigned int giving

  1030       the number of bytes in the string, and the second is that many bytes,

  1031       which are taken literally as the string content.

  1032       """),

  1033 

  1034     # Ways to spell None.

  1035 

  1036     I(name='NONE',

  1037       code='N',

  1038       arg=None,

  1039       stack_before=[],

  1040       stack_after=[pynone],

  1041       proto=0,

  1042       doc="Push None on the stack."),

  1043 

  1044     # Ways to spell bools, starting with proto 2.  See INT for how this was

  1045     # done before proto 2.

  1046 

  1047     I(name='NEWTRUE',

  1048       code='\x88',

  1049       arg=None,

  1050       stack_before=[],

  1051       stack_after=[pybool],

  1052       proto=2,

  1053       doc="""True.

  1054 

  1055       Push True onto the stack."""),

  1056 

  1057     I(name='NEWFALSE',

  1058       code='\x89',

  1059       arg=None,

  1060       stack_before=[],

  1061       stack_after=[pybool],

  1062       proto=2,

  1063       doc="""True.

  1064 

  1065       Push False onto the stack."""),

  1066 

  1067     # Ways to spell Unicode strings.

  1068 

  1069     I(name='UNICODE',

  1070       code='V',

  1071       arg=unicodestringnl,

  1072       stack_before=[],

  1073       stack_after=[pyunicode],

  1074       proto=0,  # this may be pure-text, but it's a later addition

  1075       doc="""Push a Python Unicode string object.

  1076 

  1077       The argument is a raw-unicode-escape encoding of a Unicode string,

  1078       and so may contain embedded escape sequences.  The argument extends

  1079       until the next newline character.

  1080       """),

  1081 

  1082     I(name='BINUNICODE',

  1083       code='X',

  1084       arg=unicodestring4,

  1085       stack_before=[],

  1086       stack_after=[pyunicode],

  1087       proto=1,

  1088       doc="""Push a Python Unicode string object.

  1089 

  1090       There are two arguments:  the first is a 4-byte little-endian signed int

  1091       giving the number of bytes in the string.  The second is that many

  1092       bytes, and is the UTF-8 encoding of the Unicode string.

  1093       """),

  1094 

  1095     # Ways to spell floats.

  1096 

  1097     I(name='FLOAT',

  1098       code='F',

  1099       arg=floatnl,

  1100       stack_before=[],

  1101       stack_after=[pyfloat],

  1102       proto=0,

  1103       doc="""Newline-terminated decimal float literal.

  1104 

  1105       The argument is repr(a_float), and in general requires 17 significant

  1106       digits for roundtrip conversion to be an identity (this is so for

  1107       IEEE-754 double precision values, which is what Python float maps to

  1108       on most boxes).

  1109 

  1110       In general, FLOAT cannot be used to transport infinities, NaNs, or

  1111       minus zero across boxes (or even on a single box, if the platform C

  1112       library can't read the strings it produces for such things -- Windows

  1113       is like that), but may do less damage than BINFLOAT on boxes with

  1114       greater precision or dynamic range than IEEE-754 double.

  1115       """),

  1116 

  1117     I(name='BINFLOAT',

  1118       code='G',

  1119       arg=float8,

  1120       stack_before=[],

  1121       stack_after=[pyfloat],

  1122       proto=1,

  1123       doc="""Float stored in binary form, with 8 bytes of data.

  1124 

  1125       This generally requires less than half the space of FLOAT encoding.

  1126       In general, BINFLOAT cannot be used to transport infinities, NaNs, or

  1127       minus zero, raises an exception if the exponent exceeds the range of

  1128       an IEEE-754 double, and retains no more than 53 bits of precision (if

  1129       there are more than that, "add a half and chop" rounding is used to

  1130       cut it back to 53 significant bits).

  1131       """),

  1132 

  1133     # Ways to build lists.

  1134 

  1135     I(name='EMPTY_LIST',