blob: 66f4eddbc96c2967aff2761c3305fbd56e798a01 [file] [log] [blame]
Skip Montanaro54455942003-01-29 15:41:33 +00001'''"Executable documentation" for the pickle module.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002
3Extensive comments about the pickle protocols and pickle-machine opcodes
4can be found here. Some functions meant for external use:
5
6genops(pickle)
7 Generate all the opcodes in a pickle, as (opcode, arg, position) triples.
8
Andrew M. Kuchlingd0c53fe2004-08-07 16:51:30 +00009dis(pickle, out=None, memo=None, indentlevel=4)
Tim Peters8ecfc8e2003-01-27 18:51:48 +000010 Print a symbolic disassembly of a pickle.
Skip Montanaro54455942003-01-29 15:41:33 +000011'''
Tim Peters8ecfc8e2003-01-27 18:51:48 +000012
Walter Dörwald42748a82007-06-12 16:40:17 +000013import codecs
Guido van Rossum98297ee2007-11-06 21:34:58 +000014import pickle
15import re
Walter Dörwald42748a82007-06-12 16:40:17 +000016
Christian Heimes3feef612008-02-11 06:19:17 +000017__all__ = ['dis', 'genops', 'optimize']
Tim Peters90cf2122004-11-06 23:45:48 +000018
Guido van Rossum98297ee2007-11-06 21:34:58 +000019bytes_types = pickle.bytes_types
20
Tim Peters8ecfc8e2003-01-27 18:51:48 +000021# Other ideas:
22#
23# - A pickle verifier: read a pickle and check it exhaustively for
Tim Petersc1c2b3e2003-01-29 20:12:21 +000024# well-formedness. dis() does a lot of this already.
Tim Peters8ecfc8e2003-01-27 18:51:48 +000025#
26# - A protocol identifier: examine a pickle and return its protocol number
27# (== the highest .proto attr value among all the opcodes in the pickle).
Tim Petersc1c2b3e2003-01-29 20:12:21 +000028# dis() already prints this info at the end.
Tim Peters8ecfc8e2003-01-27 18:51:48 +000029#
30# - A pickle optimizer: for example, tuple-building code is sometimes more
31# elaborate than necessary, catering for the possibility that the tuple
32# is recursive. Or lots of times a PUT is generated that's never accessed
33# by a later GET.
34
35
36"""
37"A pickle" is a program for a virtual pickle machine (PM, but more accurately
38called an unpickling machine). It's a sequence of opcodes, interpreted by the
39PM, building an arbitrarily complex Python object.
40
41For the most part, the PM is very simple: there are no looping, testing, or
42conditional instructions, no arithmetic and no function calls. Opcodes are
43executed once each, from first to last, until a STOP opcode is reached.
44
45The PM has two data areas, "the stack" and "the memo".
46
47Many opcodes push Python objects onto the stack; e.g., INT pushes a Python
48integer object on the stack, whose value is gotten from a decimal string
49literal immediately following the INT opcode in the pickle bytestream. Other
50opcodes take Python objects off the stack. The result of unpickling is
51whatever object is left on the stack when the final STOP opcode is executed.
52
53The memo is simply an array of objects, or it can be implemented as a dict
54mapping little integers to objects. The memo serves as the PM's "long term
55memory", and the little integers indexing the memo are akin to variable
56names. Some opcodes pop a stack object into the memo at a given index,
57and others push a memo object at a given index onto the stack again.
58
59At heart, that's all the PM has. Subtleties arise for these reasons:
60
61+ Object identity. Objects can be arbitrarily complex, and subobjects
62 may be shared (for example, the list [a, a] refers to the same object a
63 twice). It can be vital that unpickling recreate an isomorphic object
64 graph, faithfully reproducing sharing.
65
66+ Recursive objects. For example, after "L = []; L.append(L)", L is a
67 list, and L[0] is the same list. This is related to the object identity
68 point, and some sequences of pickle opcodes are subtle in order to
69 get the right result in all cases.
70
71+ Things pickle doesn't know everything about. Examples of things pickle
72 does know everything about are Python's builtin scalar and container
73 types, like ints and tuples. They generally have opcodes dedicated to
74 them. For things like module references and instances of user-defined
75 classes, pickle's knowledge is limited. Historically, many enhancements
76 have been made to the pickle protocol in order to do a better (faster,
77 and/or more compact) job on those.
78
79+ Backward compatibility and micro-optimization. As explained below,
80 pickle opcodes never go away, not even when better ways to do a thing
81 get invented. The repertoire of the PM just keeps growing over time.
Tim Petersfdc03462003-01-28 04:56:33 +000082 For example, protocol 0 had two opcodes for building Python integers (INT
83 and LONG), protocol 1 added three more for more-efficient pickling of short
84 integers, and protocol 2 added two more for more-efficient pickling of
85 long integers (before protocol 2, the only ways to pickle a Python long
86 took time quadratic in the number of digits, for both pickling and
87 unpickling). "Opcode bloat" isn't so much a subtlety as a source of
Tim Peters8ecfc8e2003-01-27 18:51:48 +000088 wearying complication.
89
90
91Pickle protocols:
92
93For compatibility, the meaning of a pickle opcode never changes. Instead new
94pickle opcodes get added, and each version's unpickler can handle all the
95pickle opcodes in all protocol versions to date. So old pickles continue to
96be readable forever. The pickler can generally be told to restrict itself to
97the subset of opcodes available under previous protocol versions too, so that
98users can create pickles under the current version readable by older
99versions. However, a pickle does not contain its version number embedded
100within it. If an older unpickler tries to read a pickle using a later
101protocol, the result is most likely an exception due to seeing an unknown (in
102the older unpickler) opcode.
103
104The original pickle used what's now called "protocol 0", and what was called
105"text mode" before Python 2.3. The entire pickle bytestream is made up of
106printable 7-bit ASCII characters, plus the newline character, in protocol 0.
Tim Petersfdc03462003-01-28 04:56:33 +0000107That's why it was called text mode. Protocol 0 is small and elegant, but
108sometimes painfully inefficient.
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000109
110The second major set of additions is now called "protocol 1", and was called
111"binary mode" before Python 2.3. This added many opcodes with arguments
112consisting of arbitrary bytes, including NUL bytes and unprintable "high bit"
113bytes. Binary mode pickles can be substantially smaller than equivalent
114text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte
115int as 4 bytes following the opcode, which is cheaper to unpickle than the
Tim Petersfdc03462003-01-28 04:56:33 +0000116(perhaps) 11-character decimal string attached to INT. Protocol 1 also added
117a number of opcodes that operate on many stack elements at once (like APPENDS
Tim Peters81098ac2003-01-28 05:12:08 +0000118and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE).
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000119
120The third major set of additions came in Python 2.3, and is called "protocol
Tim Petersfdc03462003-01-28 04:56:33 +00001212". This added:
122
123- A better way to pickle instances of new-style classes (NEWOBJ).
124
125- A way for a pickle to identify its protocol (PROTO).
126
127- Time- and space- efficient pickling of long ints (LONG{1,4}).
128
129- Shortcuts for small tuples (TUPLE{1,2,3}}.
130
131- Dedicated opcodes for bools (NEWTRUE, NEWFALSE).
132
133- The "extension registry", a vector of popular objects that can be pushed
134 efficiently by index (EXT{1,2,4}). This is akin to the memo and GET, but
135 the registry contents are predefined (there's nothing akin to the memo's
136 PUT).
Guido van Rossumecb11042003-01-29 06:24:30 +0000137
Skip Montanaro54455942003-01-29 15:41:33 +0000138Another independent change with Python 2.3 is the abandonment of any
139pretense that it might be safe to load pickles received from untrusted
Guido van Rossumecb11042003-01-29 06:24:30 +0000140parties -- no sufficient security analysis has been done to guarantee
Skip Montanaro54455942003-01-29 15:41:33 +0000141this and there isn't a use case that warrants the expense of such an
Guido van Rossumecb11042003-01-29 06:24:30 +0000142analysis.
143
144To this end, all tests for __safe_for_unpickling__ or for
Alexandre Vassalottif7fa63d2008-05-11 08:55:36 +0000145copyreg.safe_constructors are removed from the unpickling code.
Guido van Rossumecb11042003-01-29 06:24:30 +0000146References to these variables in the descriptions below are to be seen
147as describing unpickling in Python 2.2 and before.
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000148"""
149
150# Meta-rule: Descriptions are stored in instances of descriptor objects,
151# with plain constructors. No meta-language is defined from which
152# descriptors could be constructed. If you want, e.g., XML, write a little
153# program to generate XML from the objects.
154
155##############################################################################
156# Some pickle opcodes have an argument, following the opcode in the
157# bytestream. An argument is of a specific type, described by an instance
158# of ArgumentDescriptor. These are not to be confused with arguments taken
159# off the stack -- ArgumentDescriptor applies only to arguments embedded in
160# the opcode stream, immediately following an opcode.
161
162# Represents the number of bytes consumed by an argument delimited by the
163# next newline character.
164UP_TO_NEWLINE = -1
165
166# Represents the number of bytes consumed by a two-argument opcode where
167# the first argument gives the number of bytes in the second argument.
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000168TAKEN_FROM_ARGUMENT1 = -2 # num bytes is 1-byte unsigned int
169TAKEN_FROM_ARGUMENT4 = -3 # num bytes is 4-byte signed little-endian int
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000170
171class ArgumentDescriptor(object):
172 __slots__ = (
173 # name of descriptor record, also a module global name; a string
174 'name',
175
176 # length of argument, in bytes; an int; UP_TO_NEWLINE and
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000177 # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length
178 # cases
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000179 'n',
180
181 # a function taking a file-like object, reading this kind of argument
182 # from the object at the current position, advancing the current
183 # position by n bytes, and returning the value of the argument
184 'reader',
185
186 # human-readable docs for this arg descriptor; a string
187 'doc',
188 )
189
190 def __init__(self, name, n, reader, doc):
191 assert isinstance(name, str)
192 self.name = name
193
194 assert isinstance(n, int) and (n >= 0 or
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000195 n in (UP_TO_NEWLINE,
196 TAKEN_FROM_ARGUMENT1,
197 TAKEN_FROM_ARGUMENT4))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000198 self.n = n
199
200 self.reader = reader
201
202 assert isinstance(doc, str)
203 self.doc = doc
204
205from struct import unpack as _unpack
206
207def read_uint1(f):
Tim Peters55762f52003-01-28 16:01:25 +0000208 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000209 >>> import io
210 >>> read_uint1(io.BytesIO(b'\xff'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000211 255
212 """
213
214 data = f.read(1)
215 if data:
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000216 return data[0]
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000217 raise ValueError("not enough data in stream to read uint1")
218
219uint1 = ArgumentDescriptor(
220 name='uint1',
221 n=1,
222 reader=read_uint1,
223 doc="One-byte unsigned integer.")
224
225
226def read_uint2(f):
Tim Peters55762f52003-01-28 16:01:25 +0000227 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000228 >>> import io
229 >>> read_uint2(io.BytesIO(b'\xff\x00'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000230 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000231 >>> read_uint2(io.BytesIO(b'\xff\xff'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000232 65535
233 """
234
235 data = f.read(2)
236 if len(data) == 2:
237 return _unpack("<H", data)[0]
238 raise ValueError("not enough data in stream to read uint2")
239
240uint2 = ArgumentDescriptor(
241 name='uint2',
242 n=2,
243 reader=read_uint2,
244 doc="Two-byte unsigned integer, little-endian.")
245
246
247def read_int4(f):
Tim Peters55762f52003-01-28 16:01:25 +0000248 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000249 >>> import io
250 >>> read_int4(io.BytesIO(b'\xff\x00\x00\x00'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000251 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000252 >>> read_int4(io.BytesIO(b'\x00\x00\x00\x80')) == -(2**31)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000253 True
254 """
255
256 data = f.read(4)
257 if len(data) == 4:
258 return _unpack("<i", data)[0]
259 raise ValueError("not enough data in stream to read int4")
260
261int4 = ArgumentDescriptor(
262 name='int4',
263 n=4,
264 reader=read_int4,
265 doc="Four-byte signed integer, little-endian, 2's complement.")
266
267
268def read_stringnl(f, decode=True, stripquotes=True):
Tim Peters55762f52003-01-28 16:01:25 +0000269 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000270 >>> import io
271 >>> read_stringnl(io.BytesIO(b"'abcd'\nefg\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000272 'abcd'
273
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000274 >>> read_stringnl(io.BytesIO(b"\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000275 Traceback (most recent call last):
276 ...
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000277 ValueError: no string quotes around b''
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000278
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000279 >>> read_stringnl(io.BytesIO(b"\n"), stripquotes=False)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000280 ''
281
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000282 >>> read_stringnl(io.BytesIO(b"''\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000283 ''
284
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000285 >>> read_stringnl(io.BytesIO(b'"abcd"'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000286 Traceback (most recent call last):
287 ...
288 ValueError: no newline found when trying to read stringnl
289
290 Embedded escapes are undone in the result.
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000291 >>> read_stringnl(io.BytesIO(br"'a\n\\b\x00c\td'" + b"\n'e'"))
Tim Peters55762f52003-01-28 16:01:25 +0000292 'a\n\\b\x00c\td'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000293 """
294
Guido van Rossum26986312007-07-17 00:19:46 +0000295 data = f.readline()
Guido van Rossum26d95c32007-08-27 23:18:54 +0000296 if not data.endswith(b'\n'):
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000297 raise ValueError("no newline found when trying to read stringnl")
298 data = data[:-1] # lose the newline
299
300 if stripquotes:
Guido van Rossum26d95c32007-08-27 23:18:54 +0000301 for q in (b'"', b"'"):
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000302 if data.startswith(q):
303 if not data.endswith(q):
304 raise ValueError("strinq quote %r not found at both "
305 "ends of %r" % (q, data))
306 data = data[1:-1]
307 break
308 else:
309 raise ValueError("no string quotes around %r" % data)
310
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000311 if decode:
Guido van Rossum98297ee2007-11-06 21:34:58 +0000312 data = codecs.escape_decode(data)[0].decode("ascii")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000313 return data
314
315stringnl = ArgumentDescriptor(
316 name='stringnl',
317 n=UP_TO_NEWLINE,
318 reader=read_stringnl,
319 doc="""A newline-terminated string.
320
321 This is a repr-style string, with embedded escapes, and
322 bracketing quotes.
323 """)
324
325def read_stringnl_noescape(f):
Guido van Rossum98297ee2007-11-06 21:34:58 +0000326 return read_stringnl(f, stripquotes=False)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000327
328stringnl_noescape = ArgumentDescriptor(
329 name='stringnl_noescape',
330 n=UP_TO_NEWLINE,
331 reader=read_stringnl_noescape,
332 doc="""A newline-terminated string.
333
334 This is a str-style string, without embedded escapes,
335 or bracketing quotes. It should consist solely of
336 printable ASCII characters.
337 """)
338
339def read_stringnl_noescape_pair(f):
Tim Peters55762f52003-01-28 16:01:25 +0000340 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000341 >>> import io
342 >>> read_stringnl_noescape_pair(io.BytesIO(b"Queue\nEmpty\njunk"))
Tim Petersd916cf42003-01-27 19:01:47 +0000343 'Queue Empty'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000344 """
345
Tim Petersd916cf42003-01-27 19:01:47 +0000346 return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000347
348stringnl_noescape_pair = ArgumentDescriptor(
349 name='stringnl_noescape_pair',
350 n=UP_TO_NEWLINE,
351 reader=read_stringnl_noescape_pair,
352 doc="""A pair of newline-terminated strings.
353
354 These are str-style strings, without embedded
355 escapes, or bracketing quotes. They should
356 consist solely of printable ASCII characters.
357 The pair is returned as a single string, with
Tim Petersd916cf42003-01-27 19:01:47 +0000358 a single blank separating the two strings.
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000359 """)
360
361def read_string4(f):
Tim Peters55762f52003-01-28 16:01:25 +0000362 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000363 >>> import io
364 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x00abc"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000365 ''
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000366 >>> read_string4(io.BytesIO(b"\x03\x00\x00\x00abcdef"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000367 'abc'
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000368 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x03abcdef"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000369 Traceback (most recent call last):
370 ...
371 ValueError: expected 50331648 bytes in a string4, but only 6 remain
372 """
373
374 n = read_int4(f)
375 if n < 0:
376 raise ValueError("string4 byte count < 0: %d" % n)
377 data = f.read(n)
378 if len(data) == n:
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000379 return data.decode("latin-1")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000380 raise ValueError("expected %d bytes in a string4, but only %d remain" %
381 (n, len(data)))
382
383string4 = ArgumentDescriptor(
384 name="string4",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000385 n=TAKEN_FROM_ARGUMENT4,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000386 reader=read_string4,
387 doc="""A counted string.
388
389 The first argument is a 4-byte little-endian signed int giving
390 the number of bytes in the string, and the second argument is
391 that many bytes.
392 """)
393
394
395def read_string1(f):
Tim Peters55762f52003-01-28 16:01:25 +0000396 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000397 >>> import io
398 >>> read_string1(io.BytesIO(b"\x00"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000399 ''
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000400 >>> read_string1(io.BytesIO(b"\x03abcdef"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000401 'abc'
402 """
403
404 n = read_uint1(f)
405 assert n >= 0
406 data = f.read(n)
407 if len(data) == n:
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000408 return data.decode("latin-1")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000409 raise ValueError("expected %d bytes in a string1, but only %d remain" %
410 (n, len(data)))
411
412string1 = ArgumentDescriptor(
413 name="string1",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000414 n=TAKEN_FROM_ARGUMENT1,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000415 reader=read_string1,
416 doc="""A counted string.
417
418 The first argument is a 1-byte unsigned int giving the number
419 of bytes in the string, and the second argument is that many
420 bytes.
421 """)
422
423
424def read_unicodestringnl(f):
Tim Peters55762f52003-01-28 16:01:25 +0000425 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000426 >>> import io
427 >>> read_unicodestringnl(io.BytesIO(b"abc\\uabcd\njunk")) == 'abc\uabcd'
428 True
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000429 """
430
Guido van Rossum26986312007-07-17 00:19:46 +0000431 data = f.readline()
Guido van Rossum26d95c32007-08-27 23:18:54 +0000432 if not data.endswith(b'\n'):
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000433 raise ValueError("no newline found when trying to read "
434 "unicodestringnl")
435 data = data[:-1] # lose the newline
Guido van Rossumef87d6e2007-05-02 19:09:54 +0000436 return str(data, 'raw-unicode-escape')
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000437
438unicodestringnl = ArgumentDescriptor(
439 name='unicodestringnl',
440 n=UP_TO_NEWLINE,
441 reader=read_unicodestringnl,
442 doc="""A newline-terminated Unicode string.
443
444 This is raw-unicode-escape encoded, so consists of
445 printable ASCII characters, and may contain embedded
446 escape sequences.
447 """)
448
449def read_unicodestring4(f):
Tim Peters55762f52003-01-28 16:01:25 +0000450 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000451 >>> import io
452 >>> s = 'abcd\uabcd'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000453 >>> enc = s.encode('utf-8')
454 >>> enc
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000455 b'abcd\xea\xaf\x8d'
456 >>> n = bytes([len(enc), 0, 0, 0]) # little-endian 4-byte length
457 >>> t = read_unicodestring4(io.BytesIO(n + enc + b'junk'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000458 >>> s == t
459 True
460
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000461 >>> read_unicodestring4(io.BytesIO(n + enc[:-1]))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000462 Traceback (most recent call last):
463 ...
464 ValueError: expected 7 bytes in a unicodestring4, but only 6 remain
465 """
466
467 n = read_int4(f)
468 if n < 0:
469 raise ValueError("unicodestring4 byte count < 0: %d" % n)
470 data = f.read(n)
471 if len(data) == n:
Victor Stinner485fb562010-04-13 11:07:24 +0000472 return str(data, 'utf-8', 'surrogatepass')
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000473 raise ValueError("expected %d bytes in a unicodestring4, but only %d "
474 "remain" % (n, len(data)))
475
476unicodestring4 = ArgumentDescriptor(
477 name="unicodestring4",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000478 n=TAKEN_FROM_ARGUMENT4,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000479 reader=read_unicodestring4,
480 doc="""A counted Unicode string.
481
482 The first argument is a 4-byte little-endian signed int
483 giving the number of bytes in the string, and the second
484 argument-- the UTF-8 encoding of the Unicode string --
485 contains that many bytes.
486 """)
487
488
489def read_decimalnl_short(f):
Tim Peters55762f52003-01-28 16:01:25 +0000490 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000491 >>> import io
492 >>> read_decimalnl_short(io.BytesIO(b"1234\n56"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000493 1234
494
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000495 >>> read_decimalnl_short(io.BytesIO(b"1234L\n56"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000496 Traceback (most recent call last):
497 ...
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000498 ValueError: trailing 'L' not allowed in b'1234L'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000499 """
500
501 s = read_stringnl(f, decode=False, stripquotes=False)
Guido van Rossum26d95c32007-08-27 23:18:54 +0000502 if s.endswith(b"L"):
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000503 raise ValueError("trailing 'L' not allowed in %r" % s)
504
505 # It's not necessarily true that the result fits in a Python short int:
506 # the pickle may have been written on a 64-bit box. There's also a hack
507 # for True and False here.
Jeremy Hyltona5dc3db2007-08-29 19:07:40 +0000508 if s == b"00":
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000509 return False
Jeremy Hyltona5dc3db2007-08-29 19:07:40 +0000510 elif s == b"01":
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000511 return True
512
Florent Xicluna2bb96f52011-10-23 22:11:00 +0200513 return int(s)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000514
515def read_decimalnl_long(f):
Tim Peters55762f52003-01-28 16:01:25 +0000516 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000517 >>> import io
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000518
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000519 >>> read_decimalnl_long(io.BytesIO(b"1234L\n56"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000520 1234
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000521
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000522 >>> read_decimalnl_long(io.BytesIO(b"123456789012345678901234L\n6"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000523 123456789012345678901234
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000524 """
525
526 s = read_stringnl(f, decode=False, stripquotes=False)
Mark Dickinson8dd05142009-01-20 20:43:58 +0000527 if s[-1:] == b'L':
528 s = s[:-1]
Guido van Rossume2a383d2007-01-15 16:59:06 +0000529 return int(s)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000530
531
532decimalnl_short = ArgumentDescriptor(
533 name='decimalnl_short',
534 n=UP_TO_NEWLINE,
535 reader=read_decimalnl_short,
536 doc="""A newline-terminated decimal integer literal.
537
538 This never has a trailing 'L', and the integer fit
539 in a short Python int on the box where the pickle
540 was written -- but there's no guarantee it will fit
541 in a short Python int on the box where the pickle
542 is read.
543 """)
544
545decimalnl_long = ArgumentDescriptor(
546 name='decimalnl_long',
547 n=UP_TO_NEWLINE,
548 reader=read_decimalnl_long,
549 doc="""A newline-terminated decimal integer literal.
550
551 This has a trailing 'L', and can represent integers
552 of any size.
553 """)
554
555
556def read_floatnl(f):
Tim Peters55762f52003-01-28 16:01:25 +0000557 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000558 >>> import io
559 >>> read_floatnl(io.BytesIO(b"-1.25\n6"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000560 -1.25
561 """
562 s = read_stringnl(f, decode=False, stripquotes=False)
563 return float(s)
564
565floatnl = ArgumentDescriptor(
566 name='floatnl',
567 n=UP_TO_NEWLINE,
568 reader=read_floatnl,
569 doc="""A newline-terminated decimal floating literal.
570
571 In general this requires 17 significant digits for roundtrip
572 identity, and pickling then unpickling infinities, NaNs, and
573 minus zero doesn't work across boxes, or on some boxes even
574 on itself (e.g., Windows can't read the strings it produces
575 for infinities or NaNs).
576 """)
577
578def read_float8(f):
Tim Peters55762f52003-01-28 16:01:25 +0000579 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000580 >>> import io, struct
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000581 >>> raw = struct.pack(">d", -1.25)
582 >>> raw
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000583 b'\xbf\xf4\x00\x00\x00\x00\x00\x00'
584 >>> read_float8(io.BytesIO(raw + b"\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000585 -1.25
586 """
587
588 data = f.read(8)
589 if len(data) == 8:
590 return _unpack(">d", data)[0]
591 raise ValueError("not enough data in stream to read float8")
592
593
594float8 = ArgumentDescriptor(
595 name='float8',
596 n=8,
597 reader=read_float8,
598 doc="""An 8-byte binary representation of a float, big-endian.
599
600 The format is unique to Python, and shared with the struct
Guido van Rossum99603b02007-07-20 00:22:32 +0000601 module (format string '>d') "in theory" (the struct and pickle
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000602 implementations don't share the code -- they should). It's
603 strongly related to the IEEE-754 double format, and, in normal
604 cases, is in fact identical to the big-endian 754 double format.
605 On other boxes the dynamic range is limited to that of a 754
606 double, and "add a half and chop" rounding is used to reduce
607 the precision to 53 bits. However, even on a 754 box,
608 infinities, NaNs, and minus zero may not be handled correctly
609 (may not survive roundtrip pickling intact).
610 """)
611
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000612# Protocol 2 formats
613
Tim Petersc0c12b52003-01-29 00:56:17 +0000614from pickle import decode_long
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000615
616def read_long1(f):
617 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000618 >>> import io
619 >>> read_long1(io.BytesIO(b"\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000620 0
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000621 >>> read_long1(io.BytesIO(b"\x02\xff\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000622 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000623 >>> read_long1(io.BytesIO(b"\x02\xff\x7f"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000624 32767
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000625 >>> read_long1(io.BytesIO(b"\x02\x00\xff"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000626 -256
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000627 >>> read_long1(io.BytesIO(b"\x02\x00\x80"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000628 -32768
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000629 """
630
631 n = read_uint1(f)
632 data = f.read(n)
633 if len(data) != n:
634 raise ValueError("not enough data in stream to read long1")
635 return decode_long(data)
636
637long1 = ArgumentDescriptor(
638 name="long1",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000639 n=TAKEN_FROM_ARGUMENT1,
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000640 reader=read_long1,
641 doc="""A binary long, little-endian, using 1-byte size.
642
643 This first reads one byte as an unsigned size, then reads that
Tim Petersbdbe7412003-01-27 23:54:04 +0000644 many bytes and interprets them as a little-endian 2's-complement long.
Tim Peters4b23f2b2003-01-31 16:43:39 +0000645 If the size is 0, that's taken as a shortcut for the long 0L.
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000646 """)
647
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000648def read_long4(f):
649 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000650 >>> import io
651 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000652 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000653 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x7f"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000654 32767
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000655 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\xff"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000656 -256
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000657 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\x80"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000658 -32768
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000659 >>> read_long1(io.BytesIO(b"\x00\x00\x00\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000660 0
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000661 """
662
663 n = read_int4(f)
664 if n < 0:
Neal Norwitz784a3f52003-01-28 00:20:41 +0000665 raise ValueError("long4 byte count < 0: %d" % n)
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000666 data = f.read(n)
667 if len(data) != n:
Neal Norwitz784a3f52003-01-28 00:20:41 +0000668 raise ValueError("not enough data in stream to read long4")
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000669 return decode_long(data)
670
671long4 = ArgumentDescriptor(
672 name="long4",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000673 n=TAKEN_FROM_ARGUMENT4,
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000674 reader=read_long4,
675 doc="""A binary representation of a long, little-endian.
676
677 This first reads four bytes as a signed size (but requires the
678 size to be >= 0), then reads that many bytes and interprets them
Tim Peters4b23f2b2003-01-31 16:43:39 +0000679 as a little-endian 2's-complement long. If the size is 0, that's taken
Guido van Rossume2a383d2007-01-15 16:59:06 +0000680 as a shortcut for the int 0, although LONG1 should really be used
Tim Peters4b23f2b2003-01-31 16:43:39 +0000681 then instead (and in any case where # of bytes < 256).
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000682 """)
683
684
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000685##############################################################################
686# Object descriptors. The stack used by the pickle machine holds objects,
687# and in the stack_before and stack_after attributes of OpcodeInfo
688# descriptors we need names to describe the various types of objects that can
689# appear on the stack.
690
691class StackObject(object):
692 __slots__ = (
693 # name of descriptor record, for info only
694 'name',
695
696 # type of object, or tuple of type objects (meaning the object can
697 # be of any type in the tuple)
698 'obtype',
699
700 # human-readable docs for this kind of stack object; a string
701 'doc',
702 )
703
704 def __init__(self, name, obtype, doc):
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000705 assert isinstance(name, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000706 self.name = name
707
708 assert isinstance(obtype, type) or isinstance(obtype, tuple)
709 if isinstance(obtype, tuple):
710 for contained in obtype:
711 assert isinstance(contained, type)
712 self.obtype = obtype
713
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000714 assert isinstance(doc, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000715 self.doc = doc
716
Tim Petersc1c2b3e2003-01-29 20:12:21 +0000717 def __repr__(self):
718 return self.name
719
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000720
721pyint = StackObject(
722 name='int',
723 obtype=int,
724 doc="A short (as opposed to long) Python integer object.")
725
726pylong = StackObject(
727 name='long',
Guido van Rossume2a383d2007-01-15 16:59:06 +0000728 obtype=int,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000729 doc="A long (as opposed to short) Python integer object.")
730
731pyinteger_or_bool = StackObject(
732 name='int_or_bool',
Florent Xicluna02ea12b22010-07-28 16:39:41 +0000733 obtype=(int, bool),
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000734 doc="A Python integer object (short or long), or "
735 "a Python bool.")
736
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000737pybool = StackObject(
738 name='bool',
739 obtype=(bool,),
740 doc="A Python bool object.")
741
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000742pyfloat = StackObject(
743 name='float',
744 obtype=float,
745 doc="A Python float object.")
746
747pystring = StackObject(
Guido van Rossumf4169812008-03-17 22:56:06 +0000748 name='string',
749 obtype=bytes,
750 doc="A Python (8-bit) string object.")
751
752pybytes = StackObject(
Guido van Rossum98297ee2007-11-06 21:34:58 +0000753 name='bytes',
754 obtype=bytes,
755 doc="A Python bytes object.")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000756
757pyunicode = StackObject(
Guido van Rossum98297ee2007-11-06 21:34:58 +0000758 name='str',
Guido van Rossumef87d6e2007-05-02 19:09:54 +0000759 obtype=str,
Guido van Rossumf4169812008-03-17 22:56:06 +0000760 doc="A Python (Unicode) string object.")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000761
762pynone = StackObject(
763 name="None",
764 obtype=type(None),
765 doc="The Python None object.")
766
767pytuple = StackObject(
768 name="tuple",
769 obtype=tuple,
770 doc="A Python tuple object.")
771
772pylist = StackObject(
773 name="list",
774 obtype=list,
775 doc="A Python list object.")
776
777pydict = StackObject(
778 name="dict",
779 obtype=dict,
780 doc="A Python dict object.")
781
782anyobject = StackObject(
783 name='any',
784 obtype=object,
785 doc="Any kind of object whatsoever.")
786
787markobject = StackObject(
788 name="mark",
789 obtype=StackObject,
790 doc="""'The mark' is a unique object.
791
792 Opcodes that operate on a variable number of objects
793 generally don't embed the count of objects in the opcode,
794 or pull it off the stack. Instead the MARK opcode is used
795 to push a special marker object on the stack, and then
796 some other opcodes grab all the objects from the top of
797 the stack down to (but not including) the topmost marker
798 object.
799 """)
800
801stackslice = StackObject(
802 name="stackslice",
803 obtype=StackObject,
804 doc="""An object representing a contiguous slice of the stack.
805
806 This is used in conjuction with markobject, to represent all
807 of the stack following the topmost markobject. For example,
808 the POP_MARK opcode changes the stack from
809
810 [..., markobject, stackslice]
811 to
812 [...]
813
814 No matter how many object are on the stack after the topmost
815 markobject, POP_MARK gets rid of all of them (including the
816 topmost markobject too).
817 """)
818
819##############################################################################
820# Descriptors for pickle opcodes.
821
822class OpcodeInfo(object):
823
824 __slots__ = (
825 # symbolic name of opcode; a string
826 'name',
827
828 # the code used in a bytestream to represent the opcode; a
829 # one-character string
830 'code',
831
832 # If the opcode has an argument embedded in the byte string, an
833 # instance of ArgumentDescriptor specifying its type. Note that
834 # arg.reader(s) can be used to read and decode the argument from
835 # the bytestream s, and arg.doc documents the format of the raw
836 # argument bytes. If the opcode doesn't have an argument embedded
837 # in the bytestream, arg should be None.
838 'arg',
839
840 # what the stack looks like before this opcode runs; a list
841 'stack_before',
842
843 # what the stack looks like after this opcode runs; a list
844 'stack_after',
845
846 # the protocol number in which this opcode was introduced; an int
847 'proto',
848
849 # human-readable docs for this opcode; a string
850 'doc',
851 )
852
853 def __init__(self, name, code, arg,
854 stack_before, stack_after, proto, doc):
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000855 assert isinstance(name, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000856 self.name = name
857
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000858 assert isinstance(code, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000859 assert len(code) == 1
860 self.code = code
861
862 assert arg is None or isinstance(arg, ArgumentDescriptor)
863 self.arg = arg
864
865 assert isinstance(stack_before, list)
866 for x in stack_before:
867 assert isinstance(x, StackObject)
868 self.stack_before = stack_before
869
870 assert isinstance(stack_after, list)
871 for x in stack_after:
872 assert isinstance(x, StackObject)
873 self.stack_after = stack_after
874
Guido van Rossumf4169812008-03-17 22:56:06 +0000875 assert isinstance(proto, int) and 0 <= proto <= 3
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000876 self.proto = proto
877
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000878 assert isinstance(doc, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000879 self.doc = doc
880
881I = OpcodeInfo
882opcodes = [
883
884 # Ways to spell integers.
885
886 I(name='INT',
887 code='I',
888 arg=decimalnl_short,
889 stack_before=[],
890 stack_after=[pyinteger_or_bool],
891 proto=0,
892 doc="""Push an integer or bool.
893
894 The argument is a newline-terminated decimal literal string.
895
896 The intent may have been that this always fit in a short Python int,
897 but INT can be generated in pickles written on a 64-bit box that
898 require a Python long on a 32-bit box. The difference between this
899 and LONG then is that INT skips a trailing 'L', and produces a short
900 int whenever possible.
901
902 Another difference is due to that, when bool was introduced as a
903 distinct type in 2.3, builtin names True and False were also added to
904 2.2.2, mapping to ints 1 and 0. For compatibility in both directions,
905 True gets pickled as INT + "I01\\n", and False as INT + "I00\\n".
906 Leading zeroes are never produced for a genuine integer. The 2.3
907 (and later) unpicklers special-case these and return bool instead;
908 earlier unpicklers ignore the leading "0" and return the int.
909 """),
910
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000911 I(name='BININT',
912 code='J',
913 arg=int4,
914 stack_before=[],
915 stack_after=[pyint],
916 proto=1,
917 doc="""Push a four-byte signed integer.
918
919 This handles the full range of Python (short) integers on a 32-bit
920 box, directly as binary bytes (1 for the opcode and 4 for the integer).
921 If the integer is non-negative and fits in 1 or 2 bytes, pickling via
922 BININT1 or BININT2 saves space.
923 """),
924
925 I(name='BININT1',
926 code='K',
927 arg=uint1,
928 stack_before=[],
929 stack_after=[pyint],
930 proto=1,
931 doc="""Push a one-byte unsigned integer.
932
933 This is a space optimization for pickling very small non-negative ints,
934 in range(256).
935 """),
936
937 I(name='BININT2',
938 code='M',
939 arg=uint2,
940 stack_before=[],
941 stack_after=[pyint],
942 proto=1,
943 doc="""Push a two-byte unsigned integer.
944
945 This is a space optimization for pickling small positive ints, in
946 range(256, 2**16). Integers in range(256) can also be pickled via
947 BININT2, but BININT1 instead saves a byte.
948 """),
949
Tim Petersfdc03462003-01-28 04:56:33 +0000950 I(name='LONG',
951 code='L',
952 arg=decimalnl_long,
953 stack_before=[],
954 stack_after=[pylong],
955 proto=0,
956 doc="""Push a long integer.
957
958 The same as INT, except that the literal ends with 'L', and always
959 unpickles to a Python long. There doesn't seem a real purpose to the
960 trailing 'L'.
961
962 Note that LONG takes time quadratic in the number of digits when
963 unpickling (this is simply due to the nature of decimal->binary
964 conversion). Proto 2 added linear-time (in C; still quadratic-time
965 in Python) LONG1 and LONG4 opcodes.
966 """),
967
968 I(name="LONG1",
969 code='\x8a',
970 arg=long1,
971 stack_before=[],
972 stack_after=[pylong],
973 proto=2,
974 doc="""Long integer using one-byte length.
975
976 A more efficient encoding of a Python long; the long1 encoding
977 says it all."""),
978
979 I(name="LONG4",
980 code='\x8b',
981 arg=long4,
982 stack_before=[],
983 stack_after=[pylong],
984 proto=2,
985 doc="""Long integer using found-byte length.
986
987 A more efficient encoding of a Python long; the long4 encoding
988 says it all."""),
989
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000990 # Ways to spell strings (8-bit, not Unicode).
991
992 I(name='STRING',
993 code='S',
994 arg=stringnl,
995 stack_before=[],
996 stack_after=[pystring],
997 proto=0,
998 doc="""Push a Python string object.
999
1000 The argument is a repr-style string, with bracketing quote characters,
1001 and perhaps embedded escapes. The argument extends until the next
Guido van Rossumf4169812008-03-17 22:56:06 +00001002 newline character. (Actually, they are decoded into a str instance
1003 using the encoding given to the Unpickler constructor. or the default,
1004 'ASCII'.)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001005 """),
1006
1007 I(name='BINSTRING',
1008 code='T',
1009 arg=string4,
1010 stack_before=[],
1011 stack_after=[pystring],
1012 proto=1,
1013 doc="""Push a Python string object.
1014
1015 There are two arguments: the first is a 4-byte little-endian signed int
1016 giving the number of bytes in the string, and the second is that many
Guido van Rossumf4169812008-03-17 22:56:06 +00001017 bytes, which are taken literally as the string content. (Actually,
1018 they are decoded into a str instance using the encoding given to the
1019 Unpickler constructor. or the default, 'ASCII'.)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001020 """),
1021
1022 I(name='SHORT_BINSTRING',
1023 code='U',
1024 arg=string1,
1025 stack_before=[],
1026 stack_after=[pystring],
1027 proto=1,
1028 doc="""Push a Python string object.
1029
1030 There are two arguments: the first is a 1-byte unsigned int giving
1031 the number of bytes in the string, and the second is that many bytes,
Guido van Rossumf4169812008-03-17 22:56:06 +00001032 which are taken literally as the string content. (Actually, they
1033 are decoded into a str instance using the encoding given to the
1034 Unpickler constructor. or the default, 'ASCII'.)
1035 """),
1036
1037 # Bytes (protocol 3 only; older protocols don't support bytes at all)
1038
1039 I(name='BINBYTES',
1040 code='B',
1041 arg=string4,
1042 stack_before=[],
1043 stack_after=[pybytes],
1044 proto=3,
1045 doc="""Push a Python bytes object.
1046
1047 There are two arguments: the first is a 4-byte little-endian signed int
1048 giving the number of bytes in the string, and the second is that many
1049 bytes, which are taken literally as the bytes content.
1050 """),
1051
1052 I(name='SHORT_BINBYTES',
1053 code='C',
1054 arg=string1,
1055 stack_before=[],
1056 stack_after=[pybytes],
Collin Wintere61d4372009-05-20 17:46:47 +00001057 proto=3,
Guido van Rossumf4169812008-03-17 22:56:06 +00001058 doc="""Push a Python string object.
1059
1060 There are two arguments: the first is a 1-byte unsigned int giving
1061 the number of bytes in the string, and the second is that many bytes,
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001062 which are taken literally as the string content.
1063 """),
1064
1065 # Ways to spell None.
1066
1067 I(name='NONE',
1068 code='N',
1069 arg=None,
1070 stack_before=[],
1071 stack_after=[pynone],
1072 proto=0,
1073 doc="Push None on the stack."),
1074
Tim Petersfdc03462003-01-28 04:56:33 +00001075 # Ways to spell bools, starting with proto 2. See INT for how this was
1076 # done before proto 2.
1077
1078 I(name='NEWTRUE',
1079 code='\x88',
1080 arg=None,
1081 stack_before=[],
1082 stack_after=[pybool],
1083 proto=2,
1084 doc="""True.
1085
1086 Push True onto the stack."""),
1087
1088 I(name='NEWFALSE',
1089 code='\x89',
1090 arg=None,
1091 stack_before=[],
1092 stack_after=[pybool],
1093 proto=2,
1094 doc="""True.
1095
1096 Push False onto the stack."""),
1097
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001098 # Ways to spell Unicode strings.
1099
1100 I(name='UNICODE',
1101 code='V',
1102 arg=unicodestringnl,
1103 stack_before=[],
1104 stack_after=[pyunicode],
1105 proto=0, # this may be pure-text, but it's a later addition
1106 doc="""Push a Python Unicode string object.
1107
1108 The argument is a raw-unicode-escape encoding of a Unicode string,
1109 and so may contain embedded escape sequences. The argument extends
1110 until the next newline character.
1111 """),
1112
1113 I(name='BINUNICODE',
1114 code='X',
1115 arg=unicodestring4,
1116 stack_before=[],
1117 stack_after=[pyunicode],
1118 proto=1,
1119 doc="""Push a Python Unicode string object.
1120
1121 There are two arguments: the first is a 4-byte little-endian signed int
1122 giving the number of bytes in the string. The second is that many
1123 bytes, and is the UTF-8 encoding of the Unicode string.
1124 """),
1125
1126 # Ways to spell floats.
1127
1128 I(name='FLOAT',
1129 code='F',
1130 arg=floatnl,
1131 stack_before=[],
1132 stack_after=[pyfloat],
1133 proto=0,
1134 doc="""Newline-terminated decimal float literal.
1135
1136 The argument is repr(a_float), and in general requires 17 significant
1137 digits for roundtrip conversion to be an identity (this is so for
1138 IEEE-754 double precision values, which is what Python float maps to
1139 on most boxes).
1140
1141 In general, FLOAT cannot be used to transport infinities, NaNs, or
1142 minus zero across boxes (or even on a single box, if the platform C
1143 library can't read the strings it produces for such things -- Windows
1144 is like that), but may do less damage than BINFLOAT on boxes with
1145 greater precision or dynamic range than IEEE-754 double.
1146 """),
1147
1148 I(name='BINFLOAT',
1149 code='G',
1150 arg=float8,
1151 stack_before=[],
1152 stack_after=[pyfloat],
1153 proto=1,
1154 doc="""Float stored in binary form, with 8 bytes of data.
1155
1156 This generally requires less than half the space of FLOAT encoding.
1157 In general, BINFLOAT cannot be used to transport infinities, NaNs, or
1158 minus zero, raises an exception if the exponent exceeds the range of
1159 an IEEE-754 double, and retains no more than 53 bits of precision (if
1160 there are more than that, "add a half and chop" rounding is used to
1161 cut it back to 53 significant bits).
1162 """),
1163
1164 # Ways to build lists.
1165
1166 I(name='EMPTY_LIST',
1167 code=']',
1168 arg=None,
1169 stack_before=[],
1170 stack_after=[pylist],
1171 proto=1,
1172 doc="Push an empty list."),
1173
1174 I(name='APPEND',
1175 code='a',
1176 arg=None,
1177 stack_before=[pylist, anyobject],
1178 stack_after=[pylist],
1179 proto=0,
1180 doc="""Append an object to a list.
1181
1182 Stack before: ... pylist anyobject
1183 Stack after: ... pylist+[anyobject]
Tim Peters81098ac2003-01-28 05:12:08 +00001184
1185 although pylist is really extended in-place.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001186 """),
1187
1188 I(name='APPENDS',
1189 code='e',
1190 arg=None,
1191 stack_before=[pylist, markobject, stackslice],
1192 stack_after=[pylist],
1193 proto=1,
1194 doc="""Extend a list by a slice of stack objects.
1195
1196 Stack before: ... pylist markobject stackslice
1197 Stack after: ... pylist+stackslice
Tim Peters81098ac2003-01-28 05:12:08 +00001198
1199 although pylist is really extended in-place.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001200 """),
1201
1202 I(name='LIST',
1203 code='l',
1204 arg=None,
1205 stack_before=[markobject, stackslice],
1206 stack_after=[pylist],
1207 proto=0,
1208 doc="""Build a list out of the topmost stack slice, after markobject.
1209
1210 All the stack entries following the topmost markobject are placed into
1211 a single Python list, which single list object replaces all of the
1212 stack from the topmost markobject onward. For example,
1213
1214 Stack before: ... markobject 1 2 3 'abc'
1215 Stack after: ... [1, 2, 3, 'abc']
1216 """),
1217
1218 # Ways to build tuples.
1219
1220 I(name='EMPTY_TUPLE',
1221 code=')',
1222 arg=None,
1223 stack_before=[],
1224 stack_after=[pytuple],
1225 proto=1,
1226 doc="Push an empty tuple."),
1227
1228 I(name='TUPLE',
1229 code='t',
1230 arg=None,
1231 stack_before=[markobject, stackslice],
1232 stack_after=[pytuple],
1233 proto=0,
1234 doc="""Build a tuple out of the topmost stack slice, after markobject.
1235
1236 All the stack entries following the topmost markobject are placed into
1237 a single Python tuple, which single tuple object replaces all of the
1238 stack from the topmost markobject onward. For example,
1239
1240 Stack before: ... markobject 1 2 3 'abc'
1241 Stack after: ... (1, 2, 3, 'abc')
1242 """),
1243
Tim Petersfdc03462003-01-28 04:56:33 +00001244 I(name='TUPLE1',
1245 code='\x85',
1246 arg=None,
1247 stack_before=[anyobject],
1248 stack_after=[pytuple],
1249 proto=2,
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001250 doc="""Build a one-tuple out of the topmost item on the stack.
Tim Petersfdc03462003-01-28 04:56:33 +00001251
1252 This code pops one value off the stack and pushes a tuple of
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001253 length 1 whose one item is that value back onto it. In other
1254 words:
Tim Petersfdc03462003-01-28 04:56:33 +00001255
1256 stack[-1] = tuple(stack[-1:])
1257 """),
1258
1259 I(name='TUPLE2',
1260 code='\x86',
1261 arg=None,
1262 stack_before=[anyobject, anyobject],
1263 stack_after=[pytuple],
1264 proto=2,
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001265 doc="""Build a two-tuple out of the top two items on the stack.
Tim Petersfdc03462003-01-28 04:56:33 +00001266
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001267 This code pops two values off the stack and pushes a tuple of
1268 length 2 whose items are those values back onto it. In other
1269 words:
Tim Petersfdc03462003-01-28 04:56:33 +00001270
1271 stack[-2:] = [tuple(stack[-2:])]
1272 """),
1273
1274 I(name='TUPLE3',
1275 code='\x87',
1276 arg=None,
1277 stack_before=[anyobject, anyobject, anyobject],
1278 stack_after=[pytuple],
1279 proto=2,
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001280 doc="""Build a three-tuple out of the top three items on the stack.
Tim Petersfdc03462003-01-28 04:56:33 +00001281
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001282 This code pops three values off the stack and pushes a tuple of
1283 length 3 whose items are those values back onto it. In other
1284 words:
Tim Petersfdc03462003-01-28 04:56:33 +00001285
1286 stack[-3:] = [tuple(stack[-3:])]
1287 """),
1288
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001289 # Ways to build dicts.
1290
1291 I(name='EMPTY_DICT',
1292 code='}',
1293 arg=None,
1294 stack_before=[],
1295 stack_after=[pydict],
1296 proto=1,
1297 doc="Push an empty dict."),
1298
1299 I(name='DICT',
1300 code='d',
1301 arg=None,
1302 stack_before=[markobject, stackslice],
1303 stack_after=[pydict],
1304 proto=0,
1305 doc="""Build a dict out of the topmost stack slice, after markobject.
1306
1307 All the stack entries following the topmost markobject are placed into
1308 a single Python dict, which single dict object replaces all of the
1309 stack from the topmost markobject onward. The stack slice alternates
1310 key, value, key, value, .... For example,
1311
1312 Stack before: ... markobject 1 2 3 'abc'
1313 Stack after: ... {1: 2, 3: 'abc'}
1314 """),
1315
1316 I(name='SETITEM',
1317 code='s',
1318 arg=None,
1319 stack_before=[pydict, anyobject, anyobject],
1320 stack_after=[pydict],
1321 proto=0,
1322 doc="""Add a key+value pair to an existing dict.
1323
1324 Stack before: ... pydict key value
1325 Stack after: ... pydict
1326
1327 where pydict has been modified via pydict[key] = value.
1328 """),
1329
1330 I(name='SETITEMS',
1331 code='u',
1332 arg=None,
1333 stack_before=[pydict, markobject, stackslice],
1334 stack_after=[pydict],
1335 proto=1,
1336 doc="""Add an arbitrary number of key+value pairs to an existing dict.
1337
1338 The slice of the stack following the topmost markobject is taken as
1339 an alternating sequence of keys and values, added to the dict
1340 immediately under the topmost markobject. Everything at and after the
1341 topmost markobject is popped, leaving the mutated dict at the top
1342 of the stack.
1343
1344 Stack before: ... pydict markobject key_1 value_1 ... key_n value_n
1345 Stack after: ... pydict
1346
1347 where pydict has been modified via pydict[key_i] = value_i for i in
1348 1, 2, ..., n, and in that order.
1349 """),
1350
1351 # Stack manipulation.
1352
1353 I(name='POP',
1354 code='0',
1355 arg=None,
1356 stack_before=[anyobject],
1357 stack_after=[],
1358 proto=0,
1359 doc="Discard the top stack item, shrinking the stack by one item."),
1360
1361 I(name='DUP',
1362 code='2',
1363 arg=None,
1364 stack_before=[anyobject],
1365 stack_after=[anyobject, anyobject],
1366 proto=0,
1367 doc="Push the top stack item onto the stack again, duplicating it."),
1368
1369 I(name='MARK',
1370 code='(',
1371 arg=None,
1372 stack_before=[],
1373 stack_after=[markobject],
1374 proto=0,
1375 doc="""Push markobject onto the stack.
1376
1377 markobject is a unique object, used by other opcodes to identify a
1378 region of the stack containing a variable number of objects for them
1379 to work on. See markobject.doc for more detail.
1380 """),
1381
1382 I(name='POP_MARK',
1383 code='1',
1384 arg=None,
1385 stack_before=[markobject, stackslice],
1386 stack_after=[],
Collin Wintere61d4372009-05-20 17:46:47 +00001387 proto=1,
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001388 doc="""Pop all the stack objects at and above the topmost markobject.
1389
1390 When an opcode using a variable number of stack objects is done,
1391 POP_MARK is used to remove those objects, and to remove the markobject
1392 that delimited their starting position on the stack.
1393 """),
1394
1395 # Memo manipulation. There are really only two operations (get and put),
1396 # each in all-text, "short binary", and "long binary" flavors.
1397
1398 I(name='GET',
1399 code='g',
1400 arg=decimalnl_short,
1401 stack_before=[],
1402 stack_after=[anyobject],
1403 proto=0,
1404 doc="""Read an object from the memo and push it on the stack.
1405
Ezio Melotti13925002011-03-16 11:05:33 +02001406 The index of the memo object to push is given by the newline-terminated
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001407 decimal string following. BINGET and LONG_BINGET are space-optimized
1408 versions.
1409 """),
1410
1411 I(name='BINGET',
1412 code='h',
1413 arg=uint1,
1414 stack_before=[],
1415 stack_after=[anyobject],
1416 proto=1,
1417 doc="""Read an object from the memo and push it on the stack.
1418
1419 The index of the memo object to push is given by the 1-byte unsigned
1420 integer following.
1421 """),
1422
1423 I(name='LONG_BINGET',
1424 code='j',
1425 arg=int4,
1426 stack_before=[],
1427 stack_after=[anyobject],
1428 proto=1,
1429 doc="""Read an object from the memo and push it on the stack.
1430
1431 The index of the memo object to push is given by the 4-byte signed
1432 little-endian integer following.
1433 """),
1434
1435 I(name='PUT',
1436 code='p',
1437 arg=decimalnl_short,
1438 stack_before=[],
1439 stack_after=[],
1440 proto=0,
1441 doc="""Store the stack top into the memo. The stack is not popped.
1442
1443 The index of the memo location to write into is given by the newline-
1444 terminated decimal string following. BINPUT and LONG_BINPUT are
1445 space-optimized versions.
1446 """),
1447
1448 I(name='BINPUT',
1449 code='q',
1450 arg=uint1,
1451 stack_before=[],
1452 stack_after=[],
1453 proto=1,
1454 doc="""Store the stack top into the memo. The stack is not popped.
1455
1456 The index of the memo location to write into is given by the 1-byte
1457 unsigned integer following.
1458 """),
1459
1460 I(name='LONG_BINPUT',
1461 code='r',
1462 arg=int4,
1463 stack_before=[],
1464 stack_after=[],
1465 proto=1,
1466 doc="""Store the stack top into the memo. The stack is not popped.
1467
1468 The index of the memo location to write into is given by the 4-byte
1469 signed little-endian integer following.
1470 """),
1471
Tim Petersfdc03462003-01-28 04:56:33 +00001472 # Access the extension registry (predefined objects). Akin to the GET
1473 # family.
1474
1475 I(name='EXT1',
1476 code='\x82',
1477 arg=uint1,
1478 stack_before=[],
1479 stack_after=[anyobject],
1480 proto=2,
1481 doc="""Extension code.
1482
1483 This code and the similar EXT2 and EXT4 allow using a registry
1484 of popular objects that are pickled by name, typically classes.
1485 It is envisioned that through a global negotiation and
1486 registration process, third parties can set up a mapping between
1487 ints and object names.
1488
1489 In order to guarantee pickle interchangeability, the extension
1490 code registry ought to be global, although a range of codes may
1491 be reserved for private use.
1492
1493 EXT1 has a 1-byte integer argument. This is used to index into the
1494 extension registry, and the object at that index is pushed on the stack.
1495 """),
1496
1497 I(name='EXT2',
1498 code='\x83',
1499 arg=uint2,
1500 stack_before=[],
1501 stack_after=[anyobject],
1502 proto=2,
1503 doc="""Extension code.
1504
1505 See EXT1. EXT2 has a two-byte integer argument.
1506 """),
1507
1508 I(name='EXT4',
1509 code='\x84',
1510 arg=int4,
1511 stack_before=[],
1512 stack_after=[anyobject],
1513 proto=2,
1514 doc="""Extension code.
1515
1516 See EXT1. EXT4 has a four-byte integer argument.
1517 """),
1518
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001519 # Push a class object, or module function, on the stack, via its module
1520 # and name.
1521
1522 I(name='GLOBAL',
1523 code='c',
1524 arg=stringnl_noescape_pair,
1525 stack_before=[],
1526 stack_after=[anyobject],
1527 proto=0,
1528 doc="""Push a global object (module.attr) on the stack.
1529
1530 Two newline-terminated strings follow the GLOBAL opcode. The first is
1531 taken as a module name, and the second as a class name. The class
1532 object module.class is pushed on the stack. More accurately, the
1533 object returned by self.find_class(module, class) is pushed on the
1534 stack, so unpickling subclasses can override this form of lookup.
1535 """),
1536
1537 # Ways to build objects of classes pickle doesn't know about directly
1538 # (user-defined classes). I despair of documenting this accurately
1539 # and comprehensibly -- you really have to read the pickle code to
1540 # find all the special cases.
1541
1542 I(name='REDUCE',
1543 code='R',
1544 arg=None,
1545 stack_before=[anyobject, anyobject],
1546 stack_after=[anyobject],
1547 proto=0,
1548 doc="""Push an object built from a callable and an argument tuple.
1549
1550 The opcode is named to remind of the __reduce__() method.
1551
1552 Stack before: ... callable pytuple
1553 Stack after: ... callable(*pytuple)
1554
1555 The callable and the argument tuple are the first two items returned
1556 by a __reduce__ method. Applying the callable to the argtuple is
1557 supposed to reproduce the original object, or at least get it started.
1558 If the __reduce__ method returns a 3-tuple, the last component is an
1559 argument to be passed to the object's __setstate__, and then the REDUCE
1560 opcode is followed by code to create setstate's argument, and then a
1561 BUILD opcode to apply __setstate__ to that argument.
1562
Guido van Rossum13257902007-06-07 23:15:56 +00001563 If not isinstance(callable, type), REDUCE complains unless the
Alexandre Vassalottif7fa63d2008-05-11 08:55:36 +00001564 callable has been registered with the copyreg module's
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001565 safe_constructors dict, or the callable has a magic
1566 '__safe_for_unpickling__' attribute with a true value. I'm not sure
1567 why it does this, but I've sure seen this complaint often enough when
1568 I didn't want to <wink>.
1569 """),
1570
1571 I(name='BUILD',
1572 code='b',
1573 arg=None,
1574 stack_before=[anyobject, anyobject],
1575 stack_after=[anyobject],
1576 proto=0,
1577 doc="""Finish building an object, via __setstate__ or dict update.
1578
1579 Stack before: ... anyobject argument
1580 Stack after: ... anyobject
1581
1582 where anyobject may have been mutated, as follows:
1583
1584 If the object has a __setstate__ method,
1585
1586 anyobject.__setstate__(argument)
1587
1588 is called.
1589
1590 Else the argument must be a dict, the object must have a __dict__, and
1591 the object is updated via
1592
1593 anyobject.__dict__.update(argument)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001594 """),
1595
1596 I(name='INST',
1597 code='i',
1598 arg=stringnl_noescape_pair,
1599 stack_before=[markobject, stackslice],
1600 stack_after=[anyobject],
1601 proto=0,
1602 doc="""Build a class instance.
1603
1604 This is the protocol 0 version of protocol 1's OBJ opcode.
1605 INST is followed by two newline-terminated strings, giving a
1606 module and class name, just as for the GLOBAL opcode (and see
1607 GLOBAL for more details about that). self.find_class(module, name)
1608 is used to get a class object.
1609
1610 In addition, all the objects on the stack following the topmost
1611 markobject are gathered into a tuple and popped (along with the
1612 topmost markobject), just as for the TUPLE opcode.
1613
1614 Now it gets complicated. If all of these are true:
1615
1616 + The argtuple is empty (markobject was at the top of the stack
1617 at the start).
1618
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001619 + The class object does not have a __getinitargs__ attribute.
1620
1621 then we want to create an old-style class instance without invoking
1622 its __init__() method (pickle has waffled on this over the years; not
1623 calling __init__() is current wisdom). In this case, an instance of
1624 an old-style dummy class is created, and then we try to rebind its
1625 __class__ attribute to the desired class object. If this succeeds,
Guido van Rossuma8add0e2007-05-14 22:03:55 +00001626 the new instance object is pushed on the stack, and we're done.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001627
1628 Else (the argtuple is not empty, it's not an old-style class object,
1629 or the class object does have a __getinitargs__ attribute), the code
1630 first insists that the class object have a __safe_for_unpickling__
1631 attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE,
1632 it doesn't matter whether this attribute has a true or false value, it
Guido van Rossum99603b02007-07-20 00:22:32 +00001633 only matters whether it exists (XXX this is a bug). If
1634 __safe_for_unpickling__ doesn't exist, UnpicklingError is raised.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001635
1636 Else (the class object does have a __safe_for_unpickling__ attr),
1637 the class object obtained from INST's arguments is applied to the
1638 argtuple obtained from the stack, and the resulting instance object
1639 is pushed on the stack.
Tim Peters2b93c4c2003-01-30 16:35:08 +00001640
1641 NOTE: checks for __safe_for_unpickling__ went away in Python 2.3.
Florent Xiclunaaa6c1d22011-12-12 18:54:29 +01001642 NOTE: the distinction between old-style and new-style classes does
1643 not make sense in Python 3.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001644 """),
1645
1646 I(name='OBJ',
1647 code='o',
1648 arg=None,
1649 stack_before=[markobject, anyobject, stackslice],
1650 stack_after=[anyobject],
1651 proto=1,
1652 doc="""Build a class instance.
1653
1654 This is the protocol 1 version of protocol 0's INST opcode, and is
1655 very much like it. The major difference is that the class object
1656 is taken off the stack, allowing it to be retrieved from the memo
1657 repeatedly if several instances of the same class are created. This
1658 can be much more efficient (in both time and space) than repeatedly
1659 embedding the module and class names in INST opcodes.
1660
1661 Unlike INST, OBJ takes no arguments from the opcode stream. Instead
1662 the class object is taken off the stack, immediately above the
1663 topmost markobject:
1664
1665 Stack before: ... markobject classobject stackslice
1666 Stack after: ... new_instance_object
1667
1668 As for INST, the remainder of the stack above the markobject is
1669 gathered into an argument tuple, and then the logic seems identical,
Guido van Rossumecb11042003-01-29 06:24:30 +00001670 except that no __safe_for_unpickling__ check is done (XXX this is
Guido van Rossum99603b02007-07-20 00:22:32 +00001671 a bug). See INST for the gory details.
Tim Peters2b93c4c2003-01-30 16:35:08 +00001672
1673 NOTE: In Python 2.3, INST and OBJ are identical except for how they
1674 get the class object. That was always the intent; the implementations
1675 had diverged for accidental reasons.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001676 """),
1677
Tim Petersfdc03462003-01-28 04:56:33 +00001678 I(name='NEWOBJ',
1679 code='\x81',
1680 arg=None,
1681 stack_before=[anyobject, anyobject],
1682 stack_after=[anyobject],
1683 proto=2,
1684 doc="""Build an object instance.
1685
1686 The stack before should be thought of as containing a class
1687 object followed by an argument tuple (the tuple being the stack
1688 top). Call these cls and args. They are popped off the stack,
1689 and the value returned by cls.__new__(cls, *args) is pushed back
1690 onto the stack.
1691 """),
1692
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001693 # Machine control.
1694
Tim Petersfdc03462003-01-28 04:56:33 +00001695 I(name='PROTO',
1696 code='\x80',
1697 arg=uint1,
1698 stack_before=[],
1699 stack_after=[],
1700 proto=2,
1701 doc="""Protocol version indicator.
1702
1703 For protocol 2 and above, a pickle must start with this opcode.
1704 The argument is the protocol version, an int in range(2, 256).
1705 """),
1706
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001707 I(name='STOP',
1708 code='.',
1709 arg=None,
1710 stack_before=[anyobject],
1711 stack_after=[],
1712 proto=0,
1713 doc="""Stop the unpickling machine.
1714
1715 Every pickle ends with this opcode. The object at the top of the stack
1716 is popped, and that's the result of unpickling. The stack should be
1717 empty then.
1718 """),
1719
1720 # Ways to deal with persistent IDs.
1721
1722 I(name='PERSID',
1723 code='P',
1724 arg=stringnl_noescape,
1725 stack_before=[],
1726 stack_after=[anyobject],
1727 proto=0,
1728 doc="""Push an object identified by a persistent ID.
1729
1730 The pickle module doesn't define what a persistent ID means. PERSID's
1731 argument is a newline-terminated str-style (no embedded escapes, no
1732 bracketing quote characters) string, which *is* "the persistent ID".
1733 The unpickler passes this string to self.persistent_load(). Whatever
1734 object that returns is pushed on the stack. There is no implementation
1735 of persistent_load() in Python's unpickler: it must be supplied by an
1736 unpickler subclass.
1737 """),
1738
1739 I(name='BINPERSID',
1740 code='Q',
1741 arg=None,
1742 stack_before=[anyobject],
1743 stack_after=[anyobject],
1744 proto=1,
1745 doc="""Push an object identified by a persistent ID.
1746
1747 Like PERSID, except the persistent ID is popped off the stack (instead
1748 of being a string embedded in the opcode bytestream). The persistent
1749 ID is passed to self.persistent_load(), and whatever object that
1750 returns is pushed on the stack. See PERSID for more detail.
1751 """),
1752]
1753del I
1754
1755# Verify uniqueness of .name and .code members.
1756name2i = {}
1757code2i = {}
1758
1759for i, d in enumerate(opcodes):
1760 if d.name in name2i:
1761 raise ValueError("repeated name %r at indices %d and %d" %
1762 (d.name, name2i[d.name], i))
1763 if d.code in code2i:
1764 raise ValueError("repeated code %r at indices %d and %d" %
1765 (d.code, code2i[d.code], i))
1766
1767 name2i[d.name] = i
1768 code2i[d.code] = i
1769
1770del name2i, code2i, i, d
1771
1772##############################################################################
1773# Build a code2op dict, mapping opcode characters to OpcodeInfo records.
1774# Also ensure we've got the same stuff as pickle.py, although the
1775# introspection here is dicey.
1776
1777code2op = {}
1778for d in opcodes:
1779 code2op[d.code] = d
1780del d
1781
1782def assure_pickle_consistency(verbose=False):
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001783
1784 copy = code2op.copy()
1785 for name in pickle.__all__:
1786 if not re.match("[A-Z][A-Z0-9_]+$", name):
1787 if verbose:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001788 print("skipping %r: it doesn't look like an opcode name" % name)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001789 continue
1790 picklecode = getattr(pickle, name)
Guido van Rossum617dbc42007-05-07 23:57:08 +00001791 if not isinstance(picklecode, bytes) or len(picklecode) != 1:
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001792 if verbose:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001793 print(("skipping %r: value %r doesn't look like a pickle "
1794 "code" % (name, picklecode)))
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001795 continue
Guido van Rossum617dbc42007-05-07 23:57:08 +00001796 picklecode = picklecode.decode("latin-1")
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001797 if picklecode in copy:
1798 if verbose:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001799 print("checking name %r w/ code %r for consistency" % (
1800 name, picklecode))
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001801 d = copy[picklecode]
1802 if d.name != name:
1803 raise ValueError("for pickle code %r, pickle.py uses name %r "
1804 "but we're using name %r" % (picklecode,
1805 name,
1806 d.name))
1807 # Forget this one. Any left over in copy at the end are a problem
1808 # of a different kind.
1809 del copy[picklecode]
1810 else:
1811 raise ValueError("pickle.py appears to have a pickle opcode with "
1812 "name %r and code %r, but we don't" %
1813 (name, picklecode))
1814 if copy:
1815 msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"]
1816 for code, d in copy.items():
1817 msg.append(" name %r with code %r" % (d.name, code))
1818 raise ValueError("\n".join(msg))
1819
1820assure_pickle_consistency()
Tim Petersc0c12b52003-01-29 00:56:17 +00001821del assure_pickle_consistency
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001822
1823##############################################################################
1824# A pickle opcode generator.
1825
1826def genops(pickle):
Guido van Rossuma72ded92003-01-27 19:40:47 +00001827 """Generate all the opcodes in a pickle.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001828
1829 'pickle' is a file-like object, or string, containing the pickle.
1830
1831 Each opcode in the pickle is generated, from the current pickle position,
1832 stopping after a STOP opcode is delivered. A triple is generated for
1833 each opcode:
1834
1835 opcode, arg, pos
1836
1837 opcode is an OpcodeInfo record, describing the current opcode.
1838
1839 If the opcode has an argument embedded in the pickle, arg is its decoded
1840 value, as a Python object. If the opcode doesn't have an argument, arg
1841 is None.
1842
1843 If the pickle has a tell() method, pos was the value of pickle.tell()
Guido van Rossum34d19282007-08-09 01:03:29 +00001844 before reading the current opcode. If the pickle is a bytes object,
1845 it's wrapped in a BytesIO object, and the latter's tell() result is
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001846 used. Else (the pickle doesn't have a tell(), and it's not obvious how
1847 to query its current position) pos is None.
1848 """
1849
Guido van Rossum98297ee2007-11-06 21:34:58 +00001850 if isinstance(pickle, bytes_types):
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001851 import io
1852 pickle = io.BytesIO(pickle)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001853
1854 if hasattr(pickle, "tell"):
1855 getpos = pickle.tell
1856 else:
1857 getpos = lambda: None
1858
1859 while True:
1860 pos = getpos()
1861 code = pickle.read(1)
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001862 opcode = code2op.get(code.decode("latin-1"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001863 if opcode is None:
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001864 if code == b"":
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001865 raise ValueError("pickle exhausted before seeing STOP")
1866 else:
1867 raise ValueError("at position %s, opcode %r unknown" % (
1868 pos is None and "<unknown>" or pos,
1869 code))
1870 if opcode.arg is None:
1871 arg = None
1872 else:
1873 arg = opcode.arg.reader(pickle)
1874 yield opcode, arg, pos
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001875 if code == b'.':
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001876 assert opcode.name == 'STOP'
1877 break
1878
1879##############################################################################
Christian Heimes3feef612008-02-11 06:19:17 +00001880# A pickle optimizer.
1881
1882def optimize(p):
1883 'Optimize a pickle string by removing unused PUT opcodes'
1884 gets = set() # set of args used by a GET opcode
1885 puts = [] # (arg, startpos, stoppos) for the PUT opcodes
1886 prevpos = None # set to pos if previous opcode was a PUT
1887 for opcode, arg, pos in genops(p):
1888 if prevpos is not None:
1889 puts.append((prevarg, prevpos, pos))
1890 prevpos = None
1891 if 'PUT' in opcode.name:
1892 prevarg, prevpos = arg, pos
1893 elif 'GET' in opcode.name:
1894 gets.add(arg)
1895
1896 # Copy the pickle string except for PUTS without a corresponding GET
1897 s = []
1898 i = 0
1899 for arg, start, stop in puts:
1900 j = stop if (arg in gets) else start
1901 s.append(p[i:j])
1902 i = stop
1903 s.append(p[i:])
Christian Heimes126d29a2008-02-11 22:57:17 +00001904 return b''.join(s)
Christian Heimes3feef612008-02-11 06:19:17 +00001905
1906##############################################################################
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001907# A symbolic pickle disassembler.
1908
Alexander Belopolsky929d3842010-07-17 15:51:21 +00001909def dis(pickle, out=None, memo=None, indentlevel=4, annotate=0):
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001910 """Produce a symbolic disassembly of a pickle.
1911
1912 'pickle' is a file-like object, or string, containing a (at least one)
1913 pickle. The pickle is disassembled from the current position, through
1914 the first STOP opcode encountered.
1915
1916 Optional arg 'out' is a file-like object to which the disassembly is
1917 printed. It defaults to sys.stdout.
1918
Tim Peters62235e72003-02-05 19:55:53 +00001919 Optional arg 'memo' is a Python dict, used as the pickle's memo. It
1920 may be mutated by dis(), if the pickle contains PUT or BINPUT opcodes.
1921 Passing the same memo object to another dis() call then allows disassembly
1922 to proceed across multiple pickles that were all created by the same
1923 pickler with the same memo. Ordinarily you don't need to worry about this.
1924
Alexander Belopolsky929d3842010-07-17 15:51:21 +00001925 Optional arg 'indentlevel' is the number of blanks by which to indent
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001926 a new MARK level. It defaults to 4.
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001927
Alexander Belopolsky929d3842010-07-17 15:51:21 +00001928 Optional arg 'annotate' if nonzero instructs dis() to add short
1929 description of the opcode on each line of disassembled output.
1930 The value given to 'annotate' must be an integer and is used as a
1931 hint for the column where annotation should start. The default
1932 value is 0, meaning no annotations.
1933
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001934 In addition to printing the disassembly, some sanity checks are made:
1935
1936 + All embedded opcode arguments "make sense".
1937
1938 + Explicit and implicit pop operations have enough items on the stack.
1939
1940 + When an opcode implicitly refers to a markobject, a markobject is
1941 actually on the stack.
1942
1943 + A memo entry isn't referenced before it's defined.
1944
1945 + The markobject isn't stored in the memo.
1946
1947 + A memo entry isn't redefined.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001948 """
1949
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001950 # Most of the hair here is for sanity checks, but most of it is needed
1951 # anyway to detect when a protocol 0 POP takes a MARK off the stack
1952 # (which in turn is needed to indent MARK blocks correctly).
1953
1954 stack = [] # crude emulation of unpickler stack
Tim Peters62235e72003-02-05 19:55:53 +00001955 if memo is None:
1956 memo = {} # crude emulation of unpicker memo
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001957 maxproto = -1 # max protocol number seen
1958 markstack = [] # bytecode positions of MARK opcodes
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001959 indentchunk = ' ' * indentlevel
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001960 errormsg = None
Alexander Belopolsky929d3842010-07-17 15:51:21 +00001961 annocol = annotate # columnt hint for annotations
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001962 for opcode, arg, pos in genops(pickle):
1963 if pos is not None:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001964 print("%5d:" % pos, end=' ', file=out)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001965
Tim Petersd0f7c862003-01-28 15:27:57 +00001966 line = "%-4s %s%s" % (repr(opcode.code)[1:-1],
1967 indentchunk * len(markstack),
1968 opcode.name)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001969
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001970 maxproto = max(maxproto, opcode.proto)
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001971 before = opcode.stack_before # don't mutate
1972 after = opcode.stack_after # don't mutate
Tim Peters43277d62003-01-30 15:02:12 +00001973 numtopop = len(before)
1974
1975 # See whether a MARK should be popped.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001976 markmsg = None
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001977 if markobject in before or (opcode.name == "POP" and
1978 stack and
1979 stack[-1] is markobject):
1980 assert markobject not in after
Tim Peters43277d62003-01-30 15:02:12 +00001981 if __debug__:
1982 if markobject in before:
1983 assert before[-1] is stackslice
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001984 if markstack:
1985 markpos = markstack.pop()
1986 if markpos is None:
1987 markmsg = "(MARK at unknown opcode offset)"
1988 else:
1989 markmsg = "(MARK at %d)" % markpos
1990 # Pop everything at and after the topmost markobject.
1991 while stack[-1] is not markobject:
1992 stack.pop()
1993 stack.pop()
Tim Peters43277d62003-01-30 15:02:12 +00001994 # Stop later code from popping too much.
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001995 try:
Tim Peters43277d62003-01-30 15:02:12 +00001996 numtopop = before.index(markobject)
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001997 except ValueError:
1998 assert opcode.name == "POP"
Tim Peters43277d62003-01-30 15:02:12 +00001999 numtopop = 0
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002000 else:
2001 errormsg = markmsg = "no MARK exists on stack"
2002
2003 # Check for correct memo usage.
2004 if opcode.name in ("PUT", "BINPUT", "LONG_BINPUT"):
Tim Peters43277d62003-01-30 15:02:12 +00002005 assert arg is not None
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002006 if arg in memo:
2007 errormsg = "memo key %r already defined" % arg
2008 elif not stack:
2009 errormsg = "stack is empty -- can't store into memo"
2010 elif stack[-1] is markobject:
2011 errormsg = "can't store markobject in the memo"
2012 else:
2013 memo[arg] = stack[-1]
2014
2015 elif opcode.name in ("GET", "BINGET", "LONG_BINGET"):
2016 if arg in memo:
2017 assert len(after) == 1
2018 after = [memo[arg]] # for better stack emulation
2019 else:
2020 errormsg = "memo key %r has never been stored into" % arg
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002021
2022 if arg is not None or markmsg:
2023 # make a mild effort to align arguments
2024 line += ' ' * (10 - len(opcode.name))
2025 if arg is not None:
2026 line += ' ' + repr(arg)
2027 if markmsg:
2028 line += ' ' + markmsg
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002029 if annotate:
2030 line += ' ' * (annocol - len(line))
2031 # make a mild effort to align annotations
2032 annocol = len(line)
2033 if annocol > 50:
2034 annocol = annotate
2035 line += ' ' + opcode.doc.split('\n', 1)[0]
Guido van Rossumbe19ed72007-02-09 05:37:30 +00002036 print(line, file=out)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002037
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002038 if errormsg:
2039 # Note that we delayed complaining until the offending opcode
2040 # was printed.
2041 raise ValueError(errormsg)
2042
2043 # Emulate the stack effects.
Tim Peters43277d62003-01-30 15:02:12 +00002044 if len(stack) < numtopop:
2045 raise ValueError("tries to pop %d items from stack with "
2046 "only %d items" % (numtopop, len(stack)))
2047 if numtopop:
2048 del stack[-numtopop:]
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002049 if markobject in after:
Tim Peters43277d62003-01-30 15:02:12 +00002050 assert markobject not in before
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002051 markstack.append(pos)
2052
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002053 stack.extend(after)
2054
Guido van Rossumbe19ed72007-02-09 05:37:30 +00002055 print("highest protocol among opcodes =", maxproto, file=out)
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002056 if stack:
2057 raise ValueError("stack not empty after STOP: %r" % stack)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002058
Tim Peters90718a42005-02-15 16:22:34 +00002059# For use in the doctest, simply as an example of a class to pickle.
2060class _Example:
2061 def __init__(self, value):
2062 self.value = value
2063
Guido van Rossum03e35322003-01-28 15:37:13 +00002064_dis_test = r"""
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002065>>> import pickle
Guido van Rossumf4169812008-03-17 22:56:06 +00002066>>> x = [1, 2, (3, 4), {b'abc': "def"}]
2067>>> pkl0 = pickle.dumps(x, 0)
2068>>> dis(pkl0)
Tim Petersd0f7c862003-01-28 15:27:57 +00002069 0: ( MARK
2070 1: l LIST (MARK at 0)
2071 2: p PUT 0
Guido van Rossumf4100002007-01-15 00:21:46 +00002072 5: L LONG 1
Mark Dickinson8dd05142009-01-20 20:43:58 +00002073 9: a APPEND
2074 10: L LONG 2
2075 14: a APPEND
2076 15: ( MARK
2077 16: L LONG 3
2078 20: L LONG 4
2079 24: t TUPLE (MARK at 15)
2080 25: p PUT 1
2081 28: a APPEND
2082 29: ( MARK
2083 30: d DICT (MARK at 29)
2084 31: p PUT 2
Alexandre Vassalotti3bfc65a2011-12-13 13:08:09 -05002085 34: c GLOBAL '_codecs encode'
2086 50: p PUT 3
2087 53: ( MARK
2088 54: V UNICODE 'abc'
Antoine Pitroud9dfaa92009-06-04 20:32:06 +00002089 59: p PUT 4
Alexandre Vassalotti3bfc65a2011-12-13 13:08:09 -05002090 62: V UNICODE 'latin1'
2091 70: p PUT 5
2092 73: t TUPLE (MARK at 53)
2093 74: p PUT 6
2094 77: R REDUCE
2095 78: p PUT 7
2096 81: V UNICODE 'def'
2097 86: p PUT 8
2098 89: s SETITEM
2099 90: a APPEND
2100 91: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002101highest protocol among opcodes = 0
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002102
2103Try again with a "binary" pickle.
2104
Guido van Rossumf4169812008-03-17 22:56:06 +00002105>>> pkl1 = pickle.dumps(x, 1)
2106>>> dis(pkl1)
Tim Petersd0f7c862003-01-28 15:27:57 +00002107 0: ] EMPTY_LIST
2108 1: q BINPUT 0
2109 3: ( MARK
2110 4: K BININT1 1
2111 6: K BININT1 2
2112 8: ( MARK
2113 9: K BININT1 3
2114 11: K BININT1 4
2115 13: t TUPLE (MARK at 8)
2116 14: q BINPUT 1
2117 16: } EMPTY_DICT
2118 17: q BINPUT 2
Alexandre Vassalotti3bfc65a2011-12-13 13:08:09 -05002119 19: c GLOBAL '_codecs encode'
2120 35: q BINPUT 3
2121 37: ( MARK
2122 38: X BINUNICODE 'abc'
2123 46: q BINPUT 4
2124 48: X BINUNICODE 'latin1'
2125 59: q BINPUT 5
2126 61: t TUPLE (MARK at 37)
2127 62: q BINPUT 6
2128 64: R REDUCE
2129 65: q BINPUT 7
2130 67: X BINUNICODE 'def'
2131 75: q BINPUT 8
2132 77: s SETITEM
2133 78: e APPENDS (MARK at 3)
2134 79: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002135highest protocol among opcodes = 1
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002136
2137Exercise the INST/OBJ/BUILD family.
2138
Mark Dickinsoncddcf442009-01-24 21:46:33 +00002139>>> import pickletools
2140>>> dis(pickle.dumps(pickletools.dis, 0))
2141 0: c GLOBAL 'pickletools dis'
2142 17: p PUT 0
2143 20: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002144highest protocol among opcodes = 0
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002145
Tim Peters90718a42005-02-15 16:22:34 +00002146>>> from pickletools import _Example
2147>>> x = [_Example(42)] * 2
Guido van Rossumf29d3d62003-01-27 22:47:53 +00002148>>> dis(pickle.dumps(x, 0))
Tim Petersd0f7c862003-01-28 15:27:57 +00002149 0: ( MARK
2150 1: l LIST (MARK at 0)
2151 2: p PUT 0
Antoine Pitroud9dfaa92009-06-04 20:32:06 +00002152 5: c GLOBAL 'copy_reg _reconstructor'
2153 30: p PUT 1
2154 33: ( MARK
2155 34: c GLOBAL 'pickletools _Example'
2156 56: p PUT 2
2157 59: c GLOBAL '__builtin__ object'
2158 79: p PUT 3
2159 82: N NONE
2160 83: t TUPLE (MARK at 33)
2161 84: p PUT 4
2162 87: R REDUCE
2163 88: p PUT 5
2164 91: ( MARK
2165 92: d DICT (MARK at 91)
2166 93: p PUT 6
2167 96: V UNICODE 'value'
2168 103: p PUT 7
2169 106: L LONG 42
2170 111: s SETITEM
2171 112: b BUILD
Mark Dickinson8dd05142009-01-20 20:43:58 +00002172 113: a APPEND
Antoine Pitroud9dfaa92009-06-04 20:32:06 +00002173 114: g GET 5
2174 117: a APPEND
2175 118: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002176highest protocol among opcodes = 0
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002177
2178>>> dis(pickle.dumps(x, 1))
Tim Petersd0f7c862003-01-28 15:27:57 +00002179 0: ] EMPTY_LIST
2180 1: q BINPUT 0
2181 3: ( MARK
Antoine Pitroud9dfaa92009-06-04 20:32:06 +00002182 4: c GLOBAL 'copy_reg _reconstructor'
2183 29: q BINPUT 1
2184 31: ( MARK
2185 32: c GLOBAL 'pickletools _Example'
2186 54: q BINPUT 2
2187 56: c GLOBAL '__builtin__ object'
2188 76: q BINPUT 3
2189 78: N NONE
2190 79: t TUPLE (MARK at 31)
2191 80: q BINPUT 4
2192 82: R REDUCE
2193 83: q BINPUT 5
2194 85: } EMPTY_DICT
2195 86: q BINPUT 6
2196 88: X BINUNICODE 'value'
2197 98: q BINPUT 7
2198 100: K BININT1 42
2199 102: s SETITEM
2200 103: b BUILD
2201 104: h BINGET 5
2202 106: e APPENDS (MARK at 3)
2203 107: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002204highest protocol among opcodes = 1
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002205
2206Try "the canonical" recursive-object test.
2207
2208>>> L = []
2209>>> T = L,
2210>>> L.append(T)
2211>>> L[0] is T
2212True
2213>>> T[0] is L
2214True
2215>>> L[0][0] is L
2216True
2217>>> T[0][0] is T
2218True
Guido van Rossumf29d3d62003-01-27 22:47:53 +00002219>>> dis(pickle.dumps(L, 0))
Tim Petersd0f7c862003-01-28 15:27:57 +00002220 0: ( MARK
2221 1: l LIST (MARK at 0)
2222 2: p PUT 0
2223 5: ( MARK
2224 6: g GET 0
2225 9: t TUPLE (MARK at 5)
2226 10: p PUT 1
2227 13: a APPEND
2228 14: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002229highest protocol among opcodes = 0
2230
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002231>>> dis(pickle.dumps(L, 1))
Tim Petersd0f7c862003-01-28 15:27:57 +00002232 0: ] EMPTY_LIST
2233 1: q BINPUT 0
2234 3: ( MARK
2235 4: h BINGET 0
2236 6: t TUPLE (MARK at 3)
2237 7: q BINPUT 1
2238 9: a APPEND
2239 10: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002240highest protocol among opcodes = 1
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002241
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002242Note that, in the protocol 0 pickle of the recursive tuple, the disassembler
2243has to emulate the stack in order to realize that the POP opcode at 16 gets
2244rid of the MARK at 0.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002245
Guido van Rossumf29d3d62003-01-27 22:47:53 +00002246>>> dis(pickle.dumps(T, 0))
Tim Petersd0f7c862003-01-28 15:27:57 +00002247 0: ( MARK
2248 1: ( MARK
2249 2: l LIST (MARK at 1)
2250 3: p PUT 0
2251 6: ( MARK
2252 7: g GET 0
2253 10: t TUPLE (MARK at 6)
2254 11: p PUT 1
2255 14: a APPEND
2256 15: 0 POP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002257 16: 0 POP (MARK at 0)
2258 17: g GET 1
2259 20: . STOP
2260highest protocol among opcodes = 0
2261
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002262>>> dis(pickle.dumps(T, 1))
Tim Petersd0f7c862003-01-28 15:27:57 +00002263 0: ( MARK
2264 1: ] EMPTY_LIST
2265 2: q BINPUT 0
2266 4: ( MARK
2267 5: h BINGET 0
2268 7: t TUPLE (MARK at 4)
2269 8: q BINPUT 1
2270 10: a APPEND
2271 11: 1 POP_MARK (MARK at 0)
2272 12: h BINGET 1
2273 14: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002274highest protocol among opcodes = 1
Tim Petersd0f7c862003-01-28 15:27:57 +00002275
2276Try protocol 2.
2277
2278>>> dis(pickle.dumps(L, 2))
2279 0: \x80 PROTO 2
2280 2: ] EMPTY_LIST
2281 3: q BINPUT 0
2282 5: h BINGET 0
2283 7: \x85 TUPLE1
2284 8: q BINPUT 1
2285 10: a APPEND
2286 11: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002287highest protocol among opcodes = 2
Tim Petersd0f7c862003-01-28 15:27:57 +00002288
2289>>> dis(pickle.dumps(T, 2))
2290 0: \x80 PROTO 2
2291 2: ] EMPTY_LIST
2292 3: q BINPUT 0
2293 5: h BINGET 0
2294 7: \x85 TUPLE1
2295 8: q BINPUT 1
2296 10: a APPEND
2297 11: 0 POP
2298 12: h BINGET 1
2299 14: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002300highest protocol among opcodes = 2
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002301
2302Try protocol 3 with annotations:
2303
2304>>> dis(pickle.dumps(T, 3), annotate=1)
2305 0: \x80 PROTO 3 Protocol version indicator.
2306 2: ] EMPTY_LIST Push an empty list.
2307 3: q BINPUT 0 Store the stack top into the memo. The stack is not popped.
2308 5: h BINGET 0 Read an object from the memo and push it on the stack.
2309 7: \x85 TUPLE1 Build a one-tuple out of the topmost item on the stack.
2310 8: q BINPUT 1 Store the stack top into the memo. The stack is not popped.
2311 10: a APPEND Append an object to a list.
2312 11: 0 POP Discard the top stack item, shrinking the stack by one item.
2313 12: h BINGET 1 Read an object from the memo and push it on the stack.
2314 14: . STOP Stop the unpickling machine.
2315highest protocol among opcodes = 2
2316
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002317"""
2318
Tim Peters62235e72003-02-05 19:55:53 +00002319_memo_test = r"""
2320>>> import pickle
Guido van Rossumcfe5f202007-05-08 21:26:54 +00002321>>> import io
2322>>> f = io.BytesIO()
Tim Peters62235e72003-02-05 19:55:53 +00002323>>> p = pickle.Pickler(f, 2)
2324>>> x = [1, 2, 3]
2325>>> p.dump(x)
2326>>> p.dump(x)
2327>>> f.seek(0)
Guido van Rossumcfe5f202007-05-08 21:26:54 +000023280
Tim Peters62235e72003-02-05 19:55:53 +00002329>>> memo = {}
2330>>> dis(f, memo=memo)
2331 0: \x80 PROTO 2
2332 2: ] EMPTY_LIST
2333 3: q BINPUT 0
2334 5: ( MARK
2335 6: K BININT1 1
2336 8: K BININT1 2
2337 10: K BININT1 3
2338 12: e APPENDS (MARK at 5)
2339 13: . STOP
2340highest protocol among opcodes = 2
2341>>> dis(f, memo=memo)
2342 14: \x80 PROTO 2
2343 16: h BINGET 0
2344 18: . STOP
2345highest protocol among opcodes = 2
2346"""
2347
Guido van Rossum57028352003-01-28 15:09:10 +00002348__test__ = {'disassembler_test': _dis_test,
Tim Peters62235e72003-02-05 19:55:53 +00002349 'disassembler_memo_test': _memo_test,
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002350 }
2351
2352def _test():
2353 import doctest
2354 return doctest.testmod()
2355
2356if __name__ == "__main__":
Alexander Belopolsky60c762b2010-07-03 20:35:53 +00002357 import sys, argparse
2358 parser = argparse.ArgumentParser(
2359 description='disassemble one or more pickle files')
2360 parser.add_argument(
2361 'pickle_file', type=argparse.FileType('br'),
2362 nargs='*', help='the pickle file')
2363 parser.add_argument(
2364 '-o', '--output', default=sys.stdout, type=argparse.FileType('w'),
2365 help='the file where the output should be written')
2366 parser.add_argument(
2367 '-m', '--memo', action='store_true',
2368 help='preserve memo between disassemblies')
2369 parser.add_argument(
2370 '-l', '--indentlevel', default=4, type=int,
2371 help='the number of blanks by which to indent a new MARK level')
2372 parser.add_argument(
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002373 '-a', '--annotate', action='store_true',
2374 help='annotate each line with a short opcode description')
2375 parser.add_argument(
Alexander Belopolsky60c762b2010-07-03 20:35:53 +00002376 '-p', '--preamble', default="==> {name} <==",
2377 help='if more than one pickle file is specified, print this before'
2378 ' each disassembly')
2379 parser.add_argument(
2380 '-t', '--test', action='store_true',
2381 help='run self-test suite')
2382 parser.add_argument(
2383 '-v', action='store_true',
2384 help='run verbosely; only affects self-test run')
2385 args = parser.parse_args()
2386 if args.test:
2387 _test()
2388 else:
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002389 annotate = 30 if args.annotate else 0
Alexander Belopolsky60c762b2010-07-03 20:35:53 +00002390 if not args.pickle_file:
2391 parser.print_help()
2392 elif len(args.pickle_file) == 1:
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002393 dis(args.pickle_file[0], args.output, None,
2394 args.indentlevel, annotate)
Alexander Belopolsky60c762b2010-07-03 20:35:53 +00002395 else:
2396 memo = {} if args.memo else None
2397 for f in args.pickle_file:
2398 preamble = args.preamble.format(name=f.name)
2399 args.output.write(preamble + '\n')
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002400 dis(f, args.output, memo, args.indentlevel, annotate)