blob: 69cedf7731ceedf8a4206e7235109aceaa9f6d9e [file] [log] [blame]
Skip Montanaro54455942003-01-29 15:41:33 +00001'''"Executable documentation" for the pickle module.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002
3Extensive comments about the pickle protocols and pickle-machine opcodes
4can be found here. Some functions meant for external use:
5
6genops(pickle)
7 Generate all the opcodes in a pickle, as (opcode, arg, position) triples.
8
Andrew M. Kuchlingd0c53fe2004-08-07 16:51:30 +00009dis(pickle, out=None, memo=None, indentlevel=4)
Tim Peters8ecfc8e2003-01-27 18:51:48 +000010 Print a symbolic disassembly of a pickle.
Skip Montanaro54455942003-01-29 15:41:33 +000011'''
Tim Peters8ecfc8e2003-01-27 18:51:48 +000012
Walter Dörwald42748a82007-06-12 16:40:17 +000013import codecs
Guido van Rossum98297ee2007-11-06 21:34:58 +000014import pickle
15import re
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -070016import sys
Walter Dörwald42748a82007-06-12 16:40:17 +000017
Christian Heimes3feef612008-02-11 06:19:17 +000018__all__ = ['dis', 'genops', 'optimize']
Tim Peters90cf2122004-11-06 23:45:48 +000019
Guido van Rossum98297ee2007-11-06 21:34:58 +000020bytes_types = pickle.bytes_types
21
Tim Peters8ecfc8e2003-01-27 18:51:48 +000022# Other ideas:
23#
24# - A pickle verifier: read a pickle and check it exhaustively for
Tim Petersc1c2b3e2003-01-29 20:12:21 +000025# well-formedness. dis() does a lot of this already.
Tim Peters8ecfc8e2003-01-27 18:51:48 +000026#
27# - A protocol identifier: examine a pickle and return its protocol number
28# (== the highest .proto attr value among all the opcodes in the pickle).
Tim Petersc1c2b3e2003-01-29 20:12:21 +000029# dis() already prints this info at the end.
Tim Peters8ecfc8e2003-01-27 18:51:48 +000030#
31# - A pickle optimizer: for example, tuple-building code is sometimes more
32# elaborate than necessary, catering for the possibility that the tuple
33# is recursive. Or lots of times a PUT is generated that's never accessed
34# by a later GET.
35
36
37"""
38"A pickle" is a program for a virtual pickle machine (PM, but more accurately
39called an unpickling machine). It's a sequence of opcodes, interpreted by the
40PM, building an arbitrarily complex Python object.
41
42For the most part, the PM is very simple: there are no looping, testing, or
43conditional instructions, no arithmetic and no function calls. Opcodes are
44executed once each, from first to last, until a STOP opcode is reached.
45
46The PM has two data areas, "the stack" and "the memo".
47
48Many opcodes push Python objects onto the stack; e.g., INT pushes a Python
49integer object on the stack, whose value is gotten from a decimal string
50literal immediately following the INT opcode in the pickle bytestream. Other
51opcodes take Python objects off the stack. The result of unpickling is
52whatever object is left on the stack when the final STOP opcode is executed.
53
54The memo is simply an array of objects, or it can be implemented as a dict
55mapping little integers to objects. The memo serves as the PM's "long term
56memory", and the little integers indexing the memo are akin to variable
57names. Some opcodes pop a stack object into the memo at a given index,
58and others push a memo object at a given index onto the stack again.
59
60At heart, that's all the PM has. Subtleties arise for these reasons:
61
62+ Object identity. Objects can be arbitrarily complex, and subobjects
63 may be shared (for example, the list [a, a] refers to the same object a
64 twice). It can be vital that unpickling recreate an isomorphic object
65 graph, faithfully reproducing sharing.
66
67+ Recursive objects. For example, after "L = []; L.append(L)", L is a
68 list, and L[0] is the same list. This is related to the object identity
69 point, and some sequences of pickle opcodes are subtle in order to
70 get the right result in all cases.
71
72+ Things pickle doesn't know everything about. Examples of things pickle
73 does know everything about are Python's builtin scalar and container
74 types, like ints and tuples. They generally have opcodes dedicated to
75 them. For things like module references and instances of user-defined
76 classes, pickle's knowledge is limited. Historically, many enhancements
77 have been made to the pickle protocol in order to do a better (faster,
78 and/or more compact) job on those.
79
80+ Backward compatibility and micro-optimization. As explained below,
81 pickle opcodes never go away, not even when better ways to do a thing
82 get invented. The repertoire of the PM just keeps growing over time.
Tim Petersfdc03462003-01-28 04:56:33 +000083 For example, protocol 0 had two opcodes for building Python integers (INT
84 and LONG), protocol 1 added three more for more-efficient pickling of short
85 integers, and protocol 2 added two more for more-efficient pickling of
86 long integers (before protocol 2, the only ways to pickle a Python long
87 took time quadratic in the number of digits, for both pickling and
88 unpickling). "Opcode bloat" isn't so much a subtlety as a source of
Tim Peters8ecfc8e2003-01-27 18:51:48 +000089 wearying complication.
90
91
92Pickle protocols:
93
94For compatibility, the meaning of a pickle opcode never changes. Instead new
95pickle opcodes get added, and each version's unpickler can handle all the
96pickle opcodes in all protocol versions to date. So old pickles continue to
97be readable forever. The pickler can generally be told to restrict itself to
98the subset of opcodes available under previous protocol versions too, so that
99users can create pickles under the current version readable by older
100versions. However, a pickle does not contain its version number embedded
101within it. If an older unpickler tries to read a pickle using a later
102protocol, the result is most likely an exception due to seeing an unknown (in
103the older unpickler) opcode.
104
105The original pickle used what's now called "protocol 0", and what was called
106"text mode" before Python 2.3. The entire pickle bytestream is made up of
107printable 7-bit ASCII characters, plus the newline character, in protocol 0.
Tim Petersfdc03462003-01-28 04:56:33 +0000108That's why it was called text mode. Protocol 0 is small and elegant, but
109sometimes painfully inefficient.
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000110
111The second major set of additions is now called "protocol 1", and was called
112"binary mode" before Python 2.3. This added many opcodes with arguments
113consisting of arbitrary bytes, including NUL bytes and unprintable "high bit"
114bytes. Binary mode pickles can be substantially smaller than equivalent
115text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte
116int as 4 bytes following the opcode, which is cheaper to unpickle than the
Tim Petersfdc03462003-01-28 04:56:33 +0000117(perhaps) 11-character decimal string attached to INT. Protocol 1 also added
118a number of opcodes that operate on many stack elements at once (like APPENDS
Tim Peters81098ac2003-01-28 05:12:08 +0000119and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE).
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000120
121The third major set of additions came in Python 2.3, and is called "protocol
Tim Petersfdc03462003-01-28 04:56:33 +00001222". This added:
123
124- A better way to pickle instances of new-style classes (NEWOBJ).
125
126- A way for a pickle to identify its protocol (PROTO).
127
128- Time- and space- efficient pickling of long ints (LONG{1,4}).
129
130- Shortcuts for small tuples (TUPLE{1,2,3}}.
131
132- Dedicated opcodes for bools (NEWTRUE, NEWFALSE).
133
134- The "extension registry", a vector of popular objects that can be pushed
135 efficiently by index (EXT{1,2,4}). This is akin to the memo and GET, but
136 the registry contents are predefined (there's nothing akin to the memo's
137 PUT).
Guido van Rossumecb11042003-01-29 06:24:30 +0000138
Skip Montanaro54455942003-01-29 15:41:33 +0000139Another independent change with Python 2.3 is the abandonment of any
140pretense that it might be safe to load pickles received from untrusted
Guido van Rossumecb11042003-01-29 06:24:30 +0000141parties -- no sufficient security analysis has been done to guarantee
Skip Montanaro54455942003-01-29 15:41:33 +0000142this and there isn't a use case that warrants the expense of such an
Guido van Rossumecb11042003-01-29 06:24:30 +0000143analysis.
144
145To this end, all tests for __safe_for_unpickling__ or for
Alexandre Vassalottif7fa63d2008-05-11 08:55:36 +0000146copyreg.safe_constructors are removed from the unpickling code.
Guido van Rossumecb11042003-01-29 06:24:30 +0000147References to these variables in the descriptions below are to be seen
148as describing unpickling in Python 2.2 and before.
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000149"""
150
151# Meta-rule: Descriptions are stored in instances of descriptor objects,
152# with plain constructors. No meta-language is defined from which
153# descriptors could be constructed. If you want, e.g., XML, write a little
154# program to generate XML from the objects.
155
156##############################################################################
157# Some pickle opcodes have an argument, following the opcode in the
158# bytestream. An argument is of a specific type, described by an instance
159# of ArgumentDescriptor. These are not to be confused with arguments taken
160# off the stack -- ArgumentDescriptor applies only to arguments embedded in
161# the opcode stream, immediately following an opcode.
162
163# Represents the number of bytes consumed by an argument delimited by the
164# next newline character.
165UP_TO_NEWLINE = -1
166
167# Represents the number of bytes consumed by a two-argument opcode where
168# the first argument gives the number of bytes in the second argument.
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -0700169TAKEN_FROM_ARGUMENT1 = -2 # num bytes is 1-byte unsigned int
170TAKEN_FROM_ARGUMENT4 = -3 # num bytes is 4-byte signed little-endian int
171TAKEN_FROM_ARGUMENT4U = -4 # num bytes is 4-byte unsigned little-endian int
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000172
173class ArgumentDescriptor(object):
174 __slots__ = (
175 # name of descriptor record, also a module global name; a string
176 'name',
177
178 # length of argument, in bytes; an int; UP_TO_NEWLINE and
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000179 # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length
180 # cases
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000181 'n',
182
183 # a function taking a file-like object, reading this kind of argument
184 # from the object at the current position, advancing the current
185 # position by n bytes, and returning the value of the argument
186 'reader',
187
188 # human-readable docs for this arg descriptor; a string
189 'doc',
190 )
191
192 def __init__(self, name, n, reader, doc):
193 assert isinstance(name, str)
194 self.name = name
195
196 assert isinstance(n, int) and (n >= 0 or
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000197 n in (UP_TO_NEWLINE,
198 TAKEN_FROM_ARGUMENT1,
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -0700199 TAKEN_FROM_ARGUMENT4,
200 TAKEN_FROM_ARGUMENT4U))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000201 self.n = n
202
203 self.reader = reader
204
205 assert isinstance(doc, str)
206 self.doc = doc
207
208from struct import unpack as _unpack
209
210def read_uint1(f):
Tim Peters55762f52003-01-28 16:01:25 +0000211 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000212 >>> import io
213 >>> read_uint1(io.BytesIO(b'\xff'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000214 255
215 """
216
217 data = f.read(1)
218 if data:
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000219 return data[0]
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000220 raise ValueError("not enough data in stream to read uint1")
221
222uint1 = ArgumentDescriptor(
223 name='uint1',
224 n=1,
225 reader=read_uint1,
226 doc="One-byte unsigned integer.")
227
228
229def read_uint2(f):
Tim Peters55762f52003-01-28 16:01:25 +0000230 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000231 >>> import io
232 >>> read_uint2(io.BytesIO(b'\xff\x00'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000233 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000234 >>> read_uint2(io.BytesIO(b'\xff\xff'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000235 65535
236 """
237
238 data = f.read(2)
239 if len(data) == 2:
240 return _unpack("<H", data)[0]
241 raise ValueError("not enough data in stream to read uint2")
242
243uint2 = ArgumentDescriptor(
244 name='uint2',
245 n=2,
246 reader=read_uint2,
247 doc="Two-byte unsigned integer, little-endian.")
248
249
250def read_int4(f):
Tim Peters55762f52003-01-28 16:01:25 +0000251 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000252 >>> import io
253 >>> read_int4(io.BytesIO(b'\xff\x00\x00\x00'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000254 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000255 >>> read_int4(io.BytesIO(b'\x00\x00\x00\x80')) == -(2**31)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000256 True
257 """
258
259 data = f.read(4)
260 if len(data) == 4:
261 return _unpack("<i", data)[0]
262 raise ValueError("not enough data in stream to read int4")
263
264int4 = ArgumentDescriptor(
265 name='int4',
266 n=4,
267 reader=read_int4,
268 doc="Four-byte signed integer, little-endian, 2's complement.")
269
270
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -0700271def read_uint4(f):
272 r"""
273 >>> import io
274 >>> read_uint4(io.BytesIO(b'\xff\x00\x00\x00'))
275 255
276 >>> read_uint4(io.BytesIO(b'\x00\x00\x00\x80')) == 2**31
277 True
278 """
279
280 data = f.read(4)
281 if len(data) == 4:
282 return _unpack("<I", data)[0]
283 raise ValueError("not enough data in stream to read uint4")
284
285uint4 = ArgumentDescriptor(
286 name='uint4',
287 n=4,
288 reader=read_uint4,
289 doc="Four-byte unsigned integer, little-endian.")
290
291
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000292def read_stringnl(f, decode=True, stripquotes=True):
Tim Peters55762f52003-01-28 16:01:25 +0000293 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000294 >>> import io
295 >>> read_stringnl(io.BytesIO(b"'abcd'\nefg\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000296 'abcd'
297
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000298 >>> read_stringnl(io.BytesIO(b"\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000299 Traceback (most recent call last):
300 ...
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000301 ValueError: no string quotes around b''
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000302
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000303 >>> read_stringnl(io.BytesIO(b"\n"), stripquotes=False)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000304 ''
305
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000306 >>> read_stringnl(io.BytesIO(b"''\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000307 ''
308
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000309 >>> read_stringnl(io.BytesIO(b'"abcd"'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000310 Traceback (most recent call last):
311 ...
312 ValueError: no newline found when trying to read stringnl
313
314 Embedded escapes are undone in the result.
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000315 >>> read_stringnl(io.BytesIO(br"'a\n\\b\x00c\td'" + b"\n'e'"))
Tim Peters55762f52003-01-28 16:01:25 +0000316 'a\n\\b\x00c\td'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000317 """
318
Guido van Rossum26986312007-07-17 00:19:46 +0000319 data = f.readline()
Guido van Rossum26d95c32007-08-27 23:18:54 +0000320 if not data.endswith(b'\n'):
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000321 raise ValueError("no newline found when trying to read stringnl")
322 data = data[:-1] # lose the newline
323
324 if stripquotes:
Guido van Rossum26d95c32007-08-27 23:18:54 +0000325 for q in (b'"', b"'"):
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000326 if data.startswith(q):
327 if not data.endswith(q):
328 raise ValueError("strinq quote %r not found at both "
329 "ends of %r" % (q, data))
330 data = data[1:-1]
331 break
332 else:
333 raise ValueError("no string quotes around %r" % data)
334
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000335 if decode:
Guido van Rossum98297ee2007-11-06 21:34:58 +0000336 data = codecs.escape_decode(data)[0].decode("ascii")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000337 return data
338
339stringnl = ArgumentDescriptor(
340 name='stringnl',
341 n=UP_TO_NEWLINE,
342 reader=read_stringnl,
343 doc="""A newline-terminated string.
344
345 This is a repr-style string, with embedded escapes, and
346 bracketing quotes.
347 """)
348
349def read_stringnl_noescape(f):
Guido van Rossum98297ee2007-11-06 21:34:58 +0000350 return read_stringnl(f, stripquotes=False)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000351
352stringnl_noescape = ArgumentDescriptor(
353 name='stringnl_noescape',
354 n=UP_TO_NEWLINE,
355 reader=read_stringnl_noescape,
356 doc="""A newline-terminated string.
357
358 This is a str-style string, without embedded escapes,
359 or bracketing quotes. It should consist solely of
360 printable ASCII characters.
361 """)
362
363def read_stringnl_noescape_pair(f):
Tim Peters55762f52003-01-28 16:01:25 +0000364 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000365 >>> import io
366 >>> read_stringnl_noescape_pair(io.BytesIO(b"Queue\nEmpty\njunk"))
Tim Petersd916cf42003-01-27 19:01:47 +0000367 'Queue Empty'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000368 """
369
Tim Petersd916cf42003-01-27 19:01:47 +0000370 return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000371
372stringnl_noescape_pair = ArgumentDescriptor(
373 name='stringnl_noescape_pair',
374 n=UP_TO_NEWLINE,
375 reader=read_stringnl_noescape_pair,
376 doc="""A pair of newline-terminated strings.
377
378 These are str-style strings, without embedded
379 escapes, or bracketing quotes. They should
380 consist solely of printable ASCII characters.
381 The pair is returned as a single string, with
Tim Petersd916cf42003-01-27 19:01:47 +0000382 a single blank separating the two strings.
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000383 """)
384
385def read_string4(f):
Tim Peters55762f52003-01-28 16:01:25 +0000386 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000387 >>> import io
388 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x00abc"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000389 ''
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000390 >>> read_string4(io.BytesIO(b"\x03\x00\x00\x00abcdef"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000391 'abc'
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000392 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x03abcdef"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000393 Traceback (most recent call last):
394 ...
395 ValueError: expected 50331648 bytes in a string4, but only 6 remain
396 """
397
398 n = read_int4(f)
399 if n < 0:
400 raise ValueError("string4 byte count < 0: %d" % n)
401 data = f.read(n)
402 if len(data) == n:
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000403 return data.decode("latin-1")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000404 raise ValueError("expected %d bytes in a string4, but only %d remain" %
405 (n, len(data)))
406
407string4 = ArgumentDescriptor(
408 name="string4",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000409 n=TAKEN_FROM_ARGUMENT4,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000410 reader=read_string4,
411 doc="""A counted string.
412
413 The first argument is a 4-byte little-endian signed int giving
414 the number of bytes in the string, and the second argument is
415 that many bytes.
416 """)
417
418
419def read_string1(f):
Tim Peters55762f52003-01-28 16:01:25 +0000420 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000421 >>> import io
422 >>> read_string1(io.BytesIO(b"\x00"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000423 ''
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000424 >>> read_string1(io.BytesIO(b"\x03abcdef"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000425 'abc'
426 """
427
428 n = read_uint1(f)
429 assert n >= 0
430 data = f.read(n)
431 if len(data) == n:
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000432 return data.decode("latin-1")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000433 raise ValueError("expected %d bytes in a string1, but only %d remain" %
434 (n, len(data)))
435
436string1 = ArgumentDescriptor(
437 name="string1",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000438 n=TAKEN_FROM_ARGUMENT1,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000439 reader=read_string1,
440 doc="""A counted string.
441
442 The first argument is a 1-byte unsigned int giving the number
443 of bytes in the string, and the second argument is that many
444 bytes.
445 """)
446
447
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -0700448def read_bytes1(f):
449 r"""
450 >>> import io
451 >>> read_bytes1(io.BytesIO(b"\x00"))
452 b''
453 >>> read_bytes1(io.BytesIO(b"\x03abcdef"))
454 b'abc'
455 """
456
457 n = read_uint1(f)
458 assert n >= 0
459 data = f.read(n)
460 if len(data) == n:
461 return data
462 raise ValueError("expected %d bytes in a bytes1, but only %d remain" %
463 (n, len(data)))
464
465bytes1 = ArgumentDescriptor(
466 name="bytes1",
467 n=TAKEN_FROM_ARGUMENT1,
468 reader=read_bytes1,
469 doc="""A counted bytes string.
470
471 The first argument is a 1-byte unsigned int giving the number
472 of bytes, and the second argument is that many bytes.
473 """)
474
475
476def read_bytes4(f):
477 r"""
478 >>> import io
479 >>> read_bytes4(io.BytesIO(b"\x00\x00\x00\x00abc"))
480 b''
481 >>> read_bytes4(io.BytesIO(b"\x03\x00\x00\x00abcdef"))
482 b'abc'
483 >>> read_bytes4(io.BytesIO(b"\x00\x00\x00\x03abcdef"))
484 Traceback (most recent call last):
485 ...
486 ValueError: expected 50331648 bytes in a bytes4, but only 6 remain
487 """
488
489 n = read_uint4(f)
490 if n > sys.maxsize:
491 raise ValueError("bytes4 byte count > sys.maxsize: %d" % n)
492 data = f.read(n)
493 if len(data) == n:
494 return data
495 raise ValueError("expected %d bytes in a bytes4, but only %d remain" %
496 (n, len(data)))
497
498bytes4 = ArgumentDescriptor(
499 name="bytes4",
500 n=TAKEN_FROM_ARGUMENT4U,
501 reader=read_bytes4,
502 doc="""A counted bytes string.
503
504 The first argument is a 4-byte little-endian unsigned int giving
505 the number of bytes, and the second argument is that many bytes.
506 """)
507
508
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000509def read_unicodestringnl(f):
Tim Peters55762f52003-01-28 16:01:25 +0000510 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000511 >>> import io
512 >>> read_unicodestringnl(io.BytesIO(b"abc\\uabcd\njunk")) == 'abc\uabcd'
513 True
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000514 """
515
Guido van Rossum26986312007-07-17 00:19:46 +0000516 data = f.readline()
Guido van Rossum26d95c32007-08-27 23:18:54 +0000517 if not data.endswith(b'\n'):
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000518 raise ValueError("no newline found when trying to read "
519 "unicodestringnl")
520 data = data[:-1] # lose the newline
Guido van Rossumef87d6e2007-05-02 19:09:54 +0000521 return str(data, 'raw-unicode-escape')
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000522
523unicodestringnl = ArgumentDescriptor(
524 name='unicodestringnl',
525 n=UP_TO_NEWLINE,
526 reader=read_unicodestringnl,
527 doc="""A newline-terminated Unicode string.
528
529 This is raw-unicode-escape encoded, so consists of
530 printable ASCII characters, and may contain embedded
531 escape sequences.
532 """)
533
534def read_unicodestring4(f):
Tim Peters55762f52003-01-28 16:01:25 +0000535 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000536 >>> import io
537 >>> s = 'abcd\uabcd'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000538 >>> enc = s.encode('utf-8')
539 >>> enc
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000540 b'abcd\xea\xaf\x8d'
541 >>> n = bytes([len(enc), 0, 0, 0]) # little-endian 4-byte length
542 >>> t = read_unicodestring4(io.BytesIO(n + enc + b'junk'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000543 >>> s == t
544 True
545
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000546 >>> read_unicodestring4(io.BytesIO(n + enc[:-1]))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000547 Traceback (most recent call last):
548 ...
549 ValueError: expected 7 bytes in a unicodestring4, but only 6 remain
550 """
551
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -0700552 n = read_uint4(f)
553 if n > sys.maxsize:
554 raise ValueError("unicodestring4 byte count > sys.maxsize: %d" % n)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000555 data = f.read(n)
556 if len(data) == n:
Victor Stinner485fb562010-04-13 11:07:24 +0000557 return str(data, 'utf-8', 'surrogatepass')
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000558 raise ValueError("expected %d bytes in a unicodestring4, but only %d "
559 "remain" % (n, len(data)))
560
561unicodestring4 = ArgumentDescriptor(
562 name="unicodestring4",
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -0700563 n=TAKEN_FROM_ARGUMENT4U,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000564 reader=read_unicodestring4,
565 doc="""A counted Unicode string.
566
567 The first argument is a 4-byte little-endian signed int
568 giving the number of bytes in the string, and the second
569 argument-- the UTF-8 encoding of the Unicode string --
570 contains that many bytes.
571 """)
572
573
574def read_decimalnl_short(f):
Tim Peters55762f52003-01-28 16:01:25 +0000575 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000576 >>> import io
577 >>> read_decimalnl_short(io.BytesIO(b"1234\n56"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000578 1234
579
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000580 >>> read_decimalnl_short(io.BytesIO(b"1234L\n56"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000581 Traceback (most recent call last):
582 ...
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000583 ValueError: trailing 'L' not allowed in b'1234L'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000584 """
585
586 s = read_stringnl(f, decode=False, stripquotes=False)
Guido van Rossum26d95c32007-08-27 23:18:54 +0000587 if s.endswith(b"L"):
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000588 raise ValueError("trailing 'L' not allowed in %r" % s)
589
590 # It's not necessarily true that the result fits in a Python short int:
591 # the pickle may have been written on a 64-bit box. There's also a hack
592 # for True and False here.
Jeremy Hyltona5dc3db2007-08-29 19:07:40 +0000593 if s == b"00":
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000594 return False
Jeremy Hyltona5dc3db2007-08-29 19:07:40 +0000595 elif s == b"01":
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000596 return True
597
Florent Xicluna2bb96f52011-10-23 22:11:00 +0200598 return int(s)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000599
600def read_decimalnl_long(f):
Tim Peters55762f52003-01-28 16:01:25 +0000601 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000602 >>> import io
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000603
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000604 >>> read_decimalnl_long(io.BytesIO(b"1234L\n56"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000605 1234
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000606
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000607 >>> read_decimalnl_long(io.BytesIO(b"123456789012345678901234L\n6"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000608 123456789012345678901234
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000609 """
610
611 s = read_stringnl(f, decode=False, stripquotes=False)
Mark Dickinson8dd05142009-01-20 20:43:58 +0000612 if s[-1:] == b'L':
613 s = s[:-1]
Guido van Rossume2a383d2007-01-15 16:59:06 +0000614 return int(s)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000615
616
617decimalnl_short = ArgumentDescriptor(
618 name='decimalnl_short',
619 n=UP_TO_NEWLINE,
620 reader=read_decimalnl_short,
621 doc="""A newline-terminated decimal integer literal.
622
623 This never has a trailing 'L', and the integer fit
624 in a short Python int on the box where the pickle
625 was written -- but there's no guarantee it will fit
626 in a short Python int on the box where the pickle
627 is read.
628 """)
629
630decimalnl_long = ArgumentDescriptor(
631 name='decimalnl_long',
632 n=UP_TO_NEWLINE,
633 reader=read_decimalnl_long,
634 doc="""A newline-terminated decimal integer literal.
635
636 This has a trailing 'L', and can represent integers
637 of any size.
638 """)
639
640
641def read_floatnl(f):
Tim Peters55762f52003-01-28 16:01:25 +0000642 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000643 >>> import io
644 >>> read_floatnl(io.BytesIO(b"-1.25\n6"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000645 -1.25
646 """
647 s = read_stringnl(f, decode=False, stripquotes=False)
648 return float(s)
649
650floatnl = ArgumentDescriptor(
651 name='floatnl',
652 n=UP_TO_NEWLINE,
653 reader=read_floatnl,
654 doc="""A newline-terminated decimal floating literal.
655
656 In general this requires 17 significant digits for roundtrip
657 identity, and pickling then unpickling infinities, NaNs, and
658 minus zero doesn't work across boxes, or on some boxes even
659 on itself (e.g., Windows can't read the strings it produces
660 for infinities or NaNs).
661 """)
662
663def read_float8(f):
Tim Peters55762f52003-01-28 16:01:25 +0000664 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000665 >>> import io, struct
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000666 >>> raw = struct.pack(">d", -1.25)
667 >>> raw
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000668 b'\xbf\xf4\x00\x00\x00\x00\x00\x00'
669 >>> read_float8(io.BytesIO(raw + b"\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000670 -1.25
671 """
672
673 data = f.read(8)
674 if len(data) == 8:
675 return _unpack(">d", data)[0]
676 raise ValueError("not enough data in stream to read float8")
677
678
679float8 = ArgumentDescriptor(
680 name='float8',
681 n=8,
682 reader=read_float8,
683 doc="""An 8-byte binary representation of a float, big-endian.
684
685 The format is unique to Python, and shared with the struct
Guido van Rossum99603b02007-07-20 00:22:32 +0000686 module (format string '>d') "in theory" (the struct and pickle
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000687 implementations don't share the code -- they should). It's
688 strongly related to the IEEE-754 double format, and, in normal
689 cases, is in fact identical to the big-endian 754 double format.
690 On other boxes the dynamic range is limited to that of a 754
691 double, and "add a half and chop" rounding is used to reduce
692 the precision to 53 bits. However, even on a 754 box,
693 infinities, NaNs, and minus zero may not be handled correctly
694 (may not survive roundtrip pickling intact).
695 """)
696
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000697# Protocol 2 formats
698
Tim Petersc0c12b52003-01-29 00:56:17 +0000699from pickle import decode_long
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000700
701def read_long1(f):
702 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000703 >>> import io
704 >>> read_long1(io.BytesIO(b"\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000705 0
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000706 >>> read_long1(io.BytesIO(b"\x02\xff\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000707 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000708 >>> read_long1(io.BytesIO(b"\x02\xff\x7f"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000709 32767
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000710 >>> read_long1(io.BytesIO(b"\x02\x00\xff"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000711 -256
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000712 >>> read_long1(io.BytesIO(b"\x02\x00\x80"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000713 -32768
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000714 """
715
716 n = read_uint1(f)
717 data = f.read(n)
718 if len(data) != n:
719 raise ValueError("not enough data in stream to read long1")
720 return decode_long(data)
721
722long1 = ArgumentDescriptor(
723 name="long1",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000724 n=TAKEN_FROM_ARGUMENT1,
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000725 reader=read_long1,
726 doc="""A binary long, little-endian, using 1-byte size.
727
728 This first reads one byte as an unsigned size, then reads that
Tim Petersbdbe7412003-01-27 23:54:04 +0000729 many bytes and interprets them as a little-endian 2's-complement long.
Tim Peters4b23f2b2003-01-31 16:43:39 +0000730 If the size is 0, that's taken as a shortcut for the long 0L.
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000731 """)
732
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000733def read_long4(f):
734 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000735 >>> import io
736 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000737 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000738 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x7f"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000739 32767
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000740 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\xff"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000741 -256
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000742 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\x80"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000743 -32768
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000744 >>> read_long1(io.BytesIO(b"\x00\x00\x00\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000745 0
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000746 """
747
748 n = read_int4(f)
749 if n < 0:
Neal Norwitz784a3f52003-01-28 00:20:41 +0000750 raise ValueError("long4 byte count < 0: %d" % n)
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000751 data = f.read(n)
752 if len(data) != n:
Neal Norwitz784a3f52003-01-28 00:20:41 +0000753 raise ValueError("not enough data in stream to read long4")
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000754 return decode_long(data)
755
756long4 = ArgumentDescriptor(
757 name="long4",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000758 n=TAKEN_FROM_ARGUMENT4,
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000759 reader=read_long4,
760 doc="""A binary representation of a long, little-endian.
761
762 This first reads four bytes as a signed size (but requires the
763 size to be >= 0), then reads that many bytes and interprets them
Tim Peters4b23f2b2003-01-31 16:43:39 +0000764 as a little-endian 2's-complement long. If the size is 0, that's taken
Guido van Rossume2a383d2007-01-15 16:59:06 +0000765 as a shortcut for the int 0, although LONG1 should really be used
Tim Peters4b23f2b2003-01-31 16:43:39 +0000766 then instead (and in any case where # of bytes < 256).
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000767 """)
768
769
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000770##############################################################################
771# Object descriptors. The stack used by the pickle machine holds objects,
772# and in the stack_before and stack_after attributes of OpcodeInfo
773# descriptors we need names to describe the various types of objects that can
774# appear on the stack.
775
776class StackObject(object):
777 __slots__ = (
778 # name of descriptor record, for info only
779 'name',
780
781 # type of object, or tuple of type objects (meaning the object can
782 # be of any type in the tuple)
783 'obtype',
784
785 # human-readable docs for this kind of stack object; a string
786 'doc',
787 )
788
789 def __init__(self, name, obtype, doc):
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000790 assert isinstance(name, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000791 self.name = name
792
793 assert isinstance(obtype, type) or isinstance(obtype, tuple)
794 if isinstance(obtype, tuple):
795 for contained in obtype:
796 assert isinstance(contained, type)
797 self.obtype = obtype
798
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000799 assert isinstance(doc, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000800 self.doc = doc
801
Tim Petersc1c2b3e2003-01-29 20:12:21 +0000802 def __repr__(self):
803 return self.name
804
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000805
806pyint = StackObject(
807 name='int',
808 obtype=int,
809 doc="A short (as opposed to long) Python integer object.")
810
811pylong = StackObject(
812 name='long',
Guido van Rossume2a383d2007-01-15 16:59:06 +0000813 obtype=int,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000814 doc="A long (as opposed to short) Python integer object.")
815
816pyinteger_or_bool = StackObject(
817 name='int_or_bool',
Florent Xicluna02ea12b22010-07-28 16:39:41 +0000818 obtype=(int, bool),
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000819 doc="A Python integer object (short or long), or "
820 "a Python bool.")
821
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000822pybool = StackObject(
823 name='bool',
824 obtype=(bool,),
825 doc="A Python bool object.")
826
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000827pyfloat = StackObject(
828 name='float',
829 obtype=float,
830 doc="A Python float object.")
831
832pystring = StackObject(
Guido van Rossumf4169812008-03-17 22:56:06 +0000833 name='string',
834 obtype=bytes,
835 doc="A Python (8-bit) string object.")
836
837pybytes = StackObject(
Guido van Rossum98297ee2007-11-06 21:34:58 +0000838 name='bytes',
839 obtype=bytes,
840 doc="A Python bytes object.")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000841
842pyunicode = StackObject(
Guido van Rossum98297ee2007-11-06 21:34:58 +0000843 name='str',
Guido van Rossumef87d6e2007-05-02 19:09:54 +0000844 obtype=str,
Guido van Rossumf4169812008-03-17 22:56:06 +0000845 doc="A Python (Unicode) string object.")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000846
847pynone = StackObject(
848 name="None",
849 obtype=type(None),
850 doc="The Python None object.")
851
852pytuple = StackObject(
853 name="tuple",
854 obtype=tuple,
855 doc="A Python tuple object.")
856
857pylist = StackObject(
858 name="list",
859 obtype=list,
860 doc="A Python list object.")
861
862pydict = StackObject(
863 name="dict",
864 obtype=dict,
865 doc="A Python dict object.")
866
867anyobject = StackObject(
868 name='any',
869 obtype=object,
870 doc="Any kind of object whatsoever.")
871
872markobject = StackObject(
873 name="mark",
874 obtype=StackObject,
875 doc="""'The mark' is a unique object.
876
877 Opcodes that operate on a variable number of objects
878 generally don't embed the count of objects in the opcode,
879 or pull it off the stack. Instead the MARK opcode is used
880 to push a special marker object on the stack, and then
881 some other opcodes grab all the objects from the top of
882 the stack down to (but not including) the topmost marker
883 object.
884 """)
885
886stackslice = StackObject(
887 name="stackslice",
888 obtype=StackObject,
889 doc="""An object representing a contiguous slice of the stack.
890
891 This is used in conjuction with markobject, to represent all
892 of the stack following the topmost markobject. For example,
893 the POP_MARK opcode changes the stack from
894
895 [..., markobject, stackslice]
896 to
897 [...]
898
899 No matter how many object are on the stack after the topmost
900 markobject, POP_MARK gets rid of all of them (including the
901 topmost markobject too).
902 """)
903
904##############################################################################
905# Descriptors for pickle opcodes.
906
907class OpcodeInfo(object):
908
909 __slots__ = (
910 # symbolic name of opcode; a string
911 'name',
912
913 # the code used in a bytestream to represent the opcode; a
914 # one-character string
915 'code',
916
917 # If the opcode has an argument embedded in the byte string, an
918 # instance of ArgumentDescriptor specifying its type. Note that
919 # arg.reader(s) can be used to read and decode the argument from
920 # the bytestream s, and arg.doc documents the format of the raw
921 # argument bytes. If the opcode doesn't have an argument embedded
922 # in the bytestream, arg should be None.
923 'arg',
924
925 # what the stack looks like before this opcode runs; a list
926 'stack_before',
927
928 # what the stack looks like after this opcode runs; a list
929 'stack_after',
930
931 # the protocol number in which this opcode was introduced; an int
932 'proto',
933
934 # human-readable docs for this opcode; a string
935 'doc',
936 )
937
938 def __init__(self, name, code, arg,
939 stack_before, stack_after, proto, doc):
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000940 assert isinstance(name, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000941 self.name = name
942
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000943 assert isinstance(code, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000944 assert len(code) == 1
945 self.code = code
946
947 assert arg is None or isinstance(arg, ArgumentDescriptor)
948 self.arg = arg
949
950 assert isinstance(stack_before, list)
951 for x in stack_before:
952 assert isinstance(x, StackObject)
953 self.stack_before = stack_before
954
955 assert isinstance(stack_after, list)
956 for x in stack_after:
957 assert isinstance(x, StackObject)
958 self.stack_after = stack_after
959
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -0700960 assert isinstance(proto, int) and 0 <= proto <= pickle.HIGHEST_PROTOCOL
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000961 self.proto = proto
962
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000963 assert isinstance(doc, str)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000964 self.doc = doc
965
966I = OpcodeInfo
967opcodes = [
968
969 # Ways to spell integers.
970
971 I(name='INT',
972 code='I',
973 arg=decimalnl_short,
974 stack_before=[],
975 stack_after=[pyinteger_or_bool],
976 proto=0,
977 doc="""Push an integer or bool.
978
979 The argument is a newline-terminated decimal literal string.
980
981 The intent may have been that this always fit in a short Python int,
982 but INT can be generated in pickles written on a 64-bit box that
983 require a Python long on a 32-bit box. The difference between this
984 and LONG then is that INT skips a trailing 'L', and produces a short
985 int whenever possible.
986
987 Another difference is due to that, when bool was introduced as a
988 distinct type in 2.3, builtin names True and False were also added to
989 2.2.2, mapping to ints 1 and 0. For compatibility in both directions,
990 True gets pickled as INT + "I01\\n", and False as INT + "I00\\n".
991 Leading zeroes are never produced for a genuine integer. The 2.3
992 (and later) unpicklers special-case these and return bool instead;
993 earlier unpicklers ignore the leading "0" and return the int.
994 """),
995
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000996 I(name='BININT',
997 code='J',
998 arg=int4,
999 stack_before=[],
1000 stack_after=[pyint],
1001 proto=1,
1002 doc="""Push a four-byte signed integer.
1003
1004 This handles the full range of Python (short) integers on a 32-bit
1005 box, directly as binary bytes (1 for the opcode and 4 for the integer).
1006 If the integer is non-negative and fits in 1 or 2 bytes, pickling via
1007 BININT1 or BININT2 saves space.
1008 """),
1009
1010 I(name='BININT1',
1011 code='K',
1012 arg=uint1,
1013 stack_before=[],
1014 stack_after=[pyint],
1015 proto=1,
1016 doc="""Push a one-byte unsigned integer.
1017
1018 This is a space optimization for pickling very small non-negative ints,
1019 in range(256).
1020 """),
1021
1022 I(name='BININT2',
1023 code='M',
1024 arg=uint2,
1025 stack_before=[],
1026 stack_after=[pyint],
1027 proto=1,
1028 doc="""Push a two-byte unsigned integer.
1029
1030 This is a space optimization for pickling small positive ints, in
1031 range(256, 2**16). Integers in range(256) can also be pickled via
1032 BININT2, but BININT1 instead saves a byte.
1033 """),
1034
Tim Petersfdc03462003-01-28 04:56:33 +00001035 I(name='LONG',
1036 code='L',
1037 arg=decimalnl_long,
1038 stack_before=[],
1039 stack_after=[pylong],
1040 proto=0,
1041 doc="""Push a long integer.
1042
1043 The same as INT, except that the literal ends with 'L', and always
1044 unpickles to a Python long. There doesn't seem a real purpose to the
1045 trailing 'L'.
1046
1047 Note that LONG takes time quadratic in the number of digits when
1048 unpickling (this is simply due to the nature of decimal->binary
1049 conversion). Proto 2 added linear-time (in C; still quadratic-time
1050 in Python) LONG1 and LONG4 opcodes.
1051 """),
1052
1053 I(name="LONG1",
1054 code='\x8a',
1055 arg=long1,
1056 stack_before=[],
1057 stack_after=[pylong],
1058 proto=2,
1059 doc="""Long integer using one-byte length.
1060
1061 A more efficient encoding of a Python long; the long1 encoding
1062 says it all."""),
1063
1064 I(name="LONG4",
1065 code='\x8b',
1066 arg=long4,
1067 stack_before=[],
1068 stack_after=[pylong],
1069 proto=2,
1070 doc="""Long integer using found-byte length.
1071
1072 A more efficient encoding of a Python long; the long4 encoding
1073 says it all."""),
1074
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001075 # Ways to spell strings (8-bit, not Unicode).
1076
1077 I(name='STRING',
1078 code='S',
1079 arg=stringnl,
1080 stack_before=[],
1081 stack_after=[pystring],
1082 proto=0,
1083 doc="""Push a Python string object.
1084
1085 The argument is a repr-style string, with bracketing quote characters,
1086 and perhaps embedded escapes. The argument extends until the next
Guido van Rossumf4169812008-03-17 22:56:06 +00001087 newline character. (Actually, they are decoded into a str instance
1088 using the encoding given to the Unpickler constructor. or the default,
1089 'ASCII'.)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001090 """),
1091
1092 I(name='BINSTRING',
1093 code='T',
1094 arg=string4,
1095 stack_before=[],
1096 stack_after=[pystring],
1097 proto=1,
1098 doc="""Push a Python string object.
1099
1100 There are two arguments: the first is a 4-byte little-endian signed int
1101 giving the number of bytes in the string, and the second is that many
Guido van Rossumf4169812008-03-17 22:56:06 +00001102 bytes, which are taken literally as the string content. (Actually,
1103 they are decoded into a str instance using the encoding given to the
1104 Unpickler constructor. or the default, 'ASCII'.)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001105 """),
1106
1107 I(name='SHORT_BINSTRING',
1108 code='U',
1109 arg=string1,
1110 stack_before=[],
1111 stack_after=[pystring],
1112 proto=1,
1113 doc="""Push a Python string object.
1114
1115 There are two arguments: the first is a 1-byte unsigned int giving
1116 the number of bytes in the string, and the second is that many bytes,
Guido van Rossumf4169812008-03-17 22:56:06 +00001117 which are taken literally as the string content. (Actually, they
1118 are decoded into a str instance using the encoding given to the
1119 Unpickler constructor. or the default, 'ASCII'.)
1120 """),
1121
1122 # Bytes (protocol 3 only; older protocols don't support bytes at all)
1123
1124 I(name='BINBYTES',
1125 code='B',
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001126 arg=bytes4,
Guido van Rossumf4169812008-03-17 22:56:06 +00001127 stack_before=[],
1128 stack_after=[pybytes],
1129 proto=3,
1130 doc="""Push a Python bytes object.
1131
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001132 There are two arguments: the first is a 4-byte little-endian unsigned int
1133 giving the number of bytes, and the second is that many bytes, which are
1134 taken literally as the bytes content.
Guido van Rossumf4169812008-03-17 22:56:06 +00001135 """),
1136
1137 I(name='SHORT_BINBYTES',
1138 code='C',
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001139 arg=bytes1,
Guido van Rossumf4169812008-03-17 22:56:06 +00001140 stack_before=[],
1141 stack_after=[pybytes],
Collin Wintere61d4372009-05-20 17:46:47 +00001142 proto=3,
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001143 doc="""Push a Python bytes object.
Guido van Rossumf4169812008-03-17 22:56:06 +00001144
1145 There are two arguments: the first is a 1-byte unsigned int giving
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001146 the number of bytes, and the second is that many bytes, which are taken
1147 literally as the string content.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001148 """),
1149
1150 # Ways to spell None.
1151
1152 I(name='NONE',
1153 code='N',
1154 arg=None,
1155 stack_before=[],
1156 stack_after=[pynone],
1157 proto=0,
1158 doc="Push None on the stack."),
1159
Tim Petersfdc03462003-01-28 04:56:33 +00001160 # Ways to spell bools, starting with proto 2. See INT for how this was
1161 # done before proto 2.
1162
1163 I(name='NEWTRUE',
1164 code='\x88',
1165 arg=None,
1166 stack_before=[],
1167 stack_after=[pybool],
1168 proto=2,
1169 doc="""True.
1170
1171 Push True onto the stack."""),
1172
1173 I(name='NEWFALSE',
1174 code='\x89',
1175 arg=None,
1176 stack_before=[],
1177 stack_after=[pybool],
1178 proto=2,
1179 doc="""True.
1180
1181 Push False onto the stack."""),
1182
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001183 # Ways to spell Unicode strings.
1184
1185 I(name='UNICODE',
1186 code='V',
1187 arg=unicodestringnl,
1188 stack_before=[],
1189 stack_after=[pyunicode],
1190 proto=0, # this may be pure-text, but it's a later addition
1191 doc="""Push a Python Unicode string object.
1192
1193 The argument is a raw-unicode-escape encoding of a Unicode string,
1194 and so may contain embedded escape sequences. The argument extends
1195 until the next newline character.
1196 """),
1197
1198 I(name='BINUNICODE',
1199 code='X',
1200 arg=unicodestring4,
1201 stack_before=[],
1202 stack_after=[pyunicode],
1203 proto=1,
1204 doc="""Push a Python Unicode string object.
1205
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001206 There are two arguments: the first is a 4-byte little-endian unsigned int
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001207 giving the number of bytes in the string. The second is that many
1208 bytes, and is the UTF-8 encoding of the Unicode string.
1209 """),
1210
1211 # Ways to spell floats.
1212
1213 I(name='FLOAT',
1214 code='F',
1215 arg=floatnl,
1216 stack_before=[],
1217 stack_after=[pyfloat],
1218 proto=0,
1219 doc="""Newline-terminated decimal float literal.
1220
1221 The argument is repr(a_float), and in general requires 17 significant
1222 digits for roundtrip conversion to be an identity (this is so for
1223 IEEE-754 double precision values, which is what Python float maps to
1224 on most boxes).
1225
1226 In general, FLOAT cannot be used to transport infinities, NaNs, or
1227 minus zero across boxes (or even on a single box, if the platform C
1228 library can't read the strings it produces for such things -- Windows
1229 is like that), but may do less damage than BINFLOAT on boxes with
1230 greater precision or dynamic range than IEEE-754 double.
1231 """),
1232
1233 I(name='BINFLOAT',
1234 code='G',
1235 arg=float8,
1236 stack_before=[],
1237 stack_after=[pyfloat],
1238 proto=1,
1239 doc="""Float stored in binary form, with 8 bytes of data.
1240
1241 This generally requires less than half the space of FLOAT encoding.
1242 In general, BINFLOAT cannot be used to transport infinities, NaNs, or
1243 minus zero, raises an exception if the exponent exceeds the range of
1244 an IEEE-754 double, and retains no more than 53 bits of precision (if
1245 there are more than that, "add a half and chop" rounding is used to
1246 cut it back to 53 significant bits).
1247 """),
1248
1249 # Ways to build lists.
1250
1251 I(name='EMPTY_LIST',
1252 code=']',
1253 arg=None,
1254 stack_before=[],
1255 stack_after=[pylist],
1256 proto=1,
1257 doc="Push an empty list."),
1258
1259 I(name='APPEND',
1260 code='a',
1261 arg=None,
1262 stack_before=[pylist, anyobject],
1263 stack_after=[pylist],
1264 proto=0,
1265 doc="""Append an object to a list.
1266
1267 Stack before: ... pylist anyobject
1268 Stack after: ... pylist+[anyobject]
Tim Peters81098ac2003-01-28 05:12:08 +00001269
1270 although pylist is really extended in-place.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001271 """),
1272
1273 I(name='APPENDS',
1274 code='e',
1275 arg=None,
1276 stack_before=[pylist, markobject, stackslice],
1277 stack_after=[pylist],
1278 proto=1,
1279 doc="""Extend a list by a slice of stack objects.
1280
1281 Stack before: ... pylist markobject stackslice
1282 Stack after: ... pylist+stackslice
Tim Peters81098ac2003-01-28 05:12:08 +00001283
1284 although pylist is really extended in-place.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001285 """),
1286
1287 I(name='LIST',
1288 code='l',
1289 arg=None,
1290 stack_before=[markobject, stackslice],
1291 stack_after=[pylist],
1292 proto=0,
1293 doc="""Build a list out of the topmost stack slice, after markobject.
1294
1295 All the stack entries following the topmost markobject are placed into
1296 a single Python list, which single list object replaces all of the
1297 stack from the topmost markobject onward. For example,
1298
1299 Stack before: ... markobject 1 2 3 'abc'
1300 Stack after: ... [1, 2, 3, 'abc']
1301 """),
1302
1303 # Ways to build tuples.
1304
1305 I(name='EMPTY_TUPLE',
1306 code=')',
1307 arg=None,
1308 stack_before=[],
1309 stack_after=[pytuple],
1310 proto=1,
1311 doc="Push an empty tuple."),
1312
1313 I(name='TUPLE',
1314 code='t',
1315 arg=None,
1316 stack_before=[markobject, stackslice],
1317 stack_after=[pytuple],
1318 proto=0,
1319 doc="""Build a tuple out of the topmost stack slice, after markobject.
1320
1321 All the stack entries following the topmost markobject are placed into
1322 a single Python tuple, which single tuple object replaces all of the
1323 stack from the topmost markobject onward. For example,
1324
1325 Stack before: ... markobject 1 2 3 'abc'
1326 Stack after: ... (1, 2, 3, 'abc')
1327 """),
1328
Tim Petersfdc03462003-01-28 04:56:33 +00001329 I(name='TUPLE1',
1330 code='\x85',
1331 arg=None,
1332 stack_before=[anyobject],
1333 stack_after=[pytuple],
1334 proto=2,
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001335 doc="""Build a one-tuple out of the topmost item on the stack.
Tim Petersfdc03462003-01-28 04:56:33 +00001336
1337 This code pops one value off the stack and pushes a tuple of
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001338 length 1 whose one item is that value back onto it. In other
1339 words:
Tim Petersfdc03462003-01-28 04:56:33 +00001340
1341 stack[-1] = tuple(stack[-1:])
1342 """),
1343
1344 I(name='TUPLE2',
1345 code='\x86',
1346 arg=None,
1347 stack_before=[anyobject, anyobject],
1348 stack_after=[pytuple],
1349 proto=2,
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001350 doc="""Build a two-tuple out of the top two items on the stack.
Tim Petersfdc03462003-01-28 04:56:33 +00001351
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001352 This code pops two values off the stack and pushes a tuple of
1353 length 2 whose items are those values back onto it. In other
1354 words:
Tim Petersfdc03462003-01-28 04:56:33 +00001355
1356 stack[-2:] = [tuple(stack[-2:])]
1357 """),
1358
1359 I(name='TUPLE3',
1360 code='\x87',
1361 arg=None,
1362 stack_before=[anyobject, anyobject, anyobject],
1363 stack_after=[pytuple],
1364 proto=2,
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001365 doc="""Build a three-tuple out of the top three items on the stack.
Tim Petersfdc03462003-01-28 04:56:33 +00001366
Alexander Belopolsky44c2ffd2010-07-16 14:39:45 +00001367 This code pops three values off the stack and pushes a tuple of
1368 length 3 whose items are those values back onto it. In other
1369 words:
Tim Petersfdc03462003-01-28 04:56:33 +00001370
1371 stack[-3:] = [tuple(stack[-3:])]
1372 """),
1373
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001374 # Ways to build dicts.
1375
1376 I(name='EMPTY_DICT',
1377 code='}',
1378 arg=None,
1379 stack_before=[],
1380 stack_after=[pydict],
1381 proto=1,
1382 doc="Push an empty dict."),
1383
1384 I(name='DICT',
1385 code='d',
1386 arg=None,
1387 stack_before=[markobject, stackslice],
1388 stack_after=[pydict],
1389 proto=0,
1390 doc="""Build a dict out of the topmost stack slice, after markobject.
1391
1392 All the stack entries following the topmost markobject are placed into
1393 a single Python dict, which single dict object replaces all of the
1394 stack from the topmost markobject onward. The stack slice alternates
1395 key, value, key, value, .... For example,
1396
1397 Stack before: ... markobject 1 2 3 'abc'
1398 Stack after: ... {1: 2, 3: 'abc'}
1399 """),
1400
1401 I(name='SETITEM',
1402 code='s',
1403 arg=None,
1404 stack_before=[pydict, anyobject, anyobject],
1405 stack_after=[pydict],
1406 proto=0,
1407 doc="""Add a key+value pair to an existing dict.
1408
1409 Stack before: ... pydict key value
1410 Stack after: ... pydict
1411
1412 where pydict has been modified via pydict[key] = value.
1413 """),
1414
1415 I(name='SETITEMS',
1416 code='u',
1417 arg=None,
1418 stack_before=[pydict, markobject, stackslice],
1419 stack_after=[pydict],
1420 proto=1,
1421 doc="""Add an arbitrary number of key+value pairs to an existing dict.
1422
1423 The slice of the stack following the topmost markobject is taken as
1424 an alternating sequence of keys and values, added to the dict
1425 immediately under the topmost markobject. Everything at and after the
1426 topmost markobject is popped, leaving the mutated dict at the top
1427 of the stack.
1428
1429 Stack before: ... pydict markobject key_1 value_1 ... key_n value_n
1430 Stack after: ... pydict
1431
1432 where pydict has been modified via pydict[key_i] = value_i for i in
1433 1, 2, ..., n, and in that order.
1434 """),
1435
1436 # Stack manipulation.
1437
1438 I(name='POP',
1439 code='0',
1440 arg=None,
1441 stack_before=[anyobject],
1442 stack_after=[],
1443 proto=0,
1444 doc="Discard the top stack item, shrinking the stack by one item."),
1445
1446 I(name='DUP',
1447 code='2',
1448 arg=None,
1449 stack_before=[anyobject],
1450 stack_after=[anyobject, anyobject],
1451 proto=0,
1452 doc="Push the top stack item onto the stack again, duplicating it."),
1453
1454 I(name='MARK',
1455 code='(',
1456 arg=None,
1457 stack_before=[],
1458 stack_after=[markobject],
1459 proto=0,
1460 doc="""Push markobject onto the stack.
1461
1462 markobject is a unique object, used by other opcodes to identify a
1463 region of the stack containing a variable number of objects for them
1464 to work on. See markobject.doc for more detail.
1465 """),
1466
1467 I(name='POP_MARK',
1468 code='1',
1469 arg=None,
1470 stack_before=[markobject, stackslice],
1471 stack_after=[],
Collin Wintere61d4372009-05-20 17:46:47 +00001472 proto=1,
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001473 doc="""Pop all the stack objects at and above the topmost markobject.
1474
1475 When an opcode using a variable number of stack objects is done,
1476 POP_MARK is used to remove those objects, and to remove the markobject
1477 that delimited their starting position on the stack.
1478 """),
1479
1480 # Memo manipulation. There are really only two operations (get and put),
1481 # each in all-text, "short binary", and "long binary" flavors.
1482
1483 I(name='GET',
1484 code='g',
1485 arg=decimalnl_short,
1486 stack_before=[],
1487 stack_after=[anyobject],
1488 proto=0,
1489 doc="""Read an object from the memo and push it on the stack.
1490
Ezio Melotti13925002011-03-16 11:05:33 +02001491 The index of the memo object to push is given by the newline-terminated
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001492 decimal string following. BINGET and LONG_BINGET are space-optimized
1493 versions.
1494 """),
1495
1496 I(name='BINGET',
1497 code='h',
1498 arg=uint1,
1499 stack_before=[],
1500 stack_after=[anyobject],
1501 proto=1,
1502 doc="""Read an object from the memo and push it on the stack.
1503
1504 The index of the memo object to push is given by the 1-byte unsigned
1505 integer following.
1506 """),
1507
1508 I(name='LONG_BINGET',
1509 code='j',
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001510 arg=uint4,
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001511 stack_before=[],
1512 stack_after=[anyobject],
1513 proto=1,
1514 doc="""Read an object from the memo and push it on the stack.
1515
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001516 The index of the memo object to push is given by the 4-byte unsigned
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001517 little-endian integer following.
1518 """),
1519
1520 I(name='PUT',
1521 code='p',
1522 arg=decimalnl_short,
1523 stack_before=[],
1524 stack_after=[],
1525 proto=0,
1526 doc="""Store the stack top into the memo. The stack is not popped.
1527
1528 The index of the memo location to write into is given by the newline-
1529 terminated decimal string following. BINPUT and LONG_BINPUT are
1530 space-optimized versions.
1531 """),
1532
1533 I(name='BINPUT',
1534 code='q',
1535 arg=uint1,
1536 stack_before=[],
1537 stack_after=[],
1538 proto=1,
1539 doc="""Store the stack top into the memo. The stack is not popped.
1540
1541 The index of the memo location to write into is given by the 1-byte
1542 unsigned integer following.
1543 """),
1544
1545 I(name='LONG_BINPUT',
1546 code='r',
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001547 arg=uint4,
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001548 stack_before=[],
1549 stack_after=[],
1550 proto=1,
1551 doc="""Store the stack top into the memo. The stack is not popped.
1552
1553 The index of the memo location to write into is given by the 4-byte
Alexandre Vassalotti8db89ca2013-04-14 03:30:35 -07001554 unsigned little-endian integer following.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001555 """),
1556
Tim Petersfdc03462003-01-28 04:56:33 +00001557 # Access the extension registry (predefined objects). Akin to the GET
1558 # family.
1559
1560 I(name='EXT1',
1561 code='\x82',
1562 arg=uint1,
1563 stack_before=[],
1564 stack_after=[anyobject],
1565 proto=2,
1566 doc="""Extension code.
1567
1568 This code and the similar EXT2 and EXT4 allow using a registry
1569 of popular objects that are pickled by name, typically classes.
1570 It is envisioned that through a global negotiation and
1571 registration process, third parties can set up a mapping between
1572 ints and object names.
1573
1574 In order to guarantee pickle interchangeability, the extension
1575 code registry ought to be global, although a range of codes may
1576 be reserved for private use.
1577
1578 EXT1 has a 1-byte integer argument. This is used to index into the
1579 extension registry, and the object at that index is pushed on the stack.
1580 """),
1581
1582 I(name='EXT2',
1583 code='\x83',
1584 arg=uint2,
1585 stack_before=[],
1586 stack_after=[anyobject],
1587 proto=2,
1588 doc="""Extension code.
1589
1590 See EXT1. EXT2 has a two-byte integer argument.
1591 """),
1592
1593 I(name='EXT4',
1594 code='\x84',
1595 arg=int4,
1596 stack_before=[],
1597 stack_after=[anyobject],
1598 proto=2,
1599 doc="""Extension code.
1600
1601 See EXT1. EXT4 has a four-byte integer argument.
1602 """),
1603
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001604 # Push a class object, or module function, on the stack, via its module
1605 # and name.
1606
1607 I(name='GLOBAL',
1608 code='c',
1609 arg=stringnl_noescape_pair,
1610 stack_before=[],
1611 stack_after=[anyobject],
1612 proto=0,
1613 doc="""Push a global object (module.attr) on the stack.
1614
1615 Two newline-terminated strings follow the GLOBAL opcode. The first is
1616 taken as a module name, and the second as a class name. The class
1617 object module.class is pushed on the stack. More accurately, the
1618 object returned by self.find_class(module, class) is pushed on the
1619 stack, so unpickling subclasses can override this form of lookup.
1620 """),
1621
1622 # Ways to build objects of classes pickle doesn't know about directly
1623 # (user-defined classes). I despair of documenting this accurately
1624 # and comprehensibly -- you really have to read the pickle code to
1625 # find all the special cases.
1626
1627 I(name='REDUCE',
1628 code='R',
1629 arg=None,
1630 stack_before=[anyobject, anyobject],
1631 stack_after=[anyobject],
1632 proto=0,
1633 doc="""Push an object built from a callable and an argument tuple.
1634
1635 The opcode is named to remind of the __reduce__() method.
1636
1637 Stack before: ... callable pytuple
1638 Stack after: ... callable(*pytuple)
1639
1640 The callable and the argument tuple are the first two items returned
1641 by a __reduce__ method. Applying the callable to the argtuple is
1642 supposed to reproduce the original object, or at least get it started.
1643 If the __reduce__ method returns a 3-tuple, the last component is an
1644 argument to be passed to the object's __setstate__, and then the REDUCE
1645 opcode is followed by code to create setstate's argument, and then a
1646 BUILD opcode to apply __setstate__ to that argument.
1647
Guido van Rossum13257902007-06-07 23:15:56 +00001648 If not isinstance(callable, type), REDUCE complains unless the
Alexandre Vassalottif7fa63d2008-05-11 08:55:36 +00001649 callable has been registered with the copyreg module's
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001650 safe_constructors dict, or the callable has a magic
1651 '__safe_for_unpickling__' attribute with a true value. I'm not sure
1652 why it does this, but I've sure seen this complaint often enough when
1653 I didn't want to <wink>.
1654 """),
1655
1656 I(name='BUILD',
1657 code='b',
1658 arg=None,
1659 stack_before=[anyobject, anyobject],
1660 stack_after=[anyobject],
1661 proto=0,
1662 doc="""Finish building an object, via __setstate__ or dict update.
1663
1664 Stack before: ... anyobject argument
1665 Stack after: ... anyobject
1666
1667 where anyobject may have been mutated, as follows:
1668
1669 If the object has a __setstate__ method,
1670
1671 anyobject.__setstate__(argument)
1672
1673 is called.
1674
1675 Else the argument must be a dict, the object must have a __dict__, and
1676 the object is updated via
1677
1678 anyobject.__dict__.update(argument)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001679 """),
1680
1681 I(name='INST',
1682 code='i',
1683 arg=stringnl_noescape_pair,
1684 stack_before=[markobject, stackslice],
1685 stack_after=[anyobject],
1686 proto=0,
1687 doc="""Build a class instance.
1688
1689 This is the protocol 0 version of protocol 1's OBJ opcode.
1690 INST is followed by two newline-terminated strings, giving a
1691 module and class name, just as for the GLOBAL opcode (and see
1692 GLOBAL for more details about that). self.find_class(module, name)
1693 is used to get a class object.
1694
1695 In addition, all the objects on the stack following the topmost
1696 markobject are gathered into a tuple and popped (along with the
1697 topmost markobject), just as for the TUPLE opcode.
1698
1699 Now it gets complicated. If all of these are true:
1700
1701 + The argtuple is empty (markobject was at the top of the stack
1702 at the start).
1703
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001704 + The class object does not have a __getinitargs__ attribute.
1705
1706 then we want to create an old-style class instance without invoking
1707 its __init__() method (pickle has waffled on this over the years; not
1708 calling __init__() is current wisdom). In this case, an instance of
1709 an old-style dummy class is created, and then we try to rebind its
1710 __class__ attribute to the desired class object. If this succeeds,
Guido van Rossuma8add0e2007-05-14 22:03:55 +00001711 the new instance object is pushed on the stack, and we're done.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001712
1713 Else (the argtuple is not empty, it's not an old-style class object,
1714 or the class object does have a __getinitargs__ attribute), the code
1715 first insists that the class object have a __safe_for_unpickling__
1716 attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE,
1717 it doesn't matter whether this attribute has a true or false value, it
Guido van Rossum99603b02007-07-20 00:22:32 +00001718 only matters whether it exists (XXX this is a bug). If
1719 __safe_for_unpickling__ doesn't exist, UnpicklingError is raised.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001720
1721 Else (the class object does have a __safe_for_unpickling__ attr),
1722 the class object obtained from INST's arguments is applied to the
1723 argtuple obtained from the stack, and the resulting instance object
1724 is pushed on the stack.
Tim Peters2b93c4c2003-01-30 16:35:08 +00001725
1726 NOTE: checks for __safe_for_unpickling__ went away in Python 2.3.
Florent Xiclunaaa6c1d22011-12-12 18:54:29 +01001727 NOTE: the distinction between old-style and new-style classes does
1728 not make sense in Python 3.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001729 """),
1730
1731 I(name='OBJ',
1732 code='o',
1733 arg=None,
1734 stack_before=[markobject, anyobject, stackslice],
1735 stack_after=[anyobject],
1736 proto=1,
1737 doc="""Build a class instance.
1738
1739 This is the protocol 1 version of protocol 0's INST opcode, and is
1740 very much like it. The major difference is that the class object
1741 is taken off the stack, allowing it to be retrieved from the memo
1742 repeatedly if several instances of the same class are created. This
1743 can be much more efficient (in both time and space) than repeatedly
1744 embedding the module and class names in INST opcodes.
1745
1746 Unlike INST, OBJ takes no arguments from the opcode stream. Instead
1747 the class object is taken off the stack, immediately above the
1748 topmost markobject:
1749
1750 Stack before: ... markobject classobject stackslice
1751 Stack after: ... new_instance_object
1752
1753 As for INST, the remainder of the stack above the markobject is
1754 gathered into an argument tuple, and then the logic seems identical,
Guido van Rossumecb11042003-01-29 06:24:30 +00001755 except that no __safe_for_unpickling__ check is done (XXX this is
Guido van Rossum99603b02007-07-20 00:22:32 +00001756 a bug). See INST for the gory details.
Tim Peters2b93c4c2003-01-30 16:35:08 +00001757
1758 NOTE: In Python 2.3, INST and OBJ are identical except for how they
1759 get the class object. That was always the intent; the implementations
1760 had diverged for accidental reasons.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001761 """),
1762
Tim Petersfdc03462003-01-28 04:56:33 +00001763 I(name='NEWOBJ',
1764 code='\x81',
1765 arg=None,
1766 stack_before=[anyobject, anyobject],
1767 stack_after=[anyobject],
1768 proto=2,
1769 doc="""Build an object instance.
1770
1771 The stack before should be thought of as containing a class
1772 object followed by an argument tuple (the tuple being the stack
1773 top). Call these cls and args. They are popped off the stack,
1774 and the value returned by cls.__new__(cls, *args) is pushed back
1775 onto the stack.
1776 """),
1777
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001778 # Machine control.
1779
Tim Petersfdc03462003-01-28 04:56:33 +00001780 I(name='PROTO',
1781 code='\x80',
1782 arg=uint1,
1783 stack_before=[],
1784 stack_after=[],
1785 proto=2,
1786 doc="""Protocol version indicator.
1787
1788 For protocol 2 and above, a pickle must start with this opcode.
1789 The argument is the protocol version, an int in range(2, 256).
1790 """),
1791
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001792 I(name='STOP',
1793 code='.',
1794 arg=None,
1795 stack_before=[anyobject],
1796 stack_after=[],
1797 proto=0,
1798 doc="""Stop the unpickling machine.
1799
1800 Every pickle ends with this opcode. The object at the top of the stack
1801 is popped, and that's the result of unpickling. The stack should be
1802 empty then.
1803 """),
1804
1805 # Ways to deal with persistent IDs.
1806
1807 I(name='PERSID',
1808 code='P',
1809 arg=stringnl_noescape,
1810 stack_before=[],
1811 stack_after=[anyobject],
1812 proto=0,
1813 doc="""Push an object identified by a persistent ID.
1814
1815 The pickle module doesn't define what a persistent ID means. PERSID's
1816 argument is a newline-terminated str-style (no embedded escapes, no
1817 bracketing quote characters) string, which *is* "the persistent ID".
1818 The unpickler passes this string to self.persistent_load(). Whatever
1819 object that returns is pushed on the stack. There is no implementation
1820 of persistent_load() in Python's unpickler: it must be supplied by an
1821 unpickler subclass.
1822 """),
1823
1824 I(name='BINPERSID',
1825 code='Q',
1826 arg=None,
1827 stack_before=[anyobject],
1828 stack_after=[anyobject],
1829 proto=1,
1830 doc="""Push an object identified by a persistent ID.
1831
1832 Like PERSID, except the persistent ID is popped off the stack (instead
1833 of being a string embedded in the opcode bytestream). The persistent
1834 ID is passed to self.persistent_load(), and whatever object that
1835 returns is pushed on the stack. See PERSID for more detail.
1836 """),
1837]
1838del I
1839
1840# Verify uniqueness of .name and .code members.
1841name2i = {}
1842code2i = {}
1843
1844for i, d in enumerate(opcodes):
1845 if d.name in name2i:
1846 raise ValueError("repeated name %r at indices %d and %d" %
1847 (d.name, name2i[d.name], i))
1848 if d.code in code2i:
1849 raise ValueError("repeated code %r at indices %d and %d" %
1850 (d.code, code2i[d.code], i))
1851
1852 name2i[d.name] = i
1853 code2i[d.code] = i
1854
1855del name2i, code2i, i, d
1856
1857##############################################################################
1858# Build a code2op dict, mapping opcode characters to OpcodeInfo records.
1859# Also ensure we've got the same stuff as pickle.py, although the
1860# introspection here is dicey.
1861
1862code2op = {}
1863for d in opcodes:
1864 code2op[d.code] = d
1865del d
1866
1867def assure_pickle_consistency(verbose=False):
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001868
1869 copy = code2op.copy()
1870 for name in pickle.__all__:
1871 if not re.match("[A-Z][A-Z0-9_]+$", name):
1872 if verbose:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001873 print("skipping %r: it doesn't look like an opcode name" % name)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001874 continue
1875 picklecode = getattr(pickle, name)
Guido van Rossum617dbc42007-05-07 23:57:08 +00001876 if not isinstance(picklecode, bytes) or len(picklecode) != 1:
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001877 if verbose:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001878 print(("skipping %r: value %r doesn't look like a pickle "
1879 "code" % (name, picklecode)))
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001880 continue
Guido van Rossum617dbc42007-05-07 23:57:08 +00001881 picklecode = picklecode.decode("latin-1")
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001882 if picklecode in copy:
1883 if verbose:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001884 print("checking name %r w/ code %r for consistency" % (
1885 name, picklecode))
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001886 d = copy[picklecode]
1887 if d.name != name:
1888 raise ValueError("for pickle code %r, pickle.py uses name %r "
1889 "but we're using name %r" % (picklecode,
1890 name,
1891 d.name))
1892 # Forget this one. Any left over in copy at the end are a problem
1893 # of a different kind.
1894 del copy[picklecode]
1895 else:
1896 raise ValueError("pickle.py appears to have a pickle opcode with "
1897 "name %r and code %r, but we don't" %
1898 (name, picklecode))
1899 if copy:
1900 msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"]
1901 for code, d in copy.items():
1902 msg.append(" name %r with code %r" % (d.name, code))
1903 raise ValueError("\n".join(msg))
1904
1905assure_pickle_consistency()
Tim Petersc0c12b52003-01-29 00:56:17 +00001906del assure_pickle_consistency
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001907
1908##############################################################################
1909# A pickle opcode generator.
1910
1911def genops(pickle):
Guido van Rossuma72ded92003-01-27 19:40:47 +00001912 """Generate all the opcodes in a pickle.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001913
1914 'pickle' is a file-like object, or string, containing the pickle.
1915
1916 Each opcode in the pickle is generated, from the current pickle position,
1917 stopping after a STOP opcode is delivered. A triple is generated for
1918 each opcode:
1919
1920 opcode, arg, pos
1921
1922 opcode is an OpcodeInfo record, describing the current opcode.
1923
1924 If the opcode has an argument embedded in the pickle, arg is its decoded
1925 value, as a Python object. If the opcode doesn't have an argument, arg
1926 is None.
1927
1928 If the pickle has a tell() method, pos was the value of pickle.tell()
Guido van Rossum34d19282007-08-09 01:03:29 +00001929 before reading the current opcode. If the pickle is a bytes object,
1930 it's wrapped in a BytesIO object, and the latter's tell() result is
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001931 used. Else (the pickle doesn't have a tell(), and it's not obvious how
1932 to query its current position) pos is None.
1933 """
1934
Guido van Rossum98297ee2007-11-06 21:34:58 +00001935 if isinstance(pickle, bytes_types):
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001936 import io
1937 pickle = io.BytesIO(pickle)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001938
1939 if hasattr(pickle, "tell"):
1940 getpos = pickle.tell
1941 else:
1942 getpos = lambda: None
1943
1944 while True:
1945 pos = getpos()
1946 code = pickle.read(1)
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001947 opcode = code2op.get(code.decode("latin-1"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001948 if opcode is None:
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001949 if code == b"":
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001950 raise ValueError("pickle exhausted before seeing STOP")
1951 else:
1952 raise ValueError("at position %s, opcode %r unknown" % (
1953 pos is None and "<unknown>" or pos,
1954 code))
1955 if opcode.arg is None:
1956 arg = None
1957 else:
1958 arg = opcode.arg.reader(pickle)
1959 yield opcode, arg, pos
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001960 if code == b'.':
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001961 assert opcode.name == 'STOP'
1962 break
1963
1964##############################################################################
Christian Heimes3feef612008-02-11 06:19:17 +00001965# A pickle optimizer.
1966
1967def optimize(p):
1968 'Optimize a pickle string by removing unused PUT opcodes'
1969 gets = set() # set of args used by a GET opcode
1970 puts = [] # (arg, startpos, stoppos) for the PUT opcodes
1971 prevpos = None # set to pos if previous opcode was a PUT
1972 for opcode, arg, pos in genops(p):
1973 if prevpos is not None:
1974 puts.append((prevarg, prevpos, pos))
1975 prevpos = None
1976 if 'PUT' in opcode.name:
1977 prevarg, prevpos = arg, pos
1978 elif 'GET' in opcode.name:
1979 gets.add(arg)
1980
1981 # Copy the pickle string except for PUTS without a corresponding GET
1982 s = []
1983 i = 0
1984 for arg, start, stop in puts:
1985 j = stop if (arg in gets) else start
1986 s.append(p[i:j])
1987 i = stop
1988 s.append(p[i:])
Christian Heimes126d29a2008-02-11 22:57:17 +00001989 return b''.join(s)
Christian Heimes3feef612008-02-11 06:19:17 +00001990
1991##############################################################################
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001992# A symbolic pickle disassembler.
1993
Alexander Belopolsky929d3842010-07-17 15:51:21 +00001994def dis(pickle, out=None, memo=None, indentlevel=4, annotate=0):
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001995 """Produce a symbolic disassembly of a pickle.
1996
1997 'pickle' is a file-like object, or string, containing a (at least one)
1998 pickle. The pickle is disassembled from the current position, through
1999 the first STOP opcode encountered.
2000
2001 Optional arg 'out' is a file-like object to which the disassembly is
2002 printed. It defaults to sys.stdout.
2003
Tim Peters62235e72003-02-05 19:55:53 +00002004 Optional arg 'memo' is a Python dict, used as the pickle's memo. It
2005 may be mutated by dis(), if the pickle contains PUT or BINPUT opcodes.
2006 Passing the same memo object to another dis() call then allows disassembly
2007 to proceed across multiple pickles that were all created by the same
2008 pickler with the same memo. Ordinarily you don't need to worry about this.
2009
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002010 Optional arg 'indentlevel' is the number of blanks by which to indent
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002011 a new MARK level. It defaults to 4.
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002012
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002013 Optional arg 'annotate' if nonzero instructs dis() to add short
2014 description of the opcode on each line of disassembled output.
2015 The value given to 'annotate' must be an integer and is used as a
2016 hint for the column where annotation should start. The default
2017 value is 0, meaning no annotations.
2018
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002019 In addition to printing the disassembly, some sanity checks are made:
2020
2021 + All embedded opcode arguments "make sense".
2022
2023 + Explicit and implicit pop operations have enough items on the stack.
2024
2025 + When an opcode implicitly refers to a markobject, a markobject is
2026 actually on the stack.
2027
2028 + A memo entry isn't referenced before it's defined.
2029
2030 + The markobject isn't stored in the memo.
2031
2032 + A memo entry isn't redefined.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002033 """
2034
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002035 # Most of the hair here is for sanity checks, but most of it is needed
2036 # anyway to detect when a protocol 0 POP takes a MARK off the stack
2037 # (which in turn is needed to indent MARK blocks correctly).
2038
2039 stack = [] # crude emulation of unpickler stack
Tim Peters62235e72003-02-05 19:55:53 +00002040 if memo is None:
2041 memo = {} # crude emulation of unpicker memo
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002042 maxproto = -1 # max protocol number seen
2043 markstack = [] # bytecode positions of MARK opcodes
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002044 indentchunk = ' ' * indentlevel
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002045 errormsg = None
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002046 annocol = annotate # columnt hint for annotations
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002047 for opcode, arg, pos in genops(pickle):
2048 if pos is not None:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00002049 print("%5d:" % pos, end=' ', file=out)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002050
Tim Petersd0f7c862003-01-28 15:27:57 +00002051 line = "%-4s %s%s" % (repr(opcode.code)[1:-1],
2052 indentchunk * len(markstack),
2053 opcode.name)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002054
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002055 maxproto = max(maxproto, opcode.proto)
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002056 before = opcode.stack_before # don't mutate
2057 after = opcode.stack_after # don't mutate
Tim Peters43277d62003-01-30 15:02:12 +00002058 numtopop = len(before)
2059
2060 # See whether a MARK should be popped.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002061 markmsg = None
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002062 if markobject in before or (opcode.name == "POP" and
2063 stack and
2064 stack[-1] is markobject):
2065 assert markobject not in after
Tim Peters43277d62003-01-30 15:02:12 +00002066 if __debug__:
2067 if markobject in before:
2068 assert before[-1] is stackslice
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002069 if markstack:
2070 markpos = markstack.pop()
2071 if markpos is None:
2072 markmsg = "(MARK at unknown opcode offset)"
2073 else:
2074 markmsg = "(MARK at %d)" % markpos
2075 # Pop everything at and after the topmost markobject.
2076 while stack[-1] is not markobject:
2077 stack.pop()
2078 stack.pop()
Tim Peters43277d62003-01-30 15:02:12 +00002079 # Stop later code from popping too much.
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002080 try:
Tim Peters43277d62003-01-30 15:02:12 +00002081 numtopop = before.index(markobject)
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002082 except ValueError:
2083 assert opcode.name == "POP"
Tim Peters43277d62003-01-30 15:02:12 +00002084 numtopop = 0
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002085 else:
2086 errormsg = markmsg = "no MARK exists on stack"
2087
2088 # Check for correct memo usage.
2089 if opcode.name in ("PUT", "BINPUT", "LONG_BINPUT"):
Tim Peters43277d62003-01-30 15:02:12 +00002090 assert arg is not None
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002091 if arg in memo:
2092 errormsg = "memo key %r already defined" % arg
2093 elif not stack:
2094 errormsg = "stack is empty -- can't store into memo"
2095 elif stack[-1] is markobject:
2096 errormsg = "can't store markobject in the memo"
2097 else:
2098 memo[arg] = stack[-1]
2099
2100 elif opcode.name in ("GET", "BINGET", "LONG_BINGET"):
2101 if arg in memo:
2102 assert len(after) == 1
2103 after = [memo[arg]] # for better stack emulation
2104 else:
2105 errormsg = "memo key %r has never been stored into" % arg
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002106
2107 if arg is not None or markmsg:
2108 # make a mild effort to align arguments
2109 line += ' ' * (10 - len(opcode.name))
2110 if arg is not None:
2111 line += ' ' + repr(arg)
2112 if markmsg:
2113 line += ' ' + markmsg
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002114 if annotate:
2115 line += ' ' * (annocol - len(line))
2116 # make a mild effort to align annotations
2117 annocol = len(line)
2118 if annocol > 50:
2119 annocol = annotate
2120 line += ' ' + opcode.doc.split('\n', 1)[0]
Guido van Rossumbe19ed72007-02-09 05:37:30 +00002121 print(line, file=out)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002122
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002123 if errormsg:
2124 # Note that we delayed complaining until the offending opcode
2125 # was printed.
2126 raise ValueError(errormsg)
2127
2128 # Emulate the stack effects.
Tim Peters43277d62003-01-30 15:02:12 +00002129 if len(stack) < numtopop:
2130 raise ValueError("tries to pop %d items from stack with "
2131 "only %d items" % (numtopop, len(stack)))
2132 if numtopop:
2133 del stack[-numtopop:]
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002134 if markobject in after:
Tim Peters43277d62003-01-30 15:02:12 +00002135 assert markobject not in before
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002136 markstack.append(pos)
2137
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002138 stack.extend(after)
2139
Guido van Rossumbe19ed72007-02-09 05:37:30 +00002140 print("highest protocol among opcodes =", maxproto, file=out)
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002141 if stack:
2142 raise ValueError("stack not empty after STOP: %r" % stack)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002143
Tim Peters90718a42005-02-15 16:22:34 +00002144# For use in the doctest, simply as an example of a class to pickle.
2145class _Example:
2146 def __init__(self, value):
2147 self.value = value
2148
Guido van Rossum03e35322003-01-28 15:37:13 +00002149_dis_test = r"""
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002150>>> import pickle
Guido van Rossumf4169812008-03-17 22:56:06 +00002151>>> x = [1, 2, (3, 4), {b'abc': "def"}]
2152>>> pkl0 = pickle.dumps(x, 0)
2153>>> dis(pkl0)
Tim Petersd0f7c862003-01-28 15:27:57 +00002154 0: ( MARK
2155 1: l LIST (MARK at 0)
2156 2: p PUT 0
Guido van Rossumf4100002007-01-15 00:21:46 +00002157 5: L LONG 1
Mark Dickinson8dd05142009-01-20 20:43:58 +00002158 9: a APPEND
2159 10: L LONG 2
2160 14: a APPEND
2161 15: ( MARK
2162 16: L LONG 3
2163 20: L LONG 4
2164 24: t TUPLE (MARK at 15)
2165 25: p PUT 1
2166 28: a APPEND
2167 29: ( MARK
2168 30: d DICT (MARK at 29)
2169 31: p PUT 2
Alexandre Vassalotti3bfc65a2011-12-13 13:08:09 -05002170 34: c GLOBAL '_codecs encode'
2171 50: p PUT 3
2172 53: ( MARK
2173 54: V UNICODE 'abc'
Antoine Pitroud9dfaa92009-06-04 20:32:06 +00002174 59: p PUT 4
Alexandre Vassalotti3bfc65a2011-12-13 13:08:09 -05002175 62: V UNICODE 'latin1'
2176 70: p PUT 5
2177 73: t TUPLE (MARK at 53)
2178 74: p PUT 6
2179 77: R REDUCE
2180 78: p PUT 7
2181 81: V UNICODE 'def'
2182 86: p PUT 8
2183 89: s SETITEM
2184 90: a APPEND
2185 91: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002186highest protocol among opcodes = 0
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002187
2188Try again with a "binary" pickle.
2189
Guido van Rossumf4169812008-03-17 22:56:06 +00002190>>> pkl1 = pickle.dumps(x, 1)
2191>>> dis(pkl1)
Tim Petersd0f7c862003-01-28 15:27:57 +00002192 0: ] EMPTY_LIST
2193 1: q BINPUT 0
2194 3: ( MARK
2195 4: K BININT1 1
2196 6: K BININT1 2
2197 8: ( MARK
2198 9: K BININT1 3
2199 11: K BININT1 4
2200 13: t TUPLE (MARK at 8)
2201 14: q BINPUT 1
2202 16: } EMPTY_DICT
2203 17: q BINPUT 2
Alexandre Vassalotti3bfc65a2011-12-13 13:08:09 -05002204 19: c GLOBAL '_codecs encode'
2205 35: q BINPUT 3
2206 37: ( MARK
2207 38: X BINUNICODE 'abc'
2208 46: q BINPUT 4
2209 48: X BINUNICODE 'latin1'
2210 59: q BINPUT 5
2211 61: t TUPLE (MARK at 37)
2212 62: q BINPUT 6
2213 64: R REDUCE
2214 65: q BINPUT 7
2215 67: X BINUNICODE 'def'
2216 75: q BINPUT 8
2217 77: s SETITEM
2218 78: e APPENDS (MARK at 3)
2219 79: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002220highest protocol among opcodes = 1
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002221
2222Exercise the INST/OBJ/BUILD family.
2223
Mark Dickinsoncddcf442009-01-24 21:46:33 +00002224>>> import pickletools
2225>>> dis(pickle.dumps(pickletools.dis, 0))
2226 0: c GLOBAL 'pickletools dis'
2227 17: p PUT 0
2228 20: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002229highest protocol among opcodes = 0
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002230
Tim Peters90718a42005-02-15 16:22:34 +00002231>>> from pickletools import _Example
2232>>> x = [_Example(42)] * 2
Guido van Rossumf29d3d62003-01-27 22:47:53 +00002233>>> dis(pickle.dumps(x, 0))
Tim Petersd0f7c862003-01-28 15:27:57 +00002234 0: ( MARK
2235 1: l LIST (MARK at 0)
2236 2: p PUT 0
Antoine Pitroud9dfaa92009-06-04 20:32:06 +00002237 5: c GLOBAL 'copy_reg _reconstructor'
2238 30: p PUT 1
2239 33: ( MARK
2240 34: c GLOBAL 'pickletools _Example'
2241 56: p PUT 2
2242 59: c GLOBAL '__builtin__ object'
2243 79: p PUT 3
2244 82: N NONE
2245 83: t TUPLE (MARK at 33)
2246 84: p PUT 4
2247 87: R REDUCE
2248 88: p PUT 5
2249 91: ( MARK
2250 92: d DICT (MARK at 91)
2251 93: p PUT 6
2252 96: V UNICODE 'value'
2253 103: p PUT 7
2254 106: L LONG 42
2255 111: s SETITEM
2256 112: b BUILD
Mark Dickinson8dd05142009-01-20 20:43:58 +00002257 113: a APPEND
Antoine Pitroud9dfaa92009-06-04 20:32:06 +00002258 114: g GET 5
2259 117: a APPEND
2260 118: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002261highest protocol among opcodes = 0
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002262
2263>>> dis(pickle.dumps(x, 1))
Tim Petersd0f7c862003-01-28 15:27:57 +00002264 0: ] EMPTY_LIST
2265 1: q BINPUT 0
2266 3: ( MARK
Antoine Pitroud9dfaa92009-06-04 20:32:06 +00002267 4: c GLOBAL 'copy_reg _reconstructor'
2268 29: q BINPUT 1
2269 31: ( MARK
2270 32: c GLOBAL 'pickletools _Example'
2271 54: q BINPUT 2
2272 56: c GLOBAL '__builtin__ object'
2273 76: q BINPUT 3
2274 78: N NONE
2275 79: t TUPLE (MARK at 31)
2276 80: q BINPUT 4
2277 82: R REDUCE
2278 83: q BINPUT 5
2279 85: } EMPTY_DICT
2280 86: q BINPUT 6
2281 88: X BINUNICODE 'value'
2282 98: q BINPUT 7
2283 100: K BININT1 42
2284 102: s SETITEM
2285 103: b BUILD
2286 104: h BINGET 5
2287 106: e APPENDS (MARK at 3)
2288 107: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002289highest protocol among opcodes = 1
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002290
2291Try "the canonical" recursive-object test.
2292
2293>>> L = []
2294>>> T = L,
2295>>> L.append(T)
2296>>> L[0] is T
2297True
2298>>> T[0] is L
2299True
2300>>> L[0][0] is L
2301True
2302>>> T[0][0] is T
2303True
Guido van Rossumf29d3d62003-01-27 22:47:53 +00002304>>> dis(pickle.dumps(L, 0))
Tim Petersd0f7c862003-01-28 15:27:57 +00002305 0: ( MARK
2306 1: l LIST (MARK at 0)
2307 2: p PUT 0
2308 5: ( MARK
2309 6: g GET 0
2310 9: t TUPLE (MARK at 5)
2311 10: p PUT 1
2312 13: a APPEND
2313 14: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002314highest protocol among opcodes = 0
2315
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002316>>> dis(pickle.dumps(L, 1))
Tim Petersd0f7c862003-01-28 15:27:57 +00002317 0: ] EMPTY_LIST
2318 1: q BINPUT 0
2319 3: ( MARK
2320 4: h BINGET 0
2321 6: t TUPLE (MARK at 3)
2322 7: q BINPUT 1
2323 9: a APPEND
2324 10: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002325highest protocol among opcodes = 1
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002326
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002327Note that, in the protocol 0 pickle of the recursive tuple, the disassembler
2328has to emulate the stack in order to realize that the POP opcode at 16 gets
2329rid of the MARK at 0.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002330
Guido van Rossumf29d3d62003-01-27 22:47:53 +00002331>>> dis(pickle.dumps(T, 0))
Tim Petersd0f7c862003-01-28 15:27:57 +00002332 0: ( MARK
2333 1: ( MARK
2334 2: l LIST (MARK at 1)
2335 3: p PUT 0
2336 6: ( MARK
2337 7: g GET 0
2338 10: t TUPLE (MARK at 6)
2339 11: p PUT 1
2340 14: a APPEND
2341 15: 0 POP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002342 16: 0 POP (MARK at 0)
2343 17: g GET 1
2344 20: . STOP
2345highest protocol among opcodes = 0
2346
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002347>>> dis(pickle.dumps(T, 1))
Tim Petersd0f7c862003-01-28 15:27:57 +00002348 0: ( MARK
2349 1: ] EMPTY_LIST
2350 2: q BINPUT 0
2351 4: ( MARK
2352 5: h BINGET 0
2353 7: t TUPLE (MARK at 4)
2354 8: q BINPUT 1
2355 10: a APPEND
2356 11: 1 POP_MARK (MARK at 0)
2357 12: h BINGET 1
2358 14: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002359highest protocol among opcodes = 1
Tim Petersd0f7c862003-01-28 15:27:57 +00002360
2361Try protocol 2.
2362
2363>>> dis(pickle.dumps(L, 2))
2364 0: \x80 PROTO 2
2365 2: ] EMPTY_LIST
2366 3: q BINPUT 0
2367 5: h BINGET 0
2368 7: \x85 TUPLE1
2369 8: q BINPUT 1
2370 10: a APPEND
2371 11: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002372highest protocol among opcodes = 2
Tim Petersd0f7c862003-01-28 15:27:57 +00002373
2374>>> dis(pickle.dumps(T, 2))
2375 0: \x80 PROTO 2
2376 2: ] EMPTY_LIST
2377 3: q BINPUT 0
2378 5: h BINGET 0
2379 7: \x85 TUPLE1
2380 8: q BINPUT 1
2381 10: a APPEND
2382 11: 0 POP
2383 12: h BINGET 1
2384 14: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002385highest protocol among opcodes = 2
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002386
2387Try protocol 3 with annotations:
2388
2389>>> dis(pickle.dumps(T, 3), annotate=1)
2390 0: \x80 PROTO 3 Protocol version indicator.
2391 2: ] EMPTY_LIST Push an empty list.
2392 3: q BINPUT 0 Store the stack top into the memo. The stack is not popped.
2393 5: h BINGET 0 Read an object from the memo and push it on the stack.
2394 7: \x85 TUPLE1 Build a one-tuple out of the topmost item on the stack.
2395 8: q BINPUT 1 Store the stack top into the memo. The stack is not popped.
2396 10: a APPEND Append an object to a list.
2397 11: 0 POP Discard the top stack item, shrinking the stack by one item.
2398 12: h BINGET 1 Read an object from the memo and push it on the stack.
2399 14: . STOP Stop the unpickling machine.
2400highest protocol among opcodes = 2
2401
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002402"""
2403
Tim Peters62235e72003-02-05 19:55:53 +00002404_memo_test = r"""
2405>>> import pickle
Guido van Rossumcfe5f202007-05-08 21:26:54 +00002406>>> import io
2407>>> f = io.BytesIO()
Tim Peters62235e72003-02-05 19:55:53 +00002408>>> p = pickle.Pickler(f, 2)
2409>>> x = [1, 2, 3]
2410>>> p.dump(x)
2411>>> p.dump(x)
2412>>> f.seek(0)
Guido van Rossumcfe5f202007-05-08 21:26:54 +000024130
Tim Peters62235e72003-02-05 19:55:53 +00002414>>> memo = {}
2415>>> dis(f, memo=memo)
2416 0: \x80 PROTO 2
2417 2: ] EMPTY_LIST
2418 3: q BINPUT 0
2419 5: ( MARK
2420 6: K BININT1 1
2421 8: K BININT1 2
2422 10: K BININT1 3
2423 12: e APPENDS (MARK at 5)
2424 13: . STOP
2425highest protocol among opcodes = 2
2426>>> dis(f, memo=memo)
2427 14: \x80 PROTO 2
2428 16: h BINGET 0
2429 18: . STOP
2430highest protocol among opcodes = 2
2431"""
2432
Guido van Rossum57028352003-01-28 15:09:10 +00002433__test__ = {'disassembler_test': _dis_test,
Tim Peters62235e72003-02-05 19:55:53 +00002434 'disassembler_memo_test': _memo_test,
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002435 }
2436
2437def _test():
2438 import doctest
2439 return doctest.testmod()
2440
2441if __name__ == "__main__":
Alexander Belopolsky60c762b2010-07-03 20:35:53 +00002442 import sys, argparse
2443 parser = argparse.ArgumentParser(
2444 description='disassemble one or more pickle files')
2445 parser.add_argument(
2446 'pickle_file', type=argparse.FileType('br'),
2447 nargs='*', help='the pickle file')
2448 parser.add_argument(
2449 '-o', '--output', default=sys.stdout, type=argparse.FileType('w'),
2450 help='the file where the output should be written')
2451 parser.add_argument(
2452 '-m', '--memo', action='store_true',
2453 help='preserve memo between disassemblies')
2454 parser.add_argument(
2455 '-l', '--indentlevel', default=4, type=int,
2456 help='the number of blanks by which to indent a new MARK level')
2457 parser.add_argument(
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002458 '-a', '--annotate', action='store_true',
2459 help='annotate each line with a short opcode description')
2460 parser.add_argument(
Alexander Belopolsky60c762b2010-07-03 20:35:53 +00002461 '-p', '--preamble', default="==> {name} <==",
2462 help='if more than one pickle file is specified, print this before'
2463 ' each disassembly')
2464 parser.add_argument(
2465 '-t', '--test', action='store_true',
2466 help='run self-test suite')
2467 parser.add_argument(
2468 '-v', action='store_true',
2469 help='run verbosely; only affects self-test run')
2470 args = parser.parse_args()
2471 if args.test:
2472 _test()
2473 else:
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002474 annotate = 30 if args.annotate else 0
Alexander Belopolsky60c762b2010-07-03 20:35:53 +00002475 if not args.pickle_file:
2476 parser.print_help()
2477 elif len(args.pickle_file) == 1:
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002478 dis(args.pickle_file[0], args.output, None,
2479 args.indentlevel, annotate)
Alexander Belopolsky60c762b2010-07-03 20:35:53 +00002480 else:
2481 memo = {} if args.memo else None
2482 for f in args.pickle_file:
2483 preamble = args.preamble.format(name=f.name)
2484 args.output.write(preamble + '\n')
Alexander Belopolsky929d3842010-07-17 15:51:21 +00002485 dis(f, args.output, memo, args.indentlevel, annotate)