blob: f3e8fbce0a3ab32fbc9bb79b6eaa37ce37b1602c [file] [log] [blame]
Skip Montanaro54455942003-01-29 15:41:33 +00001'''"Executable documentation" for the pickle module.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002
3Extensive comments about the pickle protocols and pickle-machine opcodes
4can be found here. Some functions meant for external use:
5
6genops(pickle)
7 Generate all the opcodes in a pickle, as (opcode, arg, position) triples.
8
Andrew M. Kuchlingd0c53fe2004-08-07 16:51:30 +00009dis(pickle, out=None, memo=None, indentlevel=4)
Tim Peters8ecfc8e2003-01-27 18:51:48 +000010 Print a symbolic disassembly of a pickle.
Skip Montanaro54455942003-01-29 15:41:33 +000011'''
Tim Peters8ecfc8e2003-01-27 18:51:48 +000012
Walter Dörwald42748a82007-06-12 16:40:17 +000013import codecs
14
Tim Peters90cf2122004-11-06 23:45:48 +000015__all__ = ['dis',
16 'genops',
17 ]
18
Tim Peters8ecfc8e2003-01-27 18:51:48 +000019# Other ideas:
20#
21# - A pickle verifier: read a pickle and check it exhaustively for
Tim Petersc1c2b3e2003-01-29 20:12:21 +000022# well-formedness. dis() does a lot of this already.
Tim Peters8ecfc8e2003-01-27 18:51:48 +000023#
24# - A protocol identifier: examine a pickle and return its protocol number
25# (== the highest .proto attr value among all the opcodes in the pickle).
Tim Petersc1c2b3e2003-01-29 20:12:21 +000026# dis() already prints this info at the end.
Tim Peters8ecfc8e2003-01-27 18:51:48 +000027#
28# - A pickle optimizer: for example, tuple-building code is sometimes more
29# elaborate than necessary, catering for the possibility that the tuple
30# is recursive. Or lots of times a PUT is generated that's never accessed
31# by a later GET.
32
33
34"""
35"A pickle" is a program for a virtual pickle machine (PM, but more accurately
36called an unpickling machine). It's a sequence of opcodes, interpreted by the
37PM, building an arbitrarily complex Python object.
38
39For the most part, the PM is very simple: there are no looping, testing, or
40conditional instructions, no arithmetic and no function calls. Opcodes are
41executed once each, from first to last, until a STOP opcode is reached.
42
43The PM has two data areas, "the stack" and "the memo".
44
45Many opcodes push Python objects onto the stack; e.g., INT pushes a Python
46integer object on the stack, whose value is gotten from a decimal string
47literal immediately following the INT opcode in the pickle bytestream. Other
48opcodes take Python objects off the stack. The result of unpickling is
49whatever object is left on the stack when the final STOP opcode is executed.
50
51The memo is simply an array of objects, or it can be implemented as a dict
52mapping little integers to objects. The memo serves as the PM's "long term
53memory", and the little integers indexing the memo are akin to variable
54names. Some opcodes pop a stack object into the memo at a given index,
55and others push a memo object at a given index onto the stack again.
56
57At heart, that's all the PM has. Subtleties arise for these reasons:
58
59+ Object identity. Objects can be arbitrarily complex, and subobjects
60 may be shared (for example, the list [a, a] refers to the same object a
61 twice). It can be vital that unpickling recreate an isomorphic object
62 graph, faithfully reproducing sharing.
63
64+ Recursive objects. For example, after "L = []; L.append(L)", L is a
65 list, and L[0] is the same list. This is related to the object identity
66 point, and some sequences of pickle opcodes are subtle in order to
67 get the right result in all cases.
68
69+ Things pickle doesn't know everything about. Examples of things pickle
70 does know everything about are Python's builtin scalar and container
71 types, like ints and tuples. They generally have opcodes dedicated to
72 them. For things like module references and instances of user-defined
73 classes, pickle's knowledge is limited. Historically, many enhancements
74 have been made to the pickle protocol in order to do a better (faster,
75 and/or more compact) job on those.
76
77+ Backward compatibility and micro-optimization. As explained below,
78 pickle opcodes never go away, not even when better ways to do a thing
79 get invented. The repertoire of the PM just keeps growing over time.
Tim Petersfdc03462003-01-28 04:56:33 +000080 For example, protocol 0 had two opcodes for building Python integers (INT
81 and LONG), protocol 1 added three more for more-efficient pickling of short
82 integers, and protocol 2 added two more for more-efficient pickling of
83 long integers (before protocol 2, the only ways to pickle a Python long
84 took time quadratic in the number of digits, for both pickling and
85 unpickling). "Opcode bloat" isn't so much a subtlety as a source of
Tim Peters8ecfc8e2003-01-27 18:51:48 +000086 wearying complication.
87
88
89Pickle protocols:
90
91For compatibility, the meaning of a pickle opcode never changes. Instead new
92pickle opcodes get added, and each version's unpickler can handle all the
93pickle opcodes in all protocol versions to date. So old pickles continue to
94be readable forever. The pickler can generally be told to restrict itself to
95the subset of opcodes available under previous protocol versions too, so that
96users can create pickles under the current version readable by older
97versions. However, a pickle does not contain its version number embedded
98within it. If an older unpickler tries to read a pickle using a later
99protocol, the result is most likely an exception due to seeing an unknown (in
100the older unpickler) opcode.
101
102The original pickle used what's now called "protocol 0", and what was called
103"text mode" before Python 2.3. The entire pickle bytestream is made up of
104printable 7-bit ASCII characters, plus the newline character, in protocol 0.
Tim Petersfdc03462003-01-28 04:56:33 +0000105That's why it was called text mode. Protocol 0 is small and elegant, but
106sometimes painfully inefficient.
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000107
108The second major set of additions is now called "protocol 1", and was called
109"binary mode" before Python 2.3. This added many opcodes with arguments
110consisting of arbitrary bytes, including NUL bytes and unprintable "high bit"
111bytes. Binary mode pickles can be substantially smaller than equivalent
112text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte
113int as 4 bytes following the opcode, which is cheaper to unpickle than the
Tim Petersfdc03462003-01-28 04:56:33 +0000114(perhaps) 11-character decimal string attached to INT. Protocol 1 also added
115a number of opcodes that operate on many stack elements at once (like APPENDS
Tim Peters81098ac2003-01-28 05:12:08 +0000116and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE).
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000117
118The third major set of additions came in Python 2.3, and is called "protocol
Tim Petersfdc03462003-01-28 04:56:33 +00001192". This added:
120
121- A better way to pickle instances of new-style classes (NEWOBJ).
122
123- A way for a pickle to identify its protocol (PROTO).
124
125- Time- and space- efficient pickling of long ints (LONG{1,4}).
126
127- Shortcuts for small tuples (TUPLE{1,2,3}}.
128
129- Dedicated opcodes for bools (NEWTRUE, NEWFALSE).
130
131- The "extension registry", a vector of popular objects that can be pushed
132 efficiently by index (EXT{1,2,4}). This is akin to the memo and GET, but
133 the registry contents are predefined (there's nothing akin to the memo's
134 PUT).
Guido van Rossumecb11042003-01-29 06:24:30 +0000135
Skip Montanaro54455942003-01-29 15:41:33 +0000136Another independent change with Python 2.3 is the abandonment of any
137pretense that it might be safe to load pickles received from untrusted
Guido van Rossumecb11042003-01-29 06:24:30 +0000138parties -- no sufficient security analysis has been done to guarantee
Skip Montanaro54455942003-01-29 15:41:33 +0000139this and there isn't a use case that warrants the expense of such an
Guido van Rossumecb11042003-01-29 06:24:30 +0000140analysis.
141
142To this end, all tests for __safe_for_unpickling__ or for
143copy_reg.safe_constructors are removed from the unpickling code.
144References to these variables in the descriptions below are to be seen
145as describing unpickling in Python 2.2 and before.
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000146"""
147
148# Meta-rule: Descriptions are stored in instances of descriptor objects,
149# with plain constructors. No meta-language is defined from which
150# descriptors could be constructed. If you want, e.g., XML, write a little
151# program to generate XML from the objects.
152
153##############################################################################
154# Some pickle opcodes have an argument, following the opcode in the
155# bytestream. An argument is of a specific type, described by an instance
156# of ArgumentDescriptor. These are not to be confused with arguments taken
157# off the stack -- ArgumentDescriptor applies only to arguments embedded in
158# the opcode stream, immediately following an opcode.
159
160# Represents the number of bytes consumed by an argument delimited by the
161# next newline character.
162UP_TO_NEWLINE = -1
163
164# Represents the number of bytes consumed by a two-argument opcode where
165# the first argument gives the number of bytes in the second argument.
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000166TAKEN_FROM_ARGUMENT1 = -2 # num bytes is 1-byte unsigned int
167TAKEN_FROM_ARGUMENT4 = -3 # num bytes is 4-byte signed little-endian int
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000168
169class ArgumentDescriptor(object):
170 __slots__ = (
171 # name of descriptor record, also a module global name; a string
172 'name',
173
174 # length of argument, in bytes; an int; UP_TO_NEWLINE and
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000175 # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length
176 # cases
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000177 'n',
178
179 # a function taking a file-like object, reading this kind of argument
180 # from the object at the current position, advancing the current
181 # position by n bytes, and returning the value of the argument
182 'reader',
183
184 # human-readable docs for this arg descriptor; a string
185 'doc',
186 )
187
188 def __init__(self, name, n, reader, doc):
189 assert isinstance(name, str)
190 self.name = name
191
192 assert isinstance(n, int) and (n >= 0 or
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000193 n in (UP_TO_NEWLINE,
194 TAKEN_FROM_ARGUMENT1,
195 TAKEN_FROM_ARGUMENT4))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000196 self.n = n
197
198 self.reader = reader
199
200 assert isinstance(doc, str)
201 self.doc = doc
202
203from struct import unpack as _unpack
204
205def read_uint1(f):
Tim Peters55762f52003-01-28 16:01:25 +0000206 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000207 >>> import io
208 >>> read_uint1(io.BytesIO(b'\xff'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000209 255
210 """
211
212 data = f.read(1)
213 if data:
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000214 return data[0]
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000215 raise ValueError("not enough data in stream to read uint1")
216
217uint1 = ArgumentDescriptor(
218 name='uint1',
219 n=1,
220 reader=read_uint1,
221 doc="One-byte unsigned integer.")
222
223
224def read_uint2(f):
Tim Peters55762f52003-01-28 16:01:25 +0000225 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000226 >>> import io
227 >>> read_uint2(io.BytesIO(b'\xff\x00'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000228 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000229 >>> read_uint2(io.BytesIO(b'\xff\xff'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000230 65535
231 """
232
233 data = f.read(2)
234 if len(data) == 2:
235 return _unpack("<H", data)[0]
236 raise ValueError("not enough data in stream to read uint2")
237
238uint2 = ArgumentDescriptor(
239 name='uint2',
240 n=2,
241 reader=read_uint2,
242 doc="Two-byte unsigned integer, little-endian.")
243
244
245def read_int4(f):
Tim Peters55762f52003-01-28 16:01:25 +0000246 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000247 >>> import io
248 >>> read_int4(io.BytesIO(b'\xff\x00\x00\x00'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000249 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000250 >>> read_int4(io.BytesIO(b'\x00\x00\x00\x80')) == -(2**31)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000251 True
252 """
253
254 data = f.read(4)
255 if len(data) == 4:
256 return _unpack("<i", data)[0]
257 raise ValueError("not enough data in stream to read int4")
258
259int4 = ArgumentDescriptor(
260 name='int4',
261 n=4,
262 reader=read_int4,
263 doc="Four-byte signed integer, little-endian, 2's complement.")
264
265
266def read_stringnl(f, decode=True, stripquotes=True):
Tim Peters55762f52003-01-28 16:01:25 +0000267 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000268 >>> import io
269 >>> read_stringnl(io.BytesIO(b"'abcd'\nefg\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000270 'abcd'
271
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000272 >>> read_stringnl(io.BytesIO(b"\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000273 Traceback (most recent call last):
274 ...
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000275 ValueError: no string quotes around b''
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000276
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000277 >>> read_stringnl(io.BytesIO(b"\n"), stripquotes=False)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000278 ''
279
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000280 >>> read_stringnl(io.BytesIO(b"''\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000281 ''
282
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000283 >>> read_stringnl(io.BytesIO(b'"abcd"'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000284 Traceback (most recent call last):
285 ...
286 ValueError: no newline found when trying to read stringnl
287
288 Embedded escapes are undone in the result.
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000289 >>> read_stringnl(io.BytesIO(br"'a\n\\b\x00c\td'" + b"\n'e'"))
Tim Peters55762f52003-01-28 16:01:25 +0000290 'a\n\\b\x00c\td'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000291 """
292
Guido van Rossum26986312007-07-17 00:19:46 +0000293 data = f.readline()
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000294 if not data.endswith('\n'):
295 raise ValueError("no newline found when trying to read stringnl")
296 data = data[:-1] # lose the newline
297
298 if stripquotes:
299 for q in "'\"":
300 if data.startswith(q):
301 if not data.endswith(q):
302 raise ValueError("strinq quote %r not found at both "
303 "ends of %r" % (q, data))
304 data = data[1:-1]
305 break
306 else:
307 raise ValueError("no string quotes around %r" % data)
308
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000309 if decode:
Guido van Rossum26986312007-07-17 00:19:46 +0000310 data = str(codecs.escape_decode(data)[0])
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000311 return data
312
313stringnl = ArgumentDescriptor(
314 name='stringnl',
315 n=UP_TO_NEWLINE,
316 reader=read_stringnl,
317 doc="""A newline-terminated string.
318
319 This is a repr-style string, with embedded escapes, and
320 bracketing quotes.
321 """)
322
323def read_stringnl_noescape(f):
324 return read_stringnl(f, decode=False, stripquotes=False)
325
326stringnl_noescape = ArgumentDescriptor(
327 name='stringnl_noescape',
328 n=UP_TO_NEWLINE,
329 reader=read_stringnl_noescape,
330 doc="""A newline-terminated string.
331
332 This is a str-style string, without embedded escapes,
333 or bracketing quotes. It should consist solely of
334 printable ASCII characters.
335 """)
336
337def read_stringnl_noescape_pair(f):
Tim Peters55762f52003-01-28 16:01:25 +0000338 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000339 >>> import io
340 >>> read_stringnl_noescape_pair(io.BytesIO(b"Queue\nEmpty\njunk"))
Tim Petersd916cf42003-01-27 19:01:47 +0000341 'Queue Empty'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000342 """
343
Tim Petersd916cf42003-01-27 19:01:47 +0000344 return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000345
346stringnl_noescape_pair = ArgumentDescriptor(
347 name='stringnl_noescape_pair',
348 n=UP_TO_NEWLINE,
349 reader=read_stringnl_noescape_pair,
350 doc="""A pair of newline-terminated strings.
351
352 These are str-style strings, without embedded
353 escapes, or bracketing quotes. They should
354 consist solely of printable ASCII characters.
355 The pair is returned as a single string, with
Tim Petersd916cf42003-01-27 19:01:47 +0000356 a single blank separating the two strings.
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000357 """)
358
359def read_string4(f):
Tim Peters55762f52003-01-28 16:01:25 +0000360 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000361 >>> import io
362 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x00abc"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000363 ''
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000364 >>> read_string4(io.BytesIO(b"\x03\x00\x00\x00abcdef"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000365 'abc'
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000366 >>> read_string4(io.BytesIO(b"\x00\x00\x00\x03abcdef"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000367 Traceback (most recent call last):
368 ...
369 ValueError: expected 50331648 bytes in a string4, but only 6 remain
370 """
371
372 n = read_int4(f)
373 if n < 0:
374 raise ValueError("string4 byte count < 0: %d" % n)
375 data = f.read(n)
376 if len(data) == n:
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000377 return data.decode("latin-1")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000378 raise ValueError("expected %d bytes in a string4, but only %d remain" %
379 (n, len(data)))
380
381string4 = ArgumentDescriptor(
382 name="string4",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000383 n=TAKEN_FROM_ARGUMENT4,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000384 reader=read_string4,
385 doc="""A counted string.
386
387 The first argument is a 4-byte little-endian signed int giving
388 the number of bytes in the string, and the second argument is
389 that many bytes.
390 """)
391
392
393def read_string1(f):
Tim Peters55762f52003-01-28 16:01:25 +0000394 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000395 >>> import io
396 >>> read_string1(io.BytesIO(b"\x00"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000397 ''
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000398 >>> read_string1(io.BytesIO(b"\x03abcdef"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000399 'abc'
400 """
401
402 n = read_uint1(f)
403 assert n >= 0
404 data = f.read(n)
405 if len(data) == n:
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000406 return data.decode("latin-1")
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000407 raise ValueError("expected %d bytes in a string1, but only %d remain" %
408 (n, len(data)))
409
410string1 = ArgumentDescriptor(
411 name="string1",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000412 n=TAKEN_FROM_ARGUMENT1,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000413 reader=read_string1,
414 doc="""A counted string.
415
416 The first argument is a 1-byte unsigned int giving the number
417 of bytes in the string, and the second argument is that many
418 bytes.
419 """)
420
421
422def read_unicodestringnl(f):
Tim Peters55762f52003-01-28 16:01:25 +0000423 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000424 >>> import io
425 >>> read_unicodestringnl(io.BytesIO(b"abc\\uabcd\njunk")) == 'abc\uabcd'
426 True
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000427 """
428
Guido van Rossum26986312007-07-17 00:19:46 +0000429 data = f.readline()
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000430 if not data.endswith('\n'):
431 raise ValueError("no newline found when trying to read "
432 "unicodestringnl")
433 data = data[:-1] # lose the newline
Guido van Rossumef87d6e2007-05-02 19:09:54 +0000434 return str(data, 'raw-unicode-escape')
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000435
436unicodestringnl = ArgumentDescriptor(
437 name='unicodestringnl',
438 n=UP_TO_NEWLINE,
439 reader=read_unicodestringnl,
440 doc="""A newline-terminated Unicode string.
441
442 This is raw-unicode-escape encoded, so consists of
443 printable ASCII characters, and may contain embedded
444 escape sequences.
445 """)
446
447def read_unicodestring4(f):
Tim Peters55762f52003-01-28 16:01:25 +0000448 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000449 >>> import io
450 >>> s = 'abcd\uabcd'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000451 >>> enc = s.encode('utf-8')
452 >>> enc
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000453 b'abcd\xea\xaf\x8d'
454 >>> n = bytes([len(enc), 0, 0, 0]) # little-endian 4-byte length
455 >>> t = read_unicodestring4(io.BytesIO(n + enc + b'junk'))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000456 >>> s == t
457 True
458
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000459 >>> read_unicodestring4(io.BytesIO(n + enc[:-1]))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000460 Traceback (most recent call last):
461 ...
462 ValueError: expected 7 bytes in a unicodestring4, but only 6 remain
463 """
464
465 n = read_int4(f)
466 if n < 0:
467 raise ValueError("unicodestring4 byte count < 0: %d" % n)
468 data = f.read(n)
469 if len(data) == n:
Guido van Rossumef87d6e2007-05-02 19:09:54 +0000470 return str(data, 'utf-8')
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000471 raise ValueError("expected %d bytes in a unicodestring4, but only %d "
472 "remain" % (n, len(data)))
473
474unicodestring4 = ArgumentDescriptor(
475 name="unicodestring4",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000476 n=TAKEN_FROM_ARGUMENT4,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000477 reader=read_unicodestring4,
478 doc="""A counted Unicode string.
479
480 The first argument is a 4-byte little-endian signed int
481 giving the number of bytes in the string, and the second
482 argument-- the UTF-8 encoding of the Unicode string --
483 contains that many bytes.
484 """)
485
486
487def read_decimalnl_short(f):
Tim Peters55762f52003-01-28 16:01:25 +0000488 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000489 >>> import io
490 >>> read_decimalnl_short(io.BytesIO(b"1234\n56"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000491 1234
492
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000493 >>> read_decimalnl_short(io.BytesIO(b"1234L\n56"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000494 Traceback (most recent call last):
495 ...
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000496 ValueError: trailing 'L' not allowed in b'1234L'
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000497 """
498
499 s = read_stringnl(f, decode=False, stripquotes=False)
500 if s.endswith("L"):
501 raise ValueError("trailing 'L' not allowed in %r" % s)
502
503 # It's not necessarily true that the result fits in a Python short int:
504 # the pickle may have been written on a 64-bit box. There's also a hack
505 # for True and False here.
506 if s == "00":
507 return False
508 elif s == "01":
509 return True
510
511 try:
512 return int(s)
513 except OverflowError:
Guido van Rossume2a383d2007-01-15 16:59:06 +0000514 return int(s)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000515
516def read_decimalnl_long(f):
Tim Peters55762f52003-01-28 16:01:25 +0000517 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000518 >>> import io
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000519
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000520 >>> read_decimalnl_long(io.BytesIO(b"1234L\n56"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000521 1234
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000522
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000523 >>> read_decimalnl_long(io.BytesIO(b"123456789012345678901234L\n6"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000524 123456789012345678901234
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000525 """
526
527 s = read_stringnl(f, decode=False, stripquotes=False)
Guido van Rossume2a383d2007-01-15 16:59:06 +0000528 return int(s)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000529
530
531decimalnl_short = ArgumentDescriptor(
532 name='decimalnl_short',
533 n=UP_TO_NEWLINE,
534 reader=read_decimalnl_short,
535 doc="""A newline-terminated decimal integer literal.
536
537 This never has a trailing 'L', and the integer fit
538 in a short Python int on the box where the pickle
539 was written -- but there's no guarantee it will fit
540 in a short Python int on the box where the pickle
541 is read.
542 """)
543
544decimalnl_long = ArgumentDescriptor(
545 name='decimalnl_long',
546 n=UP_TO_NEWLINE,
547 reader=read_decimalnl_long,
548 doc="""A newline-terminated decimal integer literal.
549
550 This has a trailing 'L', and can represent integers
551 of any size.
552 """)
553
554
555def read_floatnl(f):
Tim Peters55762f52003-01-28 16:01:25 +0000556 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000557 >>> import io
558 >>> read_floatnl(io.BytesIO(b"-1.25\n6"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000559 -1.25
560 """
561 s = read_stringnl(f, decode=False, stripquotes=False)
562 return float(s)
563
564floatnl = ArgumentDescriptor(
565 name='floatnl',
566 n=UP_TO_NEWLINE,
567 reader=read_floatnl,
568 doc="""A newline-terminated decimal floating literal.
569
570 In general this requires 17 significant digits for roundtrip
571 identity, and pickling then unpickling infinities, NaNs, and
572 minus zero doesn't work across boxes, or on some boxes even
573 on itself (e.g., Windows can't read the strings it produces
574 for infinities or NaNs).
575 """)
576
577def read_float8(f):
Tim Peters55762f52003-01-28 16:01:25 +0000578 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000579 >>> import io, struct
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000580 >>> raw = struct.pack(">d", -1.25)
581 >>> raw
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000582 b'\xbf\xf4\x00\x00\x00\x00\x00\x00'
583 >>> read_float8(io.BytesIO(raw + b"\n"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000584 -1.25
585 """
586
587 data = f.read(8)
588 if len(data) == 8:
589 return _unpack(">d", data)[0]
590 raise ValueError("not enough data in stream to read float8")
591
592
593float8 = ArgumentDescriptor(
594 name='float8',
595 n=8,
596 reader=read_float8,
597 doc="""An 8-byte binary representation of a float, big-endian.
598
599 The format is unique to Python, and shared with the struct
Guido van Rossum99603b02007-07-20 00:22:32 +0000600 module (format string '>d') "in theory" (the struct and pickle
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000601 implementations don't share the code -- they should). It's
602 strongly related to the IEEE-754 double format, and, in normal
603 cases, is in fact identical to the big-endian 754 double format.
604 On other boxes the dynamic range is limited to that of a 754
605 double, and "add a half and chop" rounding is used to reduce
606 the precision to 53 bits. However, even on a 754 box,
607 infinities, NaNs, and minus zero may not be handled correctly
608 (may not survive roundtrip pickling intact).
609 """)
610
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000611# Protocol 2 formats
612
Tim Petersc0c12b52003-01-29 00:56:17 +0000613from pickle import decode_long
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000614
615def read_long1(f):
616 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000617 >>> import io
618 >>> read_long1(io.BytesIO(b"\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000619 0
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000620 >>> read_long1(io.BytesIO(b"\x02\xff\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000621 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000622 >>> read_long1(io.BytesIO(b"\x02\xff\x7f"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000623 32767
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000624 >>> read_long1(io.BytesIO(b"\x02\x00\xff"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000625 -256
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000626 >>> read_long1(io.BytesIO(b"\x02\x00\x80"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000627 -32768
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000628 """
629
630 n = read_uint1(f)
631 data = f.read(n)
632 if len(data) != n:
633 raise ValueError("not enough data in stream to read long1")
634 return decode_long(data)
635
636long1 = ArgumentDescriptor(
637 name="long1",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000638 n=TAKEN_FROM_ARGUMENT1,
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000639 reader=read_long1,
640 doc="""A binary long, little-endian, using 1-byte size.
641
642 This first reads one byte as an unsigned size, then reads that
Tim Petersbdbe7412003-01-27 23:54:04 +0000643 many bytes and interprets them as a little-endian 2's-complement long.
Tim Peters4b23f2b2003-01-31 16:43:39 +0000644 If the size is 0, that's taken as a shortcut for the long 0L.
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000645 """)
646
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000647def read_long4(f):
648 r"""
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000649 >>> import io
650 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000651 255
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000652 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\xff\x7f"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000653 32767
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000654 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\xff"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000655 -256
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000656 >>> read_long4(io.BytesIO(b"\x02\x00\x00\x00\x00\x80"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000657 -32768
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000658 >>> read_long1(io.BytesIO(b"\x00\x00\x00\x00"))
Guido van Rossume2b70bc2006-08-18 22:13:04 +0000659 0
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000660 """
661
662 n = read_int4(f)
663 if n < 0:
Neal Norwitz784a3f52003-01-28 00:20:41 +0000664 raise ValueError("long4 byte count < 0: %d" % n)
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000665 data = f.read(n)
666 if len(data) != n:
Neal Norwitz784a3f52003-01-28 00:20:41 +0000667 raise ValueError("not enough data in stream to read long4")
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000668 return decode_long(data)
669
670long4 = ArgumentDescriptor(
671 name="long4",
Tim Petersfdb8cfa2003-01-28 00:13:19 +0000672 n=TAKEN_FROM_ARGUMENT4,
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000673 reader=read_long4,
674 doc="""A binary representation of a long, little-endian.
675
676 This first reads four bytes as a signed size (but requires the
677 size to be >= 0), then reads that many bytes and interprets them
Tim Peters4b23f2b2003-01-31 16:43:39 +0000678 as a little-endian 2's-complement long. If the size is 0, that's taken
Guido van Rossume2a383d2007-01-15 16:59:06 +0000679 as a shortcut for the int 0, although LONG1 should really be used
Tim Peters4b23f2b2003-01-31 16:43:39 +0000680 then instead (and in any case where # of bytes < 256).
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000681 """)
682
683
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000684##############################################################################
685# Object descriptors. The stack used by the pickle machine holds objects,
686# and in the stack_before and stack_after attributes of OpcodeInfo
687# descriptors we need names to describe the various types of objects that can
688# appear on the stack.
689
690class StackObject(object):
691 __slots__ = (
692 # name of descriptor record, for info only
693 'name',
694
695 # type of object, or tuple of type objects (meaning the object can
696 # be of any type in the tuple)
697 'obtype',
698
699 # human-readable docs for this kind of stack object; a string
700 'doc',
701 )
702
703 def __init__(self, name, obtype, doc):
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000704 assert isinstance(name, basestring)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000705 self.name = name
706
707 assert isinstance(obtype, type) or isinstance(obtype, tuple)
708 if isinstance(obtype, tuple):
709 for contained in obtype:
710 assert isinstance(contained, type)
711 self.obtype = obtype
712
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000713 assert isinstance(doc, basestring)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000714 self.doc = doc
715
Tim Petersc1c2b3e2003-01-29 20:12:21 +0000716 def __repr__(self):
717 return self.name
718
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000719
720pyint = StackObject(
721 name='int',
722 obtype=int,
723 doc="A short (as opposed to long) Python integer object.")
724
725pylong = StackObject(
726 name='long',
Guido van Rossume2a383d2007-01-15 16:59:06 +0000727 obtype=int,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000728 doc="A long (as opposed to short) Python integer object.")
729
730pyinteger_or_bool = StackObject(
731 name='int_or_bool',
Guido van Rossume2a383d2007-01-15 16:59:06 +0000732 obtype=(int, int, bool),
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000733 doc="A Python integer object (short or long), or "
734 "a Python bool.")
735
Guido van Rossum5a2d8f52003-01-27 21:44:25 +0000736pybool = StackObject(
737 name='bool',
738 obtype=(bool,),
739 doc="A Python bool object.")
740
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000741pyfloat = StackObject(
742 name='float',
743 obtype=float,
744 doc="A Python float object.")
745
746pystring = StackObject(
747 name='str',
748 obtype=str,
749 doc="A Python string object.")
750
751pyunicode = StackObject(
752 name='unicode',
Guido van Rossumef87d6e2007-05-02 19:09:54 +0000753 obtype=str,
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000754 doc="A Python Unicode string object.")
755
756pynone = StackObject(
757 name="None",
758 obtype=type(None),
759 doc="The Python None object.")
760
761pytuple = StackObject(
762 name="tuple",
763 obtype=tuple,
764 doc="A Python tuple object.")
765
766pylist = StackObject(
767 name="list",
768 obtype=list,
769 doc="A Python list object.")
770
771pydict = StackObject(
772 name="dict",
773 obtype=dict,
774 doc="A Python dict object.")
775
776anyobject = StackObject(
777 name='any',
778 obtype=object,
779 doc="Any kind of object whatsoever.")
780
781markobject = StackObject(
782 name="mark",
783 obtype=StackObject,
784 doc="""'The mark' is a unique object.
785
786 Opcodes that operate on a variable number of objects
787 generally don't embed the count of objects in the opcode,
788 or pull it off the stack. Instead the MARK opcode is used
789 to push a special marker object on the stack, and then
790 some other opcodes grab all the objects from the top of
791 the stack down to (but not including) the topmost marker
792 object.
793 """)
794
795stackslice = StackObject(
796 name="stackslice",
797 obtype=StackObject,
798 doc="""An object representing a contiguous slice of the stack.
799
800 This is used in conjuction with markobject, to represent all
801 of the stack following the topmost markobject. For example,
802 the POP_MARK opcode changes the stack from
803
804 [..., markobject, stackslice]
805 to
806 [...]
807
808 No matter how many object are on the stack after the topmost
809 markobject, POP_MARK gets rid of all of them (including the
810 topmost markobject too).
811 """)
812
813##############################################################################
814# Descriptors for pickle opcodes.
815
816class OpcodeInfo(object):
817
818 __slots__ = (
819 # symbolic name of opcode; a string
820 'name',
821
822 # the code used in a bytestream to represent the opcode; a
823 # one-character string
824 'code',
825
826 # If the opcode has an argument embedded in the byte string, an
827 # instance of ArgumentDescriptor specifying its type. Note that
828 # arg.reader(s) can be used to read and decode the argument from
829 # the bytestream s, and arg.doc documents the format of the raw
830 # argument bytes. If the opcode doesn't have an argument embedded
831 # in the bytestream, arg should be None.
832 'arg',
833
834 # what the stack looks like before this opcode runs; a list
835 'stack_before',
836
837 # what the stack looks like after this opcode runs; a list
838 'stack_after',
839
840 # the protocol number in which this opcode was introduced; an int
841 'proto',
842
843 # human-readable docs for this opcode; a string
844 'doc',
845 )
846
847 def __init__(self, name, code, arg,
848 stack_before, stack_after, proto, doc):
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000849 assert isinstance(name, basestring)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000850 self.name = name
851
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000852 assert isinstance(code, basestring)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000853 assert len(code) == 1
854 self.code = code
855
856 assert arg is None or isinstance(arg, ArgumentDescriptor)
857 self.arg = arg
858
859 assert isinstance(stack_before, list)
860 for x in stack_before:
861 assert isinstance(x, StackObject)
862 self.stack_before = stack_before
863
864 assert isinstance(stack_after, list)
865 for x in stack_after:
866 assert isinstance(x, StackObject)
867 self.stack_after = stack_after
868
869 assert isinstance(proto, int) and 0 <= proto <= 2
870 self.proto = proto
871
Guido van Rossumcfe5f202007-05-08 21:26:54 +0000872 assert isinstance(doc, basestring)
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000873 self.doc = doc
874
875I = OpcodeInfo
876opcodes = [
877
878 # Ways to spell integers.
879
880 I(name='INT',
881 code='I',
882 arg=decimalnl_short,
883 stack_before=[],
884 stack_after=[pyinteger_or_bool],
885 proto=0,
886 doc="""Push an integer or bool.
887
888 The argument is a newline-terminated decimal literal string.
889
890 The intent may have been that this always fit in a short Python int,
891 but INT can be generated in pickles written on a 64-bit box that
892 require a Python long on a 32-bit box. The difference between this
893 and LONG then is that INT skips a trailing 'L', and produces a short
894 int whenever possible.
895
896 Another difference is due to that, when bool was introduced as a
897 distinct type in 2.3, builtin names True and False were also added to
898 2.2.2, mapping to ints 1 and 0. For compatibility in both directions,
899 True gets pickled as INT + "I01\\n", and False as INT + "I00\\n".
900 Leading zeroes are never produced for a genuine integer. The 2.3
901 (and later) unpicklers special-case these and return bool instead;
902 earlier unpicklers ignore the leading "0" and return the int.
903 """),
904
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000905 I(name='BININT',
906 code='J',
907 arg=int4,
908 stack_before=[],
909 stack_after=[pyint],
910 proto=1,
911 doc="""Push a four-byte signed integer.
912
913 This handles the full range of Python (short) integers on a 32-bit
914 box, directly as binary bytes (1 for the opcode and 4 for the integer).
915 If the integer is non-negative and fits in 1 or 2 bytes, pickling via
916 BININT1 or BININT2 saves space.
917 """),
918
919 I(name='BININT1',
920 code='K',
921 arg=uint1,
922 stack_before=[],
923 stack_after=[pyint],
924 proto=1,
925 doc="""Push a one-byte unsigned integer.
926
927 This is a space optimization for pickling very small non-negative ints,
928 in range(256).
929 """),
930
931 I(name='BININT2',
932 code='M',
933 arg=uint2,
934 stack_before=[],
935 stack_after=[pyint],
936 proto=1,
937 doc="""Push a two-byte unsigned integer.
938
939 This is a space optimization for pickling small positive ints, in
940 range(256, 2**16). Integers in range(256) can also be pickled via
941 BININT2, but BININT1 instead saves a byte.
942 """),
943
Tim Petersfdc03462003-01-28 04:56:33 +0000944 I(name='LONG',
945 code='L',
946 arg=decimalnl_long,
947 stack_before=[],
948 stack_after=[pylong],
949 proto=0,
950 doc="""Push a long integer.
951
952 The same as INT, except that the literal ends with 'L', and always
953 unpickles to a Python long. There doesn't seem a real purpose to the
954 trailing 'L'.
955
956 Note that LONG takes time quadratic in the number of digits when
957 unpickling (this is simply due to the nature of decimal->binary
958 conversion). Proto 2 added linear-time (in C; still quadratic-time
959 in Python) LONG1 and LONG4 opcodes.
960 """),
961
962 I(name="LONG1",
963 code='\x8a',
964 arg=long1,
965 stack_before=[],
966 stack_after=[pylong],
967 proto=2,
968 doc="""Long integer using one-byte length.
969
970 A more efficient encoding of a Python long; the long1 encoding
971 says it all."""),
972
973 I(name="LONG4",
974 code='\x8b',
975 arg=long4,
976 stack_before=[],
977 stack_after=[pylong],
978 proto=2,
979 doc="""Long integer using found-byte length.
980
981 A more efficient encoding of a Python long; the long4 encoding
982 says it all."""),
983
Tim Peters8ecfc8e2003-01-27 18:51:48 +0000984 # Ways to spell strings (8-bit, not Unicode).
985
986 I(name='STRING',
987 code='S',
988 arg=stringnl,
989 stack_before=[],
990 stack_after=[pystring],
991 proto=0,
992 doc="""Push a Python string object.
993
994 The argument is a repr-style string, with bracketing quote characters,
995 and perhaps embedded escapes. The argument extends until the next
996 newline character.
997 """),
998
999 I(name='BINSTRING',
1000 code='T',
1001 arg=string4,
1002 stack_before=[],
1003 stack_after=[pystring],
1004 proto=1,
1005 doc="""Push a Python string object.
1006
1007 There are two arguments: the first is a 4-byte little-endian signed int
1008 giving the number of bytes in the string, and the second is that many
1009 bytes, which are taken literally as the string content.
1010 """),
1011
1012 I(name='SHORT_BINSTRING',
1013 code='U',
1014 arg=string1,
1015 stack_before=[],
1016 stack_after=[pystring],
1017 proto=1,
1018 doc="""Push a Python string object.
1019
1020 There are two arguments: the first is a 1-byte unsigned int giving
1021 the number of bytes in the string, and the second is that many bytes,
1022 which are taken literally as the string content.
1023 """),
1024
1025 # Ways to spell None.
1026
1027 I(name='NONE',
1028 code='N',
1029 arg=None,
1030 stack_before=[],
1031 stack_after=[pynone],
1032 proto=0,
1033 doc="Push None on the stack."),
1034
Tim Petersfdc03462003-01-28 04:56:33 +00001035 # Ways to spell bools, starting with proto 2. See INT for how this was
1036 # done before proto 2.
1037
1038 I(name='NEWTRUE',
1039 code='\x88',
1040 arg=None,
1041 stack_before=[],
1042 stack_after=[pybool],
1043 proto=2,
1044 doc="""True.
1045
1046 Push True onto the stack."""),
1047
1048 I(name='NEWFALSE',
1049 code='\x89',
1050 arg=None,
1051 stack_before=[],
1052 stack_after=[pybool],
1053 proto=2,
1054 doc="""True.
1055
1056 Push False onto the stack."""),
1057
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001058 # Ways to spell Unicode strings.
1059
1060 I(name='UNICODE',
1061 code='V',
1062 arg=unicodestringnl,
1063 stack_before=[],
1064 stack_after=[pyunicode],
1065 proto=0, # this may be pure-text, but it's a later addition
1066 doc="""Push a Python Unicode string object.
1067
1068 The argument is a raw-unicode-escape encoding of a Unicode string,
1069 and so may contain embedded escape sequences. The argument extends
1070 until the next newline character.
1071 """),
1072
1073 I(name='BINUNICODE',
1074 code='X',
1075 arg=unicodestring4,
1076 stack_before=[],
1077 stack_after=[pyunicode],
1078 proto=1,
1079 doc="""Push a Python Unicode string object.
1080
1081 There are two arguments: the first is a 4-byte little-endian signed int
1082 giving the number of bytes in the string. The second is that many
1083 bytes, and is the UTF-8 encoding of the Unicode string.
1084 """),
1085
1086 # Ways to spell floats.
1087
1088 I(name='FLOAT',
1089 code='F',
1090 arg=floatnl,
1091 stack_before=[],
1092 stack_after=[pyfloat],
1093 proto=0,
1094 doc="""Newline-terminated decimal float literal.
1095
1096 The argument is repr(a_float), and in general requires 17 significant
1097 digits for roundtrip conversion to be an identity (this is so for
1098 IEEE-754 double precision values, which is what Python float maps to
1099 on most boxes).
1100
1101 In general, FLOAT cannot be used to transport infinities, NaNs, or
1102 minus zero across boxes (or even on a single box, if the platform C
1103 library can't read the strings it produces for such things -- Windows
1104 is like that), but may do less damage than BINFLOAT on boxes with
1105 greater precision or dynamic range than IEEE-754 double.
1106 """),
1107
1108 I(name='BINFLOAT',
1109 code='G',
1110 arg=float8,
1111 stack_before=[],
1112 stack_after=[pyfloat],
1113 proto=1,
1114 doc="""Float stored in binary form, with 8 bytes of data.
1115
1116 This generally requires less than half the space of FLOAT encoding.
1117 In general, BINFLOAT cannot be used to transport infinities, NaNs, or
1118 minus zero, raises an exception if the exponent exceeds the range of
1119 an IEEE-754 double, and retains no more than 53 bits of precision (if
1120 there are more than that, "add a half and chop" rounding is used to
1121 cut it back to 53 significant bits).
1122 """),
1123
1124 # Ways to build lists.
1125
1126 I(name='EMPTY_LIST',
1127 code=']',
1128 arg=None,
1129 stack_before=[],
1130 stack_after=[pylist],
1131 proto=1,
1132 doc="Push an empty list."),
1133
1134 I(name='APPEND',
1135 code='a',
1136 arg=None,
1137 stack_before=[pylist, anyobject],
1138 stack_after=[pylist],
1139 proto=0,
1140 doc="""Append an object to a list.
1141
1142 Stack before: ... pylist anyobject
1143 Stack after: ... pylist+[anyobject]
Tim Peters81098ac2003-01-28 05:12:08 +00001144
1145 although pylist is really extended in-place.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001146 """),
1147
1148 I(name='APPENDS',
1149 code='e',
1150 arg=None,
1151 stack_before=[pylist, markobject, stackslice],
1152 stack_after=[pylist],
1153 proto=1,
1154 doc="""Extend a list by a slice of stack objects.
1155
1156 Stack before: ... pylist markobject stackslice
1157 Stack after: ... pylist+stackslice
Tim Peters81098ac2003-01-28 05:12:08 +00001158
1159 although pylist is really extended in-place.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001160 """),
1161
1162 I(name='LIST',
1163 code='l',
1164 arg=None,
1165 stack_before=[markobject, stackslice],
1166 stack_after=[pylist],
1167 proto=0,
1168 doc="""Build a list out of the topmost stack slice, after markobject.
1169
1170 All the stack entries following the topmost markobject are placed into
1171 a single Python list, which single list object replaces all of the
1172 stack from the topmost markobject onward. For example,
1173
1174 Stack before: ... markobject 1 2 3 'abc'
1175 Stack after: ... [1, 2, 3, 'abc']
1176 """),
1177
1178 # Ways to build tuples.
1179
1180 I(name='EMPTY_TUPLE',
1181 code=')',
1182 arg=None,
1183 stack_before=[],
1184 stack_after=[pytuple],
1185 proto=1,
1186 doc="Push an empty tuple."),
1187
1188 I(name='TUPLE',
1189 code='t',
1190 arg=None,
1191 stack_before=[markobject, stackslice],
1192 stack_after=[pytuple],
1193 proto=0,
1194 doc="""Build a tuple out of the topmost stack slice, after markobject.
1195
1196 All the stack entries following the topmost markobject are placed into
1197 a single Python tuple, which single tuple object replaces all of the
1198 stack from the topmost markobject onward. For example,
1199
1200 Stack before: ... markobject 1 2 3 'abc'
1201 Stack after: ... (1, 2, 3, 'abc')
1202 """),
1203
Tim Petersfdc03462003-01-28 04:56:33 +00001204 I(name='TUPLE1',
1205 code='\x85',
1206 arg=None,
1207 stack_before=[anyobject],
1208 stack_after=[pytuple],
1209 proto=2,
1210 doc="""One-tuple.
1211
1212 This code pops one value off the stack and pushes a tuple of
1213 length 1 whose one item is that value back onto it. IOW:
1214
1215 stack[-1] = tuple(stack[-1:])
1216 """),
1217
1218 I(name='TUPLE2',
1219 code='\x86',
1220 arg=None,
1221 stack_before=[anyobject, anyobject],
1222 stack_after=[pytuple],
1223 proto=2,
1224 doc="""One-tuple.
1225
1226 This code pops two values off the stack and pushes a tuple
1227 of length 2 whose items are those values back onto it. IOW:
1228
1229 stack[-2:] = [tuple(stack[-2:])]
1230 """),
1231
1232 I(name='TUPLE3',
1233 code='\x87',
1234 arg=None,
1235 stack_before=[anyobject, anyobject, anyobject],
1236 stack_after=[pytuple],
1237 proto=2,
1238 doc="""One-tuple.
1239
1240 This code pops three values off the stack and pushes a tuple
1241 of length 3 whose items are those values back onto it. IOW:
1242
1243 stack[-3:] = [tuple(stack[-3:])]
1244 """),
1245
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001246 # Ways to build dicts.
1247
1248 I(name='EMPTY_DICT',
1249 code='}',
1250 arg=None,
1251 stack_before=[],
1252 stack_after=[pydict],
1253 proto=1,
1254 doc="Push an empty dict."),
1255
1256 I(name='DICT',
1257 code='d',
1258 arg=None,
1259 stack_before=[markobject, stackslice],
1260 stack_after=[pydict],
1261 proto=0,
1262 doc="""Build a dict out of the topmost stack slice, after markobject.
1263
1264 All the stack entries following the topmost markobject are placed into
1265 a single Python dict, which single dict object replaces all of the
1266 stack from the topmost markobject onward. The stack slice alternates
1267 key, value, key, value, .... For example,
1268
1269 Stack before: ... markobject 1 2 3 'abc'
1270 Stack after: ... {1: 2, 3: 'abc'}
1271 """),
1272
1273 I(name='SETITEM',
1274 code='s',
1275 arg=None,
1276 stack_before=[pydict, anyobject, anyobject],
1277 stack_after=[pydict],
1278 proto=0,
1279 doc="""Add a key+value pair to an existing dict.
1280
1281 Stack before: ... pydict key value
1282 Stack after: ... pydict
1283
1284 where pydict has been modified via pydict[key] = value.
1285 """),
1286
1287 I(name='SETITEMS',
1288 code='u',
1289 arg=None,
1290 stack_before=[pydict, markobject, stackslice],
1291 stack_after=[pydict],
1292 proto=1,
1293 doc="""Add an arbitrary number of key+value pairs to an existing dict.
1294
1295 The slice of the stack following the topmost markobject is taken as
1296 an alternating sequence of keys and values, added to the dict
1297 immediately under the topmost markobject. Everything at and after the
1298 topmost markobject is popped, leaving the mutated dict at the top
1299 of the stack.
1300
1301 Stack before: ... pydict markobject key_1 value_1 ... key_n value_n
1302 Stack after: ... pydict
1303
1304 where pydict has been modified via pydict[key_i] = value_i for i in
1305 1, 2, ..., n, and in that order.
1306 """),
1307
1308 # Stack manipulation.
1309
1310 I(name='POP',
1311 code='0',
1312 arg=None,
1313 stack_before=[anyobject],
1314 stack_after=[],
1315 proto=0,
1316 doc="Discard the top stack item, shrinking the stack by one item."),
1317
1318 I(name='DUP',
1319 code='2',
1320 arg=None,
1321 stack_before=[anyobject],
1322 stack_after=[anyobject, anyobject],
1323 proto=0,
1324 doc="Push the top stack item onto the stack again, duplicating it."),
1325
1326 I(name='MARK',
1327 code='(',
1328 arg=None,
1329 stack_before=[],
1330 stack_after=[markobject],
1331 proto=0,
1332 doc="""Push markobject onto the stack.
1333
1334 markobject is a unique object, used by other opcodes to identify a
1335 region of the stack containing a variable number of objects for them
1336 to work on. See markobject.doc for more detail.
1337 """),
1338
1339 I(name='POP_MARK',
1340 code='1',
1341 arg=None,
1342 stack_before=[markobject, stackslice],
1343 stack_after=[],
1344 proto=0,
1345 doc="""Pop all the stack objects at and above the topmost markobject.
1346
1347 When an opcode using a variable number of stack objects is done,
1348 POP_MARK is used to remove those objects, and to remove the markobject
1349 that delimited their starting position on the stack.
1350 """),
1351
1352 # Memo manipulation. There are really only two operations (get and put),
1353 # each in all-text, "short binary", and "long binary" flavors.
1354
1355 I(name='GET',
1356 code='g',
1357 arg=decimalnl_short,
1358 stack_before=[],
1359 stack_after=[anyobject],
1360 proto=0,
1361 doc="""Read an object from the memo and push it on the stack.
1362
1363 The index of the memo object to push is given by the newline-teriminated
1364 decimal string following. BINGET and LONG_BINGET are space-optimized
1365 versions.
1366 """),
1367
1368 I(name='BINGET',
1369 code='h',
1370 arg=uint1,
1371 stack_before=[],
1372 stack_after=[anyobject],
1373 proto=1,
1374 doc="""Read an object from the memo and push it on the stack.
1375
1376 The index of the memo object to push is given by the 1-byte unsigned
1377 integer following.
1378 """),
1379
1380 I(name='LONG_BINGET',
1381 code='j',
1382 arg=int4,
1383 stack_before=[],
1384 stack_after=[anyobject],
1385 proto=1,
1386 doc="""Read an object from the memo and push it on the stack.
1387
1388 The index of the memo object to push is given by the 4-byte signed
1389 little-endian integer following.
1390 """),
1391
1392 I(name='PUT',
1393 code='p',
1394 arg=decimalnl_short,
1395 stack_before=[],
1396 stack_after=[],
1397 proto=0,
1398 doc="""Store the stack top into the memo. The stack is not popped.
1399
1400 The index of the memo location to write into is given by the newline-
1401 terminated decimal string following. BINPUT and LONG_BINPUT are
1402 space-optimized versions.
1403 """),
1404
1405 I(name='BINPUT',
1406 code='q',
1407 arg=uint1,
1408 stack_before=[],
1409 stack_after=[],
1410 proto=1,
1411 doc="""Store the stack top into the memo. The stack is not popped.
1412
1413 The index of the memo location to write into is given by the 1-byte
1414 unsigned integer following.
1415 """),
1416
1417 I(name='LONG_BINPUT',
1418 code='r',
1419 arg=int4,
1420 stack_before=[],
1421 stack_after=[],
1422 proto=1,
1423 doc="""Store the stack top into the memo. The stack is not popped.
1424
1425 The index of the memo location to write into is given by the 4-byte
1426 signed little-endian integer following.
1427 """),
1428
Tim Petersfdc03462003-01-28 04:56:33 +00001429 # Access the extension registry (predefined objects). Akin to the GET
1430 # family.
1431
1432 I(name='EXT1',
1433 code='\x82',
1434 arg=uint1,
1435 stack_before=[],
1436 stack_after=[anyobject],
1437 proto=2,
1438 doc="""Extension code.
1439
1440 This code and the similar EXT2 and EXT4 allow using a registry
1441 of popular objects that are pickled by name, typically classes.
1442 It is envisioned that through a global negotiation and
1443 registration process, third parties can set up a mapping between
1444 ints and object names.
1445
1446 In order to guarantee pickle interchangeability, the extension
1447 code registry ought to be global, although a range of codes may
1448 be reserved for private use.
1449
1450 EXT1 has a 1-byte integer argument. This is used to index into the
1451 extension registry, and the object at that index is pushed on the stack.
1452 """),
1453
1454 I(name='EXT2',
1455 code='\x83',
1456 arg=uint2,
1457 stack_before=[],
1458 stack_after=[anyobject],
1459 proto=2,
1460 doc="""Extension code.
1461
1462 See EXT1. EXT2 has a two-byte integer argument.
1463 """),
1464
1465 I(name='EXT4',
1466 code='\x84',
1467 arg=int4,
1468 stack_before=[],
1469 stack_after=[anyobject],
1470 proto=2,
1471 doc="""Extension code.
1472
1473 See EXT1. EXT4 has a four-byte integer argument.
1474 """),
1475
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001476 # Push a class object, or module function, on the stack, via its module
1477 # and name.
1478
1479 I(name='GLOBAL',
1480 code='c',
1481 arg=stringnl_noescape_pair,
1482 stack_before=[],
1483 stack_after=[anyobject],
1484 proto=0,
1485 doc="""Push a global object (module.attr) on the stack.
1486
1487 Two newline-terminated strings follow the GLOBAL opcode. The first is
1488 taken as a module name, and the second as a class name. The class
1489 object module.class is pushed on the stack. More accurately, the
1490 object returned by self.find_class(module, class) is pushed on the
1491 stack, so unpickling subclasses can override this form of lookup.
1492 """),
1493
1494 # Ways to build objects of classes pickle doesn't know about directly
1495 # (user-defined classes). I despair of documenting this accurately
1496 # and comprehensibly -- you really have to read the pickle code to
1497 # find all the special cases.
1498
1499 I(name='REDUCE',
1500 code='R',
1501 arg=None,
1502 stack_before=[anyobject, anyobject],
1503 stack_after=[anyobject],
1504 proto=0,
1505 doc="""Push an object built from a callable and an argument tuple.
1506
1507 The opcode is named to remind of the __reduce__() method.
1508
1509 Stack before: ... callable pytuple
1510 Stack after: ... callable(*pytuple)
1511
1512 The callable and the argument tuple are the first two items returned
1513 by a __reduce__ method. Applying the callable to the argtuple is
1514 supposed to reproduce the original object, or at least get it started.
1515 If the __reduce__ method returns a 3-tuple, the last component is an
1516 argument to be passed to the object's __setstate__, and then the REDUCE
1517 opcode is followed by code to create setstate's argument, and then a
1518 BUILD opcode to apply __setstate__ to that argument.
1519
Guido van Rossum13257902007-06-07 23:15:56 +00001520 If not isinstance(callable, type), REDUCE complains unless the
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001521 callable has been registered with the copy_reg module's
1522 safe_constructors dict, or the callable has a magic
1523 '__safe_for_unpickling__' attribute with a true value. I'm not sure
1524 why it does this, but I've sure seen this complaint often enough when
1525 I didn't want to <wink>.
1526 """),
1527
1528 I(name='BUILD',
1529 code='b',
1530 arg=None,
1531 stack_before=[anyobject, anyobject],
1532 stack_after=[anyobject],
1533 proto=0,
1534 doc="""Finish building an object, via __setstate__ or dict update.
1535
1536 Stack before: ... anyobject argument
1537 Stack after: ... anyobject
1538
1539 where anyobject may have been mutated, as follows:
1540
1541 If the object has a __setstate__ method,
1542
1543 anyobject.__setstate__(argument)
1544
1545 is called.
1546
1547 Else the argument must be a dict, the object must have a __dict__, and
1548 the object is updated via
1549
1550 anyobject.__dict__.update(argument)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001551 """),
1552
1553 I(name='INST',
1554 code='i',
1555 arg=stringnl_noescape_pair,
1556 stack_before=[markobject, stackslice],
1557 stack_after=[anyobject],
1558 proto=0,
1559 doc="""Build a class instance.
1560
1561 This is the protocol 0 version of protocol 1's OBJ opcode.
1562 INST is followed by two newline-terminated strings, giving a
1563 module and class name, just as for the GLOBAL opcode (and see
1564 GLOBAL for more details about that). self.find_class(module, name)
1565 is used to get a class object.
1566
1567 In addition, all the objects on the stack following the topmost
1568 markobject are gathered into a tuple and popped (along with the
1569 topmost markobject), just as for the TUPLE opcode.
1570
1571 Now it gets complicated. If all of these are true:
1572
1573 + The argtuple is empty (markobject was at the top of the stack
1574 at the start).
1575
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001576 + The class object does not have a __getinitargs__ attribute.
1577
1578 then we want to create an old-style class instance without invoking
1579 its __init__() method (pickle has waffled on this over the years; not
1580 calling __init__() is current wisdom). In this case, an instance of
1581 an old-style dummy class is created, and then we try to rebind its
1582 __class__ attribute to the desired class object. If this succeeds,
Guido van Rossuma8add0e2007-05-14 22:03:55 +00001583 the new instance object is pushed on the stack, and we're done.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001584
1585 Else (the argtuple is not empty, it's not an old-style class object,
1586 or the class object does have a __getinitargs__ attribute), the code
1587 first insists that the class object have a __safe_for_unpickling__
1588 attribute. Unlike as for the __safe_for_unpickling__ check in REDUCE,
1589 it doesn't matter whether this attribute has a true or false value, it
Guido van Rossum99603b02007-07-20 00:22:32 +00001590 only matters whether it exists (XXX this is a bug). If
1591 __safe_for_unpickling__ doesn't exist, UnpicklingError is raised.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001592
1593 Else (the class object does have a __safe_for_unpickling__ attr),
1594 the class object obtained from INST's arguments is applied to the
1595 argtuple obtained from the stack, and the resulting instance object
1596 is pushed on the stack.
Tim Peters2b93c4c2003-01-30 16:35:08 +00001597
1598 NOTE: checks for __safe_for_unpickling__ went away in Python 2.3.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001599 """),
1600
1601 I(name='OBJ',
1602 code='o',
1603 arg=None,
1604 stack_before=[markobject, anyobject, stackslice],
1605 stack_after=[anyobject],
1606 proto=1,
1607 doc="""Build a class instance.
1608
1609 This is the protocol 1 version of protocol 0's INST opcode, and is
1610 very much like it. The major difference is that the class object
1611 is taken off the stack, allowing it to be retrieved from the memo
1612 repeatedly if several instances of the same class are created. This
1613 can be much more efficient (in both time and space) than repeatedly
1614 embedding the module and class names in INST opcodes.
1615
1616 Unlike INST, OBJ takes no arguments from the opcode stream. Instead
1617 the class object is taken off the stack, immediately above the
1618 topmost markobject:
1619
1620 Stack before: ... markobject classobject stackslice
1621 Stack after: ... new_instance_object
1622
1623 As for INST, the remainder of the stack above the markobject is
1624 gathered into an argument tuple, and then the logic seems identical,
Guido van Rossumecb11042003-01-29 06:24:30 +00001625 except that no __safe_for_unpickling__ check is done (XXX this is
Guido van Rossum99603b02007-07-20 00:22:32 +00001626 a bug). See INST for the gory details.
Tim Peters2b93c4c2003-01-30 16:35:08 +00001627
1628 NOTE: In Python 2.3, INST and OBJ are identical except for how they
1629 get the class object. That was always the intent; the implementations
1630 had diverged for accidental reasons.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001631 """),
1632
Tim Petersfdc03462003-01-28 04:56:33 +00001633 I(name='NEWOBJ',
1634 code='\x81',
1635 arg=None,
1636 stack_before=[anyobject, anyobject],
1637 stack_after=[anyobject],
1638 proto=2,
1639 doc="""Build an object instance.
1640
1641 The stack before should be thought of as containing a class
1642 object followed by an argument tuple (the tuple being the stack
1643 top). Call these cls and args. They are popped off the stack,
1644 and the value returned by cls.__new__(cls, *args) is pushed back
1645 onto the stack.
1646 """),
1647
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001648 # Machine control.
1649
Tim Petersfdc03462003-01-28 04:56:33 +00001650 I(name='PROTO',
1651 code='\x80',
1652 arg=uint1,
1653 stack_before=[],
1654 stack_after=[],
1655 proto=2,
1656 doc="""Protocol version indicator.
1657
1658 For protocol 2 and above, a pickle must start with this opcode.
1659 The argument is the protocol version, an int in range(2, 256).
1660 """),
1661
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001662 I(name='STOP',
1663 code='.',
1664 arg=None,
1665 stack_before=[anyobject],
1666 stack_after=[],
1667 proto=0,
1668 doc="""Stop the unpickling machine.
1669
1670 Every pickle ends with this opcode. The object at the top of the stack
1671 is popped, and that's the result of unpickling. The stack should be
1672 empty then.
1673 """),
1674
1675 # Ways to deal with persistent IDs.
1676
1677 I(name='PERSID',
1678 code='P',
1679 arg=stringnl_noescape,
1680 stack_before=[],
1681 stack_after=[anyobject],
1682 proto=0,
1683 doc="""Push an object identified by a persistent ID.
1684
1685 The pickle module doesn't define what a persistent ID means. PERSID's
1686 argument is a newline-terminated str-style (no embedded escapes, no
1687 bracketing quote characters) string, which *is* "the persistent ID".
1688 The unpickler passes this string to self.persistent_load(). Whatever
1689 object that returns is pushed on the stack. There is no implementation
1690 of persistent_load() in Python's unpickler: it must be supplied by an
1691 unpickler subclass.
1692 """),
1693
1694 I(name='BINPERSID',
1695 code='Q',
1696 arg=None,
1697 stack_before=[anyobject],
1698 stack_after=[anyobject],
1699 proto=1,
1700 doc="""Push an object identified by a persistent ID.
1701
1702 Like PERSID, except the persistent ID is popped off the stack (instead
1703 of being a string embedded in the opcode bytestream). The persistent
1704 ID is passed to self.persistent_load(), and whatever object that
1705 returns is pushed on the stack. See PERSID for more detail.
1706 """),
1707]
1708del I
1709
1710# Verify uniqueness of .name and .code members.
1711name2i = {}
1712code2i = {}
1713
1714for i, d in enumerate(opcodes):
1715 if d.name in name2i:
1716 raise ValueError("repeated name %r at indices %d and %d" %
1717 (d.name, name2i[d.name], i))
1718 if d.code in code2i:
1719 raise ValueError("repeated code %r at indices %d and %d" %
1720 (d.code, code2i[d.code], i))
1721
1722 name2i[d.name] = i
1723 code2i[d.code] = i
1724
1725del name2i, code2i, i, d
1726
1727##############################################################################
1728# Build a code2op dict, mapping opcode characters to OpcodeInfo records.
1729# Also ensure we've got the same stuff as pickle.py, although the
1730# introspection here is dicey.
1731
1732code2op = {}
1733for d in opcodes:
1734 code2op[d.code] = d
1735del d
1736
1737def assure_pickle_consistency(verbose=False):
1738 import pickle, re
1739
1740 copy = code2op.copy()
1741 for name in pickle.__all__:
1742 if not re.match("[A-Z][A-Z0-9_]+$", name):
1743 if verbose:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001744 print("skipping %r: it doesn't look like an opcode name" % name)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001745 continue
1746 picklecode = getattr(pickle, name)
Guido van Rossum617dbc42007-05-07 23:57:08 +00001747 if not isinstance(picklecode, bytes) or len(picklecode) != 1:
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001748 if verbose:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001749 print(("skipping %r: value %r doesn't look like a pickle "
1750 "code" % (name, picklecode)))
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001751 continue
Guido van Rossum617dbc42007-05-07 23:57:08 +00001752 picklecode = picklecode.decode("latin-1")
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001753 if picklecode in copy:
1754 if verbose:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001755 print("checking name %r w/ code %r for consistency" % (
1756 name, picklecode))
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001757 d = copy[picklecode]
1758 if d.name != name:
1759 raise ValueError("for pickle code %r, pickle.py uses name %r "
1760 "but we're using name %r" % (picklecode,
1761 name,
1762 d.name))
1763 # Forget this one. Any left over in copy at the end are a problem
1764 # of a different kind.
1765 del copy[picklecode]
1766 else:
1767 raise ValueError("pickle.py appears to have a pickle opcode with "
1768 "name %r and code %r, but we don't" %
1769 (name, picklecode))
1770 if copy:
1771 msg = ["we appear to have pickle opcodes that pickle.py doesn't have:"]
1772 for code, d in copy.items():
1773 msg.append(" name %r with code %r" % (d.name, code))
1774 raise ValueError("\n".join(msg))
1775
1776assure_pickle_consistency()
Tim Petersc0c12b52003-01-29 00:56:17 +00001777del assure_pickle_consistency
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001778
1779##############################################################################
1780# A pickle opcode generator.
1781
1782def genops(pickle):
Guido van Rossuma72ded92003-01-27 19:40:47 +00001783 """Generate all the opcodes in a pickle.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001784
1785 'pickle' is a file-like object, or string, containing the pickle.
1786
1787 Each opcode in the pickle is generated, from the current pickle position,
1788 stopping after a STOP opcode is delivered. A triple is generated for
1789 each opcode:
1790
1791 opcode, arg, pos
1792
1793 opcode is an OpcodeInfo record, describing the current opcode.
1794
1795 If the opcode has an argument embedded in the pickle, arg is its decoded
1796 value, as a Python object. If the opcode doesn't have an argument, arg
1797 is None.
1798
1799 If the pickle has a tell() method, pos was the value of pickle.tell()
Guido van Rossum34d19282007-08-09 01:03:29 +00001800 before reading the current opcode. If the pickle is a bytes object,
1801 it's wrapped in a BytesIO object, and the latter's tell() result is
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001802 used. Else (the pickle doesn't have a tell(), and it's not obvious how
1803 to query its current position) pos is None.
1804 """
1805
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001806 if isinstance(pickle, bytes):
1807 import io
1808 pickle = io.BytesIO(pickle)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001809
1810 if hasattr(pickle, "tell"):
1811 getpos = pickle.tell
1812 else:
1813 getpos = lambda: None
1814
1815 while True:
1816 pos = getpos()
1817 code = pickle.read(1)
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001818 opcode = code2op.get(code.decode("latin-1"))
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001819 if opcode is None:
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001820 if code == b"":
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001821 raise ValueError("pickle exhausted before seeing STOP")
1822 else:
1823 raise ValueError("at position %s, opcode %r unknown" % (
1824 pos is None and "<unknown>" or pos,
1825 code))
1826 if opcode.arg is None:
1827 arg = None
1828 else:
1829 arg = opcode.arg.reader(pickle)
1830 yield opcode, arg, pos
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001831 if code == b'.':
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001832 assert opcode.name == 'STOP'
1833 break
1834
1835##############################################################################
1836# A symbolic pickle disassembler.
1837
Tim Peters62235e72003-02-05 19:55:53 +00001838def dis(pickle, out=None, memo=None, indentlevel=4):
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001839 """Produce a symbolic disassembly of a pickle.
1840
1841 'pickle' is a file-like object, or string, containing a (at least one)
1842 pickle. The pickle is disassembled from the current position, through
1843 the first STOP opcode encountered.
1844
1845 Optional arg 'out' is a file-like object to which the disassembly is
1846 printed. It defaults to sys.stdout.
1847
Tim Peters62235e72003-02-05 19:55:53 +00001848 Optional arg 'memo' is a Python dict, used as the pickle's memo. It
1849 may be mutated by dis(), if the pickle contains PUT or BINPUT opcodes.
1850 Passing the same memo object to another dis() call then allows disassembly
1851 to proceed across multiple pickles that were all created by the same
1852 pickler with the same memo. Ordinarily you don't need to worry about this.
1853
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001854 Optional arg indentlevel is the number of blanks by which to indent
1855 a new MARK level. It defaults to 4.
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001856
1857 In addition to printing the disassembly, some sanity checks are made:
1858
1859 + All embedded opcode arguments "make sense".
1860
1861 + Explicit and implicit pop operations have enough items on the stack.
1862
1863 + When an opcode implicitly refers to a markobject, a markobject is
1864 actually on the stack.
1865
1866 + A memo entry isn't referenced before it's defined.
1867
1868 + The markobject isn't stored in the memo.
1869
1870 + A memo entry isn't redefined.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001871 """
1872
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001873 # Most of the hair here is for sanity checks, but most of it is needed
1874 # anyway to detect when a protocol 0 POP takes a MARK off the stack
1875 # (which in turn is needed to indent MARK blocks correctly).
1876
1877 stack = [] # crude emulation of unpickler stack
Tim Peters62235e72003-02-05 19:55:53 +00001878 if memo is None:
1879 memo = {} # crude emulation of unpicker memo
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001880 maxproto = -1 # max protocol number seen
1881 markstack = [] # bytecode positions of MARK opcodes
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001882 indentchunk = ' ' * indentlevel
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001883 errormsg = None
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001884 for opcode, arg, pos in genops(pickle):
1885 if pos is not None:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001886 print("%5d:" % pos, end=' ', file=out)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001887
Tim Petersd0f7c862003-01-28 15:27:57 +00001888 line = "%-4s %s%s" % (repr(opcode.code)[1:-1],
1889 indentchunk * len(markstack),
1890 opcode.name)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001891
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001892 maxproto = max(maxproto, opcode.proto)
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001893 before = opcode.stack_before # don't mutate
1894 after = opcode.stack_after # don't mutate
Tim Peters43277d62003-01-30 15:02:12 +00001895 numtopop = len(before)
1896
1897 # See whether a MARK should be popped.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001898 markmsg = None
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001899 if markobject in before or (opcode.name == "POP" and
1900 stack and
1901 stack[-1] is markobject):
1902 assert markobject not in after
Tim Peters43277d62003-01-30 15:02:12 +00001903 if __debug__:
1904 if markobject in before:
1905 assert before[-1] is stackslice
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001906 if markstack:
1907 markpos = markstack.pop()
1908 if markpos is None:
1909 markmsg = "(MARK at unknown opcode offset)"
1910 else:
1911 markmsg = "(MARK at %d)" % markpos
1912 # Pop everything at and after the topmost markobject.
1913 while stack[-1] is not markobject:
1914 stack.pop()
1915 stack.pop()
Tim Peters43277d62003-01-30 15:02:12 +00001916 # Stop later code from popping too much.
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001917 try:
Tim Peters43277d62003-01-30 15:02:12 +00001918 numtopop = before.index(markobject)
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001919 except ValueError:
1920 assert opcode.name == "POP"
Tim Peters43277d62003-01-30 15:02:12 +00001921 numtopop = 0
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001922 else:
1923 errormsg = markmsg = "no MARK exists on stack"
1924
1925 # Check for correct memo usage.
1926 if opcode.name in ("PUT", "BINPUT", "LONG_BINPUT"):
Tim Peters43277d62003-01-30 15:02:12 +00001927 assert arg is not None
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001928 if arg in memo:
1929 errormsg = "memo key %r already defined" % arg
1930 elif not stack:
1931 errormsg = "stack is empty -- can't store into memo"
1932 elif stack[-1] is markobject:
1933 errormsg = "can't store markobject in the memo"
1934 else:
1935 memo[arg] = stack[-1]
1936
1937 elif opcode.name in ("GET", "BINGET", "LONG_BINGET"):
1938 if arg in memo:
1939 assert len(after) == 1
1940 after = [memo[arg]] # for better stack emulation
1941 else:
1942 errormsg = "memo key %r has never been stored into" % arg
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001943
1944 if arg is not None or markmsg:
1945 # make a mild effort to align arguments
1946 line += ' ' * (10 - len(opcode.name))
1947 if arg is not None:
1948 line += ' ' + repr(arg)
1949 if markmsg:
1950 line += ' ' + markmsg
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001951 print(line, file=out)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001952
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001953 if errormsg:
1954 # Note that we delayed complaining until the offending opcode
1955 # was printed.
1956 raise ValueError(errormsg)
1957
1958 # Emulate the stack effects.
Tim Peters43277d62003-01-30 15:02:12 +00001959 if len(stack) < numtopop:
1960 raise ValueError("tries to pop %d items from stack with "
1961 "only %d items" % (numtopop, len(stack)))
1962 if numtopop:
1963 del stack[-numtopop:]
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001964 if markobject in after:
Tim Peters43277d62003-01-30 15:02:12 +00001965 assert markobject not in before
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001966 markstack.append(pos)
1967
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001968 stack.extend(after)
1969
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001970 print("highest protocol among opcodes =", maxproto, file=out)
Tim Petersc1c2b3e2003-01-29 20:12:21 +00001971 if stack:
1972 raise ValueError("stack not empty after STOP: %r" % stack)
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001973
Tim Peters90718a42005-02-15 16:22:34 +00001974# For use in the doctest, simply as an example of a class to pickle.
1975class _Example:
1976 def __init__(self, value):
1977 self.value = value
1978
Guido van Rossum03e35322003-01-28 15:37:13 +00001979_dis_test = r"""
Tim Peters8ecfc8e2003-01-27 18:51:48 +00001980>>> import pickle
Guido van Rossumcfe5f202007-05-08 21:26:54 +00001981>>> x = [1, 2, (3, 4), {str8('abc'): "def"}]
Guido van Rossum57028352003-01-28 15:09:10 +00001982>>> pkl = pickle.dumps(x, 0)
1983>>> dis(pkl)
Tim Petersd0f7c862003-01-28 15:27:57 +00001984 0: ( MARK
1985 1: l LIST (MARK at 0)
1986 2: p PUT 0
Guido van Rossumf4100002007-01-15 00:21:46 +00001987 5: L LONG 1
Tim Petersd0f7c862003-01-28 15:27:57 +00001988 8: a APPEND
Guido van Rossumf4100002007-01-15 00:21:46 +00001989 9: L LONG 2
Tim Petersd0f7c862003-01-28 15:27:57 +00001990 12: a APPEND
1991 13: ( MARK
Guido van Rossumf4100002007-01-15 00:21:46 +00001992 14: L LONG 3
1993 17: L LONG 4
Tim Petersd0f7c862003-01-28 15:27:57 +00001994 20: t TUPLE (MARK at 13)
1995 21: p PUT 1
1996 24: a APPEND
1997 25: ( MARK
1998 26: d DICT (MARK at 25)
1999 27: p PUT 2
2000 30: S STRING 'abc'
2001 37: p PUT 3
Guido van Rossumcfe5f202007-05-08 21:26:54 +00002002 40: V UNICODE 'def'
Tim Petersd0f7c862003-01-28 15:27:57 +00002003 45: p PUT 4
2004 48: s SETITEM
2005 49: a APPEND
2006 50: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002007highest protocol among opcodes = 0
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002008
2009Try again with a "binary" pickle.
2010
Guido van Rossum57028352003-01-28 15:09:10 +00002011>>> pkl = pickle.dumps(x, 1)
2012>>> dis(pkl)
Tim Petersd0f7c862003-01-28 15:27:57 +00002013 0: ] EMPTY_LIST
2014 1: q BINPUT 0
2015 3: ( MARK
2016 4: K BININT1 1
2017 6: K BININT1 2
2018 8: ( MARK
2019 9: K BININT1 3
2020 11: K BININT1 4
2021 13: t TUPLE (MARK at 8)
2022 14: q BINPUT 1
2023 16: } EMPTY_DICT
2024 17: q BINPUT 2
2025 19: U SHORT_BINSTRING 'abc'
2026 24: q BINPUT 3
Guido van Rossumcfe5f202007-05-08 21:26:54 +00002027 26: X BINUNICODE 'def'
Tim Petersd0f7c862003-01-28 15:27:57 +00002028 34: q BINPUT 4
2029 36: s SETITEM
2030 37: e APPENDS (MARK at 3)
2031 38: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002032highest protocol among opcodes = 1
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002033
2034Exercise the INST/OBJ/BUILD family.
2035
2036>>> import random
Guido van Rossum4f7ac2e2007-02-26 15:59:50 +00002037>>> dis(pickle.dumps(random.getrandbits, 0))
2038 0: c GLOBAL 'random getrandbits'
2039 20: p PUT 0
2040 23: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002041highest protocol among opcodes = 0
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002042
Tim Peters90718a42005-02-15 16:22:34 +00002043>>> from pickletools import _Example
2044>>> x = [_Example(42)] * 2
Guido van Rossumf29d3d62003-01-27 22:47:53 +00002045>>> dis(pickle.dumps(x, 0))
Tim Petersd0f7c862003-01-28 15:27:57 +00002046 0: ( MARK
2047 1: l LIST (MARK at 0)
2048 2: p PUT 0
Guido van Rossum65810fe2006-05-26 19:12:38 +00002049 5: c GLOBAL 'copy_reg _reconstructor'
2050 30: p PUT 1
2051 33: ( MARK
2052 34: c GLOBAL 'pickletools _Example'
2053 56: p PUT 2
2054 59: c GLOBAL '__builtin__ object'
2055 79: p PUT 3
2056 82: N NONE
2057 83: t TUPLE (MARK at 33)
2058 84: p PUT 4
2059 87: R REDUCE
2060 88: p PUT 5
2061 91: ( MARK
2062 92: d DICT (MARK at 91)
2063 93: p PUT 6
Guido van Rossum26986312007-07-17 00:19:46 +00002064 96: V UNICODE 'value'
2065 103: p PUT 7
2066 106: L LONG 42
2067 110: s SETITEM
2068 111: b BUILD
2069 112: a APPEND
2070 113: g GET 5
2071 116: a APPEND
2072 117: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002073highest protocol among opcodes = 0
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002074
2075>>> dis(pickle.dumps(x, 1))
Tim Petersd0f7c862003-01-28 15:27:57 +00002076 0: ] EMPTY_LIST
2077 1: q BINPUT 0
2078 3: ( MARK
Guido van Rossum65810fe2006-05-26 19:12:38 +00002079 4: c GLOBAL 'copy_reg _reconstructor'
2080 29: q BINPUT 1
2081 31: ( MARK
2082 32: c GLOBAL 'pickletools _Example'
2083 54: q BINPUT 2
2084 56: c GLOBAL '__builtin__ object'
2085 76: q BINPUT 3
2086 78: N NONE
2087 79: t TUPLE (MARK at 31)
2088 80: q BINPUT 4
2089 82: R REDUCE
2090 83: q BINPUT 5
2091 85: } EMPTY_DICT
2092 86: q BINPUT 6
Guido van Rossum26986312007-07-17 00:19:46 +00002093 88: X BINUNICODE 'value'
2094 98: q BINPUT 7
2095 100: K BININT1 42
2096 102: s SETITEM
2097 103: b BUILD
2098 104: h BINGET 5
2099 106: e APPENDS (MARK at 3)
2100 107: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002101highest protocol among opcodes = 1
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002102
2103Try "the canonical" recursive-object test.
2104
2105>>> L = []
2106>>> T = L,
2107>>> L.append(T)
2108>>> L[0] is T
2109True
2110>>> T[0] is L
2111True
2112>>> L[0][0] is L
2113True
2114>>> T[0][0] is T
2115True
Guido van Rossumf29d3d62003-01-27 22:47:53 +00002116>>> dis(pickle.dumps(L, 0))
Tim Petersd0f7c862003-01-28 15:27:57 +00002117 0: ( MARK
2118 1: l LIST (MARK at 0)
2119 2: p PUT 0
2120 5: ( MARK
2121 6: g GET 0
2122 9: t TUPLE (MARK at 5)
2123 10: p PUT 1
2124 13: a APPEND
2125 14: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002126highest protocol among opcodes = 0
2127
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002128>>> dis(pickle.dumps(L, 1))
Tim Petersd0f7c862003-01-28 15:27:57 +00002129 0: ] EMPTY_LIST
2130 1: q BINPUT 0
2131 3: ( MARK
2132 4: h BINGET 0
2133 6: t TUPLE (MARK at 3)
2134 7: q BINPUT 1
2135 9: a APPEND
2136 10: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002137highest protocol among opcodes = 1
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002138
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002139Note that, in the protocol 0 pickle of the recursive tuple, the disassembler
2140has to emulate the stack in order to realize that the POP opcode at 16 gets
2141rid of the MARK at 0.
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002142
Guido van Rossumf29d3d62003-01-27 22:47:53 +00002143>>> dis(pickle.dumps(T, 0))
Tim Petersd0f7c862003-01-28 15:27:57 +00002144 0: ( MARK
2145 1: ( MARK
2146 2: l LIST (MARK at 1)
2147 3: p PUT 0
2148 6: ( MARK
2149 7: g GET 0
2150 10: t TUPLE (MARK at 6)
2151 11: p PUT 1
2152 14: a APPEND
2153 15: 0 POP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002154 16: 0 POP (MARK at 0)
2155 17: g GET 1
2156 20: . STOP
2157highest protocol among opcodes = 0
2158
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002159>>> dis(pickle.dumps(T, 1))
Tim Petersd0f7c862003-01-28 15:27:57 +00002160 0: ( MARK
2161 1: ] EMPTY_LIST
2162 2: q BINPUT 0
2163 4: ( MARK
2164 5: h BINGET 0
2165 7: t TUPLE (MARK at 4)
2166 8: q BINPUT 1
2167 10: a APPEND
2168 11: 1 POP_MARK (MARK at 0)
2169 12: h BINGET 1
2170 14: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002171highest protocol among opcodes = 1
Tim Petersd0f7c862003-01-28 15:27:57 +00002172
2173Try protocol 2.
2174
2175>>> dis(pickle.dumps(L, 2))
2176 0: \x80 PROTO 2
2177 2: ] EMPTY_LIST
2178 3: q BINPUT 0
2179 5: h BINGET 0
2180 7: \x85 TUPLE1
2181 8: q BINPUT 1
2182 10: a APPEND
2183 11: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002184highest protocol among opcodes = 2
Tim Petersd0f7c862003-01-28 15:27:57 +00002185
2186>>> dis(pickle.dumps(T, 2))
2187 0: \x80 PROTO 2
2188 2: ] EMPTY_LIST
2189 3: q BINPUT 0
2190 5: h BINGET 0
2191 7: \x85 TUPLE1
2192 8: q BINPUT 1
2193 10: a APPEND
2194 11: 0 POP
2195 12: h BINGET 1
2196 14: . STOP
Tim Petersc1c2b3e2003-01-29 20:12:21 +00002197highest protocol among opcodes = 2
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002198"""
2199
Tim Peters62235e72003-02-05 19:55:53 +00002200_memo_test = r"""
2201>>> import pickle
Guido van Rossumcfe5f202007-05-08 21:26:54 +00002202>>> import io
2203>>> f = io.BytesIO()
Tim Peters62235e72003-02-05 19:55:53 +00002204>>> p = pickle.Pickler(f, 2)
2205>>> x = [1, 2, 3]
2206>>> p.dump(x)
2207>>> p.dump(x)
2208>>> f.seek(0)
Guido van Rossumcfe5f202007-05-08 21:26:54 +000022090
Tim Peters62235e72003-02-05 19:55:53 +00002210>>> memo = {}
2211>>> dis(f, memo=memo)
2212 0: \x80 PROTO 2
2213 2: ] EMPTY_LIST
2214 3: q BINPUT 0
2215 5: ( MARK
2216 6: K BININT1 1
2217 8: K BININT1 2
2218 10: K BININT1 3
2219 12: e APPENDS (MARK at 5)
2220 13: . STOP
2221highest protocol among opcodes = 2
2222>>> dis(f, memo=memo)
2223 14: \x80 PROTO 2
2224 16: h BINGET 0
2225 18: . STOP
2226highest protocol among opcodes = 2
2227"""
2228
Guido van Rossum57028352003-01-28 15:09:10 +00002229__test__ = {'disassembler_test': _dis_test,
Tim Peters62235e72003-02-05 19:55:53 +00002230 'disassembler_memo_test': _memo_test,
Tim Peters8ecfc8e2003-01-27 18:51:48 +00002231 }
2232
2233def _test():
2234 import doctest
2235 return doctest.testmod()
2236
2237if __name__ == "__main__":
2238 _test()