blob: 387420c17bd152800543ab0cae713b137f846db9 [file] [log] [blame]
Georg Brandld7413152009-10-11 21:25:26 +00001======================
2Design and History FAQ
3======================
4
Andrés Delfino38cf49b2018-06-23 15:27:16 -03005.. only:: html
6
7 .. contents::
8
9
Georg Brandld7413152009-10-11 21:25:26 +000010Why does Python use indentation for grouping of statements?
11-----------------------------------------------------------
12
13Guido van Rossum believes that using indentation for grouping is extremely
14elegant and contributes a lot to the clarity of the average Python program.
Georg Brandl99de4882009-12-20 14:24:06 +000015Most people learn to love this feature after a while.
Georg Brandld7413152009-10-11 21:25:26 +000016
17Since there are no begin/end brackets there cannot be a disagreement between
18grouping perceived by the parser and the human reader. Occasionally C
19programmers will encounter a fragment of code like this::
20
21 if (x <= y)
22 x++;
23 y--;
24 z++;
25
26Only the ``x++`` statement is executed if the condition is true, but the
Aerosd0068002019-06-21 00:43:07 -040027indentation leads many to believe otherwise. Even experienced C programmers will
28sometimes stare at it a long time wondering as to why ``y`` is being decremented even
Georg Brandld7413152009-10-11 21:25:26 +000029for ``x > y``.
30
31Because there are no begin/end brackets, Python is much less prone to
32coding-style conflicts. In C there are many different ways to place the braces.
Aerosd0068002019-06-21 00:43:07 -040033After becoming used to reading and writing code using a particular style,
34it is normal to feel somewhat uneasy when reading (or being required to write)
35in a different one.
36
Georg Brandld7413152009-10-11 21:25:26 +000037
Georg Brandl6faee4e2010-09-21 14:48:28 +000038Many coding styles place begin/end brackets on a line by themselves. This makes
Georg Brandld7413152009-10-11 21:25:26 +000039programs considerably longer and wastes valuable screen space, making it harder
40to get a good overview of a program. Ideally, a function should fit on one
Serhiy Storchakac7b1a0b2016-11-26 13:43:28 +020041screen (say, 20--30 lines). 20 lines of Python can do a lot more work than 20
Georg Brandld7413152009-10-11 21:25:26 +000042lines of C. This is not solely due to the lack of begin/end brackets -- the
43lack of declarations and the high-level data types are also responsible -- but
44the indentation-based syntax certainly helps.
45
46
47Why am I getting strange results with simple arithmetic operations?
48-------------------------------------------------------------------
49
50See the next question.
51
52
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010053Why are floating-point calculations so inaccurate?
Georg Brandld7413152009-10-11 21:25:26 +000054--------------------------------------------------
55
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010056Users are often surprised by results like this::
Georg Brandld7413152009-10-11 21:25:26 +000057
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010058 >>> 1.2 - 1.0
Georg Brandl9205e9e2014-10-06 17:51:09 +020059 0.19999999999999996
Georg Brandld7413152009-10-11 21:25:26 +000060
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010061and think it is a bug in Python. It's not. This has little to do with Python,
62and much more to do with how the underlying platform handles floating-point
63numbers.
Georg Brandld7413152009-10-11 21:25:26 +000064
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010065The :class:`float` type in CPython uses a C ``double`` for storage. A
66:class:`float` object's value is stored in binary floating-point with a fixed
67precision (typically 53 bits) and Python uses C operations, which in turn rely
68on the hardware implementation in the processor, to perform floating-point
69operations. This means that as far as floating-point operations are concerned,
70Python behaves like many popular languages including C and Java.
Georg Brandld7413152009-10-11 21:25:26 +000071
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010072Many numbers that can be written easily in decimal notation cannot be expressed
73exactly in binary floating-point. For example, after::
Georg Brandld7413152009-10-11 21:25:26 +000074
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010075 >>> x = 1.2
Georg Brandld7413152009-10-11 21:25:26 +000076
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010077the value stored for ``x`` is a (very good) approximation to the decimal value
78``1.2``, but is not exactly equal to it. On a typical machine, the actual
79stored value is::
Georg Brandld7413152009-10-11 21:25:26 +000080
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010081 1.0011001100110011001100110011001100110011001100110011 (binary)
Georg Brandld7413152009-10-11 21:25:26 +000082
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010083which is exactly::
Georg Brandld7413152009-10-11 21:25:26 +000084
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010085 1.1999999999999999555910790149937383830547332763671875 (decimal)
Georg Brandld7413152009-10-11 21:25:26 +000086
Serhiy Storchakac7b1a0b2016-11-26 13:43:28 +020087The typical precision of 53 bits provides Python floats with 15--16
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010088decimal digits of accuracy.
Georg Brandld7413152009-10-11 21:25:26 +000089
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010090For a fuller explanation, please see the :ref:`floating point arithmetic
91<tut-fp-issues>` chapter in the Python tutorial.
Georg Brandld7413152009-10-11 21:25:26 +000092
93
94Why are Python strings immutable?
95---------------------------------
96
97There are several advantages.
98
99One is performance: knowing that a string is immutable means we can allocate
100space for it at creation time, and the storage requirements are fixed and
101unchanging. This is also one of the reasons for the distinction between tuples
102and lists.
103
104Another advantage is that strings in Python are considered as "elemental" as
105numbers. No amount of activity will change the value 8 to anything else, and in
106Python, no amount of activity will change the string "eight" to anything else.
107
108
109.. _why-self:
110
111Why must 'self' be used explicitly in method definitions and calls?
112-------------------------------------------------------------------
113
114The idea was borrowed from Modula-3. It turns out to be very useful, for a
115variety of reasons.
116
117First, it's more obvious that you are using a method or instance attribute
118instead of a local variable. Reading ``self.x`` or ``self.meth()`` makes it
119absolutely clear that an instance variable or method is used even if you don't
120know the class definition by heart. In C++, you can sort of tell by the lack of
121a local variable declaration (assuming globals are rare or easily recognizable)
122-- but in Python, there are no local variable declarations, so you'd have to
123look up the class definition to be sure. Some C++ and Java coding standards
124call for instance attributes to have an ``m_`` prefix, so this explicitness is
125still useful in those languages, too.
126
127Second, it means that no special syntax is necessary if you want to explicitly
128reference or call the method from a particular class. In C++, if you want to
129use a method from a base class which is overridden in a derived class, you have
Georg Brandl99de4882009-12-20 14:24:06 +0000130to use the ``::`` operator -- in Python you can write
131``baseclass.methodname(self, <argument list>)``. This is particularly useful
132for :meth:`__init__` methods, and in general in cases where a derived class
133method wants to extend the base class method of the same name and thus has to
134call the base class method somehow.
Georg Brandld7413152009-10-11 21:25:26 +0000135
136Finally, for instance variables it solves a syntactic problem with assignment:
137since local variables in Python are (by definition!) those variables to which a
Georg Brandl99de4882009-12-20 14:24:06 +0000138value is assigned in a function body (and that aren't explicitly declared
139global), there has to be some way to tell the interpreter that an assignment was
140meant to assign to an instance variable instead of to a local variable, and it
141should preferably be syntactic (for efficiency reasons). C++ does this through
Georg Brandld7413152009-10-11 21:25:26 +0000142declarations, but Python doesn't have declarations and it would be a pity having
Georg Brandl99de4882009-12-20 14:24:06 +0000143to introduce them just for this purpose. Using the explicit ``self.var`` solves
Georg Brandld7413152009-10-11 21:25:26 +0000144this nicely. Similarly, for using instance variables, having to write
Georg Brandl99de4882009-12-20 14:24:06 +0000145``self.var`` means that references to unqualified names inside a method don't
146have to search the instance's directories. To put it another way, local
147variables and instance variables live in two different namespaces, and you need
148to tell Python which namespace to use.
Georg Brandld7413152009-10-11 21:25:26 +0000149
150
151Why can't I use an assignment in an expression?
152-----------------------------------------------
153
154Many people used to C or Perl complain that they want to use this C idiom:
155
156.. code-block:: c
157
158 while (line = readline(f)) {
159 // do something with line
160 }
161
162where in Python you're forced to write this::
163
164 while True:
165 line = f.readline()
166 if not line:
167 break
Serhiy Storchakadba90392016-05-10 12:01:23 +0300168 ... # do something with line
Georg Brandld7413152009-10-11 21:25:26 +0000169
170The reason for not allowing assignment in Python expressions is a common,
171hard-to-find bug in those other languages, caused by this construct:
172
173.. code-block:: c
174
175 if (x = 0) {
176 // error handling
177 }
178 else {
179 // code that only works for nonzero x
180 }
181
182The error is a simple typo: ``x = 0``, which assigns 0 to the variable ``x``,
183was written while the comparison ``x == 0`` is certainly what was intended.
184
185Many alternatives have been proposed. Most are hacks that save some typing but
186use arbitrary or cryptic syntax or keywords, and fail the simple criterion for
187language change proposals: it should intuitively suggest the proper meaning to a
188human reader who has not yet been introduced to the construct.
189
190An interesting phenomenon is that most experienced Python programmers recognize
191the ``while True`` idiom and don't seem to be missing the assignment in
192expression construct much; it's only newcomers who express a strong desire to
193add this to the language.
194
195There's an alternative way of spelling this that seems attractive but is
196generally less robust than the "while True" solution::
197
198 line = f.readline()
199 while line:
Serhiy Storchakadba90392016-05-10 12:01:23 +0300200 ... # do something with line...
Georg Brandld7413152009-10-11 21:25:26 +0000201 line = f.readline()
202
203The problem with this is that if you change your mind about exactly how you get
204the next line (e.g. you want to change it into ``sys.stdin.readline()``) you
205have to remember to change two places in your program -- the second occurrence
206is hidden at the bottom of the loop.
207
208The best approach is to use iterators, making it possible to loop through
Antoine Pitrou11cb9612010-09-15 11:11:28 +0000209objects using the ``for`` statement. For example, :term:`file objects
210<file object>` support the iterator protocol, so you can write simply::
Georg Brandld7413152009-10-11 21:25:26 +0000211
212 for line in f:
Serhiy Storchakadba90392016-05-10 12:01:23 +0300213 ... # do something with line...
Georg Brandld7413152009-10-11 21:25:26 +0000214
215
216
217Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?
218----------------------------------------------------------------------------------------------------------------
219
INADA Naokic48e26d2018-07-31 14:49:22 +0900220As Guido said:
Georg Brandld7413152009-10-11 21:25:26 +0000221
INADA Naokic48e26d2018-07-31 14:49:22 +0900222 (a) For some operations, prefix notation just reads better than
223 postfix -- prefix (and infix!) operations have a long tradition in
224 mathematics which likes notations where the visuals help the
225 mathematician thinking about a problem. Compare the easy with which we
226 rewrite a formula like x*(a+b) into x*a + x*b to the clumsiness of
227 doing the same thing using a raw OO notation.
Georg Brandld7413152009-10-11 21:25:26 +0000228
INADA Naokic48e26d2018-07-31 14:49:22 +0900229 (b) When I read code that says len(x) I *know* that it is asking for
230 the length of something. This tells me two things: the result is an
231 integer, and the argument is some kind of container. To the contrary,
232 when I read x.len(), I have to already know that x is some kind of
233 container implementing an interface or inheriting from a class that
234 has a standard len(). Witness the confusion we occasionally have when
235 a class that is not implementing a mapping has a get() or keys()
236 method, or something that isn't a file has a write() method.
Georg Brandld7413152009-10-11 21:25:26 +0000237
INADA Naokic48e26d2018-07-31 14:49:22 +0900238 -- https://mail.python.org/pipermail/python-3000/2006-November/004643.html
Georg Brandld7413152009-10-11 21:25:26 +0000239
240
241Why is join() a string method instead of a list or tuple method?
242----------------------------------------------------------------
243
244Strings became much more like other standard types starting in Python 1.6, when
245methods were added which give the same functionality that has always been
246available using the functions of the string module. Most of these new methods
247have been widely accepted, but the one which appears to make some programmers
248feel uncomfortable is::
249
250 ", ".join(['1', '2', '4', '8', '16'])
251
252which gives the result::
253
254 "1, 2, 4, 8, 16"
255
256There are two common arguments against this usage.
257
258The first runs along the lines of: "It looks really ugly using a method of a
259string literal (string constant)", to which the answer is that it might, but a
260string literal is just a fixed value. If the methods are to be allowed on names
261bound to strings there is no logical reason to make them unavailable on
262literals.
263
264The second objection is typically cast as: "I am really telling a sequence to
265join its members together with a string constant". Sadly, you aren't. For some
266reason there seems to be much less difficulty with having :meth:`~str.split` as
267a string method, since in that case it is easy to see that ::
268
269 "1, 2, 4, 8, 16".split(", ")
270
271is an instruction to a string literal to return the substrings delimited by the
Georg Brandl99de4882009-12-20 14:24:06 +0000272given separator (or, by default, arbitrary runs of white space).
Georg Brandld7413152009-10-11 21:25:26 +0000273
274:meth:`~str.join` is a string method because in using it you are telling the
275separator string to iterate over a sequence of strings and insert itself between
276adjacent elements. This method can be used with any argument which obeys the
277rules for sequence objects, including any new classes you might define yourself.
Georg Brandl99de4882009-12-20 14:24:06 +0000278Similar methods exist for bytes and bytearray objects.
Georg Brandld7413152009-10-11 21:25:26 +0000279
280
281How fast are exceptions?
282------------------------
283
Georg Brandl12c3cd72012-03-17 16:58:05 +0100284A try/except block is extremely efficient if no exceptions are raised. Actually
285catching an exception is expensive. In versions of Python prior to 2.0 it was
286common to use this idiom::
Georg Brandld7413152009-10-11 21:25:26 +0000287
288 try:
Georg Brandl99de4882009-12-20 14:24:06 +0000289 value = mydict[key]
Georg Brandld7413152009-10-11 21:25:26 +0000290 except KeyError:
Georg Brandl99de4882009-12-20 14:24:06 +0000291 mydict[key] = getvalue(key)
292 value = mydict[key]
Georg Brandld7413152009-10-11 21:25:26 +0000293
294This only made sense when you expected the dict to have the key almost all the
295time. If that wasn't the case, you coded it like this::
296
Georg Brandl12c3cd72012-03-17 16:58:05 +0100297 if key in mydict:
Georg Brandl99de4882009-12-20 14:24:06 +0000298 value = mydict[key]
Georg Brandld7413152009-10-11 21:25:26 +0000299 else:
Georg Brandl12c3cd72012-03-17 16:58:05 +0100300 value = mydict[key] = getvalue(key)
Georg Brandld7413152009-10-11 21:25:26 +0000301
Georg Brandlbfe95ac2009-12-19 17:46:40 +0000302For this specific case, you could also use ``value = dict.setdefault(key,
303getvalue(key))``, but only if the ``getvalue()`` call is cheap enough because it
304is evaluated in all cases.
Georg Brandld7413152009-10-11 21:25:26 +0000305
306
307Why isn't there a switch or case statement in Python?
308-----------------------------------------------------
309
310You can do this easily enough with a sequence of ``if... elif... elif... else``.
311There have been some proposals for switch statement syntax, but there is no
312consensus (yet) on whether and how to do range tests. See :pep:`275` for
313complete details and the current status.
314
315For cases where you need to choose from a very large number of possibilities,
316you can create a dictionary mapping case values to functions to call. For
317example::
318
319 def function_1(...):
320 ...
321
322 functions = {'a': function_1,
323 'b': function_2,
324 'c': self.method_1, ...}
325
326 func = functions[value]
327 func()
328
329For calling methods on objects, you can simplify yet further by using the
330:func:`getattr` built-in to retrieve methods with a particular name::
331
332 def visit_a(self, ...):
333 ...
334 ...
335
336 def dispatch(self, value):
337 method_name = 'visit_' + str(value)
338 method = getattr(self, method_name)
339 method()
340
341It's suggested that you use a prefix for the method names, such as ``visit_`` in
342this example. Without such a prefix, if values are coming from an untrusted
343source, an attacker would be able to call any method on your object.
344
345
346Can't you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?
347--------------------------------------------------------------------------------------------------------
348
349Answer 1: Unfortunately, the interpreter pushes at least one C stack frame for
350each Python stack frame. Also, extensions can call back into Python at almost
351random moments. Therefore, a complete threads implementation requires thread
352support for C.
353
Julien Palarda6e1e412018-07-05 06:31:38 +0200354Answer 2: Fortunately, there is `Stackless Python <https://github.com/stackless-dev/stackless/wiki>`_,
Georg Brandld7413152009-10-11 21:25:26 +0000355which has a completely redesigned interpreter loop that avoids the C stack.
Georg Brandld7413152009-10-11 21:25:26 +0000356
357
Georg Brandl242e6a02013-10-06 10:28:39 +0200358Why can't lambda expressions contain statements?
359------------------------------------------------
Georg Brandld7413152009-10-11 21:25:26 +0000360
Georg Brandl242e6a02013-10-06 10:28:39 +0200361Python lambda expressions cannot contain statements because Python's syntactic
Georg Brandld7413152009-10-11 21:25:26 +0000362framework can't handle statements nested inside expressions. However, in
363Python, this is not a serious problem. Unlike lambda forms in other languages,
364where they add functionality, Python lambdas are only a shorthand notation if
365you're too lazy to define a function.
366
367Functions are already first class objects in Python, and can be declared in a
Georg Brandl242e6a02013-10-06 10:28:39 +0200368local scope. Therefore the only advantage of using a lambda instead of a
Georg Brandld7413152009-10-11 21:25:26 +0000369locally-defined function is that you don't need to invent a name for the
370function -- but that's just a local variable to which the function object (which
Georg Brandl242e6a02013-10-06 10:28:39 +0200371is exactly the same type of object that a lambda expression yields) is assigned!
Georg Brandld7413152009-10-11 21:25:26 +0000372
373
374Can Python be compiled to machine code, C or some other language?
375-----------------------------------------------------------------
376
Brett Cannon78ffd6c2016-11-18 10:41:28 -0800377`Cython <http://cython.org/>`_ compiles a modified version of Python with
378optional annotations into C extensions. `Nuitka <http://www.nuitka.net/>`_ is
379an up-and-coming compiler of Python into C++ code, aiming to support the full
380Python language. For compiling to Java you can consider
381`VOC <https://voc.readthedocs.io>`_.
Georg Brandld7413152009-10-11 21:25:26 +0000382
Georg Brandld7413152009-10-11 21:25:26 +0000383
384How does Python manage memory?
385------------------------------
386
387The details of Python memory management depend on the implementation. The
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100388standard implementation of Python, :term:`CPython`, uses reference counting to
389detect inaccessible objects, and another mechanism to collect reference cycles,
Georg Brandld7413152009-10-11 21:25:26 +0000390periodically executing a cycle detection algorithm which looks for inaccessible
391cycles and deletes the objects involved. The :mod:`gc` module provides functions
392to perform a garbage collection, obtain debugging statistics, and tune the
393collector's parameters.
394
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100395Other implementations (such as `Jython <http://www.jython.org>`_ or
396`PyPy <http://www.pypy.org>`_), however, can rely on a different mechanism
397such as a full-blown garbage collector. This difference can cause some
398subtle porting problems if your Python code depends on the behavior of the
399reference counting implementation.
Georg Brandld7413152009-10-11 21:25:26 +0000400
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100401In some Python implementations, the following code (which is fine in CPython)
402will probably run out of file descriptors::
Georg Brandld7413152009-10-11 21:25:26 +0000403
Georg Brandl99de4882009-12-20 14:24:06 +0000404 for file in very_long_list_of_files:
Georg Brandld7413152009-10-11 21:25:26 +0000405 f = open(file)
406 c = f.read(1)
407
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100408Indeed, using CPython's reference counting and destructor scheme, each new
409assignment to *f* closes the previous file. With a traditional GC, however,
410those file objects will only get collected (and closed) at varying and possibly
411long intervals.
412
413If you want to write code that will work with any Python implementation,
414you should explicitly close the file or use the :keyword:`with` statement;
415this will work regardless of memory management scheme::
Georg Brandld7413152009-10-11 21:25:26 +0000416
Georg Brandl99de4882009-12-20 14:24:06 +0000417 for file in very_long_list_of_files:
418 with open(file) as f:
419 c = f.read(1)
Georg Brandld7413152009-10-11 21:25:26 +0000420
421
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100422Why doesn't CPython use a more traditional garbage collection scheme?
423---------------------------------------------------------------------
424
425For one thing, this is not a C standard feature and hence it's not portable.
426(Yes, we know about the Boehm GC library. It has bits of assembler code for
427*most* common platforms, not for all of them, and although it is mostly
428transparent, it isn't completely transparent; patches are required to get
429Python to work with it.)
430
431Traditional GC also becomes a problem when Python is embedded into other
432applications. While in a standalone Python it's fine to replace the standard
433malloc() and free() with versions provided by the GC library, an application
434embedding Python may want to have its *own* substitute for malloc() and free(),
435and may not want Python's. Right now, CPython works with anything that
436implements malloc() and free() properly.
437
438
439Why isn't all memory freed when CPython exits?
440----------------------------------------------
Georg Brandld7413152009-10-11 21:25:26 +0000441
442Objects referenced from the global namespaces of Python modules are not always
443deallocated when Python exits. This may happen if there are circular
444references. There are also certain bits of memory that are allocated by the C
445library that are impossible to free (e.g. a tool like Purify will complain about
446these). Python is, however, aggressive about cleaning up memory on exit and
447does try to destroy every single object.
448
449If you want to force Python to delete certain things on deallocation use the
450:mod:`atexit` module to run a function that will force those deletions.
451
452
453Why are there separate tuple and list data types?
454-------------------------------------------------
455
456Lists and tuples, while similar in many respects, are generally used in
457fundamentally different ways. Tuples can be thought of as being similar to
458Pascal records or C structs; they're small collections of related data which may
459be of different types which are operated on as a group. For example, a
460Cartesian coordinate is appropriately represented as a tuple of two or three
461numbers.
462
463Lists, on the other hand, are more like arrays in other languages. They tend to
464hold a varying number of objects all of which have the same type and which are
465operated on one-by-one. For example, ``os.listdir('.')`` returns a list of
466strings representing the files in the current directory. Functions which
467operate on this output would generally not break if you added another file or
468two to the directory.
469
470Tuples are immutable, meaning that once a tuple has been created, you can't
471replace any of its elements with a new value. Lists are mutable, meaning that
472you can always change a list's elements. Only immutable elements can be used as
473dictionary keys, and hence only tuples and not lists can be used as keys.
474
475
Andrés Delfino8d412782018-07-07 20:25:47 -0300476How are lists implemented in CPython?
477-------------------------------------
Georg Brandld7413152009-10-11 21:25:26 +0000478
Andrés Delfino8d412782018-07-07 20:25:47 -0300479CPython's lists are really variable-length arrays, not Lisp-style linked lists.
Georg Brandld7413152009-10-11 21:25:26 +0000480The implementation uses a contiguous array of references to other objects, and
481keeps a pointer to this array and the array's length in a list head structure.
482
483This makes indexing a list ``a[i]`` an operation whose cost is independent of
484the size of the list or the value of the index.
485
486When items are appended or inserted, the array of references is resized. Some
487cleverness is applied to improve the performance of appending items repeatedly;
488when the array must be grown, some extra space is allocated so the next few
489times don't require an actual resize.
490
491
Andrés Delfino8d412782018-07-07 20:25:47 -0300492How are dictionaries implemented in CPython?
493--------------------------------------------
Georg Brandld7413152009-10-11 21:25:26 +0000494
Andrés Delfino8d412782018-07-07 20:25:47 -0300495CPython's dictionaries are implemented as resizable hash tables. Compared to
Georg Brandld7413152009-10-11 21:25:26 +0000496B-trees, this gives better performance for lookup (the most common operation by
497far) under most circumstances, and the implementation is simpler.
498
499Dictionaries work by computing a hash code for each key stored in the dictionary
500using the :func:`hash` built-in function. The hash code varies widely depending
Georg Brandlb20a0192012-03-14 07:50:17 +0100501on the key and a per-process seed; for example, "Python" could hash to
502-539294296 while "python", a string that differs by a single bit, could hash
503to 1142331976. The hash code is then used to calculate a location in an
504internal array where the value will be stored. Assuming that you're storing
505keys that all have different hash values, this means that dictionaries take
Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి)866c1682018-06-26 13:57:05 +0530506constant time -- O(1), in Big-O notation -- to retrieve a key.
Georg Brandld7413152009-10-11 21:25:26 +0000507
508
509Why must dictionary keys be immutable?
510--------------------------------------
511
512The hash table implementation of dictionaries uses a hash value calculated from
513the key value to find the key. If the key were a mutable object, its value
514could change, and thus its hash could also change. But since whoever changes
515the key object can't tell that it was being used as a dictionary key, it can't
516move the entry around in the dictionary. Then, when you try to look up the same
517object in the dictionary it won't be found because its hash value is different.
518If you tried to look up the old value it wouldn't be found either, because the
519value of the object found in that hash bin would be different.
520
521If you want a dictionary indexed with a list, simply convert the list to a tuple
522first; the function ``tuple(L)`` creates a tuple with the same entries as the
523list ``L``. Tuples are immutable and can therefore be used as dictionary keys.
524
525Some unacceptable solutions that have been proposed:
526
527- Hash lists by their address (object ID). This doesn't work because if you
528 construct a new list with the same value it won't be found; e.g.::
529
Georg Brandl99de4882009-12-20 14:24:06 +0000530 mydict = {[1, 2]: '12'}
531 print(mydict[[1, 2]])
Georg Brandld7413152009-10-11 21:25:26 +0000532
Stéphane Wirtele483f022018-10-26 12:52:11 +0200533 would raise a :exc:`KeyError` exception because the id of the ``[1, 2]`` used in the
Georg Brandld7413152009-10-11 21:25:26 +0000534 second line differs from that in the first line. In other words, dictionary
535 keys should be compared using ``==``, not using :keyword:`is`.
536
537- Make a copy when using a list as a key. This doesn't work because the list,
538 being a mutable object, could contain a reference to itself, and then the
539 copying code would run into an infinite loop.
540
541- Allow lists as keys but tell the user not to modify them. This would allow a
542 class of hard-to-track bugs in programs when you forgot or modified a list by
543 accident. It also invalidates an important invariant of dictionaries: every
544 value in ``d.keys()`` is usable as a key of the dictionary.
545
546- Mark lists as read-only once they are used as a dictionary key. The problem
547 is that it's not just the top-level object that could change its value; you
548 could use a tuple containing a list as a key. Entering anything as a key into
549 a dictionary would require marking all objects reachable from there as
550 read-only -- and again, self-referential objects could cause an infinite loop.
551
552There is a trick to get around this if you need to, but use it at your own risk:
553You can wrap a mutable structure inside a class instance which has both a
Georg Brandl99de4882009-12-20 14:24:06 +0000554:meth:`__eq__` and a :meth:`__hash__` method. You must then make sure that the
Georg Brandld7413152009-10-11 21:25:26 +0000555hash value for all such wrapper objects that reside in a dictionary (or other
556hash based structure), remain fixed while the object is in the dictionary (or
557other structure). ::
558
559 class ListWrapper:
560 def __init__(self, the_list):
561 self.the_list = the_list
Serhiy Storchakadba90392016-05-10 12:01:23 +0300562
Georg Brandl99de4882009-12-20 14:24:06 +0000563 def __eq__(self, other):
Georg Brandld7413152009-10-11 21:25:26 +0000564 return self.the_list == other.the_list
Serhiy Storchakadba90392016-05-10 12:01:23 +0300565
Georg Brandld7413152009-10-11 21:25:26 +0000566 def __hash__(self):
567 l = self.the_list
568 result = 98767 - len(l)*555
Georg Brandl99de4882009-12-20 14:24:06 +0000569 for i, el in enumerate(l):
Georg Brandld7413152009-10-11 21:25:26 +0000570 try:
Georg Brandl99de4882009-12-20 14:24:06 +0000571 result = result + (hash(el) % 9999999) * 1001 + i
572 except Exception:
Georg Brandld7413152009-10-11 21:25:26 +0000573 result = (result % 7777777) + i * 333
574 return result
575
576Note that the hash computation is complicated by the possibility that some
577members of the list may be unhashable and also by the possibility of arithmetic
578overflow.
579
Georg Brandl99de4882009-12-20 14:24:06 +0000580Furthermore it must always be the case that if ``o1 == o2`` (ie ``o1.__eq__(o2)
581is True``) then ``hash(o1) == hash(o2)`` (ie, ``o1.__hash__() == o2.__hash__()``),
Georg Brandld7413152009-10-11 21:25:26 +0000582regardless of whether the object is in a dictionary or not. If you fail to meet
583these restrictions dictionaries and other hash based structures will misbehave.
584
585In the case of ListWrapper, whenever the wrapper object is in a dictionary the
586wrapped list must not change to avoid anomalies. Don't do this unless you are
587prepared to think hard about the requirements and the consequences of not
588meeting them correctly. Consider yourself warned.
589
590
591Why doesn't list.sort() return the sorted list?
592-----------------------------------------------
593
594In situations where performance matters, making a copy of the list just to sort
595it would be wasteful. Therefore, :meth:`list.sort` sorts the list in place. In
596order to remind you of that fact, it does not return the sorted list. This way,
597you won't be fooled into accidentally overwriting a list when you need a sorted
598copy but also need to keep the unsorted version around.
599
Antoine Pitroudec0f212011-12-03 23:08:57 +0100600If you want to return a new list, use the built-in :func:`sorted` function
601instead. This function creates a new list from a provided iterable, sorts
602it and returns it. For example, here's how to iterate over the keys of a
603dictionary in sorted order::
Georg Brandld7413152009-10-11 21:25:26 +0000604
Georg Brandl99de4882009-12-20 14:24:06 +0000605 for key in sorted(mydict):
Serhiy Storchakadba90392016-05-10 12:01:23 +0300606 ... # do whatever with mydict[key]...
Georg Brandld7413152009-10-11 21:25:26 +0000607
608
609How do you specify and enforce an interface spec in Python?
610-----------------------------------------------------------
611
612An interface specification for a module as provided by languages such as C++ and
613Java describes the prototypes for the methods and functions of the module. Many
614feel that compile-time enforcement of interface specifications helps in the
615construction of large programs.
616
617Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
618(ABCs). You can then use :func:`isinstance` and :func:`issubclass` to check
619whether an instance or a class implements a particular ABC. The
Éric Araujob8edbdf2011-09-01 05:57:12 +0200620:mod:`collections.abc` module defines a set of useful ABCs such as
Serhiy Storchakabfdcd432013-10-13 23:09:14 +0300621:class:`~collections.abc.Iterable`, :class:`~collections.abc.Container`, and
622:class:`~collections.abc.MutableMapping`.
Georg Brandld7413152009-10-11 21:25:26 +0000623
624For Python, many of the advantages of interface specifications can be obtained
625by an appropriate test discipline for components. There is also a tool,
626PyChecker, which can be used to find problems due to subclassing.
627
628A good test suite for a module can both provide a regression test and serve as a
629module interface specification and a set of examples. Many Python modules can
630be run as a script to provide a simple "self test." Even modules which use
631complex external interfaces can often be tested in isolation using trivial
632"stub" emulations of the external interface. The :mod:`doctest` and
633:mod:`unittest` modules or third-party test frameworks can be used to construct
634exhaustive test suites that exercise every line of code in a module.
635
636An appropriate testing discipline can help build large complex applications in
637Python as well as having interface specifications would. In fact, it can be
638better because an interface specification cannot test certain properties of a
639program. For example, the :meth:`append` method is expected to add new elements
640to the end of some internal list; an interface specification cannot test that
641your :meth:`append` implementation will actually do this correctly, but it's
642trivial to check this property in a test suite.
643
644Writing test suites is very helpful, and you might want to design your code with
645an eye to making it easily tested. One increasingly popular technique,
646test-directed development, calls for writing parts of the test suite first,
647before you write any of the actual code. Of course Python allows you to be
648sloppy and not write test cases at all.
649
650
Georg Brandld7413152009-10-11 21:25:26 +0000651Why is there no goto?
652---------------------
653
654You can use exceptions to provide a "structured goto" that even works across
655function calls. Many feel that exceptions can conveniently emulate all
656reasonable uses of the "go" or "goto" constructs of C, Fortran, and other
657languages. For example::
658
Ezio Melotti19cdee82013-01-05 06:53:27 +0200659 class label(Exception): pass # declare a label
Georg Brandld7413152009-10-11 21:25:26 +0000660
661 try:
Serhiy Storchakadba90392016-05-10 12:01:23 +0300662 ...
663 if condition: raise label() # goto label
664 ...
Georg Brandl99de4882009-12-20 14:24:06 +0000665 except label: # where to goto
Serhiy Storchakadba90392016-05-10 12:01:23 +0300666 pass
Georg Brandld7413152009-10-11 21:25:26 +0000667 ...
668
669This doesn't allow you to jump into the middle of a loop, but that's usually
670considered an abuse of goto anyway. Use sparingly.
671
672
673Why can't raw strings (r-strings) end with a backslash?
674-------------------------------------------------------
675
676More precisely, they can't end with an odd number of backslashes: the unpaired
677backslash at the end escapes the closing quote character, leaving an
678unterminated string.
679
680Raw strings were designed to ease creating input for processors (chiefly regular
681expression engines) that want to do their own backslash escape processing. Such
682processors consider an unmatched trailing backslash to be an error anyway, so
683raw strings disallow that. In return, they allow you to pass on the string
684quote character by escaping it with a backslash. These rules work well when
685r-strings are used for their intended purpose.
686
687If you're trying to build Windows pathnames, note that all Windows system calls
688accept forward slashes too::
689
Georg Brandl99de4882009-12-20 14:24:06 +0000690 f = open("/mydir/file.txt") # works fine!
Georg Brandld7413152009-10-11 21:25:26 +0000691
692If you're trying to build a pathname for a DOS command, try e.g. one of ::
693
694 dir = r"\this\is\my\dos\dir" "\\"
695 dir = r"\this\is\my\dos\dir\ "[:-1]
696 dir = "\\this\\is\\my\\dos\\dir\\"
697
698
699Why doesn't Python have a "with" statement for attribute assignments?
700---------------------------------------------------------------------
701
702Python has a 'with' statement that wraps the execution of a block, calling code
703on the entrance and exit from the block. Some language have a construct that
704looks like this::
705
706 with obj:
Benjamin Peterson1baf4652009-12-31 03:11:23 +0000707 a = 1 # equivalent to obj.a = 1
Georg Brandld7413152009-10-11 21:25:26 +0000708 total = total + 1 # obj.total = obj.total + 1
709
710In Python, such a construct would be ambiguous.
711
712Other languages, such as Object Pascal, Delphi, and C++, use static types, so
713it's possible to know, in an unambiguous way, what member is being assigned
714to. This is the main point of static typing -- the compiler *always* knows the
715scope of every variable at compile time.
716
717Python uses dynamic types. It is impossible to know in advance which attribute
718will be referenced at runtime. Member attributes may be added or removed from
719objects on the fly. This makes it impossible to know, from a simple reading,
720what attribute is being referenced: a local one, a global one, or a member
721attribute?
722
723For instance, take the following incomplete snippet::
724
725 def foo(a):
726 with a:
Georg Brandl99de4882009-12-20 14:24:06 +0000727 print(x)
Georg Brandld7413152009-10-11 21:25:26 +0000728
729The snippet assumes that "a" must have a member attribute called "x". However,
730there is nothing in Python that tells the interpreter this. What should happen
731if "a" is, let us say, an integer? If there is a global variable named "x",
732will it be used inside the with block? As you see, the dynamic nature of Python
733makes such choices much harder.
734
735The primary benefit of "with" and similar language features (reduction of code
736volume) can, however, easily be achieved in Python by assignment. Instead of::
737
Georg Brandl99de4882009-12-20 14:24:06 +0000738 function(args).mydict[index][index].a = 21
739 function(args).mydict[index][index].b = 42
740 function(args).mydict[index][index].c = 63
Georg Brandld7413152009-10-11 21:25:26 +0000741
742write this::
743
Georg Brandl99de4882009-12-20 14:24:06 +0000744 ref = function(args).mydict[index][index]
Georg Brandld7413152009-10-11 21:25:26 +0000745 ref.a = 21
746 ref.b = 42
747 ref.c = 63
748
749This also has the side-effect of increasing execution speed because name
750bindings are resolved at run-time in Python, and the second version only needs
Georg Brandl99de4882009-12-20 14:24:06 +0000751to perform the resolution once.
Georg Brandld7413152009-10-11 21:25:26 +0000752
753
754Why are colons required for the if/while/def/class statements?
755--------------------------------------------------------------
756
757The colon is required primarily to enhance readability (one of the results of
758the experimental ABC language). Consider this::
759
760 if a == b
Georg Brandl99de4882009-12-20 14:24:06 +0000761 print(a)
Georg Brandld7413152009-10-11 21:25:26 +0000762
763versus ::
764
765 if a == b:
Georg Brandl99de4882009-12-20 14:24:06 +0000766 print(a)
Georg Brandld7413152009-10-11 21:25:26 +0000767
768Notice how the second one is slightly easier to read. Notice further how a
769colon sets off the example in this FAQ answer; it's a standard usage in English.
770
771Another minor reason is that the colon makes it easier for editors with syntax
772highlighting; they can look for colons to decide when indentation needs to be
773increased instead of having to do a more elaborate parsing of the program text.
774
775
776Why does Python allow commas at the end of lists and tuples?
777------------------------------------------------------------
778
779Python lets you add a trailing comma at the end of lists, tuples, and
780dictionaries::
781
782 [1, 2, 3,]
783 ('a', 'b', 'c',)
784 d = {
785 "A": [1, 5],
786 "B": [6, 7], # last trailing comma is optional but good style
787 }
788
789
790There are several reasons to allow this.
791
792When you have a literal value for a list, tuple, or dictionary spread across
793multiple lines, it's easier to add more elements because you don't have to
Georg Brandl7b8c1322013-04-14 10:31:06 +0200794remember to add a comma to the previous line. The lines can also be reordered
795without creating a syntax error.
Georg Brandld7413152009-10-11 21:25:26 +0000796
797Accidentally omitting the comma can lead to errors that are hard to diagnose.
798For example::
799
800 x = [
801 "fee",
802 "fie"
803 "foo",
804 "fum"
805 ]
806
807This list looks like it has four elements, but it actually contains three:
808"fee", "fiefoo" and "fum". Always adding the comma avoids this source of error.
809
810Allowing the trailing comma may also make programmatic code generation easier.