blob: 10fa490931c558929bb2a927ed1ab4366d97bec2 [file] [log] [blame]
Georg Brandld7413152009-10-11 21:25:26 +00001======================
2Design and History FAQ
3======================
4
Andrés Delfino38cf49b2018-06-23 15:27:16 -03005.. only:: html
6
7 .. contents::
8
9
Georg Brandld7413152009-10-11 21:25:26 +000010Why does Python use indentation for grouping of statements?
11-----------------------------------------------------------
12
13Guido van Rossum believes that using indentation for grouping is extremely
14elegant and contributes a lot to the clarity of the average Python program.
Georg Brandl99de4882009-12-20 14:24:06 +000015Most people learn to love this feature after a while.
Georg Brandld7413152009-10-11 21:25:26 +000016
17Since there are no begin/end brackets there cannot be a disagreement between
18grouping perceived by the parser and the human reader. Occasionally C
19programmers will encounter a fragment of code like this::
20
21 if (x <= y)
22 x++;
23 y--;
24 z++;
25
26Only the ``x++`` statement is executed if the condition is true, but the
27indentation leads you to believe otherwise. Even experienced C programmers will
28sometimes stare at it a long time wondering why ``y`` is being decremented even
29for ``x > y``.
30
31Because there are no begin/end brackets, Python is much less prone to
32coding-style conflicts. In C there are many different ways to place the braces.
33If you're used to reading and writing code that uses one style, you will feel at
34least slightly uneasy when reading (or being required to write) another style.
35
Georg Brandl6faee4e2010-09-21 14:48:28 +000036Many coding styles place begin/end brackets on a line by themselves. This makes
Georg Brandld7413152009-10-11 21:25:26 +000037programs considerably longer and wastes valuable screen space, making it harder
38to get a good overview of a program. Ideally, a function should fit on one
Serhiy Storchakac7b1a0b2016-11-26 13:43:28 +020039screen (say, 20--30 lines). 20 lines of Python can do a lot more work than 20
Georg Brandld7413152009-10-11 21:25:26 +000040lines of C. This is not solely due to the lack of begin/end brackets -- the
41lack of declarations and the high-level data types are also responsible -- but
42the indentation-based syntax certainly helps.
43
44
45Why am I getting strange results with simple arithmetic operations?
46-------------------------------------------------------------------
47
48See the next question.
49
50
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010051Why are floating-point calculations so inaccurate?
Georg Brandld7413152009-10-11 21:25:26 +000052--------------------------------------------------
53
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010054Users are often surprised by results like this::
Georg Brandld7413152009-10-11 21:25:26 +000055
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010056 >>> 1.2 - 1.0
Georg Brandl9205e9e2014-10-06 17:51:09 +020057 0.19999999999999996
Georg Brandld7413152009-10-11 21:25:26 +000058
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010059and think it is a bug in Python. It's not. This has little to do with Python,
60and much more to do with how the underlying platform handles floating-point
61numbers.
Georg Brandld7413152009-10-11 21:25:26 +000062
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010063The :class:`float` type in CPython uses a C ``double`` for storage. A
64:class:`float` object's value is stored in binary floating-point with a fixed
65precision (typically 53 bits) and Python uses C operations, which in turn rely
66on the hardware implementation in the processor, to perform floating-point
67operations. This means that as far as floating-point operations are concerned,
68Python behaves like many popular languages including C and Java.
Georg Brandld7413152009-10-11 21:25:26 +000069
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010070Many numbers that can be written easily in decimal notation cannot be expressed
71exactly in binary floating-point. For example, after::
Georg Brandld7413152009-10-11 21:25:26 +000072
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010073 >>> x = 1.2
Georg Brandld7413152009-10-11 21:25:26 +000074
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010075the value stored for ``x`` is a (very good) approximation to the decimal value
76``1.2``, but is not exactly equal to it. On a typical machine, the actual
77stored value is::
Georg Brandld7413152009-10-11 21:25:26 +000078
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010079 1.0011001100110011001100110011001100110011001100110011 (binary)
Georg Brandld7413152009-10-11 21:25:26 +000080
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010081which is exactly::
Georg Brandld7413152009-10-11 21:25:26 +000082
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010083 1.1999999999999999555910790149937383830547332763671875 (decimal)
Georg Brandld7413152009-10-11 21:25:26 +000084
Serhiy Storchakac7b1a0b2016-11-26 13:43:28 +020085The typical precision of 53 bits provides Python floats with 15--16
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010086decimal digits of accuracy.
Georg Brandld7413152009-10-11 21:25:26 +000087
Mark Dickinsonba3b0d82012-05-13 21:00:35 +010088For a fuller explanation, please see the :ref:`floating point arithmetic
89<tut-fp-issues>` chapter in the Python tutorial.
Georg Brandld7413152009-10-11 21:25:26 +000090
91
92Why are Python strings immutable?
93---------------------------------
94
95There are several advantages.
96
97One is performance: knowing that a string is immutable means we can allocate
98space for it at creation time, and the storage requirements are fixed and
99unchanging. This is also one of the reasons for the distinction between tuples
100and lists.
101
102Another advantage is that strings in Python are considered as "elemental" as
103numbers. No amount of activity will change the value 8 to anything else, and in
104Python, no amount of activity will change the string "eight" to anything else.
105
106
107.. _why-self:
108
109Why must 'self' be used explicitly in method definitions and calls?
110-------------------------------------------------------------------
111
112The idea was borrowed from Modula-3. It turns out to be very useful, for a
113variety of reasons.
114
115First, it's more obvious that you are using a method or instance attribute
116instead of a local variable. Reading ``self.x`` or ``self.meth()`` makes it
117absolutely clear that an instance variable or method is used even if you don't
118know the class definition by heart. In C++, you can sort of tell by the lack of
119a local variable declaration (assuming globals are rare or easily recognizable)
120-- but in Python, there are no local variable declarations, so you'd have to
121look up the class definition to be sure. Some C++ and Java coding standards
122call for instance attributes to have an ``m_`` prefix, so this explicitness is
123still useful in those languages, too.
124
125Second, it means that no special syntax is necessary if you want to explicitly
126reference or call the method from a particular class. In C++, if you want to
127use a method from a base class which is overridden in a derived class, you have
Georg Brandl99de4882009-12-20 14:24:06 +0000128to use the ``::`` operator -- in Python you can write
129``baseclass.methodname(self, <argument list>)``. This is particularly useful
130for :meth:`__init__` methods, and in general in cases where a derived class
131method wants to extend the base class method of the same name and thus has to
132call the base class method somehow.
Georg Brandld7413152009-10-11 21:25:26 +0000133
134Finally, for instance variables it solves a syntactic problem with assignment:
135since local variables in Python are (by definition!) those variables to which a
Georg Brandl99de4882009-12-20 14:24:06 +0000136value is assigned in a function body (and that aren't explicitly declared
137global), there has to be some way to tell the interpreter that an assignment was
138meant to assign to an instance variable instead of to a local variable, and it
139should preferably be syntactic (for efficiency reasons). C++ does this through
Georg Brandld7413152009-10-11 21:25:26 +0000140declarations, but Python doesn't have declarations and it would be a pity having
Georg Brandl99de4882009-12-20 14:24:06 +0000141to introduce them just for this purpose. Using the explicit ``self.var`` solves
Georg Brandld7413152009-10-11 21:25:26 +0000142this nicely. Similarly, for using instance variables, having to write
Georg Brandl99de4882009-12-20 14:24:06 +0000143``self.var`` means that references to unqualified names inside a method don't
144have to search the instance's directories. To put it another way, local
145variables and instance variables live in two different namespaces, and you need
146to tell Python which namespace to use.
Georg Brandld7413152009-10-11 21:25:26 +0000147
148
149Why can't I use an assignment in an expression?
150-----------------------------------------------
151
152Many people used to C or Perl complain that they want to use this C idiom:
153
154.. code-block:: c
155
156 while (line = readline(f)) {
157 // do something with line
158 }
159
160where in Python you're forced to write this::
161
162 while True:
163 line = f.readline()
164 if not line:
165 break
Serhiy Storchakadba90392016-05-10 12:01:23 +0300166 ... # do something with line
Georg Brandld7413152009-10-11 21:25:26 +0000167
168The reason for not allowing assignment in Python expressions is a common,
169hard-to-find bug in those other languages, caused by this construct:
170
171.. code-block:: c
172
173 if (x = 0) {
174 // error handling
175 }
176 else {
177 // code that only works for nonzero x
178 }
179
180The error is a simple typo: ``x = 0``, which assigns 0 to the variable ``x``,
181was written while the comparison ``x == 0`` is certainly what was intended.
182
183Many alternatives have been proposed. Most are hacks that save some typing but
184use arbitrary or cryptic syntax or keywords, and fail the simple criterion for
185language change proposals: it should intuitively suggest the proper meaning to a
186human reader who has not yet been introduced to the construct.
187
188An interesting phenomenon is that most experienced Python programmers recognize
189the ``while True`` idiom and don't seem to be missing the assignment in
190expression construct much; it's only newcomers who express a strong desire to
191add this to the language.
192
193There's an alternative way of spelling this that seems attractive but is
194generally less robust than the "while True" solution::
195
196 line = f.readline()
197 while line:
Serhiy Storchakadba90392016-05-10 12:01:23 +0300198 ... # do something with line...
Georg Brandld7413152009-10-11 21:25:26 +0000199 line = f.readline()
200
201The problem with this is that if you change your mind about exactly how you get
202the next line (e.g. you want to change it into ``sys.stdin.readline()``) you
203have to remember to change two places in your program -- the second occurrence
204is hidden at the bottom of the loop.
205
206The best approach is to use iterators, making it possible to loop through
Antoine Pitrou11cb9612010-09-15 11:11:28 +0000207objects using the ``for`` statement. For example, :term:`file objects
208<file object>` support the iterator protocol, so you can write simply::
Georg Brandld7413152009-10-11 21:25:26 +0000209
210 for line in f:
Serhiy Storchakadba90392016-05-10 12:01:23 +0300211 ... # do something with line...
Georg Brandld7413152009-10-11 21:25:26 +0000212
213
214
215Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?
216----------------------------------------------------------------------------------------------------------------
217
218The major reason is history. Functions were used for those operations that were
219generic for a group of types and which were intended to work even for objects
220that didn't have methods at all (e.g. tuples). It is also convenient to have a
221function that can readily be applied to an amorphous collection of objects when
Ezio Melotti9beeefb2013-01-05 07:36:54 +0200222you use the functional features of Python (``map()``, ``zip()`` et al).
Georg Brandld7413152009-10-11 21:25:26 +0000223
224In fact, implementing ``len()``, ``max()``, ``min()`` as a built-in function is
225actually less code than implementing them as methods for each type. One can
226quibble about individual cases but it's a part of Python, and it's too late to
227make such fundamental changes now. The functions have to remain to avoid massive
228code breakage.
229
230.. XXX talk about protocols?
231
Georg Brandlbfe95ac2009-12-19 17:46:40 +0000232.. note::
233
234 For string operations, Python has moved from external functions (the
235 ``string`` module) to methods. However, ``len()`` is still a function.
Georg Brandld7413152009-10-11 21:25:26 +0000236
237
238Why is join() a string method instead of a list or tuple method?
239----------------------------------------------------------------
240
241Strings became much more like other standard types starting in Python 1.6, when
242methods were added which give the same functionality that has always been
243available using the functions of the string module. Most of these new methods
244have been widely accepted, but the one which appears to make some programmers
245feel uncomfortable is::
246
247 ", ".join(['1', '2', '4', '8', '16'])
248
249which gives the result::
250
251 "1, 2, 4, 8, 16"
252
253There are two common arguments against this usage.
254
255The first runs along the lines of: "It looks really ugly using a method of a
256string literal (string constant)", to which the answer is that it might, but a
257string literal is just a fixed value. If the methods are to be allowed on names
258bound to strings there is no logical reason to make them unavailable on
259literals.
260
261The second objection is typically cast as: "I am really telling a sequence to
262join its members together with a string constant". Sadly, you aren't. For some
263reason there seems to be much less difficulty with having :meth:`~str.split` as
264a string method, since in that case it is easy to see that ::
265
266 "1, 2, 4, 8, 16".split(", ")
267
268is an instruction to a string literal to return the substrings delimited by the
Georg Brandl99de4882009-12-20 14:24:06 +0000269given separator (or, by default, arbitrary runs of white space).
Georg Brandld7413152009-10-11 21:25:26 +0000270
271:meth:`~str.join` is a string method because in using it you are telling the
272separator string to iterate over a sequence of strings and insert itself between
273adjacent elements. This method can be used with any argument which obeys the
274rules for sequence objects, including any new classes you might define yourself.
Georg Brandl99de4882009-12-20 14:24:06 +0000275Similar methods exist for bytes and bytearray objects.
Georg Brandld7413152009-10-11 21:25:26 +0000276
277
278How fast are exceptions?
279------------------------
280
Georg Brandl12c3cd72012-03-17 16:58:05 +0100281A try/except block is extremely efficient if no exceptions are raised. Actually
282catching an exception is expensive. In versions of Python prior to 2.0 it was
283common to use this idiom::
Georg Brandld7413152009-10-11 21:25:26 +0000284
285 try:
Georg Brandl99de4882009-12-20 14:24:06 +0000286 value = mydict[key]
Georg Brandld7413152009-10-11 21:25:26 +0000287 except KeyError:
Georg Brandl99de4882009-12-20 14:24:06 +0000288 mydict[key] = getvalue(key)
289 value = mydict[key]
Georg Brandld7413152009-10-11 21:25:26 +0000290
291This only made sense when you expected the dict to have the key almost all the
292time. If that wasn't the case, you coded it like this::
293
Georg Brandl12c3cd72012-03-17 16:58:05 +0100294 if key in mydict:
Georg Brandl99de4882009-12-20 14:24:06 +0000295 value = mydict[key]
Georg Brandld7413152009-10-11 21:25:26 +0000296 else:
Georg Brandl12c3cd72012-03-17 16:58:05 +0100297 value = mydict[key] = getvalue(key)
Georg Brandld7413152009-10-11 21:25:26 +0000298
Georg Brandlbfe95ac2009-12-19 17:46:40 +0000299For this specific case, you could also use ``value = dict.setdefault(key,
300getvalue(key))``, but only if the ``getvalue()`` call is cheap enough because it
301is evaluated in all cases.
Georg Brandld7413152009-10-11 21:25:26 +0000302
303
304Why isn't there a switch or case statement in Python?
305-----------------------------------------------------
306
307You can do this easily enough with a sequence of ``if... elif... elif... else``.
308There have been some proposals for switch statement syntax, but there is no
309consensus (yet) on whether and how to do range tests. See :pep:`275` for
310complete details and the current status.
311
312For cases where you need to choose from a very large number of possibilities,
313you can create a dictionary mapping case values to functions to call. For
314example::
315
316 def function_1(...):
317 ...
318
319 functions = {'a': function_1,
320 'b': function_2,
321 'c': self.method_1, ...}
322
323 func = functions[value]
324 func()
325
326For calling methods on objects, you can simplify yet further by using the
327:func:`getattr` built-in to retrieve methods with a particular name::
328
329 def visit_a(self, ...):
330 ...
331 ...
332
333 def dispatch(self, value):
334 method_name = 'visit_' + str(value)
335 method = getattr(self, method_name)
336 method()
337
338It's suggested that you use a prefix for the method names, such as ``visit_`` in
339this example. Without such a prefix, if values are coming from an untrusted
340source, an attacker would be able to call any method on your object.
341
342
343Can't you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?
344--------------------------------------------------------------------------------------------------------
345
346Answer 1: Unfortunately, the interpreter pushes at least one C stack frame for
347each Python stack frame. Also, extensions can call back into Python at almost
348random moments. Therefore, a complete threads implementation requires thread
349support for C.
350
Julien Palarda6e1e412018-07-05 06:31:38 +0200351Answer 2: Fortunately, there is `Stackless Python <https://github.com/stackless-dev/stackless/wiki>`_,
Georg Brandld7413152009-10-11 21:25:26 +0000352which has a completely redesigned interpreter loop that avoids the C stack.
Georg Brandld7413152009-10-11 21:25:26 +0000353
354
Georg Brandl242e6a02013-10-06 10:28:39 +0200355Why can't lambda expressions contain statements?
356------------------------------------------------
Georg Brandld7413152009-10-11 21:25:26 +0000357
Georg Brandl242e6a02013-10-06 10:28:39 +0200358Python lambda expressions cannot contain statements because Python's syntactic
Georg Brandld7413152009-10-11 21:25:26 +0000359framework can't handle statements nested inside expressions. However, in
360Python, this is not a serious problem. Unlike lambda forms in other languages,
361where they add functionality, Python lambdas are only a shorthand notation if
362you're too lazy to define a function.
363
364Functions are already first class objects in Python, and can be declared in a
Georg Brandl242e6a02013-10-06 10:28:39 +0200365local scope. Therefore the only advantage of using a lambda instead of a
Georg Brandld7413152009-10-11 21:25:26 +0000366locally-defined function is that you don't need to invent a name for the
367function -- but that's just a local variable to which the function object (which
Georg Brandl242e6a02013-10-06 10:28:39 +0200368is exactly the same type of object that a lambda expression yields) is assigned!
Georg Brandld7413152009-10-11 21:25:26 +0000369
370
371Can Python be compiled to machine code, C or some other language?
372-----------------------------------------------------------------
373
Brett Cannon78ffd6c2016-11-18 10:41:28 -0800374`Cython <http://cython.org/>`_ compiles a modified version of Python with
375optional annotations into C extensions. `Nuitka <http://www.nuitka.net/>`_ is
376an up-and-coming compiler of Python into C++ code, aiming to support the full
377Python language. For compiling to Java you can consider
378`VOC <https://voc.readthedocs.io>`_.
Georg Brandld7413152009-10-11 21:25:26 +0000379
Georg Brandld7413152009-10-11 21:25:26 +0000380
381How does Python manage memory?
382------------------------------
383
384The details of Python memory management depend on the implementation. The
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100385standard implementation of Python, :term:`CPython`, uses reference counting to
386detect inaccessible objects, and another mechanism to collect reference cycles,
Georg Brandld7413152009-10-11 21:25:26 +0000387periodically executing a cycle detection algorithm which looks for inaccessible
388cycles and deletes the objects involved. The :mod:`gc` module provides functions
389to perform a garbage collection, obtain debugging statistics, and tune the
390collector's parameters.
391
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100392Other implementations (such as `Jython <http://www.jython.org>`_ or
393`PyPy <http://www.pypy.org>`_), however, can rely on a different mechanism
394such as a full-blown garbage collector. This difference can cause some
395subtle porting problems if your Python code depends on the behavior of the
396reference counting implementation.
Georg Brandld7413152009-10-11 21:25:26 +0000397
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100398In some Python implementations, the following code (which is fine in CPython)
399will probably run out of file descriptors::
Georg Brandld7413152009-10-11 21:25:26 +0000400
Georg Brandl99de4882009-12-20 14:24:06 +0000401 for file in very_long_list_of_files:
Georg Brandld7413152009-10-11 21:25:26 +0000402 f = open(file)
403 c = f.read(1)
404
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100405Indeed, using CPython's reference counting and destructor scheme, each new
406assignment to *f* closes the previous file. With a traditional GC, however,
407those file objects will only get collected (and closed) at varying and possibly
408long intervals.
409
410If you want to write code that will work with any Python implementation,
411you should explicitly close the file or use the :keyword:`with` statement;
412this will work regardless of memory management scheme::
Georg Brandld7413152009-10-11 21:25:26 +0000413
Georg Brandl99de4882009-12-20 14:24:06 +0000414 for file in very_long_list_of_files:
415 with open(file) as f:
416 c = f.read(1)
Georg Brandld7413152009-10-11 21:25:26 +0000417
418
Antoine Pitrouc561a9a2011-12-03 23:06:50 +0100419Why doesn't CPython use a more traditional garbage collection scheme?
420---------------------------------------------------------------------
421
422For one thing, this is not a C standard feature and hence it's not portable.
423(Yes, we know about the Boehm GC library. It has bits of assembler code for
424*most* common platforms, not for all of them, and although it is mostly
425transparent, it isn't completely transparent; patches are required to get
426Python to work with it.)
427
428Traditional GC also becomes a problem when Python is embedded into other
429applications. While in a standalone Python it's fine to replace the standard
430malloc() and free() with versions provided by the GC library, an application
431embedding Python may want to have its *own* substitute for malloc() and free(),
432and may not want Python's. Right now, CPython works with anything that
433implements malloc() and free() properly.
434
435
436Why isn't all memory freed when CPython exits?
437----------------------------------------------
Georg Brandld7413152009-10-11 21:25:26 +0000438
439Objects referenced from the global namespaces of Python modules are not always
440deallocated when Python exits. This may happen if there are circular
441references. There are also certain bits of memory that are allocated by the C
442library that are impossible to free (e.g. a tool like Purify will complain about
443these). Python is, however, aggressive about cleaning up memory on exit and
444does try to destroy every single object.
445
446If you want to force Python to delete certain things on deallocation use the
447:mod:`atexit` module to run a function that will force those deletions.
448
449
450Why are there separate tuple and list data types?
451-------------------------------------------------
452
453Lists and tuples, while similar in many respects, are generally used in
454fundamentally different ways. Tuples can be thought of as being similar to
455Pascal records or C structs; they're small collections of related data which may
456be of different types which are operated on as a group. For example, a
457Cartesian coordinate is appropriately represented as a tuple of two or three
458numbers.
459
460Lists, on the other hand, are more like arrays in other languages. They tend to
461hold a varying number of objects all of which have the same type and which are
462operated on one-by-one. For example, ``os.listdir('.')`` returns a list of
463strings representing the files in the current directory. Functions which
464operate on this output would generally not break if you added another file or
465two to the directory.
466
467Tuples are immutable, meaning that once a tuple has been created, you can't
468replace any of its elements with a new value. Lists are mutable, meaning that
469you can always change a list's elements. Only immutable elements can be used as
470dictionary keys, and hence only tuples and not lists can be used as keys.
471
472
Andrés Delfino8d412782018-07-07 20:25:47 -0300473How are lists implemented in CPython?
474-------------------------------------
Georg Brandld7413152009-10-11 21:25:26 +0000475
Andrés Delfino8d412782018-07-07 20:25:47 -0300476CPython's lists are really variable-length arrays, not Lisp-style linked lists.
Georg Brandld7413152009-10-11 21:25:26 +0000477The implementation uses a contiguous array of references to other objects, and
478keeps a pointer to this array and the array's length in a list head structure.
479
480This makes indexing a list ``a[i]`` an operation whose cost is independent of
481the size of the list or the value of the index.
482
483When items are appended or inserted, the array of references is resized. Some
484cleverness is applied to improve the performance of appending items repeatedly;
485when the array must be grown, some extra space is allocated so the next few
486times don't require an actual resize.
487
488
Andrés Delfino8d412782018-07-07 20:25:47 -0300489How are dictionaries implemented in CPython?
490--------------------------------------------
Georg Brandld7413152009-10-11 21:25:26 +0000491
Andrés Delfino8d412782018-07-07 20:25:47 -0300492CPython's dictionaries are implemented as resizable hash tables. Compared to
Georg Brandld7413152009-10-11 21:25:26 +0000493B-trees, this gives better performance for lookup (the most common operation by
494far) under most circumstances, and the implementation is simpler.
495
496Dictionaries work by computing a hash code for each key stored in the dictionary
497using the :func:`hash` built-in function. The hash code varies widely depending
Georg Brandlb20a0192012-03-14 07:50:17 +0100498on the key and a per-process seed; for example, "Python" could hash to
499-539294296 while "python", a string that differs by a single bit, could hash
500to 1142331976. The hash code is then used to calculate a location in an
501internal array where the value will be stored. Assuming that you're storing
502keys that all have different hash values, this means that dictionaries take
Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి)866c1682018-06-26 13:57:05 +0530503constant time -- O(1), in Big-O notation -- to retrieve a key.
Georg Brandld7413152009-10-11 21:25:26 +0000504
505
506Why must dictionary keys be immutable?
507--------------------------------------
508
509The hash table implementation of dictionaries uses a hash value calculated from
510the key value to find the key. If the key were a mutable object, its value
511could change, and thus its hash could also change. But since whoever changes
512the key object can't tell that it was being used as a dictionary key, it can't
513move the entry around in the dictionary. Then, when you try to look up the same
514object in the dictionary it won't be found because its hash value is different.
515If you tried to look up the old value it wouldn't be found either, because the
516value of the object found in that hash bin would be different.
517
518If you want a dictionary indexed with a list, simply convert the list to a tuple
519first; the function ``tuple(L)`` creates a tuple with the same entries as the
520list ``L``. Tuples are immutable and can therefore be used as dictionary keys.
521
522Some unacceptable solutions that have been proposed:
523
524- Hash lists by their address (object ID). This doesn't work because if you
525 construct a new list with the same value it won't be found; e.g.::
526
Georg Brandl99de4882009-12-20 14:24:06 +0000527 mydict = {[1, 2]: '12'}
528 print(mydict[[1, 2]])
Georg Brandld7413152009-10-11 21:25:26 +0000529
Georg Brandl99de4882009-12-20 14:24:06 +0000530 would raise a KeyError exception because the id of the ``[1, 2]`` used in the
Georg Brandld7413152009-10-11 21:25:26 +0000531 second line differs from that in the first line. In other words, dictionary
532 keys should be compared using ``==``, not using :keyword:`is`.
533
534- Make a copy when using a list as a key. This doesn't work because the list,
535 being a mutable object, could contain a reference to itself, and then the
536 copying code would run into an infinite loop.
537
538- Allow lists as keys but tell the user not to modify them. This would allow a
539 class of hard-to-track bugs in programs when you forgot or modified a list by
540 accident. It also invalidates an important invariant of dictionaries: every
541 value in ``d.keys()`` is usable as a key of the dictionary.
542
543- Mark lists as read-only once they are used as a dictionary key. The problem
544 is that it's not just the top-level object that could change its value; you
545 could use a tuple containing a list as a key. Entering anything as a key into
546 a dictionary would require marking all objects reachable from there as
547 read-only -- and again, self-referential objects could cause an infinite loop.
548
549There is a trick to get around this if you need to, but use it at your own risk:
550You can wrap a mutable structure inside a class instance which has both a
Georg Brandl99de4882009-12-20 14:24:06 +0000551:meth:`__eq__` and a :meth:`__hash__` method. You must then make sure that the
Georg Brandld7413152009-10-11 21:25:26 +0000552hash value for all such wrapper objects that reside in a dictionary (or other
553hash based structure), remain fixed while the object is in the dictionary (or
554other structure). ::
555
556 class ListWrapper:
557 def __init__(self, the_list):
558 self.the_list = the_list
Serhiy Storchakadba90392016-05-10 12:01:23 +0300559
Georg Brandl99de4882009-12-20 14:24:06 +0000560 def __eq__(self, other):
Georg Brandld7413152009-10-11 21:25:26 +0000561 return self.the_list == other.the_list
Serhiy Storchakadba90392016-05-10 12:01:23 +0300562
Georg Brandld7413152009-10-11 21:25:26 +0000563 def __hash__(self):
564 l = self.the_list
565 result = 98767 - len(l)*555
Georg Brandl99de4882009-12-20 14:24:06 +0000566 for i, el in enumerate(l):
Georg Brandld7413152009-10-11 21:25:26 +0000567 try:
Georg Brandl99de4882009-12-20 14:24:06 +0000568 result = result + (hash(el) % 9999999) * 1001 + i
569 except Exception:
Georg Brandld7413152009-10-11 21:25:26 +0000570 result = (result % 7777777) + i * 333
571 return result
572
573Note that the hash computation is complicated by the possibility that some
574members of the list may be unhashable and also by the possibility of arithmetic
575overflow.
576
Georg Brandl99de4882009-12-20 14:24:06 +0000577Furthermore it must always be the case that if ``o1 == o2`` (ie ``o1.__eq__(o2)
578is True``) then ``hash(o1) == hash(o2)`` (ie, ``o1.__hash__() == o2.__hash__()``),
Georg Brandld7413152009-10-11 21:25:26 +0000579regardless of whether the object is in a dictionary or not. If you fail to meet
580these restrictions dictionaries and other hash based structures will misbehave.
581
582In the case of ListWrapper, whenever the wrapper object is in a dictionary the
583wrapped list must not change to avoid anomalies. Don't do this unless you are
584prepared to think hard about the requirements and the consequences of not
585meeting them correctly. Consider yourself warned.
586
587
588Why doesn't list.sort() return the sorted list?
589-----------------------------------------------
590
591In situations where performance matters, making a copy of the list just to sort
592it would be wasteful. Therefore, :meth:`list.sort` sorts the list in place. In
593order to remind you of that fact, it does not return the sorted list. This way,
594you won't be fooled into accidentally overwriting a list when you need a sorted
595copy but also need to keep the unsorted version around.
596
Antoine Pitroudec0f212011-12-03 23:08:57 +0100597If you want to return a new list, use the built-in :func:`sorted` function
598instead. This function creates a new list from a provided iterable, sorts
599it and returns it. For example, here's how to iterate over the keys of a
600dictionary in sorted order::
Georg Brandld7413152009-10-11 21:25:26 +0000601
Georg Brandl99de4882009-12-20 14:24:06 +0000602 for key in sorted(mydict):
Serhiy Storchakadba90392016-05-10 12:01:23 +0300603 ... # do whatever with mydict[key]...
Georg Brandld7413152009-10-11 21:25:26 +0000604
605
606How do you specify and enforce an interface spec in Python?
607-----------------------------------------------------------
608
609An interface specification for a module as provided by languages such as C++ and
610Java describes the prototypes for the methods and functions of the module. Many
611feel that compile-time enforcement of interface specifications helps in the
612construction of large programs.
613
614Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes
615(ABCs). You can then use :func:`isinstance` and :func:`issubclass` to check
616whether an instance or a class implements a particular ABC. The
Éric Araujob8edbdf2011-09-01 05:57:12 +0200617:mod:`collections.abc` module defines a set of useful ABCs such as
Serhiy Storchakabfdcd432013-10-13 23:09:14 +0300618:class:`~collections.abc.Iterable`, :class:`~collections.abc.Container`, and
619:class:`~collections.abc.MutableMapping`.
Georg Brandld7413152009-10-11 21:25:26 +0000620
621For Python, many of the advantages of interface specifications can be obtained
622by an appropriate test discipline for components. There is also a tool,
623PyChecker, which can be used to find problems due to subclassing.
624
625A good test suite for a module can both provide a regression test and serve as a
626module interface specification and a set of examples. Many Python modules can
627be run as a script to provide a simple "self test." Even modules which use
628complex external interfaces can often be tested in isolation using trivial
629"stub" emulations of the external interface. The :mod:`doctest` and
630:mod:`unittest` modules or third-party test frameworks can be used to construct
631exhaustive test suites that exercise every line of code in a module.
632
633An appropriate testing discipline can help build large complex applications in
634Python as well as having interface specifications would. In fact, it can be
635better because an interface specification cannot test certain properties of a
636program. For example, the :meth:`append` method is expected to add new elements
637to the end of some internal list; an interface specification cannot test that
638your :meth:`append` implementation will actually do this correctly, but it's
639trivial to check this property in a test suite.
640
641Writing test suites is very helpful, and you might want to design your code with
642an eye to making it easily tested. One increasingly popular technique,
643test-directed development, calls for writing parts of the test suite first,
644before you write any of the actual code. Of course Python allows you to be
645sloppy and not write test cases at all.
646
647
Georg Brandld7413152009-10-11 21:25:26 +0000648Why is there no goto?
649---------------------
650
651You can use exceptions to provide a "structured goto" that even works across
652function calls. Many feel that exceptions can conveniently emulate all
653reasonable uses of the "go" or "goto" constructs of C, Fortran, and other
654languages. For example::
655
Ezio Melotti19cdee82013-01-05 06:53:27 +0200656 class label(Exception): pass # declare a label
Georg Brandld7413152009-10-11 21:25:26 +0000657
658 try:
Serhiy Storchakadba90392016-05-10 12:01:23 +0300659 ...
660 if condition: raise label() # goto label
661 ...
Georg Brandl99de4882009-12-20 14:24:06 +0000662 except label: # where to goto
Serhiy Storchakadba90392016-05-10 12:01:23 +0300663 pass
Georg Brandld7413152009-10-11 21:25:26 +0000664 ...
665
666This doesn't allow you to jump into the middle of a loop, but that's usually
667considered an abuse of goto anyway. Use sparingly.
668
669
670Why can't raw strings (r-strings) end with a backslash?
671-------------------------------------------------------
672
673More precisely, they can't end with an odd number of backslashes: the unpaired
674backslash at the end escapes the closing quote character, leaving an
675unterminated string.
676
677Raw strings were designed to ease creating input for processors (chiefly regular
678expression engines) that want to do their own backslash escape processing. Such
679processors consider an unmatched trailing backslash to be an error anyway, so
680raw strings disallow that. In return, they allow you to pass on the string
681quote character by escaping it with a backslash. These rules work well when
682r-strings are used for their intended purpose.
683
684If you're trying to build Windows pathnames, note that all Windows system calls
685accept forward slashes too::
686
Georg Brandl99de4882009-12-20 14:24:06 +0000687 f = open("/mydir/file.txt") # works fine!
Georg Brandld7413152009-10-11 21:25:26 +0000688
689If you're trying to build a pathname for a DOS command, try e.g. one of ::
690
691 dir = r"\this\is\my\dos\dir" "\\"
692 dir = r"\this\is\my\dos\dir\ "[:-1]
693 dir = "\\this\\is\\my\\dos\\dir\\"
694
695
696Why doesn't Python have a "with" statement for attribute assignments?
697---------------------------------------------------------------------
698
699Python has a 'with' statement that wraps the execution of a block, calling code
700on the entrance and exit from the block. Some language have a construct that
701looks like this::
702
703 with obj:
Benjamin Peterson1baf4652009-12-31 03:11:23 +0000704 a = 1 # equivalent to obj.a = 1
Georg Brandld7413152009-10-11 21:25:26 +0000705 total = total + 1 # obj.total = obj.total + 1
706
707In Python, such a construct would be ambiguous.
708
709Other languages, such as Object Pascal, Delphi, and C++, use static types, so
710it's possible to know, in an unambiguous way, what member is being assigned
711to. This is the main point of static typing -- the compiler *always* knows the
712scope of every variable at compile time.
713
714Python uses dynamic types. It is impossible to know in advance which attribute
715will be referenced at runtime. Member attributes may be added or removed from
716objects on the fly. This makes it impossible to know, from a simple reading,
717what attribute is being referenced: a local one, a global one, or a member
718attribute?
719
720For instance, take the following incomplete snippet::
721
722 def foo(a):
723 with a:
Georg Brandl99de4882009-12-20 14:24:06 +0000724 print(x)
Georg Brandld7413152009-10-11 21:25:26 +0000725
726The snippet assumes that "a" must have a member attribute called "x". However,
727there is nothing in Python that tells the interpreter this. What should happen
728if "a" is, let us say, an integer? If there is a global variable named "x",
729will it be used inside the with block? As you see, the dynamic nature of Python
730makes such choices much harder.
731
732The primary benefit of "with" and similar language features (reduction of code
733volume) can, however, easily be achieved in Python by assignment. Instead of::
734
Georg Brandl99de4882009-12-20 14:24:06 +0000735 function(args).mydict[index][index].a = 21
736 function(args).mydict[index][index].b = 42
737 function(args).mydict[index][index].c = 63
Georg Brandld7413152009-10-11 21:25:26 +0000738
739write this::
740
Georg Brandl99de4882009-12-20 14:24:06 +0000741 ref = function(args).mydict[index][index]
Georg Brandld7413152009-10-11 21:25:26 +0000742 ref.a = 21
743 ref.b = 42
744 ref.c = 63
745
746This also has the side-effect of increasing execution speed because name
747bindings are resolved at run-time in Python, and the second version only needs
Georg Brandl99de4882009-12-20 14:24:06 +0000748to perform the resolution once.
Georg Brandld7413152009-10-11 21:25:26 +0000749
750
751Why are colons required for the if/while/def/class statements?
752--------------------------------------------------------------
753
754The colon is required primarily to enhance readability (one of the results of
755the experimental ABC language). Consider this::
756
757 if a == b
Georg Brandl99de4882009-12-20 14:24:06 +0000758 print(a)
Georg Brandld7413152009-10-11 21:25:26 +0000759
760versus ::
761
762 if a == b:
Georg Brandl99de4882009-12-20 14:24:06 +0000763 print(a)
Georg Brandld7413152009-10-11 21:25:26 +0000764
765Notice how the second one is slightly easier to read. Notice further how a
766colon sets off the example in this FAQ answer; it's a standard usage in English.
767
768Another minor reason is that the colon makes it easier for editors with syntax
769highlighting; they can look for colons to decide when indentation needs to be
770increased instead of having to do a more elaborate parsing of the program text.
771
772
773Why does Python allow commas at the end of lists and tuples?
774------------------------------------------------------------
775
776Python lets you add a trailing comma at the end of lists, tuples, and
777dictionaries::
778
779 [1, 2, 3,]
780 ('a', 'b', 'c',)
781 d = {
782 "A": [1, 5],
783 "B": [6, 7], # last trailing comma is optional but good style
784 }
785
786
787There are several reasons to allow this.
788
789When you have a literal value for a list, tuple, or dictionary spread across
790multiple lines, it's easier to add more elements because you don't have to
Georg Brandl7b8c1322013-04-14 10:31:06 +0200791remember to add a comma to the previous line. The lines can also be reordered
792without creating a syntax error.
Georg Brandld7413152009-10-11 21:25:26 +0000793
794Accidentally omitting the comma can lead to errors that are hard to diagnose.
795For example::
796
797 x = [
798 "fee",
799 "fie"
800 "foo",
801 "fum"
802 ]
803
804This list looks like it has four elements, but it actually contains three:
805"fee", "fiefoo" and "fum". Always adding the comma avoids this source of error.
806
807Allowing the trailing comma may also make programmatic code generation easier.