Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 1 | :tocdepth: 2 |
| 2 | |
| 3 | ========================= |
| 4 | Library and Extension FAQ |
| 5 | ========================= |
| 6 | |
Georg Brandl | 44ea77b | 2013-03-28 13:28:44 +0100 | [diff] [blame] | 7 | .. only:: html |
| 8 | |
| 9 | .. contents:: |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 10 | |
| 11 | General Library Questions |
| 12 | ========================= |
| 13 | |
| 14 | How do I find a module or application to perform task X? |
| 15 | -------------------------------------------------------- |
| 16 | |
| 17 | Check :ref:`the Library Reference <library-index>` to see if there's a relevant |
| 18 | standard library module. (Eventually you'll learn what's in the standard |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 19 | library and will be able to skip this step.) |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 20 | |
Georg Brandl | 495f7b5 | 2009-10-27 15:28:25 +0000 | [diff] [blame] | 21 | For third-party packages, search the `Python Package Index |
| 22 | <http://pypi.python.org/pypi>`_ or try `Google <http://www.google.com>`_ or |
| 23 | another Web search engine. Searching for "Python" plus a keyword or two for |
| 24 | your topic of interest will usually find something helpful. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 25 | |
| 26 | |
| 27 | Where is the math.py (socket.py, regex.py, etc.) source file? |
| 28 | ------------------------------------------------------------- |
| 29 | |
Georg Brandl | c4a55fc | 2010-02-06 18:46:57 +0000 | [diff] [blame] | 30 | If you can't find a source file for a module it may be a built-in or |
| 31 | dynamically loaded module implemented in C, C++ or other compiled language. |
| 32 | In this case you may not have the source file or it may be something like |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 33 | :file:`mathmodule.c`, somewhere in a C source directory (not on the Python Path). |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 34 | |
| 35 | There are (at least) three kinds of modules in Python: |
| 36 | |
| 37 | 1) modules written in Python (.py); |
| 38 | 2) modules written in C and dynamically loaded (.dll, .pyd, .so, .sl, etc); |
| 39 | 3) modules written in C and linked with the interpreter; to get a list of these, |
| 40 | type:: |
| 41 | |
| 42 | import sys |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 43 | print(sys.builtin_module_names) |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 44 | |
| 45 | |
| 46 | How do I make a Python script executable on Unix? |
| 47 | ------------------------------------------------- |
| 48 | |
| 49 | You need to do two things: the script file's mode must be executable and the |
| 50 | first line must begin with ``#!`` followed by the path of the Python |
| 51 | interpreter. |
| 52 | |
| 53 | The first is done by executing ``chmod +x scriptfile`` or perhaps ``chmod 755 |
| 54 | scriptfile``. |
| 55 | |
| 56 | The second can be done in a number of ways. The most straightforward way is to |
| 57 | write :: |
| 58 | |
| 59 | #!/usr/local/bin/python |
| 60 | |
| 61 | as the very first line of your file, using the pathname for where the Python |
| 62 | interpreter is installed on your platform. |
| 63 | |
| 64 | If you would like the script to be independent of where the Python interpreter |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 65 | lives, you can use the :program:`env` program. Almost all Unix variants support |
| 66 | the following, assuming the Python interpreter is in a directory on the user's |
| 67 | :envvar:`PATH`:: |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 68 | |
| 69 | #!/usr/bin/env python |
| 70 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 71 | *Don't* do this for CGI scripts. The :envvar:`PATH` variable for CGI scripts is |
| 72 | often very minimal, so you need to use the actual absolute pathname of the |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 73 | interpreter. |
| 74 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 75 | Occasionally, a user's environment is so full that the :program:`/usr/bin/env` |
| 76 | program fails; or there's no env program at all. In that case, you can try the |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 77 | following hack (due to Alex Rezinsky):: |
| 78 | |
| 79 | #! /bin/sh |
| 80 | """:" |
| 81 | exec python $0 ${1+"$@"} |
| 82 | """ |
| 83 | |
| 84 | The minor disadvantage is that this defines the script's __doc__ string. |
| 85 | However, you can fix that by adding :: |
| 86 | |
| 87 | __doc__ = """...Whatever...""" |
| 88 | |
| 89 | |
| 90 | |
| 91 | Is there a curses/termcap package for Python? |
| 92 | --------------------------------------------- |
| 93 | |
| 94 | .. XXX curses *is* built by default, isn't it? |
| 95 | |
| 96 | For Unix variants: The standard Python source distribution comes with a curses |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 97 | module in the :source:`Modules` subdirectory, though it's not compiled by default. |
| 98 | (Note that this is not available in the Windows distribution -- there is no |
| 99 | curses module for Windows.) |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 100 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 101 | The :mod:`curses` module supports basic curses features as well as many additional |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 102 | functions from ncurses and SYSV curses such as colour, alternative character set |
| 103 | support, pads, and mouse support. This means the module isn't compatible with |
| 104 | operating systems that only have BSD curses, but there don't seem to be any |
| 105 | currently maintained OSes that fall into this category. |
| 106 | |
| 107 | For Windows: use `the consolelib module |
| 108 | <http://effbot.org/zone/console-index.htm>`_. |
| 109 | |
| 110 | |
| 111 | Is there an equivalent to C's onexit() in Python? |
| 112 | ------------------------------------------------- |
| 113 | |
| 114 | The :mod:`atexit` module provides a register function that is similar to C's |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 115 | :c:func:`onexit`. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 116 | |
| 117 | |
| 118 | Why don't my signal handlers work? |
| 119 | ---------------------------------- |
| 120 | |
| 121 | The most common problem is that the signal handler is declared with the wrong |
| 122 | argument list. It is called as :: |
| 123 | |
| 124 | handler(signum, frame) |
| 125 | |
| 126 | so it should be declared with two arguments:: |
| 127 | |
| 128 | def handler(signum, frame): |
| 129 | ... |
| 130 | |
| 131 | |
| 132 | Common tasks |
| 133 | ============ |
| 134 | |
| 135 | How do I test a Python program or component? |
| 136 | -------------------------------------------- |
| 137 | |
| 138 | Python comes with two testing frameworks. The :mod:`doctest` module finds |
| 139 | examples in the docstrings for a module and runs them, comparing the output with |
| 140 | the expected output given in the docstring. |
| 141 | |
| 142 | The :mod:`unittest` module is a fancier testing framework modelled on Java and |
| 143 | Smalltalk testing frameworks. |
| 144 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 145 | To make testing easier, you should use good modular design in your program. |
| 146 | Your program should have almost all functionality |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 147 | encapsulated in either functions or class methods -- and this sometimes has the |
| 148 | surprising and delightful effect of making the program run faster (because local |
| 149 | variable accesses are faster than global accesses). Furthermore the program |
| 150 | should avoid depending on mutating global variables, since this makes testing |
| 151 | much more difficult to do. |
| 152 | |
| 153 | The "global main logic" of your program may be as simple as :: |
| 154 | |
| 155 | if __name__ == "__main__": |
| 156 | main_logic() |
| 157 | |
| 158 | at the bottom of the main module of your program. |
| 159 | |
| 160 | Once your program is organized as a tractable collection of functions and class |
| 161 | behaviours you should write test functions that exercise the behaviours. A test |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 162 | suite that automates a sequence of tests can be associated with each module. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 163 | This sounds like a lot of work, but since Python is so terse and flexible it's |
| 164 | surprisingly easy. You can make coding much more pleasant and fun by writing |
| 165 | your test functions in parallel with the "production code", since this makes it |
| 166 | easy to find bugs and even design flaws earlier. |
| 167 | |
| 168 | "Support modules" that are not intended to be the main module of a program may |
| 169 | include a self-test of the module. :: |
| 170 | |
| 171 | if __name__ == "__main__": |
| 172 | self_test() |
| 173 | |
| 174 | Even programs that interact with complex external interfaces may be tested when |
| 175 | the external interfaces are unavailable by using "fake" interfaces implemented |
| 176 | in Python. |
| 177 | |
| 178 | |
| 179 | How do I create documentation from doc strings? |
| 180 | ----------------------------------------------- |
| 181 | |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 182 | The :mod:`pydoc` module can create HTML from the doc strings in your Python |
Georg Brandl | 495f7b5 | 2009-10-27 15:28:25 +0000 | [diff] [blame] | 183 | source code. An alternative for creating API documentation purely from |
| 184 | docstrings is `epydoc <http://epydoc.sf.net/>`_. `Sphinx |
| 185 | <http://sphinx.pocoo.org>`_ can also include docstring content. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 186 | |
| 187 | |
| 188 | How do I get a single keypress at a time? |
| 189 | ----------------------------------------- |
| 190 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 191 | For Unix variants there are several solutions. It's straightforward to do this |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 192 | using curses, but curses is a fairly large module to learn. |
| 193 | |
| 194 | .. XXX this doesn't work out of the box, some IO expert needs to check why |
| 195 | |
| 196 | Here's a solution without curses:: |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 197 | |
| 198 | import termios, fcntl, sys, os |
| 199 | fd = sys.stdin.fileno() |
| 200 | |
| 201 | oldterm = termios.tcgetattr(fd) |
| 202 | newattr = termios.tcgetattr(fd) |
| 203 | newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO |
| 204 | termios.tcsetattr(fd, termios.TCSANOW, newattr) |
| 205 | |
| 206 | oldflags = fcntl.fcntl(fd, fcntl.F_GETFL) |
| 207 | fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK) |
| 208 | |
| 209 | try: |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 210 | while True: |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 211 | try: |
| 212 | c = sys.stdin.read(1) |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 213 | print("Got character", repr(c)) |
Andrew Svetlov | 5f11a00 | 2012-12-18 23:16:44 +0200 | [diff] [blame] | 214 | except OSError: |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 215 | pass |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 216 | finally: |
| 217 | termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm) |
| 218 | fcntl.fcntl(fd, fcntl.F_SETFL, oldflags) |
| 219 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 220 | You need the :mod:`termios` and the :mod:`fcntl` module for any of this to |
| 221 | work, and I've only tried it on Linux, though it should work elsewhere. In |
| 222 | this code, characters are read and printed one at a time. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 223 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 224 | :func:`termios.tcsetattr` turns off stdin's echoing and disables canonical |
| 225 | mode. :func:`fcntl.fnctl` is used to obtain stdin's file descriptor flags |
| 226 | and modify them for non-blocking mode. Since reading stdin when it is empty |
Andrew Svetlov | 5f11a00 | 2012-12-18 23:16:44 +0200 | [diff] [blame] | 227 | results in an :exc:`OSError`, this error is caught and ignored. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 228 | |
Andrew Svetlov | 8a045cb | 2012-12-19 13:45:30 +0200 | [diff] [blame] | 229 | .. versionchanged:: 3.3 |
| 230 | *sys.stdin.read* used to raise :exc:`IOError`. Starting from Python 3.3 |
| 231 | :exc:`IOError` is alias for :exc:`OSError`. |
| 232 | |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 233 | |
| 234 | Threads |
| 235 | ======= |
| 236 | |
| 237 | How do I program using threads? |
| 238 | ------------------------------- |
| 239 | |
Georg Brandl | d404fa6 | 2009-10-13 16:55:12 +0000 | [diff] [blame] | 240 | Be sure to use the :mod:`threading` module and not the :mod:`_thread` module. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 241 | The :mod:`threading` module builds convenient abstractions on top of the |
Georg Brandl | d404fa6 | 2009-10-13 16:55:12 +0000 | [diff] [blame] | 242 | low-level primitives provided by the :mod:`_thread` module. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 243 | |
| 244 | Aahz has a set of slides from his threading tutorial that are helpful; see |
Georg Brandl | 495f7b5 | 2009-10-27 15:28:25 +0000 | [diff] [blame] | 245 | http://www.pythoncraft.com/OSCON2001/. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 246 | |
| 247 | |
| 248 | None of my threads seem to run: why? |
| 249 | ------------------------------------ |
| 250 | |
| 251 | As soon as the main thread exits, all threads are killed. Your main thread is |
| 252 | running too quickly, giving the threads no time to do any work. |
| 253 | |
| 254 | A simple fix is to add a sleep to the end of the program that's long enough for |
| 255 | all the threads to finish:: |
| 256 | |
| 257 | import threading, time |
| 258 | |
| 259 | def thread_task(name, n): |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 260 | for i in range(n): print(name, i) |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 261 | |
| 262 | for i in range(10): |
| 263 | T = threading.Thread(target=thread_task, args=(str(i), i)) |
| 264 | T.start() |
| 265 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 266 | time.sleep(10) # <---------------------------! |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 267 | |
| 268 | But now (on many platforms) the threads don't run in parallel, but appear to run |
| 269 | sequentially, one at a time! The reason is that the OS thread scheduler doesn't |
| 270 | start a new thread until the previous thread is blocked. |
| 271 | |
| 272 | A simple fix is to add a tiny sleep to the start of the run function:: |
| 273 | |
| 274 | def thread_task(name, n): |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 275 | time.sleep(0.001) # <--------------------! |
| 276 | for i in range(n): print(name, i) |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 277 | |
| 278 | for i in range(10): |
| 279 | T = threading.Thread(target=thread_task, args=(str(i), i)) |
| 280 | T.start() |
| 281 | |
| 282 | time.sleep(10) |
| 283 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 284 | Instead of trying to guess a good delay value for :func:`time.sleep`, |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 285 | it's better to use some kind of semaphore mechanism. One idea is to use the |
Georg Brandl | d404fa6 | 2009-10-13 16:55:12 +0000 | [diff] [blame] | 286 | :mod:`queue` module to create a queue object, let each thread append a token to |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 287 | the queue when it finishes, and let the main thread read as many tokens from the |
| 288 | queue as there are threads. |
| 289 | |
| 290 | |
| 291 | How do I parcel out work among a bunch of worker threads? |
| 292 | --------------------------------------------------------- |
| 293 | |
Antoine Pitrou | 11480b6 | 2011-02-05 11:18:34 +0000 | [diff] [blame] | 294 | The easiest way is to use the new :mod:`concurrent.futures` module, |
| 295 | especially the :mod:`~concurrent.futures.ThreadPoolExecutor` class. |
| 296 | |
| 297 | Or, if you want fine control over the dispatching algorithm, you can write |
| 298 | your own logic manually. Use the :mod:`queue` module to create a queue |
| 299 | containing a list of jobs. The :class:`~queue.Queue` class maintains a |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 300 | list of objects and has a ``.put(obj)`` method that adds items to the queue and |
| 301 | a ``.get()`` method to return them. The class will take care of the locking |
| 302 | necessary to ensure that each job is handed out exactly once. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 303 | |
| 304 | Here's a trivial example:: |
| 305 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 306 | import threading, queue, time |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 307 | |
| 308 | # The worker thread gets jobs off the queue. When the queue is empty, it |
| 309 | # assumes there will be no more work and exits. |
| 310 | # (Realistically workers will run until terminated.) |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 311 | def worker(): |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 312 | print('Running worker') |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 313 | time.sleep(0.1) |
| 314 | while True: |
| 315 | try: |
| 316 | arg = q.get(block=False) |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 317 | except queue.Empty: |
| 318 | print('Worker', threading.currentThread(), end=' ') |
| 319 | print('queue empty') |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 320 | break |
| 321 | else: |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 322 | print('Worker', threading.currentThread(), end=' ') |
| 323 | print('running with argument', arg) |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 324 | time.sleep(0.5) |
| 325 | |
| 326 | # Create queue |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 327 | q = queue.Queue() |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 328 | |
| 329 | # Start a pool of 5 workers |
| 330 | for i in range(5): |
| 331 | t = threading.Thread(target=worker, name='worker %i' % (i+1)) |
| 332 | t.start() |
| 333 | |
| 334 | # Begin adding work to the queue |
| 335 | for i in range(50): |
| 336 | q.put(i) |
| 337 | |
| 338 | # Give threads time to run |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 339 | print('Main thread sleeping') |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 340 | time.sleep(5) |
| 341 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 342 | When run, this will produce the following output: |
| 343 | |
| 344 | .. code-block:: none |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 345 | |
| 346 | Running worker |
| 347 | Running worker |
| 348 | Running worker |
| 349 | Running worker |
| 350 | Running worker |
| 351 | Main thread sleeping |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 352 | Worker <Thread(worker 1, started 130283832797456)> running with argument 0 |
| 353 | Worker <Thread(worker 2, started 130283824404752)> running with argument 1 |
| 354 | Worker <Thread(worker 3, started 130283816012048)> running with argument 2 |
| 355 | Worker <Thread(worker 4, started 130283807619344)> running with argument 3 |
| 356 | Worker <Thread(worker 5, started 130283799226640)> running with argument 4 |
| 357 | Worker <Thread(worker 1, started 130283832797456)> running with argument 5 |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 358 | ... |
| 359 | |
Georg Brandl | 3539afd | 2012-05-30 22:03:20 +0200 | [diff] [blame] | 360 | Consult the module's documentation for more details; the :class:`~queue.Queue` |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 361 | class provides a featureful interface. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 362 | |
| 363 | |
| 364 | What kinds of global value mutation are thread-safe? |
| 365 | ---------------------------------------------------- |
| 366 | |
Antoine Pitrou | 11480b6 | 2011-02-05 11:18:34 +0000 | [diff] [blame] | 367 | A :term:`global interpreter lock` (GIL) is used internally to ensure that only one |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 368 | thread runs in the Python VM at a time. In general, Python offers to switch |
| 369 | among threads only between bytecode instructions; how frequently it switches can |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 370 | be set via :func:`sys.setswitchinterval`. Each bytecode instruction and |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 371 | therefore all the C implementation code reached from each instruction is |
| 372 | therefore atomic from the point of view of a Python program. |
| 373 | |
| 374 | In theory, this means an exact accounting requires an exact understanding of the |
| 375 | PVM bytecode implementation. In practice, it means that operations on shared |
Georg Brandl | c4a55fc | 2010-02-06 18:46:57 +0000 | [diff] [blame] | 376 | variables of built-in data types (ints, lists, dicts, etc) that "look atomic" |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 377 | really are. |
| 378 | |
| 379 | For example, the following operations are all atomic (L, L1, L2 are lists, D, |
| 380 | D1, D2 are dicts, x, y are objects, i, j are ints):: |
| 381 | |
| 382 | L.append(x) |
| 383 | L1.extend(L2) |
| 384 | x = L[i] |
| 385 | x = L.pop() |
| 386 | L1[i:j] = L2 |
| 387 | L.sort() |
| 388 | x = y |
| 389 | x.field = y |
| 390 | D[x] = y |
| 391 | D1.update(D2) |
| 392 | D.keys() |
| 393 | |
| 394 | These aren't:: |
| 395 | |
| 396 | i = i+1 |
| 397 | L.append(L[-1]) |
| 398 | L[i] = L[j] |
| 399 | D[x] = D[x] + 1 |
| 400 | |
| 401 | Operations that replace other objects may invoke those other objects' |
| 402 | :meth:`__del__` method when their reference count reaches zero, and that can |
| 403 | affect things. This is especially true for the mass updates to dictionaries and |
| 404 | lists. When in doubt, use a mutex! |
| 405 | |
| 406 | |
| 407 | Can't we get rid of the Global Interpreter Lock? |
| 408 | ------------------------------------------------ |
| 409 | |
Georg Brandl | 495f7b5 | 2009-10-27 15:28:25 +0000 | [diff] [blame] | 410 | .. XXX link to dbeazley's talk about GIL? |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 411 | |
Antoine Pitrou | 11480b6 | 2011-02-05 11:18:34 +0000 | [diff] [blame] | 412 | The :term:`global interpreter lock` (GIL) is often seen as a hindrance to Python's |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 413 | deployment on high-end multiprocessor server machines, because a multi-threaded |
| 414 | Python program effectively only uses one CPU, due to the insistence that |
| 415 | (almost) all Python code can only run while the GIL is held. |
| 416 | |
| 417 | Back in the days of Python 1.5, Greg Stein actually implemented a comprehensive |
| 418 | patch set (the "free threading" patches) that removed the GIL and replaced it |
Antoine Pitrou | 11480b6 | 2011-02-05 11:18:34 +0000 | [diff] [blame] | 419 | with fine-grained locking. Adam Olsen recently did a similar experiment |
| 420 | in his `python-safethread <http://code.google.com/p/python-safethread/>`_ |
| 421 | project. Unfortunately, both experiments exhibited a sharp drop in single-thread |
| 422 | performance (at least 30% slower), due to the amount of fine-grained locking |
| 423 | necessary to compensate for the removal of the GIL. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 424 | |
| 425 | This doesn't mean that you can't make good use of Python on multi-CPU machines! |
| 426 | You just have to be creative with dividing the work up between multiple |
Antoine Pitrou | 11480b6 | 2011-02-05 11:18:34 +0000 | [diff] [blame] | 427 | *processes* rather than multiple *threads*. The |
| 428 | :class:`~concurrent.futures.ProcessPoolExecutor` class in the new |
| 429 | :mod:`concurrent.futures` module provides an easy way of doing so; the |
| 430 | :mod:`multiprocessing` module provides a lower-level API in case you want |
| 431 | more control over dispatching of tasks. |
| 432 | |
| 433 | Judicious use of C extensions will also help; if you use a C extension to |
| 434 | perform a time-consuming task, the extension can release the GIL while the |
| 435 | thread of execution is in the C code and allow other threads to get some work |
| 436 | done. Some standard library modules such as :mod:`zlib` and :mod:`hashlib` |
| 437 | already do this. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 438 | |
| 439 | It has been suggested that the GIL should be a per-interpreter-state lock rather |
| 440 | than truly global; interpreters then wouldn't be able to share objects. |
| 441 | Unfortunately, this isn't likely to happen either. It would be a tremendous |
| 442 | amount of work, because many object implementations currently have global state. |
| 443 | For example, small integers and short strings are cached; these caches would |
| 444 | have to be moved to the interpreter state. Other object types have their own |
| 445 | free list; these free lists would have to be moved to the interpreter state. |
| 446 | And so on. |
| 447 | |
| 448 | And I doubt that it can even be done in finite time, because the same problem |
| 449 | exists for 3rd party extensions. It is likely that 3rd party extensions are |
| 450 | being written at a faster rate than you can convert them to store all their |
| 451 | global state in the interpreter state. |
| 452 | |
| 453 | And finally, once you have multiple interpreters not sharing any state, what |
| 454 | have you gained over running each interpreter in a separate process? |
| 455 | |
| 456 | |
| 457 | Input and Output |
| 458 | ================ |
| 459 | |
| 460 | How do I delete a file? (And other file questions...) |
| 461 | ----------------------------------------------------- |
| 462 | |
| 463 | Use ``os.remove(filename)`` or ``os.unlink(filename)``; for documentation, see |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 464 | the :mod:`os` module. The two functions are identical; :func:`~os.unlink` is simply |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 465 | the name of the Unix system call for this function. |
| 466 | |
| 467 | To remove a directory, use :func:`os.rmdir`; use :func:`os.mkdir` to create one. |
| 468 | ``os.makedirs(path)`` will create any intermediate directories in ``path`` that |
| 469 | don't exist. ``os.removedirs(path)`` will remove intermediate directories as |
| 470 | long as they're empty; if you want to delete an entire directory tree and its |
| 471 | contents, use :func:`shutil.rmtree`. |
| 472 | |
| 473 | To rename a file, use ``os.rename(old_path, new_path)``. |
| 474 | |
Antoine Pitrou | 6a11a98 | 2010-09-15 10:08:31 +0000 | [diff] [blame] | 475 | To truncate a file, open it using ``f = open(filename, "rb+")``, and use |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 476 | ``f.truncate(offset)``; offset defaults to the current seek position. There's |
Georg Brandl | 682d7e0 | 2010-10-06 10:26:05 +0000 | [diff] [blame] | 477 | also ``os.ftruncate(fd, offset)`` for files opened with :func:`os.open`, where |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 478 | *fd* is the file descriptor (a small integer). |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 479 | |
| 480 | The :mod:`shutil` module also contains a number of functions to work on files |
| 481 | including :func:`~shutil.copyfile`, :func:`~shutil.copytree`, and |
| 482 | :func:`~shutil.rmtree`. |
| 483 | |
| 484 | |
| 485 | How do I copy a file? |
| 486 | --------------------- |
| 487 | |
| 488 | The :mod:`shutil` module contains a :func:`~shutil.copyfile` function. Note |
| 489 | that on MacOS 9 it doesn't copy the resource fork and Finder info. |
| 490 | |
| 491 | |
| 492 | How do I read (or write) binary data? |
| 493 | ------------------------------------- |
| 494 | |
| 495 | To read or write complex binary data formats, it's best to use the :mod:`struct` |
| 496 | module. It allows you to take a string containing binary data (usually numbers) |
| 497 | and convert it to Python objects; and vice versa. |
| 498 | |
| 499 | For example, the following code reads two 2-byte integers and one 4-byte integer |
| 500 | in big-endian format from a file:: |
| 501 | |
| 502 | import struct |
| 503 | |
Antoine Pitrou | 6a11a98 | 2010-09-15 10:08:31 +0000 | [diff] [blame] | 504 | with open(filename, "rb") as f: |
| 505 | s = f.read(8) |
| 506 | x, y, z = struct.unpack(">hhl", s) |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 507 | |
| 508 | The '>' in the format string forces big-endian data; the letter 'h' reads one |
| 509 | "short integer" (2 bytes), and 'l' reads one "long integer" (4 bytes) from the |
| 510 | string. |
| 511 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 512 | For data that is more regular (e.g. a homogeneous list of ints or floats), |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 513 | you can also use the :mod:`array` module. |
| 514 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 515 | .. note:: |
Éric Araujo | fa5e6e4 | 2014-03-12 19:51:00 -0400 | [diff] [blame] | 516 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 517 | To read and write binary data, it is mandatory to open the file in |
| 518 | binary mode (here, passing ``"rb"`` to :func:`open`). If you use |
| 519 | ``"r"`` instead (the default), the file will be open in text mode |
| 520 | and ``f.read()`` will return :class:`str` objects rather than |
| 521 | :class:`bytes` objects. |
Antoine Pitrou | 6a11a98 | 2010-09-15 10:08:31 +0000 | [diff] [blame] | 522 | |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 523 | |
| 524 | I can't seem to use os.read() on a pipe created with os.popen(); why? |
| 525 | --------------------------------------------------------------------- |
| 526 | |
| 527 | :func:`os.read` is a low-level function which takes a file descriptor, a small |
| 528 | integer representing the opened file. :func:`os.popen` creates a high-level |
Georg Brandl | c4a55fc | 2010-02-06 18:46:57 +0000 | [diff] [blame] | 529 | file object, the same type returned by the built-in :func:`open` function. |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 530 | Thus, to read *n* bytes from a pipe *p* created with :func:`os.popen`, you need to |
Georg Brandl | c4a55fc | 2010-02-06 18:46:57 +0000 | [diff] [blame] | 531 | use ``p.read(n)``. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 532 | |
| 533 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 534 | .. XXX update to use subprocess. See the :ref:`subprocess-replacements` section. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 535 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 536 | How do I run a subprocess with pipes connected to both input and output? |
| 537 | ------------------------------------------------------------------------ |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 538 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 539 | Use the :mod:`popen2` module. For example:: |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 540 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 541 | import popen2 |
| 542 | fromchild, tochild = popen2.popen2("command") |
| 543 | tochild.write("input\n") |
| 544 | tochild.flush() |
| 545 | output = fromchild.readline() |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 546 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 547 | Warning: in general it is unwise to do this because you can easily cause a |
| 548 | deadlock where your process is blocked waiting for output from the child |
| 549 | while the child is blocked waiting for input from you. This can be caused |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 550 | by the parent expecting the child to output more text than it does or |
| 551 | by data being stuck in stdio buffers due to lack of flushing. |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 552 | The Python parent can of course explicitly flush the data it sends to the |
| 553 | child before it reads any output, but if the child is a naive C program it |
| 554 | may have been written to never explicitly flush its output, even if it is |
| 555 | interactive, since flushing is normally automatic. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 556 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 557 | Note that a deadlock is also possible if you use :func:`popen3` to read |
| 558 | stdout and stderr. If one of the two is too large for the internal buffer |
| 559 | (increasing the buffer size does not help) and you ``read()`` the other one |
| 560 | first, there is a deadlock, too. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 561 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 562 | Note on a bug in popen2: unless your program calls ``wait()`` or |
| 563 | ``waitpid()``, finished child processes are never removed, and eventually |
| 564 | calls to popen2 will fail because of a limit on the number of child |
| 565 | processes. Calling :func:`os.waitpid` with the :data:`os.WNOHANG` option can |
| 566 | prevent this; a good place to insert such a call would be before calling |
| 567 | ``popen2`` again. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 568 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 569 | In many cases, all you really need is to run some data through a command and |
| 570 | get the result back. Unless the amount of data is very large, the easiest |
| 571 | way to do this is to write it to a temporary file and run the command with |
| 572 | that temporary file as input. The standard module :mod:`tempfile` exports a |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 573 | :func:`~tempfile.mktemp` function to generate unique temporary file names. :: |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 574 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 575 | import tempfile |
| 576 | import os |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 577 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 578 | class Popen3: |
| 579 | """ |
| 580 | This is a deadlock-safe version of popen that returns |
| 581 | an object with errorlevel, out (a string) and err (a string). |
| 582 | (capturestderr may not work under windows.) |
| 583 | Example: print(Popen3('grep spam','\n\nhere spam\n\n').out) |
| 584 | """ |
| 585 | def __init__(self,command,input=None,capturestderr=None): |
| 586 | outfile=tempfile.mktemp() |
| 587 | command="( %s ) > %s" % (command,outfile) |
| 588 | if input: |
| 589 | infile=tempfile.mktemp() |
| 590 | open(infile,"w").write(input) |
| 591 | command=command+" <"+infile |
| 592 | if capturestderr: |
| 593 | errfile=tempfile.mktemp() |
| 594 | command=command+" 2>"+errfile |
| 595 | self.errorlevel=os.system(command) >> 8 |
| 596 | self.out=open(outfile,"r").read() |
| 597 | os.remove(outfile) |
| 598 | if input: |
| 599 | os.remove(infile) |
| 600 | if capturestderr: |
| 601 | self.err=open(errfile,"r").read() |
| 602 | os.remove(errfile) |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 603 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 604 | Note that many interactive programs (e.g. vi) don't work well with pipes |
| 605 | substituted for standard input and output. You will have to use pseudo ttys |
| 606 | ("ptys") instead of pipes. Or you can use a Python interface to Don Libes' |
| 607 | "expect" library. A Python extension that interfaces to expect is called |
| 608 | "expy" and available from http://expectpy.sourceforge.net. A pure Python |
| 609 | solution that works like expect is `pexpect |
| 610 | <http://pypi.python.org/pypi/pexpect/>`_. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 611 | |
| 612 | |
| 613 | How do I access the serial (RS232) port? |
| 614 | ---------------------------------------- |
| 615 | |
| 616 | For Win32, POSIX (Linux, BSD, etc.), Jython: |
| 617 | |
| 618 | http://pyserial.sourceforge.net |
| 619 | |
| 620 | For Unix, see a Usenet post by Mitch Chapman: |
| 621 | |
| 622 | http://groups.google.com/groups?selm=34A04430.CF9@ohioee.com |
| 623 | |
| 624 | |
| 625 | Why doesn't closing sys.stdout (stdin, stderr) really close it? |
| 626 | --------------------------------------------------------------- |
| 627 | |
Antoine Pitrou | 6a11a98 | 2010-09-15 10:08:31 +0000 | [diff] [blame] | 628 | Python :term:`file objects <file object>` are a high-level layer of |
| 629 | abstraction on low-level C file descriptors. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 630 | |
Antoine Pitrou | 6a11a98 | 2010-09-15 10:08:31 +0000 | [diff] [blame] | 631 | For most file objects you create in Python via the built-in :func:`open` |
| 632 | function, ``f.close()`` marks the Python file object as being closed from |
| 633 | Python's point of view, and also arranges to close the underlying C file |
| 634 | descriptor. This also happens automatically in ``f``'s destructor, when |
| 635 | ``f`` becomes garbage. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 636 | |
| 637 | But stdin, stdout and stderr are treated specially by Python, because of the |
| 638 | special status also given to them by C. Running ``sys.stdout.close()`` marks |
| 639 | the Python-level file object as being closed, but does *not* close the |
Antoine Pitrou | 6a11a98 | 2010-09-15 10:08:31 +0000 | [diff] [blame] | 640 | associated C file descriptor. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 641 | |
Antoine Pitrou | 6a11a98 | 2010-09-15 10:08:31 +0000 | [diff] [blame] | 642 | To close the underlying C file descriptor for one of these three, you should |
| 643 | first be sure that's what you really want to do (e.g., you may confuse |
| 644 | extension modules trying to do I/O). If it is, use :func:`os.close`:: |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 645 | |
Antoine Pitrou | 6a11a98 | 2010-09-15 10:08:31 +0000 | [diff] [blame] | 646 | os.close(stdin.fileno()) |
| 647 | os.close(stdout.fileno()) |
| 648 | os.close(stderr.fileno()) |
| 649 | |
| 650 | Or you can use the numeric constants 0, 1 and 2, respectively. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 651 | |
| 652 | |
| 653 | Network/Internet Programming |
| 654 | ============================ |
| 655 | |
| 656 | What WWW tools are there for Python? |
| 657 | ------------------------------------ |
| 658 | |
| 659 | See the chapters titled :ref:`internet` and :ref:`netdata` in the Library |
| 660 | Reference Manual. Python has many modules that will help you build server-side |
| 661 | and client-side web systems. |
| 662 | |
| 663 | .. XXX check if wiki page is still up to date |
| 664 | |
| 665 | A summary of available frameworks is maintained by Paul Boddie at |
Serhiy Storchaka | a4d170d | 2013-12-23 18:20:51 +0200 | [diff] [blame] | 666 | http://wiki.python.org/moin/WebProgramming\ . |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 667 | |
| 668 | Cameron Laird maintains a useful set of pages about Python web technologies at |
| 669 | http://phaseit.net/claird/comp.lang.python/web_python. |
| 670 | |
| 671 | |
| 672 | How can I mimic CGI form submission (METHOD=POST)? |
| 673 | -------------------------------------------------- |
| 674 | |
| 675 | I would like to retrieve web pages that are the result of POSTing a form. Is |
| 676 | there existing code that would let me do this easily? |
| 677 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 678 | Yes. Here's a simple example that uses urllib.request:: |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 679 | |
| 680 | #!/usr/local/bin/python |
| 681 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 682 | import urllib.request |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 683 | |
| 684 | ### build the query string |
| 685 | qs = "First=Josephine&MI=Q&Last=Public" |
| 686 | |
| 687 | ### connect and send the server a path |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 688 | req = urllib.request.urlopen('http://www.some-server.out-there' |
| 689 | '/cgi-bin/some-cgi-script', data=qs) |
| 690 | msg, hdrs = req.read(), req.info() |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 691 | |
Georg Brandl | 54ebb78 | 2010-08-14 15:48:49 +0000 | [diff] [blame] | 692 | Note that in general for percent-encoded POST operations, query strings must be |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 693 | quoted using :func:`urllib.parse.urlencode`. For example, to send |
| 694 | ``name=Guy Steele, Jr.``:: |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 695 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 696 | >>> import urllib.parse |
| 697 | >>> urllib.parse.urlencode({'name': 'Guy Steele, Jr.'}) |
| 698 | 'name=Guy+Steele%2C+Jr.' |
| 699 | |
| 700 | .. seealso:: :ref:`urllib-howto` for extensive examples. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 701 | |
| 702 | |
| 703 | What module should I use to help with generating HTML? |
| 704 | ------------------------------------------------------ |
| 705 | |
| 706 | .. XXX add modern template languages |
| 707 | |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 708 | You can find a collection of useful links on the `Web Programming wiki page |
| 709 | <http://wiki.python.org/moin/WebProgramming>`_. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 710 | |
| 711 | |
| 712 | How do I send mail from a Python script? |
| 713 | ---------------------------------------- |
| 714 | |
| 715 | Use the standard library module :mod:`smtplib`. |
| 716 | |
| 717 | Here's a very simple interactive mail sender that uses it. This method will |
| 718 | work on any host that supports an SMTP listener. :: |
| 719 | |
| 720 | import sys, smtplib |
| 721 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 722 | fromaddr = input("From: ") |
| 723 | toaddrs = input("To: ").split(',') |
| 724 | print("Enter message, end with ^D:") |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 725 | msg = '' |
| 726 | while True: |
| 727 | line = sys.stdin.readline() |
| 728 | if not line: |
| 729 | break |
| 730 | msg += line |
| 731 | |
| 732 | # The actual mail send |
| 733 | server = smtplib.SMTP('localhost') |
| 734 | server.sendmail(fromaddr, toaddrs, msg) |
| 735 | server.quit() |
| 736 | |
| 737 | A Unix-only alternative uses sendmail. The location of the sendmail program |
Ezio Melotti | b35480e | 2012-05-13 20:14:04 +0300 | [diff] [blame] | 738 | varies between systems; sometimes it is ``/usr/lib/sendmail``, sometimes |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 739 | ``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's |
| 740 | some sample code:: |
| 741 | |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 742 | SENDMAIL = "/usr/sbin/sendmail" # sendmail location |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 743 | import os |
| 744 | p = os.popen("%s -t -i" % SENDMAIL, "w") |
| 745 | p.write("To: receiver@example.com\n") |
| 746 | p.write("Subject: test\n") |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 747 | p.write("\n") # blank line separating headers from body |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 748 | p.write("Some text\n") |
| 749 | p.write("some more text\n") |
| 750 | sts = p.close() |
| 751 | if sts != 0: |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 752 | print("Sendmail exit status", sts) |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 753 | |
| 754 | |
| 755 | How do I avoid blocking in the connect() method of a socket? |
| 756 | ------------------------------------------------------------ |
| 757 | |
Antoine Pitrou | 7095721 | 2011-02-05 11:24:15 +0000 | [diff] [blame] | 758 | The :mod:`select` module is commonly used to help with asynchronous I/O on |
| 759 | sockets. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 760 | |
| 761 | To prevent the TCP connect from blocking, you can set the socket to non-blocking |
| 762 | mode. Then when you do the ``connect()``, you will either connect immediately |
| 763 | (unlikely) or get an exception that contains the error number as ``.errno``. |
| 764 | ``errno.EINPROGRESS`` indicates that the connection is in progress, but hasn't |
| 765 | finished yet. Different OSes will return different values, so you're going to |
| 766 | have to check what's returned on your system. |
| 767 | |
| 768 | You can use the ``connect_ex()`` method to avoid creating an exception. It will |
| 769 | just return the errno value. To poll, you can call ``connect_ex()`` again later |
Georg Brandl | 9e4ff75 | 2009-12-19 17:57:51 +0000 | [diff] [blame] | 770 | -- ``0`` or ``errno.EISCONN`` indicate that you're connected -- or you can pass this |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 771 | socket to select to check if it's writable. |
| 772 | |
Antoine Pitrou | 7095721 | 2011-02-05 11:24:15 +0000 | [diff] [blame] | 773 | .. note:: |
| 774 | The :mod:`asyncore` module presents a framework-like approach to the problem |
| 775 | of writing non-blocking networking code. |
| 776 | The third-party `Twisted <http://twistedmatrix.com/>`_ library is |
| 777 | a popular and feature-rich alternative. |
| 778 | |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 779 | |
| 780 | Databases |
| 781 | ========= |
| 782 | |
| 783 | Are there any interfaces to database packages in Python? |
| 784 | -------------------------------------------------------- |
| 785 | |
| 786 | Yes. |
| 787 | |
Georg Brandl | d404fa6 | 2009-10-13 16:55:12 +0000 | [diff] [blame] | 788 | Interfaces to disk-based hashes such as :mod:`DBM <dbm.ndbm>` and :mod:`GDBM |
| 789 | <dbm.gnu>` are also included with standard Python. There is also the |
| 790 | :mod:`sqlite3` module, which provides a lightweight disk-based relational |
| 791 | database. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 792 | |
| 793 | Support for most relational databases is available. See the |
| 794 | `DatabaseProgramming wiki page |
| 795 | <http://wiki.python.org/moin/DatabaseProgramming>`_ for details. |
| 796 | |
| 797 | |
| 798 | How do you implement persistent objects in Python? |
| 799 | -------------------------------------------------- |
| 800 | |
| 801 | The :mod:`pickle` library module solves this in a very general way (though you |
| 802 | still can't store things like open files, sockets or windows), and the |
| 803 | :mod:`shelve` library module uses pickle and (g)dbm to create persistent |
Georg Brandl | d404fa6 | 2009-10-13 16:55:12 +0000 | [diff] [blame] | 804 | mappings containing arbitrary Python objects. |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 805 | |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 806 | |
Georg Brandl | d741315 | 2009-10-11 21:25:26 +0000 | [diff] [blame] | 807 | Mathematics and Numerics |
| 808 | ======================== |
| 809 | |
| 810 | How do I generate random numbers in Python? |
| 811 | ------------------------------------------- |
| 812 | |
| 813 | The standard module :mod:`random` implements a random number generator. Usage |
| 814 | is simple:: |
| 815 | |
| 816 | import random |
| 817 | random.random() |
| 818 | |
| 819 | This returns a random floating point number in the range [0, 1). |
| 820 | |
| 821 | There are also many other specialized generators in this module, such as: |
| 822 | |
| 823 | * ``randrange(a, b)`` chooses an integer in the range [a, b). |
| 824 | * ``uniform(a, b)`` chooses a floating point number in the range [a, b). |
| 825 | * ``normalvariate(mean, sdev)`` samples the normal (Gaussian) distribution. |
| 826 | |
| 827 | Some higher-level functions operate on sequences directly, such as: |
| 828 | |
| 829 | * ``choice(S)`` chooses random element from a given sequence |
| 830 | * ``shuffle(L)`` shuffles a list in-place, i.e. permutes it randomly |
| 831 | |
| 832 | There's also a ``Random`` class you can instantiate to create independent |
| 833 | multiple random number generators. |