blob: 9ce90952c061be3a691aeff1cec6c12b0e356ab7 [file] [log] [blame]
Benjamin Petersonae5360b2008-09-08 23:05:23 +00001*******************************
2 HOWTO Use Python in the web
3*******************************
4
5:Author: Marek Kubica
6
7.. topic:: Abstract
8
Ezio Melotti079d5ed2010-04-05 04:04:35 +00009 This document shows how Python fits into the web. It presents some ways
10 to integrate Python with a web server, and general practices useful for
Benjamin Petersonae5360b2008-09-08 23:05:23 +000011 developing web sites.
12
13
Ezio Melotti079d5ed2010-04-05 04:04:35 +000014Programming for the Web has become a hot topic since the rise of "Web 2.0",
Benjamin Petersonae5360b2008-09-08 23:05:23 +000015which focuses on user-generated content on web sites. It has always been
16possible to use Python for creating web sites, but it was a rather tedious task.
Ezio Melotti079d5ed2010-04-05 04:04:35 +000017Therefore, many frameworks and helper tools have been created to assist
18developers in creating faster and more robust sites. This HOWTO describes
19some of the methods used to combine Python with a web server to create
20dynamic content. It is not meant as a complete introduction, as this topic is
21far too broad to be covered in one single document. However, a short overview
22of the most popular libraries is provided.
Benjamin Petersonae5360b2008-09-08 23:05:23 +000023
24.. seealso::
25
Ezio Melotti079d5ed2010-04-05 04:04:35 +000026 While this HOWTO tries to give an overview of Python in the web, it cannot
27 always be as up to date as desired. Web development in Python is rapidly
28 moving forward, so the wiki page on `Web Programming
Georg Brandle73778c2014-10-29 08:36:35 +010029 <https://wiki.python.org/moin/WebProgramming>`_ may be more in sync with
Benjamin Petersonae5360b2008-09-08 23:05:23 +000030 recent development.
31
32
Ezio Melotti079d5ed2010-04-05 04:04:35 +000033The Low-Level View
Benjamin Petersonae5360b2008-09-08 23:05:23 +000034==================
35
Ezio Melotti079d5ed2010-04-05 04:04:35 +000036When a user enters a web site, their browser makes a connection to the site's
37web server (this is called the *request*). The server looks up the file in the
Benjamin Petersonae5360b2008-09-08 23:05:23 +000038file system and sends it back to the user's browser, which displays it (this is
Ezio Melotti845994d2010-03-23 23:29:39 +000039the *response*). This is roughly how the underlying protocol, HTTP, works.
Benjamin Petersonae5360b2008-09-08 23:05:23 +000040
Ezio Melotti079d5ed2010-04-05 04:04:35 +000041Dynamic web sites are not based on files in the file system, but rather on
42programs which are run by the web server when a request comes in, and which
43*generate* the content that is returned to the user. They can do all sorts of
44useful things, like display the postings of a bulletin board, show your email,
45configure software, or just display the current time. These programs can be
46written in any programming language the server supports. Since most servers
47support Python, it is easy to use Python to create dynamic web sites.
Benjamin Petersonae5360b2008-09-08 23:05:23 +000048
Ezio Melotti079d5ed2010-04-05 04:04:35 +000049Most HTTP servers are written in C or C++, so they cannot execute Python code
50directly -- a bridge is needed between the server and the program. These
51bridges, or rather interfaces, define how programs interact with the server.
52There have been numerous attempts to create the best possible interface, but
53there are only a few worth mentioning.
Benjamin Petersonae5360b2008-09-08 23:05:23 +000054
Ezio Melotti079d5ed2010-04-05 04:04:35 +000055Not every web server supports every interface. Many web servers only support
56old, now-obsolete interfaces; however, they can often be extended using
57third-party modules to support newer ones.
Benjamin Petersonae5360b2008-09-08 23:05:23 +000058
59
60Common Gateway Interface
61------------------------
62
Ezio Melotti079d5ed2010-04-05 04:04:35 +000063This interface, most commonly referred to as "CGI", is the oldest, and is
64supported by nearly every web server out of the box. Programs using CGI to
65communicate with their web server need to be started by the server for every
66request. So, every request starts a new Python interpreter -- which takes some
67time to start up -- thus making the whole interface only usable for low load
68situations.
Benjamin Petersonae5360b2008-09-08 23:05:23 +000069
Ezio Melotti079d5ed2010-04-05 04:04:35 +000070The upside of CGI is that it is simple -- writing a Python program which uses
71CGI is a matter of about three lines of code. This simplicity comes at a
72price: it does very few things to help the developer.
Benjamin Petersonae5360b2008-09-08 23:05:23 +000073
Ezio Melotti079d5ed2010-04-05 04:04:35 +000074Writing CGI programs, while still possible, is no longer recommended. With
75:ref:`WSGI <WSGI>`, a topic covered later in this document, it is possible to write
76programs that emulate CGI, so they can be run as CGI if no better option is
77available.
Benjamin Petersonae5360b2008-09-08 23:05:23 +000078
79.. seealso::
80
81 The Python standard library includes some modules that are helpful for
82 creating plain CGI programs:
83
84 * :mod:`cgi` -- Handling of user input in CGI scripts
Ezio Melotti079d5ed2010-04-05 04:04:35 +000085 * :mod:`cgitb` -- Displays nice tracebacks when errors happen in CGI
Benjamin Petersonae5360b2008-09-08 23:05:23 +000086 applications, instead of presenting a "500 Internal Server Error" message
87
88 The Python wiki features a page on `CGI scripts
Georg Brandle73778c2014-10-29 08:36:35 +010089 <https://wiki.python.org/moin/CgiScripts>`_ with some additional information
Benjamin Petersonae5360b2008-09-08 23:05:23 +000090 about CGI in Python.
91
Georg Brandl48310cd2009-01-03 21:18:54 +000092
Benjamin Petersonae5360b2008-09-08 23:05:23 +000093Simple script for testing CGI
94^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
95
96To test whether your web server works with CGI, you can use this short and
97simple CGI program::
98
99 #!/usr/bin/env python
100 # -*- coding: UTF-8 -*-
101
102 # enable debugging
Benjamin Petersonad3d5c22009-02-26 03:38:59 +0000103 import cgitb
104 cgitb.enable()
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000105
Georg Brandl6d204bf2008-12-05 18:04:41 +0000106 print("Content-Type: text/plain;charset=utf-8")
107 print()
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000108
Georg Brandl6d204bf2008-12-05 18:04:41 +0000109 print("Hello World!")
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000110
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000111Depending on your web server configuration, you may need to save this code with
112a ``.py`` or ``.cgi`` extension. Additionally, this file may also need to be
113in a ``cgi-bin`` folder, for security reasons.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000114
115You might wonder what the ``cgitb`` line is about. This line makes it possible
116to display a nice traceback instead of just crashing and displaying an "Internal
117Server Error" in the user's browser. This is useful for debugging, but it might
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000118risk exposing some confidential data to the user. You should not use ``cgitb``
119in production code for this reason. You should *always* catch exceptions, and
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000120display proper error pages -- end-users don't like to see nondescript "Internal
121Server Errors" in their browsers.
122
123
124Setting up CGI on your own server
125^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
126
127If you don't have your own web server, this does not apply to you. You can
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000128check whether it works as-is, and if not you will need to talk to the
129administrator of your web server. If it is a big host, you can try filing a
130ticket asking for Python support.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000131
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000132If you are your own administrator or want to set up CGI for testing purposes on
133your own computers, you have to configure it by yourself. There is no single
134way to configure CGI, as there are many web servers with different
135configuration options. Currently the most widely used free web server is
136`Apache HTTPd <http://httpd.apache.org/>`_, or Apache for short. Apache can be
137easily installed on nearly every system using the system's package management
138tool. `lighttpd <http://www.lighttpd.net>`_ is another alternative and is
139said to have better performance. On many systems this server can also be
140installed using the package management tool, so manually compiling the web
141server may not be needed.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000142
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000143* On Apache you can take a look at the `Dynamic Content with CGI
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000144 <http://httpd.apache.org/docs/2.2/howto/cgi.html>`_ tutorial, where everything
145 is described. Most of the time it is enough just to set ``+ExecCGI``. The
146 tutorial also describes the most common gotchas that might arise.
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000147
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000148* On lighttpd you need to use the `CGI module
Georg Brandl9bdcb3b2014-10-29 09:37:43 +0100149 <http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModCGI>`_\ , which can be configured
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000150 in a straightforward way. It boils down to setting ``cgi.assign`` properly.
151
152
153Common problems with CGI scripts
154^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
155
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000156Using CGI sometimes leads to small annoyances while trying to get these
157scripts to run. Sometimes a seemingly correct script does not work as
158expected, the cause being some small hidden problem that's difficult to spot.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000159
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000160Some of these potential problems are:
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000161
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000162* The Python script is not marked as executable. When CGI scripts are not
163 executable most web servers will let the user download it, instead of
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000164 running it and sending the output to the user. For CGI scripts to run
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000165 properly on Unix-like operating systems, the ``+x`` bit needs to be set.
166 Using ``chmod a+x your_script.py`` may solve this problem.
167
168* On a Unix-like system, The line endings in the program file must be Unix
169 style line endings. This is important because the web server checks the
170 first line of the script (called shebang) and tries to run the program
171 specified there. It gets easily confused by Windows line endings (Carriage
172 Return & Line Feed, also called CRLF), so you have to convert the file to
173 Unix line endings (only Line Feed, LF). This can be done automatically by
174 uploading the file via FTP in text mode instead of binary mode, but the
175 preferred way is just telling your editor to save the files with Unix line
176 endings. Most editors support this.
177
178* Your web server must be able to read the file, and you need to make sure the
179 permissions are correct. On unix-like systems, the server often runs as user
180 and group ``www-data``, so it might be worth a try to change the file
181 ownership, or making the file world readable by using ``chmod a+r
182 your_script.py``.
183
184* The web server must know that the file you're trying to access is a CGI script.
185 Check the configuration of your web server, as it may be configured
186 to expect a specific file extension for CGI scripts.
187
188* On Unix-like systems, the path to the interpreter in the shebang
189 (``#!/usr/bin/env python``) must be correct. This line calls
190 ``/usr/bin/env`` to find Python, but it will fail if there is no
191 ``/usr/bin/env``, or if Python is not in the web server's path. If you know
192 where your Python is installed, you can also use that full path. The
193 commands ``whereis python`` and ``type -p python`` could help you find
194 where it is installed. Once you know the path, you can change the shebang
195 accordingly: ``#!/usr/bin/python``.
196
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000197* The file must not contain a BOM (Byte Order Mark). The BOM is meant for
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000198 determining the byte order of UTF-16 and UTF-32 encodings, but some editors
199 write this also into UTF-8 files. The BOM interferes with the shebang line,
200 so be sure to tell your editor not to write the BOM.
201
202* If the web server is using :ref:`mod-python`, ``mod_python`` may be having
203 problems. ``mod_python`` is able to handle CGI scripts by itself, but it can
204 also be a source of issues.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000205
206
207.. _mod-python:
208
209mod_python
210----------
211
212People coming from PHP often find it hard to grasp how to use Python in the web.
Georg Brandl9bdcb3b2014-10-29 09:37:43 +0100213Their first thought is mostly `mod_python <http://modpython.org/>`_\ ,
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000214because they think that this is the equivalent to ``mod_php``. Actually, there
215are many differences. What ``mod_python`` does is embed the interpreter into
216the Apache process, thus speeding up requests by not having to start a Python
217interpreter for each request. On the other hand, it is not "Python intermixed
218with HTML" in the way that PHP is often intermixed with HTML. The Python
219equivalent of that is a template engine. ``mod_python`` itself is much more
220powerful and provides more access to Apache internals. It can emulate CGI,
221work in a "Python Server Pages" mode (similar to JSP) which is "HTML
222intermingled with Python", and it has a "Publisher" which designates one file
223to accept all requests and decide what to do with them.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000224
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000225``mod_python`` does have some problems. Unlike the PHP interpreter, the Python
226interpreter uses caching when executing files, so changes to a file will
227require the web server to be restarted. Another problem is the basic concept
228-- Apache starts child processes to handle the requests, and unfortunately
229every child process needs to load the whole Python interpreter even if it does
230not use it. This makes the whole web server slower. Another problem is that,
231because ``mod_python`` is linked against a specific version of ``libpython``,
232it is not possible to switch from an older version to a newer (e.g. 2.4 to 2.5)
233without recompiling ``mod_python``. ``mod_python`` is also bound to the Apache
234web server, so programs written for ``mod_python`` cannot easily run on other
235web servers.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000236
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000237These are the reasons why ``mod_python`` should be avoided when writing new
238programs. In some circumstances it still might be a good idea to use
239``mod_python`` for deployment, but WSGI makes it possible to run WSGI programs
240under ``mod_python`` as well.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000241
242
243FastCGI and SCGI
244----------------
245
246FastCGI and SCGI try to solve the performance problem of CGI in another way.
247Instead of embedding the interpreter into the web server, they create
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000248long-running background processes. There is still a module in the web server
249which makes it possible for the web server to "speak" with the background
250process. As the background process is independent of the server, it can be
251written in any language, including Python. The language just needs to have a
252library which handles the communication with the webserver.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000253
254The difference between FastCGI and SCGI is very small, as SCGI is essentially
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000255just a "simpler FastCGI". As the web server support for SCGI is limited,
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000256most people use FastCGI instead, which works the same way. Almost everything
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000257that applies to SCGI also applies to FastCGI as well, so we'll only cover
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000258the latter.
259
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000260These days, FastCGI is never used directly. Just like ``mod_python``, it is only
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000261used for the deployment of WSGI applications.
262
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000263
264Setting up FastCGI
265^^^^^^^^^^^^^^^^^^
266
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000267Each web server requires a specific module.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000268
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000269* Apache has both `mod_fastcgi <http://www.fastcgi.com/drupal/>`_ and `mod_fcgid
Sandro Tosi1abde362011-12-31 18:46:50 +0100270 <http://httpd.apache.org/mod_fcgid/>`_. ``mod_fastcgi`` is the original one, but it
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000271 has some licensing issues, which is why it is sometimes considered non-free.
272 ``mod_fcgid`` is a smaller, compatible alternative. One of these modules needs
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000273 to be loaded by Apache.
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000274
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000275* lighttpd ships its own `FastCGI module
Georg Brandl9bdcb3b2014-10-29 09:37:43 +0100276 <http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModFastCGI>`_ as well as an
277 `SCGI module <http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModSCGI>`_.
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000278
279* `nginx <http://nginx.org/>`_ also supports `FastCGI
280 <http://wiki.nginx.org/NginxSimplePythonFCGI>`_.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000281
282Once you have installed and configured the module, you can test it with the
283following WSGI-application::
284
285 #!/usr/bin/env python
286 # -*- coding: UTF-8 -*-
287
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000288 import sys, os
Georg Brandl1f7fffb2010-10-15 15:57:45 +0000289 from html import escape
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000290 from flup.server.fcgi import WSGIServer
291
292 def app(environ, start_response):
293 start_response('200 OK', [('Content-Type', 'text/html')])
294
295 yield '<h1>FastCGI Environment</h1>'
296 yield '<table>'
297 for k, v in sorted(environ.items()):
Georg Brandl6d204bf2008-12-05 18:04:41 +0000298 yield '<tr><th>{0}</th><td>{1}</td></tr>'.format(
299 escape(k), escape(v))
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000300 yield '</table>'
301
302 WSGIServer(app).run()
303
304This is a simple WSGI application, but you need to install `flup
Georg Brandle73778c2014-10-29 08:36:35 +0100305<https://pypi.python.org/pypi/flup/1.0>`_ first, as flup handles the low level
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000306FastCGI access.
307
308.. seealso::
309
310 There is some documentation on `setting up Django with FastCGI
Georg Brandl9bdcb3b2014-10-29 09:37:43 +0100311 <https://docs.djangoproject.com/en/dev/howto/deployment/fastcgi/>`_, most of
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000312 which can be reused for other WSGI-compliant frameworks and libraries.
313 Only the ``manage.py`` part has to be changed, the example used here can be
314 used instead. Django does more or less the exact same thing.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000315
316
317mod_wsgi
318--------
319
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000320`mod_wsgi <http://code.google.com/p/modwsgi/>`_ is an attempt to get rid of the
321low level gateways. Given that FastCGI, SCGI, and mod_python are mostly used to
322deploy WSGI applications, mod_wsgi was started to directly embed WSGI applications
323into the Apache web server. mod_wsgi is specifically designed to host WSGI
324applications. It makes the deployment of WSGI applications much easier than
325deployment using other low level methods, which need glue code. The downside
326is that mod_wsgi is limited to the Apache web server; other servers would need
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000327their own implementations of mod_wsgi.
328
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000329mod_wsgi supports two modes: embedded mode, in which it integrates with the
330Apache process, and daemon mode, which is more FastCGI-like. Unlike FastCGI,
331mod_wsgi handles the worker-processes by itself, which makes administration
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000332easier.
333
334
335.. _WSGI:
336
337Step back: WSGI
338===============
339
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000340WSGI has already been mentioned several times, so it has to be something
341important. In fact it really is, and now it is time to explain it.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000342
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000343The *Web Server Gateway Interface*, or WSGI for short, is defined in
344:pep:`333` and is currently the best way to do Python web programming. While
345it is great for programmers writing frameworks, a normal web developer does not
346need to get in direct contact with it. When choosing a framework for web
347development it is a good idea to choose one which supports WSGI.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000348
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000349The big benefit of WSGI is the unification of the application programming
350interface. When your program is compatible with WSGI -- which at the outer
351level means that the framework you are using has support for WSGI -- your
352program can be deployed via any web server interface for which there are WSGI
353wrappers. You do not need to care about whether the application user uses
354mod_python or FastCGI or mod_wsgi -- with WSGI your application will work on
355any gateway interface. The Python standard library contains its own WSGI
356server, :mod:`wsgiref`, which is a small web server that can be used for
357testing.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000358
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000359A really great WSGI feature is middleware. Middleware is a layer around your
360program which can add various functionality to it. There is quite a bit of
Sandro Tosi1abde362011-12-31 18:46:50 +0100361`middleware <http://www.wsgi.org/en/latest/libraries.html>`_ already
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000362available. For example, instead of writing your own session management (HTTP
363is a stateless protocol, so to associate multiple HTTP requests with a single
364user your application must create and manage such state via a session), you can
365just download middleware which does that, plug it in, and get on with coding
366the unique parts of your application. The same thing with compression -- there
367is existing middleware which handles compressing your HTML using gzip to save
368on your server's bandwidth. Authentication is another a problem easily solved
369using existing middleware.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000370
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000371Although WSGI may seem complex, the initial phase of learning can be very
372rewarding because WSGI and the associated middleware already have solutions to
373many problems that might arise while developing web sites.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000374
375
376WSGI Servers
377------------
378
379The code that is used to connect to various low level gateways like CGI or
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000380mod_python is called a *WSGI server*. One of these servers is ``flup``, which
381supports FastCGI and SCGI, as well as `AJP
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000382<http://en.wikipedia.org/wiki/Apache_JServ_Protocol>`_. Some of these servers
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000383are written in Python, as ``flup`` is, but there also exist others which are
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000384written in C and can be used as drop-in replacements.
385
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000386There are many servers already available, so a Python web application
387can be deployed nearly anywhere. This is one big advantage that Python has
388compared with other web technologies.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000389
390.. seealso::
391
Sandro Tosi1abde362011-12-31 18:46:50 +0100392 A good overview of WSGI-related code can be found in the `WSGI homepage
393 <http://www.wsgi.org/en/latest/index.html>`_, which contains an extensive list of `WSGI servers
394 <http://www.wsgi.org/en/latest/servers.html>`_ which can be used by *any* application
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000395 supporting WSGI.
396
397 You might be interested in some WSGI-supporting modules already contained in
398 the standard library, namely:
Georg Brandl48310cd2009-01-03 21:18:54 +0000399
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000400 * :mod:`wsgiref` -- some tiny utilities and servers for WSGI
401
402
403Case study: MoinMoin
404--------------------
405
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000406What does WSGI give the web application developer? Let's take a look at
407an application that's been around for a while, which was written in
408Python without using WSGI.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000409
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000410One of the most widely used wiki software packages is `MoinMoin
411<http://moinmo.in/>`_. It was created in 2000, so it predates WSGI by about
412three years. Older versions needed separate code to run on CGI, mod_python,
413FastCGI and standalone.
414
415It now includes support for WSGI. Using WSGI, it is possible to deploy
416MoinMoin on any WSGI compliant server, with no additional glue code.
417Unlike the pre-WSGI versions, this could include WSGI servers that the
418authors of MoinMoin know nothing about.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000419
420
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000421Model-View-Controller
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000422=====================
423
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000424The term *MVC* is often encountered in statements such as "framework *foo*
425supports MVC". MVC is more about the overall organization of code, rather than
426any particular API. Many web frameworks use this model to help the developer
427bring structure to their program. Bigger web applications can have lots of
428code, so it is a good idea to have an effective structure right from the beginning.
429That way, even users of other frameworks (or even other languages, since MVC is
430not Python-specific) can easily understand the code, given that they are
431already familiar with the MVC structure.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000432
433MVC stands for three components:
434
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000435* The *model*. This is the data that will be displayed and modified. In
436 Python frameworks, this component is often represented by the classes used by
437 an object-relational mapper.
438
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000439* The *view*. This component's job is to display the data of the model to the
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000440 user. Typically this component is implemented via templates.
441
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000442* The *controller*. This is the layer between the user and the model. The
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000443 controller reacts to user actions (like opening some specific URL), tells
444 the model to modify the data if necessary, and tells the view code what to
445 display,
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000446
447While one might think that MVC is a complex design pattern, in fact it is not.
448It is used in Python because it has turned out to be useful for creating clean,
449maintainable web sites.
450
451.. note::
452
453 While not all Python frameworks explicitly support MVC, it is often trivial
Mark Dickinson934896d2009-02-21 20:59:32 +0000454 to create a web site which uses the MVC pattern by separating the data logic
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000455 (the model) from the user interaction logic (the controller) and the
Mark Dickinson934896d2009-02-21 20:59:32 +0000456 templates (the view). That's why it is important not to write unnecessary
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000457 Python code in the templates -- it works against the MVC model and creates
458 chaos in the code base, making it harder to understand and modify.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000459
460.. seealso::
461
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000462 The English Wikipedia has an article about the `Model-View-Controller pattern
463 <http://en.wikipedia.org/wiki/Model-view-controller>`_. It includes a long
464 list of web frameworks for various programming languages.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000465
466
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000467Ingredients for Websites
468========================
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000469
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000470Websites are complex constructs, so tools have been created to help web
471developers make their code easier to write and more maintainable. Tools like
472these exist for all web frameworks in all languages. Developers are not forced
473to use these tools, and often there is no "best" tool. It is worth learning
474about the available tools because they can greatly simplify the process of
475developing a web site.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000476
477
478.. seealso::
479
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000480 There are far more components than can be presented here. The Python wiki
481 has a page about these components, called
Georg Brandle73778c2014-10-29 08:36:35 +0100482 `Web Components <https://wiki.python.org/moin/WebComponents>`_.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000483
484
485Templates
486---------
487
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000488Mixing of HTML and Python code is made possible by a few libraries. While
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000489convenient at first, it leads to horribly unmaintainable code. That's why
490templates exist. Templates are, in the simplest case, just HTML files with
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000491placeholders. The HTML is sent to the user's browser after filling in the
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000492placeholders.
493
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000494Python already includes a way to build simple templates::
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000495
496 # a simple template
Georg Brandl6d204bf2008-12-05 18:04:41 +0000497 template = "<html><body><h1>Hello {who}!</h1></body></html>"
498 print(template.format(who="Reader"))
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000499
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000500To generate complex HTML based on non-trivial model data, conditional
501and looping constructs like Python's *for* and *if* are generally needed.
502*Template engines* support templates of this complexity.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000503
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000504There are a lot of template engines available for Python which can be used with
505or without a `framework`_. Some of these define a plain-text programming
506language which is easy to learn, partly because it is limited in scope.
507Others use XML, and the template output is guaranteed to be always be valid
508XML. There are many other variations.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000509
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000510Some `frameworks`_ ship their own template engine or recommend one in
511particular. In the absence of a reason to use a different template engine,
512using the one provided by or recommended by the framework is a good idea.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000513
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000514Popular template engines include:
Georg Brandl48310cd2009-01-03 21:18:54 +0000515
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000516 * `Mako <http://www.makotemplates.org/>`_
517 * `Genshi <http://genshi.edgewall.org/>`_
Georg Brandlb7354a62014-10-29 10:57:37 +0100518 * `Jinja <http://jinja.pocoo.org/>`_
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000519
520.. seealso::
521
Ezio Melotti42dd5242010-11-16 20:57:59 +0000522 There are many template engines competing for attention, because it is
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000523 pretty easy to create them in Python. The page `Templating
Georg Brandle73778c2014-10-29 08:36:35 +0100524 <https://wiki.python.org/moin/Templating>`_ in the wiki lists a big,
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000525 ever-growing number of these. The three listed above are considered "second
526 generation" template engines and are a good place to start.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000527
528
529Data persistence
530----------------
531
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000532*Data persistence*, while sounding very complicated, is just about storing data.
533This data might be the text of blog entries, the postings on a bulletin board or
534the text of a wiki page. There are, of course, a number of different ways to store
535information on a web server.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000536
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000537Often, relational database engines like `MySQL <http://www.mysql.com/>`_ or
538`PostgreSQL <http://www.postgresql.org/>`_ are used because of their good
539performance when handling very large databases consisting of millions of
540entries. There is also a small database engine called `SQLite
541<http://www.sqlite.org/>`_, which is bundled with Python in the :mod:`sqlite3`
542module, and which uses only one file. It has no other dependencies. For
543smaller sites SQLite is just enough.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000544
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000545Relational databases are *queried* using a language called `SQL
546<http://en.wikipedia.org/wiki/SQL>`_. Python programmers in general do not
547like SQL too much, as they prefer to work with objects. It is possible to save
548Python objects into a database using a technology called `ORM
549<http://en.wikipedia.org/wiki/Object-relational_mapping>`_ (Object Relational
550Mapping). ORM translates all object-oriented access into SQL code under the
551hood, so the developer does not need to think about it. Most `frameworks`_ use
552ORMs, and it works quite well.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000553
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000554A second possibility is storing data in normal, plain text files (some
555times called "flat files"). This is very easy for simple sites,
556but can be difficult to get right if the web site is performing many
557updates to the stored data.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000558
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000559A third possibility are object oriented databases (also called "object
560databases"). These databases store the object data in a form that closely
561parallels the way the objects are structured in memory during program
562execution. (By contrast, ORMs store the object data as rows of data in tables
563and relations between those rows.) Storing the objects directly has the
564advantage that nearly all objects can be saved in a straightforward way, unlike
565in relational databases where some objects are very hard to represent.
566
567`Frameworks`_ often give hints on which data storage method to choose. It is
568usually a good idea to stick to the data store recommended by the framework
569unless the application has special requirements better satisfied by an
570alternate storage mechanism.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000571
572.. seealso::
573
Georg Brandle73778c2014-10-29 08:36:35 +0100574 * `Persistence Tools <https://wiki.python.org/moin/PersistenceTools>`_ lists
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000575 possibilities on how to save data in the file system. Some of these
576 modules are part of the standard library
577
Georg Brandle73778c2014-10-29 08:36:35 +0100578 * `Database Programming <https://wiki.python.org/moin/DatabaseProgramming>`_
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000579 helps with choosing a method for saving data
580
581 * `SQLAlchemy <http://www.sqlalchemy.org/>`_, the most powerful OR-Mapper
582 for Python, and `Elixir <http://elixir.ematia.de/>`_, which makes
583 SQLAlchemy easier to use
584
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000585 * `SQLObject <http://www.sqlobject.org/>`_, another popular OR-Mapper
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000586
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000587 * `ZODB <https://launchpad.net/zodb>`_ and `Durus
588 <http://www.mems-exchange.org/software/durus/>`_, two object oriented
589 databases
590
591
592.. _framework:
593
594Frameworks
595==========
596
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000597The process of creating code to run web sites involves writing code to provide
598various services. The code to provide a particular service often works the
599same way regardless of the complexity or purpose of the web site in question.
600Abstracting these common solutions into reusable code produces what are called
601"frameworks" for web development. Perhaps the most well-known framework for
602web development is Ruby on Rails, but Python has its own frameworks. Some of
603these were partly inspired by Rails, or borrowed ideas from Rails, but many
604existed a long time before Rails.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000605
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000606Originally Python web frameworks tended to incorporate all of the services
607needed to develop web sites as a giant, integrated set of tools. No two web
608frameworks were interoperable: a program developed for one could not be
609deployed on a different one without considerable re-engineering work. This led
610to the development of "minimalist" web frameworks that provided just the tools
611to communicate between the Python code and the http protocol, with all other
612services to be added on top via separate components. Some ad hoc standards
613were developed that allowed for limited interoperability between frameworks,
614such as a standard that allowed different template engines to be used
615interchangeably.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000616
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000617Since the advent of WSGI, the Python web framework world has been evolving
618toward interoperability based on the WSGI standard. Now many web frameworks,
619whether "full stack" (providing all the tools one needs to deploy the most
620complex web sites) or minimalist, or anything in between, are built from
621collections of reusable components that can be used with more than one
622framework.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000623
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000624The majority of users will probably want to select a "full stack" framework
625that has an active community. These frameworks tend to be well documented,
626and provide the easiest path to producing a fully functional web site in
627minimal time.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000628
629
630Some notable frameworks
631-----------------------
632
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000633There are an incredible number of frameworks, so they cannot all be covered
634here. Instead we will briefly touch on some of the most popular.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000635
636
637Django
638^^^^^^
639
Georg Brandl9bdcb3b2014-10-29 09:37:43 +0100640`Django <https://www.djangoproject.com/>`_ is a framework consisting of several
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000641tightly coupled elements which were written from scratch and work together very
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000642well. It includes an ORM which is quite powerful while being simple to use,
643and has a great online administration interface which makes it possible to edit
644the data in the database with a browser. The template engine is text-based and
645is designed to be usable for page designers who cannot write Python. It
646supports template inheritance and filters (which work like Unix pipes). Django
647has many handy features bundled, such as creation of RSS feeds or generic views,
648which make it possible to create web sites almost without writing any Python code.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000649
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000650It has a big, international community, the members of which have created many
651web sites. There are also a lot of add-on projects which extend Django's normal
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000652functionality. This is partly due to Django's well written `online
Georg Brandl9bdcb3b2014-10-29 09:37:43 +0100653documentation <https://docs.djangoproject.com/>`_ and the `Django book
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000654<http://www.djangobook.com/>`_.
655
656
657.. note::
658
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000659 Although Django is an MVC-style framework, it names the elements
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000660 differently, which is described in the `Django FAQ
Georg Brandl9bdcb3b2014-10-29 09:37:43 +0100661 <https://docs.djangoproject.com/en/dev/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names>`_.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000662
663
664TurboGears
665^^^^^^^^^^
666
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000667Another popular web framework for Python is `TurboGears
668<http://www.turbogears.org/>`_. TurboGears takes the approach of using already
669existing components and combining them with glue code to create a seamless
670experience. TurboGears gives the user flexibility in choosing components. For
671example the ORM and template engine can be changed to use packages different
672from those used by default.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000673
674The documentation can be found in the `TurboGears wiki
675<http://docs.turbogears.org/>`_, where links to screencasts can be found.
676TurboGears has also an active user community which can respond to most related
677questions. There is also a `TurboGears book <http://turbogearsbook.com/>`_
678published, which is a good starting point.
679
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000680The newest version of TurboGears, version 2.0, moves even further in direction
681of WSGI support and a component-based architecture. TurboGears 2 is based on
682the WSGI stack of another popular component-based web framework, `Pylons
Senthil Kumarane9738232014-06-11 06:19:21 -0700683<http://www.pylonsproject.org/>`_.
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000684
685
686Zope
687^^^^
688
689The Zope framework is one of the "old original" frameworks. Its current
690incarnation in Zope2 is a tightly integrated full-stack framework. One of its
691most interesting feature is its tight integration with a powerful object
692database called the `ZODB <https://launchpad.net/zodb>`_ (Zope Object Database).
693Because of its highly integrated nature, Zope wound up in a somewhat isolated
694ecosystem: code written for Zope wasn't very usable outside of Zope, and
695vice-versa. To solve this problem the Zope 3 effort was started. Zope 3
696re-engineers Zope as a set of more cleanly isolated components. This effort
697was started before the advent of the WSGI standard, but there is WSGI support
698for Zope 3 from the `Repoze <http://repoze.org/>`_ project. Zope components
699have many years of production use behind them, and the Zope 3 project gives
700access to these components to the wider Python community. There is even a
701separate framework based on the Zope components: `Grok
702<http://grok.zope.org/>`_.
703
Georg Brandl9bdcb3b2014-10-29 09:37:43 +0100704Zope is also the infrastructure used by the `Plone <https://plone.org/>`_ content
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000705management system, one of the most powerful and popular content management
706systems available.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000707
708
709Other notable frameworks
710^^^^^^^^^^^^^^^^^^^^^^^^
711
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000712Of course these are not the only frameworks that are available. There are
713many other frameworks worth mentioning.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000714
715Another framework that's already been mentioned is `Pylons`_. Pylons is much
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000716like TurboGears, but with an even stronger emphasis on flexibility, which comes
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000717at the cost of being more difficult to use. Nearly every component can be
Mark Dickinson934896d2009-02-21 20:59:32 +0000718exchanged, which makes it necessary to use the documentation of every single
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000719component, of which there are many. Pylons builds upon `Paste
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000720<http://pythonpaste.org/>`_, an extensive set of tools which are handy for WSGI.
721
722And that's still not everything. The most up-to-date information can always be
723found in the Python wiki.
724
725.. seealso::
Georg Brandl48310cd2009-01-03 21:18:54 +0000726
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000727 The Python wiki contains an extensive list of `web frameworks
Georg Brandle73778c2014-10-29 08:36:35 +0100728 <https://wiki.python.org/moin/WebFrameworks>`_.
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000729
730 Most frameworks also have their own mailing lists and IRC channels, look out
Ezio Melotti079d5ed2010-04-05 04:04:35 +0000731 for these on the projects' web sites. There is also a general "Python in the
Benjamin Petersonae5360b2008-09-08 23:05:23 +0000732 Web" IRC channel on freenode called `#python.web
Georg Brandle73778c2014-10-29 08:36:35 +0100733 <https://wiki.python.org/moin/PoundPythonWeb>`_.