blob: 58b88d55b4b2dd37815e4724dd99cc092fa0cf8c [file] [log] [blame]
Greg Wardabc52162000-02-26 00:52:48 +00001\documentclass{howto}
2\usepackage{ltxmarkup}
Greg Wardfacb8db2000-04-09 04:32:40 +00003\usepackage{times}
Greg Ward16aafcd2000-04-09 04:06:44 +00004\usepackage{distutils}
Greg Wardabc52162000-02-26 00:52:48 +00005
Greg Ward16aafcd2000-04-09 04:06:44 +00006\title{Distributing Python Modules}
Greg Wardabc52162000-02-26 00:52:48 +00007
Greg Wardabc52162000-02-26 00:52:48 +00008\author{Greg Ward}
9\authoraddress{E-mail: \email{gward@python.net}}
10
Greg Ward16aafcd2000-04-09 04:06:44 +000011
Greg Wardabc52162000-02-26 00:52:48 +000012\begin{document}
13
Greg Wardfacb8db2000-04-09 04:32:40 +000014\maketitle
15\tableofcontents
Greg Ward16aafcd2000-04-09 04:06:44 +000016
17\section{Introduction}
Greg Warde78298a2000-04-28 17:12:24 +000018\label{intro}
Greg Ward16aafcd2000-04-09 04:06:44 +000019
20In the past, Python module developers have not had much infrastructure
21support for distributing modules, nor have Python users had much support
22for installing and maintaining third-party modules. With the
23introduction of the Python Distribution Utilities (Distutils for short)
24in Python 1.6, this situation should start to improve.
25
26This document only covers using the Distutils to distribute your Python
27modules. Using the Distutils does not tie you to Python 1.6, though:
28the Distutils work just fine with Python 1.5, and it is reasonable (and
29expected to become commonplace) to expect users of Python 1.5 to
30download and install the Distutils separately before they can install
31your modules. Python 1.6 users, of course, won't have to add anything
32to their Python installation in order to use the Distutils to install
33third-party modules.
34
35This document concentrates on the role of developer/distributor: if
36you're looking for information on installing Python modules, you should
37refer to the ``Installing Python Modules'' manual.
38
39
Greg Wardfacb8db2000-04-09 04:32:40 +000040\section{Concepts \& Terminology}
Greg Warde78298a2000-04-28 17:12:24 +000041\label{concepts}
Greg Ward16aafcd2000-04-09 04:06:44 +000042
43Using the Distutils is quite simple, both for module developers and for
44users/administrators installing third-party modules. As a developer,
45your responsibilites (apart from writing solid, well-documented and
46well-tested code, of course!) are:
47\begin{itemize}
48\item write a setup script (\file{setup.py} by convention)
49\item (optional) write a setup configuration file
50\item create a source distribution
51\item (optional) create one or more built (binary) distributions
52\end{itemize}
53Each of these tasks is covered in this document.
54
55Not all module developers have access to a multitude of platforms, so
56it's not always feasible to expect them to create a multitude of built
57distributions. It is hoped that a class of intermediaries, called
Greg Ward19c67f82000-06-24 01:33:16 +000058\emph{packagers}, will arise to address this need. Packagers will take
59source distributions released by module developers, build them on one or
60more platforms, and release the resulting built distributions. Thus,
61users on the most popular platforms will be able to install most popular
62Python module distributions in the most natural way for their platform,
63without having to run a single setup script or compile a line of code.
Greg Ward16aafcd2000-04-09 04:06:44 +000064
65
66\subsection{A simple example}
Greg Warde78298a2000-04-28 17:12:24 +000067\label{simple-example}
Greg Ward16aafcd2000-04-09 04:06:44 +000068
69The setup script is usually quite simple, although since it's written in
70Python, there are no arbitrary limits to what you can do. If all you
71want to do is distribute a module called \module{foo}, contained in a
Greg Ward370248d2000-06-24 01:45:47 +000072file \file{foo.py}, then your setup script can be as little as this:
Greg Ward16aafcd2000-04-09 04:06:44 +000073\begin{verbatim}
74from distutils.core import setup
75setup (name = "foo",
76 version = "1.0",
77 py_modules = ["foo"])
78\end{verbatim}
Greg Ward370248d2000-06-24 01:45:47 +000079
Greg Ward16aafcd2000-04-09 04:06:44 +000080Some observations:
81\begin{itemize}
Greg Ward370248d2000-06-24 01:45:47 +000082\item most information that you supply to the Distutils is supplied as
Greg Wardfacb8db2000-04-09 04:32:40 +000083 keyword arguments to the \function{setup()} function
Greg Ward16aafcd2000-04-09 04:06:44 +000084\item those keyword arguments fall into two categories: package
85 meta-data (name, version number) and information about what's in the
Greg Ward370248d2000-06-24 01:45:47 +000086 package (a list of pure Python modules, in this case)
Greg Ward16aafcd2000-04-09 04:06:44 +000087\item modules are specified by module name, not filename (the same will
88 hold true for packages and extensions)
89\item it's recommended that you supply a little more meta-data, in
90 particular your name, email address and a URL for the project
91\end{itemize}
92
Greg Ward370248d2000-06-24 01:45:47 +000093To create a source distribution for this module, you would create a
94setup script, \file{setup.py}, containing the above code, and run:
Greg Ward16aafcd2000-04-09 04:06:44 +000095\begin{verbatim}
96python setup.py sdist
97\end{verbatim}
98which will create an archive file (e.g., tarball on Unix, zip file on
99Windows) containing your setup script, \file{setup.py}, and your module,
100\file{foo.py}. The archive file will be named \file{Foo-1.0.tar.gz} (or
101\file{.zip}), and will unpack into a directory \file{Foo-1.0}.
102
103If an end-user wishes to install your \module{foo} module, all she has
Greg Ward59d382e2000-05-26 01:04:47 +0000104to do is download \file{Foo-1.0.tar.gz} (or \file{.zip}), unpack it,
Greg Ward16aafcd2000-04-09 04:06:44 +0000105and---from the \file{Foo-1.0} directory---run
106\begin{verbatim}
107python setup.py install
108\end{verbatim}
109which will ultimately copy \file{foo.py} to the appropriate directory
110for third-party modules in their Python installation.
111
112This simple example demonstrates some fundamental concepts of the
113Distutils: first, both developers and installers have the same basic
114user interface, i.e. the setup script. The difference is which
115Distutils \emph{commands} they use: the \command{sdist} command is
116almost exclusively for module developers, while \command{install} is
117more often for installers (although most developers will want to install
118their own code occasionally).
119
120\XXX{only partially implemented}%
121If you want to make things really easy for your users, you can create
122one or more built distributions for them. For instance, if you are
123running on a Windows machine, and want to make things easy for other
124Windows users, you can create an executable installer (the most
125appropriate type of built distribution for this platform) with the
Greg Ward59d382e2000-05-26 01:04:47 +0000126\command{bdist\_wininst} command. For example:
Greg Ward16aafcd2000-04-09 04:06:44 +0000127\begin{verbatim}
Greg Ward59d382e2000-05-26 01:04:47 +0000128python setup.py bdist_wininst
Greg Ward16aafcd2000-04-09 04:06:44 +0000129\end{verbatim}
Greg Wardfacb8db2000-04-09 04:32:40 +0000130will create an executable installer, \file{Foo-1\_0.exe}, in the current
Greg Ward16aafcd2000-04-09 04:06:44 +0000131directory.
132
Greg Ward59d382e2000-05-26 01:04:47 +0000133(Another way to create executable installers for Windows is with the
134\command{bdist\_wise} command, which uses Wise---the commercial
135installer-generator used to create Python's own installer---to create
136the installer. Wise-based installers are more appropriate for large,
137industrial-strength applications that need the full capabilities of a
138``real'' installer. \command{bdist\_wininst} creates a self-extracting
139zip file with a minimal user interface, which is enough for small- to
140medium-sized module collections. You'll need to have version XXX of
Greg Ward370248d2000-06-24 01:45:47 +0000141Wise installed on your system for the \command{bdist\_wise} command to
142work; it's available from \url{http://foo/bar/baz}.)
Greg Ward59d382e2000-05-26 01:04:47 +0000143
144Other \command{bdist} commands exist for other platforms: for example,
145\command{bdist\_rpm} for RPM-based Linux systems, (\command{bdist\_deb})
146for Debian-based Linux systems, and so forth. See
147section~\ref{bdist-cmds} for details on all the \command{bdist}
148commands.
Greg Ward16aafcd2000-04-09 04:06:44 +0000149
150
151\subsection{General Python terminology}
Greg Warde78298a2000-04-28 17:12:24 +0000152\label{python-terms}
Greg Ward16aafcd2000-04-09 04:06:44 +0000153
154If you're reading this document, you probably have a good idea of what
155modules, extensions, and so forth are. Nevertheless, just to be sure
156that everyone is operating from a common starting point, we offer the
157following glossary of common Python terms:
158\begin{description}
159\item[module] the basic unit of code reusability in Python: a block of
160 code imported by some other code. There are three types of modules
161 that concern us here: pure Python modules, extension modules, and
162 packages.
163\item[pure Python module] a module written in Python and contained in a
164 single \file{.py} file (and possibly associated \file{.pyc} and/or
165 \file{.pyo} files). Sometimes referred to as a ``pure module.''
166\item[extension module] a module written in the low-level language of
167 the Python implemention: C/C++ for CPython, Java for JPython.
168 Typically contained in a single dynamically loadable pre-compiled
169 file, e.g. a shared object (\file{.so}) file for CPython extensions on
170 Unix, a DLL (given the \file{.pyd} extension) for CPython extensions
Greg Ward1bbe3292000-06-25 03:14:13 +0000171 on Windows, or a Java class file for JPython extensions. (Note that
172 currently, the Distutils only handles C/C++ extensions for CPython.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000173\item[package] a module that contains other modules; typically contained
174 in a directory in the filesystem and distinguished from other
175 directories by the presence of a file \file{\_\_init\_\_.py}.
Greg Ward6153fa12000-05-26 02:24:28 +0000176\item[root package] the root of the hierarchy of packages. (This isn't
177 really a package, since it doesn't have an \file{\_\_init\_\_.py}
178 file. But we have to call it something.) The vast majority of the
179 standard library is in the root package, as are many small, standalone
180 third-party modules that don't belong to a larger module collection.
181 Unlike regular packages, modules in the root package can be found in
182 many directories: in fact, every directory listed in \code{sys.path}
183 can contribute modules to the root package.
Greg Ward16aafcd2000-04-09 04:06:44 +0000184\end{description}
185
186
187\subsection{Distutils-specific terminology}
Greg Warde78298a2000-04-28 17:12:24 +0000188\label{distutils-term}
Greg Ward16aafcd2000-04-09 04:06:44 +0000189
190The following terms apply more specifically to the domain of
191distributing Python modules using the Distutils:
192\begin{description}
193\item[module distribution] a collection of Python modules distributed
194 together as a single downloadable resource and meant to be installed
195 \emph{en masse}. Examples of some well-known module distributions are
196 Numeric Python, PyXML, PIL (the Python Imaging Library), or
197 mxDateTime. (This would be called a \emph{package}, except that term
Greg Ward59d382e2000-05-26 01:04:47 +0000198 is already taken in the Python context: a single module distribution
199 may contain zero, one, or many Python packages.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000200\item[pure module distribution] a module distribution that contains only
201 pure Python modules and packages. Sometimes referred to as a ``pure
202 distribution.''
203\item[non-pure module distribution] a module distribution that contains
204 at least one extension module. Sometimes referred to as a ``non-pure
205 distribution.''
206\item[distribution root] the top-level directory of your source tree (or
207 source distribution); the directory where \file{setup.py} exists and
208 is run from
209\end{description}
210
211
212\section{Writing the Setup Script}
Greg Warde78298a2000-04-28 17:12:24 +0000213\label{setup-script}
Greg Ward16aafcd2000-04-09 04:06:44 +0000214
215The setup script is the centre of all activity in building,
216distributing, and installing modules using the Distutils. The main
217purpose of the setup script is to describe your module distribution to
Greg Wardd5767a52000-04-19 22:48:09 +0000218the Distutils, so that the various commands that operate on your modules
Greg Ward59d382e2000-05-26 01:04:47 +0000219do the right thing. As we saw in section~\ref{simple-example} above,
220the setup script consists mainly of a call to \function{setup()}, and
Greg Ward1bbe3292000-06-25 03:14:13 +0000221most information supplied to the Distutils by the module developer is
222supplied as keyword arguments to \function{setup()}.
Greg Ward16aafcd2000-04-09 04:06:44 +0000223
224Here's a slightly more involved example, which we'll follow for the next
225couple of sections: the Distutils' own setup script. (Keep in mind that
226although the Distutils are included with Python 1.6, they also have an
227independent existence so that Python 1.5 users can use them to install
Greg Ward59d382e2000-05-26 01:04:47 +0000228other module distributions. The Distutils' own setup script is used to
229install the package into Python 1.5.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000230
231\begin{verbatim}
232#!/usr/bin/env python
233
234from distutils.core import setup
235
236setup (name = "Distutils",
237 version = "1.0",
238 description = "Python Module Distribution Utilities",
239 author = "Greg Ward",
240 author_email = "gward@python.net",
241 url = "http://www.python.org/sigs/distutils-sig/",
242
243 packages = ['distutils', 'distutils.command'],
244 )
245\end{verbatim}
246There are only two differences between this and the trivial one-file
Greg Warde78298a2000-04-28 17:12:24 +0000247distribution presented in section~\ref{simple-example}: more
Greg Ward16aafcd2000-04-09 04:06:44 +0000248meta-data, and the specification of pure Python modules by package,
249rather than by module. This is important since the Distutils consist of
250a couple of dozen modules split into (so far) two packages; an explicit
251list of every module would be tedious to generate and difficult to
252maintain.
253
Greg Ward46b98e32000-04-14 01:53:36 +0000254Note that any pathnames (files or directories) supplied in the setup
255script should be written using the Unix convention, i.e.
256slash-separated. The Distutils will take care of converting this
Greg Ward59d382e2000-05-26 01:04:47 +0000257platform-neutral representation into whatever is appropriate on your
Greg Ward46b98e32000-04-14 01:53:36 +0000258current platform before actually using the pathname. This makes your
259setup script portable across operating systems, which of course is one
260of the major goals of the Distutils. In this spirit, all pathnames in
Greg Ward59d382e2000-05-26 01:04:47 +0000261this document are slash-separated (Mac OS programmers should keep in
262mind that the \emph{absence} of a leading slash indicates a relative
263path, the opposite of the Mac OS convention with colons).
Greg Ward46b98e32000-04-14 01:53:36 +0000264
Greg Ward16aafcd2000-04-09 04:06:44 +0000265
266\subsection{Package directories}
Greg Warde78298a2000-04-28 17:12:24 +0000267\label{package-dirs}
Greg Ward16aafcd2000-04-09 04:06:44 +0000268
269The \option{packages} option tells the Distutils to process (build,
270distribute, install, etc.) all pure Python modules found in each package
271mentioned in the \option{packages} list. In order to do this, of
272course, there has to be a correspondence between package names and
273directories in the filesystem. The default correspondence is the most
Greg Ward1ecc2512000-04-19 22:36:24 +0000274obvious one, i.e. package \module{distutils} is found in the directory
Greg Ward16aafcd2000-04-09 04:06:44 +0000275\file{distutils} relative to the distribution root. Thus, when you say
276\code{packages = ['foo']} in your setup script, you are promising that
277the Distutils will find a file \file{foo/\_\_init\_\_.py} (which might
278be spelled differently on your system, but you get the idea) relative to
279the directory where your setup script lives. (If you break this
280promise, the Distutils will issue a warning but process the broken
281package anyways.)
282
283If you use a different convention to lay out your source directory,
284that's no problem: you just have to supply the \option{package\_dir}
285option to tell the Distutils about your convention. For example, say
286you keep all Python source under \file{lib}, so that modules not in any
Greg Ward1ecc2512000-04-19 22:36:24 +0000287package are right in \file{lib}, modules in the \module{foo} package
Greg Ward16aafcd2000-04-09 04:06:44 +0000288are in \file{lib/foo}, and so forth. Then you would put
289\begin{verbatim}
290package_dir = {'': 'lib'}
291\end{verbatim}
292in your setup script. (The keys to this dictionary are package names,
293and an empty package name stands for the ``root package,'' i.e. no
294package at all. The values are directory names relative to your
295distribution root.) In this case, when you say
296\code{packages = ['foo']}, you are promising that the file
297\file{lib/foo/\_\_init\_\_.py} exists.
298
Greg Ward1ecc2512000-04-19 22:36:24 +0000299Another possible convention is to put the \module{foo} package right in
300\file{lib}, the \module{foo.bar} package in \file{lib/bar}, etc. This
Greg Ward16aafcd2000-04-09 04:06:44 +0000301would be written in the setup script as
302\begin{verbatim}
303package_dir = {'foo': 'lib'}
304\end{verbatim}
Greg Ward59d382e2000-05-26 01:04:47 +0000305A \code{\var{package}: \var{dir}} entry in the \option{package\_dir}
306dictionary implicitly applies to all packages below \var{package}, so
307the \module{foo.bar} case is automatically handled here. In this
308example, having \code{packages = ['foo', 'foo.bar']} tells the Distutils
309to look for \file{lib/\_\_init\_\_.py} and
310\file{lib/bar/\_\_init\_\_.py}. (Keep in mind that although
311\option{package\_dir} applies recursively, you must explicitly list all
312packages in \option{packages}: the Distutils will \emph{not} recursively
313scan your source tree looking for any directory with an
314\file{\_\_init\_\_.py} file.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000315
316
317\subsection{Listing individual modules}
Greg Warde78298a2000-04-28 17:12:24 +0000318\label{listing-modules}
Greg Ward16aafcd2000-04-09 04:06:44 +0000319
320For a small module distribution, you might prefer to list all modules
321rather than listing packages---especially the case of a single module
322that goes in the ``root package'' (i.e., no package at all). This
Greg Warde78298a2000-04-28 17:12:24 +0000323simplest case was shown in section~\ref{simple-example}; here is a
Greg Ward16aafcd2000-04-09 04:06:44 +0000324slightly more involved example:
325\begin{verbatim}
326py_modules = ['mod1', 'pkg.mod2']
327\end{verbatim}
328This describes two modules, one of them in the ``root'' package, the
Greg Wardd5767a52000-04-19 22:48:09 +0000329other in the \module{pkg} package. Again, the default package/directory
330layout implies that these two modules can be found in \file{mod1.py} and
331\file{pkg/mod2.py}, and that \file{pkg/\_\_init\_\_.py} exists as well.
332And again, you can override the package/directory layout using the
Greg Ward59d382e2000-05-26 01:04:47 +0000333\option{package\_dir} option.
334
335
336\subsection{Describing extension modules}
337\label{sec:describing-extensions}
338
339\XXX{be sure to describe the whole \code{build\_info} dict, including
340 \code{extra\_compile\_args} and \code{extra\_link\_args}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000341
342
343\section{Writing the Setup Configuration File}
Greg Warde78298a2000-04-28 17:12:24 +0000344\label{setup-config}
Greg Ward16aafcd2000-04-09 04:06:44 +0000345
346\XXX{not implemented yet!}
347
348Often, it's not possible to write down everything needed to build a
349distribution \emph{a priori}. You need to get some information from the
350user, or from the user's system, in order to proceed. For example, you
351might include an optional extension module that provides an interface to
352a particular C library. If that library is installed on the user's
353system, then you can build your optional extension---but you need to
354know where to find the header and library file. If it's not installed,
355you need to know this so you can omit your optional extension.
356
357The preferred way to do this, of course, would be for you to tell the
358Distutils which optional features (C libraries, system calls, external
359utilities, etc.) you're looking for, and it would inspect the user's
360system and try to find them. This functionality may appear in a future
361version of the Distutils, but it isn't there now. So, for the time
362being, we rely on the user building and installing your software to
363provide the necessary information. The vehicle for doing so is the
364setup configuration file, \file{setup.cfg}.
365
366\XXX{need more here!}
367
368
369\section{Creating a Source Distribution}
Greg Warde78298a2000-04-28 17:12:24 +0000370\label{source-dist}
Greg Ward16aafcd2000-04-09 04:06:44 +0000371
Greg Warde78298a2000-04-28 17:12:24 +0000372As shown in section~\ref{simple-example}, you use the
Greg Ward16aafcd2000-04-09 04:06:44 +0000373\command{sdist} command to create a source distribution. In the
374simplest case,
375\begin{verbatim}
376python setup.py sdist
377\end{verbatim}
Greg Ward19c67f82000-06-24 01:33:16 +0000378(assuming you haven't specified any \command{sdist} options in the setup
379script or config file), \command{sdist} creates the archive of the
Greg Ward16aafcd2000-04-09 04:06:44 +0000380default format for the current platform. The default formats are:
381\begin{tableii}{ll}{textrm}%
382 {Platform}{Default archive format for source distributions}
383 \lineii{Unix}{gzipped tar file (\file{.tar.gz})}
384 \lineii{Windows}{zip file}
385\end{tableii}
Greg Wardd5767a52000-04-19 22:48:09 +0000386You can specify as many formats as you like using the
387\longprogramopt{formats} option, for example:
Greg Ward16aafcd2000-04-09 04:06:44 +0000388\begin{verbatim}
389python setup.py sdist --formats=gztar,zip
390\end{verbatim}
391to create a gzipped tarball and a zip file. The available formats are:
Greg Ward46b98e32000-04-14 01:53:36 +0000392\begin{tableiii}{l|l|c}{code}%
393 {Format}{Description}{Notes}
394 \lineiii{zip}{zip file (\file{.zip})}{(1)}
395 \lineiii{gztar}{gzipped tar file (\file{.tar.gz})}{(2)}
396 \lineiii{ztar}{compressed tar file (\file{.tar.Z})}{}
397 \lineiii{tar}{tar file (\file{.tar})}{}
398\end{tableiii}
399
400\noindent Notes:
401\begin{description}
402\item[(1)] default on Windows
403\item[(2)] default on Unix
404\end{description}
Greg Ward16aafcd2000-04-09 04:06:44 +0000405
406
407\subsection{The manifest and manifest template}
Greg Warde78298a2000-04-28 17:12:24 +0000408\label{manifest}
Greg Ward16aafcd2000-04-09 04:06:44 +0000409
410Without any additional information, the \command{sdist} command puts a
411minimal set of files into the source distribution:
412\begin{itemize}
Greg Wardfacb8db2000-04-09 04:32:40 +0000413\item all Python source files implied by the \option{py\_modules} and
Greg Ward16aafcd2000-04-09 04:06:44 +0000414 \option{packages} options
Greg Wardfacb8db2000-04-09 04:32:40 +0000415\item all C source files mentioned in the \option{ext\_modules} or
Greg Ward16aafcd2000-04-09 04:06:44 +0000416 \option{libraries} options (\XXX{getting C library sources currently
Greg Wardfacb8db2000-04-09 04:32:40 +0000417 broken -- no get\_source\_files() method in build\_clib.py!})
Greg Ward16aafcd2000-04-09 04:06:44 +0000418\item anything that looks like a test script: \file{test/test*.py}
419 (currently, the Distutils don't do anything with test scripts except
420 include them in source distributions, but in the future there will be
421 a standard for testing Python module distributions)
422\item \file{README.txt} (or \file{README}) and \file{setup.py}
423\end{itemize}
424Sometimes this is enough, but usually you will want to specify
425additional files to distribute. The typical way to do this is to write
426a \emph{manifest template}, called \file{MANIFEST.in} by default. The
427\command{sdist} command processes this template and generates a manifest
428file, \file{MANIFEST}. (If you prefer, you can skip the manifest
429template and generate the manifest yourself: it just lists one file per
430line.)
431
432The manifest template has one command per line, where each command
433specifies a set of files to include or exclude from the source
434distribution. For an example, again we turn to the Distutils' own
435manifest template:
436\begin{verbatim}
437include *.txt
Greg Ward87da1ea2000-04-21 04:35:25 +0000438recursive-include examples *.txt *.py
Greg Ward16aafcd2000-04-09 04:06:44 +0000439prune examples/sample?/build
440\end{verbatim}
441The meanings should be fairly clear: include all files in the
442distribution root matching \code{*.txt}, all files anywhere under the
443\file{examples} directory matching \code{*.txt} or \code{*.py}, and
444exclude all directories matching \code{examples/sample?/build}. There
445are several other commands available in the manifest template
Greg Warde78298a2000-04-28 17:12:24 +0000446mini-language; see section~\ref{sdist-cmd}.
Greg Ward16aafcd2000-04-09 04:06:44 +0000447
448The order of commands in the manifest template very much matters:
449initially, we have the list of default files as described above, and
450each command in the template adds to or removes from that list of files.
451When we have fully processed the manifest template, we have our complete
452list of files. This list is written to the manifest for future
453reference, and then used to build the source distribution archive(s).
454
Greg Ward46b98e32000-04-14 01:53:36 +0000455Following the Distutils' own manifest template, let's trace how the
456\command{sdist} command will build the list of files to include in the
457Distutils source distribution:
Greg Ward16aafcd2000-04-09 04:06:44 +0000458\begin{enumerate}
459\item include all Python source files in the \file{distutils} and
460 \file{distutils/command} subdirectories (because packages
461 corresponding to those two directories were mentioned in the
462 \option{packages} option in the setup script)
463\item include \file{test/test*.py} (always included)
464\item include \file{README.txt} and \file{setup.py} (always included)
465\item include \file{*.txt} in the distribution root (this will find
466 \file{README.txt} a second time, but such redundancies are weeded out
467 later)
468\item in the sub-tree under \file{examples}, include anything matching
469 \file{*.txt}
470\item in the sub-tree under \file{examples}, include anything matching
471 \file{*.py}
472\item remove all files in the sub-trees starting at directories matching
473 \file{examples/sample?/build}---this may exclude files included by the
474 previous two steps, so it's important that the \code{prune} command in
475 the manifest template comes after the two \code{recursive-include}
476 commands
Greg Wardfacb8db2000-04-09 04:32:40 +0000477\end{enumerate}
Greg Ward16aafcd2000-04-09 04:06:44 +0000478
Greg Ward46b98e32000-04-14 01:53:36 +0000479Just like in the setup script, file and directory names in the manifest
480template should always be slash-separated; the Distutils will take care
481of converting them to the standard representation on your platform.
482That way, the manifest template is portable across operating systems.
483
Greg Ward16aafcd2000-04-09 04:06:44 +0000484
485\subsection{Manifest-related options}
Greg Warde78298a2000-04-28 17:12:24 +0000486\label{manifest-options}
Greg Ward16aafcd2000-04-09 04:06:44 +0000487
488The normal course of operations for the \command{sdist} command is as
489follows:
490\begin{itemize}
Greg Ward46b98e32000-04-14 01:53:36 +0000491\item if the manifest file, \file{MANIFEST} doesn't exist, read
492 \file{MANIFEST.in} and create the manifest
493\item if \file{MANIFEST.in} is more recent than \file{MANIFEST},
494 recreate \file{MANIFEST} by reading \file{MANIFEST.in}
Greg Ward16aafcd2000-04-09 04:06:44 +0000495\item use the list of files now in \file{MANIFEST} (either just
496 generated or read in) to create the source distribution archive(s)
497\end{itemize}
498There are a couple of options that modify this behaviour.
499
500First, you might want to force the manifest to be regenerated---for
501example, if you have added or removed files or directories that match an
502existing pattern in the manifest template, you should regenerate the
503manifest:
504\begin{verbatim}
505python setup.py sdist --force-manifest
506\end{verbatim}
507\XXX{this is stupid, but is there a better way to do it without
508 reprocessing MANIFEST.in every single bloody time?}
509
510Or, you might just want to (re)generate the manifest, but not create a
511source distribution:
512\begin{verbatim}
513python setup.py sdist --manifest-only
514\end{verbatim}
Greg Warda021aca2000-04-19 22:34:11 +0000515(\longprogramopt{manifest-only} implies \longprogramopt{force-manifest}.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000516
517If you don't want to use the default file set, you can supply the
Greg Wardd5767a52000-04-19 22:48:09 +0000518\longprogramopt{no-defaults} option. If you use
519\longprogramopt{no-defaults} and don't supply a manifest template (or
520it's empty, or nothing matches the patterns in it), then your source
521distribution will be empty.
Greg Ward16aafcd2000-04-09 04:06:44 +0000522
523
524\section{Creating Built Distributions}
Greg Warde78298a2000-04-28 17:12:24 +0000525\label{built-dist}
Greg Ward16aafcd2000-04-09 04:06:44 +0000526
Greg Ward46b98e32000-04-14 01:53:36 +0000527A ``built distribution'' is what you're probably used to thinking of
528either as a ``binary package'' or an ``installer'' (depending on your
529background). It's not necessarily binary, though, because it might
530contain only Python source code and/or byte-code; and we don't call it a
531package, because that word is already spoken for in Python. (And
532``installer'' is a term specific to the Windows world. \XXX{do Mac
533 people use it?})
Greg Ward16aafcd2000-04-09 04:06:44 +0000534
Greg Ward46b98e32000-04-14 01:53:36 +0000535A built distribution is how you make life as easy as possible for
536installers of your module distribution: for users of RPM-based Linux
537systems, it's a binary RPM; for Windows users, it's an executable
538installer; for Debian-based Linux users, it's a Debian package; and so
539forth. Obviously, no one person will be able to create built
540distributions for every platform under the sun, so the Distutils is
541designed to enable module developers to concentrate on their
542specialty---writing code and creating source distributions---while an
543intermediary species of \emph{packager} springs up to turn source
Greg Ward19c67f82000-06-24 01:33:16 +0000544distributions into built distributions for as many platforms as there
Greg Ward46b98e32000-04-14 01:53:36 +0000545are packagers.
546
547Of course, the module developer could be his own packager; or the
548packager could be a volunteer ``out there'' somewhere who has access to
549a platform which the original developer does not; or it could be
550software periodically grabbing new source distributions and turning them
551into built distributions for as many platforms as the software has
552access to. Regardless of the nature of the beast, a packager uses the
553setup script and the \command{bdist} command family to generate built
554distributions.
555
556As a simple example, if I run the following command in the Distutils
557source tree:
558\begin{verbatim}
559python setup.py bdist
560\end{verbatim}
561then the Distutils builds my module distribution (the Distutils itself
562in this case), does a ``fake'' installation (also in the \file{build}
563directory), and creates the default type of built distribution for my
564platform. In Distutils 0.8, only two types of built distribution are
565supported: \code{gztar} (default on non-Linux Unix) and \code{zip}
566(default on Windows). Thus, the above command on a Unix system creates
567\file{Distutils-0.8.built-posix.tar.gz}; unpacking this tarball from
568Python's \filevar{prefix} directory installs the Distutils just as
569though you had downloaded the source distribution and run \code{python
570 setup.py install}. Obviously, for pure Python distributions, this
571isn't a huge win---but for non-pure distributions, which include
572extensions that would need to be compiled, it can mean the difference
573between someone being able to use your extensions or not.
574
575\XXX{filenames are inaccurate here!}
576
Greg Wardd5767a52000-04-19 22:48:09 +0000577The \command{bdist} command has a \longprogramopt{format} option,
578similar to the \command{sdist} command, that you can use to select which
579formats to generate: for example,
Greg Ward46b98e32000-04-14 01:53:36 +0000580\begin{verbatim}
581python setup.py bdist --format=zip
582\end{verbatim}
583would, when run on a Unix system, create
584\file{Distutils-0.8.built-posix.tar.gz}---again, this archive would be
585unpacked from Python's \filevar{prefix} directory to install the
586Distutils.
587
588The available formats for built distributions are:
589\begin{tableiii}{l|l|c}{code}%
590 {Format}{Description}{Notes}
591 \lineiii{zip}{zip file (\file{.zip})}{(1)}
592 \lineiii{gztar}{gzipped tar file (\file{.tar.gz})}{(2)}
593 \lineiii{ztar}{compressed tar file (\file{.tar.Z})}{}
594 \lineiii{tar}{tar file (\file{.tar})}{}
595 \lineiii{rpm}{RPM}{(3)}
596 \lineiii{srpm}{source RPM}{}
597 \lineiii{wise}{Wise installer for Windows}{}
598\end{tableiii}
599
600\noindent Notes:
601\begin{description}
602\item[(1)] default on Windows
603\item[(2)] default on Unix
604\item[(3)] not implemented yet; will be default on RPM-based Linux
605 systems
606\item[(5)] not implemented yet; will be default on Windows
607\end{description}
608
609You don't have to use the \command{bdist} command with the
Greg Wardd5767a52000-04-19 22:48:09 +0000610\longprogramopt{formats} option; you can also use the command that
611directly implements the format you're interested in. Many of these
Greg Ward46b98e32000-04-14 01:53:36 +0000612\command{bdist} ``sub-commands'' actually generate several similar
613formats; for instance, the \command{bdist\_dumb} command generates all
614the ``dumb'' archive formats (\code{tar}, \code{ztar}, \code{gztar}, and
615\code{zip}), and \command{bdist\_rpm} generates both binary and source
616RPMs. The \command{bdist} sub-commands, and the formats generated by
617each, are:
618\begin{tableii}{l|l}{command}%
619 {Command}{Formats}
620 \lineii{bdist\_dumb}{tar, ztar, gztar, zip}
621 \lineii{bdist\_rpm}{rpm, srpm}
622 \lineii{bdist\_wise}{wise}
623\end{tableii}
Greg Ward16aafcd2000-04-09 04:06:44 +0000624
625\section{Examples}
Greg Warde78298a2000-04-28 17:12:24 +0000626\label{examples}
Greg Ward16aafcd2000-04-09 04:06:44 +0000627
628
629\subsection{Pure Python distribution (by module)}
Greg Warde78298a2000-04-28 17:12:24 +0000630\label{pure-mod}
Greg Ward16aafcd2000-04-09 04:06:44 +0000631
632
633\subsection{Pure Python distribution (by package)}
Greg Warde78298a2000-04-28 17:12:24 +0000634\label{pure-pkg}
Greg Ward16aafcd2000-04-09 04:06:44 +0000635
636
637\subsection{Single extension module}
Greg Warde78298a2000-04-28 17:12:24 +0000638\label{single-ext}
Greg Ward16aafcd2000-04-09 04:06:44 +0000639
640
641\subsection{Multiple extension modules}
Greg Warde78298a2000-04-28 17:12:24 +0000642\label{multiple-ext}
Greg Ward16aafcd2000-04-09 04:06:44 +0000643
644
645\subsection{Putting it all together}
646
647
Greg Ward4a9e7222000-04-25 02:57:36 +0000648
649\section{Extending the Distutils}
Greg Warde78298a2000-04-28 17:12:24 +0000650\label{extending}
Greg Ward4a9e7222000-04-25 02:57:36 +0000651
652
653\subsection{Extending existing commands}
Greg Warde78298a2000-04-28 17:12:24 +0000654\label{extend-existing}
Greg Ward4a9e7222000-04-25 02:57:36 +0000655
656
657\subsection{Writing new commands}
Greg Warde78298a2000-04-28 17:12:24 +0000658\label{new-commands}
Greg Ward4a9e7222000-04-25 02:57:36 +0000659
660
661
Greg Ward16aafcd2000-04-09 04:06:44 +0000662\section{Reference}
Greg Warde78298a2000-04-28 17:12:24 +0000663\label{ref}
Greg Ward16aafcd2000-04-09 04:06:44 +0000664
665
Greg Wardfacb8db2000-04-09 04:32:40 +0000666\subsection{Building modules: the \protect\command{build} command family}
Greg Warde78298a2000-04-28 17:12:24 +0000667\label{build-cmds}
Greg Ward16aafcd2000-04-09 04:06:44 +0000668
Greg Wardfacb8db2000-04-09 04:32:40 +0000669\subsubsection{\protect\command{build}}
Greg Warde78298a2000-04-28 17:12:24 +0000670\label{build-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000671
Greg Wardfacb8db2000-04-09 04:32:40 +0000672\subsubsection{\protect\command{build\_py}}
Greg Warde78298a2000-04-28 17:12:24 +0000673\label{build-py-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000674
Greg Wardfacb8db2000-04-09 04:32:40 +0000675\subsubsection{\protect\command{build\_ext}}
Greg Warde78298a2000-04-28 17:12:24 +0000676\label{build-ext-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000677
Greg Wardfacb8db2000-04-09 04:32:40 +0000678\subsubsection{\protect\command{build\_clib}}
Greg Warde78298a2000-04-28 17:12:24 +0000679\label{build-clib-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000680
681
Greg Wardfacb8db2000-04-09 04:32:40 +0000682\subsection{Installing modules: the \protect\command{install} command family}
Greg Warde78298a2000-04-28 17:12:24 +0000683\label{install-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000684
Gregory P. Smith147e5f32000-05-12 00:58:18 +0000685The install command ensures that the build commands have been run and then
686runs the subcommands \command{install\_lib},
687\command{install\_data} and
688\command{install\_scripts}.
689
690\subsubsection{\protect\command{install\_lib}}
691\label{sec:install-lib-cmd}
692
693\subsubsection{\protect\command{install\_data}}
694\label{sec:install-data-cmd}
695This command installs all data files provided with the distribution.
696
697\subsubsection{\protect\command{install\_scripts}}
698\label{sec:install-scripts-cmd}
699This command installs all (Python) scripts in the distribution.
700
Greg Ward16aafcd2000-04-09 04:06:44 +0000701
Greg Wardfacb8db2000-04-09 04:32:40 +0000702\subsection{Cleaning up: the \protect\command{clean} command}
Greg Warde78298a2000-04-28 17:12:24 +0000703\label{clean-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000704
705
Greg Wardfacb8db2000-04-09 04:32:40 +0000706\subsection{Creating a source distribution: the \protect\command{sdist} command}
Greg Warde78298a2000-04-28 17:12:24 +0000707\label{sdist-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000708
709
710\XXX{fragment moved down from above: needs context!}
711The manifest template commands are:
712\begin{tableii}{ll}{command}{Command}{Description}
Greg Ward87da1ea2000-04-21 04:35:25 +0000713 \lineii{include \var{pat1} \var{pat2} ... }
714 {include all files matching any of the listed patterns}
715 \lineii{exclude \var{pat1} \var{pat2} ... }
716 {exclude all files matching any of the listed patterns}
717 \lineii{recursive-include \var{dir} \var{pat1} \var{pat2} ... }
718 {include all files under \var{dir} matching any of the listed patterns}
719 \lineii{recursive-exclude \var{dir} \var{pat1} \var{pat2} ...}
720 {exclude all files under \var{dir} matching any of the listed patterns}
721 \lineii{global-include \var{pat1} \var{pat2} ...}
Greg Ward1bbe3292000-06-25 03:14:13 +0000722 {include all files anywhere in the source tree matching\\&
Greg Ward87da1ea2000-04-21 04:35:25 +0000723 any of the listed patterns}
724 \lineii{global-exclude \var{pat1} \var{pat2} ...}
Greg Ward1bbe3292000-06-25 03:14:13 +0000725 {exclude all files anywhere in the source tree matching\\&
Greg Ward87da1ea2000-04-21 04:35:25 +0000726 any of the listed patterns}
Greg Ward16aafcd2000-04-09 04:06:44 +0000727 \lineii{prune \var{dir}}{exclude all files under \var{dir}}
728 \lineii{graft \var{dir}}{include all files under \var{dir}}
729\end{tableii}
730The patterns here are Unix-style ``glob'' patterns: \code{*} matches any
731sequence of regular filename characters, \code{?} matches any single
732regular filename character, and \code{[\var{range}]} matches any of the
733characters in \var{range} (e.g., \code{a-z}, \code{a-zA-Z},
Greg Wardfacb8db2000-04-09 04:32:40 +0000734\code{a-f0-9\_.}). The definition of ``regular filename character'' is
Greg Ward16aafcd2000-04-09 04:06:44 +0000735platform-specific: on Unix it is anything except slash; on Windows
736anything except backslash or colon; on Mac OS anything except colon.
737\XXX{Windows and Mac OS support not there yet}
738
739
Greg Wardd5767a52000-04-19 22:48:09 +0000740\subsection{Creating a ``built'' distribution: the
741 \protect\command{bdist} command family}
Greg Warde78298a2000-04-28 17:12:24 +0000742\label{bdist-cmds}
Greg Ward16aafcd2000-04-09 04:06:44 +0000743
744
Greg Wardfacb8db2000-04-09 04:32:40 +0000745\subsubsection{\protect\command{blib}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000746
Greg Wardfacb8db2000-04-09 04:32:40 +0000747\subsubsection{\protect\command{blib\_dumb}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000748
Greg Wardfacb8db2000-04-09 04:32:40 +0000749\subsubsection{\protect\command{blib\_rpm}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000750
Greg Wardfacb8db2000-04-09 04:32:40 +0000751\subsubsection{\protect\command{blib\_wise}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000752
753
754
755
756
757
758
759
Greg Wardabc52162000-02-26 00:52:48 +0000760\end{document}