blob: e7d9f449eb7d0567e179360de76fd881ea816ce1 [file] [log] [blame]
Greg Wardabc52162000-02-26 00:52:48 +00001\documentclass{howto}
2\usepackage{ltxmarkup}
Greg Wardfacb8db2000-04-09 04:32:40 +00003\usepackage{times}
Greg Ward16aafcd2000-04-09 04:06:44 +00004\usepackage{distutils}
Greg Wardabc52162000-02-26 00:52:48 +00005
Greg Ward16aafcd2000-04-09 04:06:44 +00006\title{Distributing Python Modules}
Greg Wardabc52162000-02-26 00:52:48 +00007
Greg Wardabc52162000-02-26 00:52:48 +00008\author{Greg Ward}
9\authoraddress{E-mail: \email{gward@python.net}}
10
Greg Ward16aafcd2000-04-09 04:06:44 +000011
Greg Wardabc52162000-02-26 00:52:48 +000012\begin{document}
13
Greg Wardfacb8db2000-04-09 04:32:40 +000014\maketitle
15\tableofcontents
Greg Ward16aafcd2000-04-09 04:06:44 +000016
17\section{Introduction}
Greg Warde78298a2000-04-28 17:12:24 +000018\label{intro}
Greg Ward16aafcd2000-04-09 04:06:44 +000019
20In the past, Python module developers have not had much infrastructure
21support for distributing modules, nor have Python users had much support
22for installing and maintaining third-party modules. With the
23introduction of the Python Distribution Utilities (Distutils for short)
Greg Ward1d8f57a2000-08-05 00:43:11 +000024in Python 1.6, this situation should start to improve.
Greg Ward16aafcd2000-04-09 04:06:44 +000025
26This document only covers using the Distutils to distribute your Python
Greg Ward1d8f57a2000-08-05 00:43:11 +000027modules. Using the Distutils does not tie you to Python 1.6, though:
28the Distutils work just fine with Python 1.5.2, and it is reasonable
29(and expected to become commonplace) to expect users of Python 1.5.2 to
Greg Ward16aafcd2000-04-09 04:06:44 +000030download and install the Distutils separately before they can install
Greg Ward1d8f57a2000-08-05 00:43:11 +000031your modules. Python 1.6 (or later) users, of course, won't have to add
32anything to their Python installation in order to use the Distutils to
33install third-party modules.
Greg Ward16aafcd2000-04-09 04:06:44 +000034
35This document concentrates on the role of developer/distributor: if
Fred Drake01df4532000-06-30 03:36:41 +000036you're looking for information on installing Python modules, you
37should refer to the \citetitle[../inst/inst.html]{Installing Python
38Modules} manual.
Greg Ward16aafcd2000-04-09 04:06:44 +000039
40
Greg Wardfacb8db2000-04-09 04:32:40 +000041\section{Concepts \& Terminology}
Greg Warde78298a2000-04-28 17:12:24 +000042\label{concepts}
Greg Ward16aafcd2000-04-09 04:06:44 +000043
44Using the Distutils is quite simple, both for module developers and for
45users/administrators installing third-party modules. As a developer,
46your responsibilites (apart from writing solid, well-documented and
47well-tested code, of course!) are:
48\begin{itemize}
49\item write a setup script (\file{setup.py} by convention)
50\item (optional) write a setup configuration file
51\item create a source distribution
52\item (optional) create one or more built (binary) distributions
53\end{itemize}
54Each of these tasks is covered in this document.
55
56Not all module developers have access to a multitude of platforms, so
57it's not always feasible to expect them to create a multitude of built
58distributions. It is hoped that a class of intermediaries, called
Greg Ward19c67f82000-06-24 01:33:16 +000059\emph{packagers}, will arise to address this need. Packagers will take
60source distributions released by module developers, build them on one or
61more platforms, and release the resulting built distributions. Thus,
62users on the most popular platforms will be able to install most popular
63Python module distributions in the most natural way for their platform,
64without having to run a single setup script or compile a line of code.
Greg Ward16aafcd2000-04-09 04:06:44 +000065
66
67\subsection{A simple example}
Greg Warde78298a2000-04-28 17:12:24 +000068\label{simple-example}
Greg Ward16aafcd2000-04-09 04:06:44 +000069
70The setup script is usually quite simple, although since it's written in
Greg Ward1d8f57a2000-08-05 00:43:11 +000071Python, there are no arbitrary limits to what you can do with it. If
72all you want to do is distribute a module called \module{foo}, contained
73in a file \file{foo.py}, then your setup script can be as little as
74this:
Greg Ward16aafcd2000-04-09 04:06:44 +000075\begin{verbatim}
76from distutils.core import setup
77setup (name = "foo",
78 version = "1.0",
79 py_modules = ["foo"])
80\end{verbatim}
Greg Ward370248d2000-06-24 01:45:47 +000081
Greg Ward16aafcd2000-04-09 04:06:44 +000082Some observations:
83\begin{itemize}
Greg Ward370248d2000-06-24 01:45:47 +000084\item most information that you supply to the Distutils is supplied as
Greg Wardfacb8db2000-04-09 04:32:40 +000085 keyword arguments to the \function{setup()} function
Greg Ward16aafcd2000-04-09 04:06:44 +000086\item those keyword arguments fall into two categories: package
87 meta-data (name, version number) and information about what's in the
Greg Ward370248d2000-06-24 01:45:47 +000088 package (a list of pure Python modules, in this case)
Greg Ward16aafcd2000-04-09 04:06:44 +000089\item modules are specified by module name, not filename (the same will
90 hold true for packages and extensions)
91\item it's recommended that you supply a little more meta-data, in
92 particular your name, email address and a URL for the project
93\end{itemize}
94
Greg Ward370248d2000-06-24 01:45:47 +000095To create a source distribution for this module, you would create a
96setup script, \file{setup.py}, containing the above code, and run:
Greg Ward16aafcd2000-04-09 04:06:44 +000097\begin{verbatim}
98python setup.py sdist
99\end{verbatim}
100which will create an archive file (e.g., tarball on Unix, zip file on
101Windows) containing your setup script, \file{setup.py}, and your module,
102\file{foo.py}. The archive file will be named \file{Foo-1.0.tar.gz} (or
103\file{.zip}), and will unpack into a directory \file{Foo-1.0}.
104
105If an end-user wishes to install your \module{foo} module, all she has
Greg Ward59d382e2000-05-26 01:04:47 +0000106to do is download \file{Foo-1.0.tar.gz} (or \file{.zip}), unpack it,
Greg Ward16aafcd2000-04-09 04:06:44 +0000107and---from the \file{Foo-1.0} directory---run
108\begin{verbatim}
109python setup.py install
110\end{verbatim}
111which will ultimately copy \file{foo.py} to the appropriate directory
112for third-party modules in their Python installation.
113
114This simple example demonstrates some fundamental concepts of the
115Distutils: first, both developers and installers have the same basic
116user interface, i.e. the setup script. The difference is which
117Distutils \emph{commands} they use: the \command{sdist} command is
118almost exclusively for module developers, while \command{install} is
119more often for installers (although most developers will want to install
120their own code occasionally).
121
Greg Ward16aafcd2000-04-09 04:06:44 +0000122If you want to make things really easy for your users, you can create
123one or more built distributions for them. For instance, if you are
124running on a Windows machine, and want to make things easy for other
125Windows users, you can create an executable installer (the most
126appropriate type of built distribution for this platform) with the
Greg Ward59d382e2000-05-26 01:04:47 +0000127\command{bdist\_wininst} command. For example:
Greg Ward16aafcd2000-04-09 04:06:44 +0000128\begin{verbatim}
Greg Ward59d382e2000-05-26 01:04:47 +0000129python setup.py bdist_wininst
Greg Ward16aafcd2000-04-09 04:06:44 +0000130\end{verbatim}
Greg Ward1d8f57a2000-08-05 00:43:11 +0000131will create an executable installer, \file{Foo-1.0.win32.exe}, in the
132current directory.
Greg Ward16aafcd2000-04-09 04:06:44 +0000133
Greg Ward1d8f57a2000-08-05 00:43:11 +0000134\XXX{not implemented yet}
Greg Ward59d382e2000-05-26 01:04:47 +0000135(Another way to create executable installers for Windows is with the
136\command{bdist\_wise} command, which uses Wise---the commercial
137installer-generator used to create Python's own installer---to create
138the installer. Wise-based installers are more appropriate for large,
139industrial-strength applications that need the full capabilities of a
140``real'' installer. \command{bdist\_wininst} creates a self-extracting
141zip file with a minimal user interface, which is enough for small- to
142medium-sized module collections. You'll need to have version XXX of
Greg Ward370248d2000-06-24 01:45:47 +0000143Wise installed on your system for the \command{bdist\_wise} command to
144work; it's available from \url{http://foo/bar/baz}.)
Greg Ward59d382e2000-05-26 01:04:47 +0000145
Greg Ward1d8f57a2000-08-05 00:43:11 +0000146Currently (Distutils 0.9.1), the are only other useful built
147distribution format is RPM, implemented by the \command{bdist\_rpm}
148command. For example, the following command will create an RPM file
149called \file{Foo-1.0.noarch.rpm}:
150\begin{verbatim}
151python setup.py bdist_rpm
152\end{verbatim}
153(This uses the \command{rpm} command, so has to be run on an RPM-based
154system such as Red Hat Linux, SuSE Linux, or Mandrake Linux.)
155
156You can find out what distribution formats are available at any time by
157running
158\begin{verbatim}
159python setup.py bdist --help-formats
160\end{verbatim}
Greg Ward16aafcd2000-04-09 04:06:44 +0000161
162
163\subsection{General Python terminology}
Greg Warde78298a2000-04-28 17:12:24 +0000164\label{python-terms}
Greg Ward16aafcd2000-04-09 04:06:44 +0000165
166If you're reading this document, you probably have a good idea of what
167modules, extensions, and so forth are. Nevertheless, just to be sure
168that everyone is operating from a common starting point, we offer the
169following glossary of common Python terms:
170\begin{description}
171\item[module] the basic unit of code reusability in Python: a block of
Greg Ward1d8f57a2000-08-05 00:43:11 +0000172 code imported by some other code. Three types of modules concern us
173 here: pure Python modules, extension modules, and packages.
Greg Ward16aafcd2000-04-09 04:06:44 +0000174\item[pure Python module] a module written in Python and contained in a
175 single \file{.py} file (and possibly associated \file{.pyc} and/or
176 \file{.pyo} files). Sometimes referred to as a ``pure module.''
177\item[extension module] a module written in the low-level language of
178 the Python implemention: C/C++ for CPython, Java for JPython.
179 Typically contained in a single dynamically loadable pre-compiled
180 file, e.g. a shared object (\file{.so}) file for CPython extensions on
181 Unix, a DLL (given the \file{.pyd} extension) for CPython extensions
Greg Ward1bbe3292000-06-25 03:14:13 +0000182 on Windows, or a Java class file for JPython extensions. (Note that
183 currently, the Distutils only handles C/C++ extensions for CPython.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000184\item[package] a module that contains other modules; typically contained
185 in a directory in the filesystem and distinguished from other
186 directories by the presence of a file \file{\_\_init\_\_.py}.
Greg Ward6153fa12000-05-26 02:24:28 +0000187\item[root package] the root of the hierarchy of packages. (This isn't
188 really a package, since it doesn't have an \file{\_\_init\_\_.py}
189 file. But we have to call it something.) The vast majority of the
190 standard library is in the root package, as are many small, standalone
191 third-party modules that don't belong to a larger module collection.
192 Unlike regular packages, modules in the root package can be found in
193 many directories: in fact, every directory listed in \code{sys.path}
194 can contribute modules to the root package.
Greg Ward16aafcd2000-04-09 04:06:44 +0000195\end{description}
196
197
198\subsection{Distutils-specific terminology}
Greg Warde78298a2000-04-28 17:12:24 +0000199\label{distutils-term}
Greg Ward16aafcd2000-04-09 04:06:44 +0000200
201The following terms apply more specifically to the domain of
202distributing Python modules using the Distutils:
203\begin{description}
204\item[module distribution] a collection of Python modules distributed
205 together as a single downloadable resource and meant to be installed
206 \emph{en masse}. Examples of some well-known module distributions are
207 Numeric Python, PyXML, PIL (the Python Imaging Library), or
208 mxDateTime. (This would be called a \emph{package}, except that term
Greg Ward59d382e2000-05-26 01:04:47 +0000209 is already taken in the Python context: a single module distribution
210 may contain zero, one, or many Python packages.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000211\item[pure module distribution] a module distribution that contains only
212 pure Python modules and packages. Sometimes referred to as a ``pure
213 distribution.''
214\item[non-pure module distribution] a module distribution that contains
215 at least one extension module. Sometimes referred to as a ``non-pure
216 distribution.''
217\item[distribution root] the top-level directory of your source tree (or
218 source distribution); the directory where \file{setup.py} exists and
219 is run from
220\end{description}
221
222
223\section{Writing the Setup Script}
Greg Warde78298a2000-04-28 17:12:24 +0000224\label{setup-script}
Greg Ward16aafcd2000-04-09 04:06:44 +0000225
226The setup script is the centre of all activity in building,
227distributing, and installing modules using the Distutils. The main
228purpose of the setup script is to describe your module distribution to
Greg Wardd5767a52000-04-19 22:48:09 +0000229the Distutils, so that the various commands that operate on your modules
Greg Ward59d382e2000-05-26 01:04:47 +0000230do the right thing. As we saw in section~\ref{simple-example} above,
231the setup script consists mainly of a call to \function{setup()}, and
Greg Ward1bbe3292000-06-25 03:14:13 +0000232most information supplied to the Distutils by the module developer is
233supplied as keyword arguments to \function{setup()}.
Greg Ward16aafcd2000-04-09 04:06:44 +0000234
235Here's a slightly more involved example, which we'll follow for the next
236couple of sections: the Distutils' own setup script. (Keep in mind that
Greg Ward1d8f57a2000-08-05 00:43:11 +0000237although the Distutils are included with Python 1.6 and later, they also
238have an independent existence so that Python 1.5.2 users can use them to
239install other module distributions. The Distutils' own setup script,
240shown here, is used to install the package into Python 1.5.2.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000241
242\begin{verbatim}
243#!/usr/bin/env python
244
245from distutils.core import setup
246
247setup (name = "Distutils",
248 version = "1.0",
Greg Ward1d8f57a2000-08-05 00:43:11 +0000249 description = "Python Distribution Utilities",
Greg Ward16aafcd2000-04-09 04:06:44 +0000250 author = "Greg Ward",
251 author_email = "gward@python.net",
252 url = "http://www.python.org/sigs/distutils-sig/",
253
254 packages = ['distutils', 'distutils.command'],
255 )
256\end{verbatim}
257There are only two differences between this and the trivial one-file
Greg Warde78298a2000-04-28 17:12:24 +0000258distribution presented in section~\ref{simple-example}: more
Greg Ward16aafcd2000-04-09 04:06:44 +0000259meta-data, and the specification of pure Python modules by package,
260rather than by module. This is important since the Distutils consist of
261a couple of dozen modules split into (so far) two packages; an explicit
262list of every module would be tedious to generate and difficult to
263maintain.
264
Greg Ward46b98e32000-04-14 01:53:36 +0000265Note that any pathnames (files or directories) supplied in the setup
266script should be written using the Unix convention, i.e.
267slash-separated. The Distutils will take care of converting this
Greg Ward59d382e2000-05-26 01:04:47 +0000268platform-neutral representation into whatever is appropriate on your
Greg Ward46b98e32000-04-14 01:53:36 +0000269current platform before actually using the pathname. This makes your
270setup script portable across operating systems, which of course is one
271of the major goals of the Distutils. In this spirit, all pathnames in
Greg Ward59d382e2000-05-26 01:04:47 +0000272this document are slash-separated (Mac OS programmers should keep in
273mind that the \emph{absence} of a leading slash indicates a relative
274path, the opposite of the Mac OS convention with colons).
Greg Ward46b98e32000-04-14 01:53:36 +0000275
Greg Ward16aafcd2000-04-09 04:06:44 +0000276
277\subsection{Package directories}
Greg Warde78298a2000-04-28 17:12:24 +0000278\label{package-dirs}
Greg Ward16aafcd2000-04-09 04:06:44 +0000279
280The \option{packages} option tells the Distutils to process (build,
281distribute, install, etc.) all pure Python modules found in each package
282mentioned in the \option{packages} list. In order to do this, of
283course, there has to be a correspondence between package names and
284directories in the filesystem. The default correspondence is the most
Greg Ward1ecc2512000-04-19 22:36:24 +0000285obvious one, i.e. package \module{distutils} is found in the directory
Greg Ward16aafcd2000-04-09 04:06:44 +0000286\file{distutils} relative to the distribution root. Thus, when you say
287\code{packages = ['foo']} in your setup script, you are promising that
288the Distutils will find a file \file{foo/\_\_init\_\_.py} (which might
289be spelled differently on your system, but you get the idea) relative to
290the directory where your setup script lives. (If you break this
291promise, the Distutils will issue a warning but process the broken
292package anyways.)
293
294If you use a different convention to lay out your source directory,
295that's no problem: you just have to supply the \option{package\_dir}
296option to tell the Distutils about your convention. For example, say
Greg Ward1d8f57a2000-08-05 00:43:11 +0000297you keep all Python source under \file{lib}, so that modules in the
298``root package'' (i.e., not in any package at all) are right in
299\file{lib}, modules in the \module{foo} package are in \file{lib/foo},
300and so forth. Then you would put
Greg Ward16aafcd2000-04-09 04:06:44 +0000301\begin{verbatim}
302package_dir = {'': 'lib'}
303\end{verbatim}
304in your setup script. (The keys to this dictionary are package names,
Greg Ward1d8f57a2000-08-05 00:43:11 +0000305and an empty package name stands for the root package. The values are
306directory names relative to your distribution root.) In this case, when
307you say \code{packages = ['foo']}, you are promising that the file
Greg Ward16aafcd2000-04-09 04:06:44 +0000308\file{lib/foo/\_\_init\_\_.py} exists.
309
Greg Ward1ecc2512000-04-19 22:36:24 +0000310Another possible convention is to put the \module{foo} package right in
311\file{lib}, the \module{foo.bar} package in \file{lib/bar}, etc. This
Greg Ward16aafcd2000-04-09 04:06:44 +0000312would be written in the setup script as
313\begin{verbatim}
314package_dir = {'foo': 'lib'}
315\end{verbatim}
Greg Ward59d382e2000-05-26 01:04:47 +0000316A \code{\var{package}: \var{dir}} entry in the \option{package\_dir}
317dictionary implicitly applies to all packages below \var{package}, so
318the \module{foo.bar} case is automatically handled here. In this
319example, having \code{packages = ['foo', 'foo.bar']} tells the Distutils
320to look for \file{lib/\_\_init\_\_.py} and
321\file{lib/bar/\_\_init\_\_.py}. (Keep in mind that although
322\option{package\_dir} applies recursively, you must explicitly list all
323packages in \option{packages}: the Distutils will \emph{not} recursively
324scan your source tree looking for any directory with an
325\file{\_\_init\_\_.py} file.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000326
327
328\subsection{Listing individual modules}
Greg Warde78298a2000-04-28 17:12:24 +0000329\label{listing-modules}
Greg Ward16aafcd2000-04-09 04:06:44 +0000330
331For a small module distribution, you might prefer to list all modules
332rather than listing packages---especially the case of a single module
333that goes in the ``root package'' (i.e., no package at all). This
Greg Warde78298a2000-04-28 17:12:24 +0000334simplest case was shown in section~\ref{simple-example}; here is a
Greg Ward16aafcd2000-04-09 04:06:44 +0000335slightly more involved example:
336\begin{verbatim}
337py_modules = ['mod1', 'pkg.mod2']
338\end{verbatim}
339This describes two modules, one of them in the ``root'' package, the
Greg Wardd5767a52000-04-19 22:48:09 +0000340other in the \module{pkg} package. Again, the default package/directory
341layout implies that these two modules can be found in \file{mod1.py} and
342\file{pkg/mod2.py}, and that \file{pkg/\_\_init\_\_.py} exists as well.
343And again, you can override the package/directory layout using the
Greg Ward59d382e2000-05-26 01:04:47 +0000344\option{package\_dir} option.
345
346
347\subsection{Describing extension modules}
348\label{sec:describing-extensions}
349
Greg Ward16aafcd2000-04-09 04:06:44 +0000350
351
352\section{Writing the Setup Configuration File}
Greg Warde78298a2000-04-28 17:12:24 +0000353\label{setup-config}
Greg Ward16aafcd2000-04-09 04:06:44 +0000354
Greg Ward16aafcd2000-04-09 04:06:44 +0000355Often, it's not possible to write down everything needed to build a
356distribution \emph{a priori}. You need to get some information from the
357user, or from the user's system, in order to proceed. For example, you
358might include an optional extension module that provides an interface to
359a particular C library. If that library is installed on the user's
360system, then you can build your optional extension---but you need to
361know where to find the header and library file. If it's not installed,
362you need to know this so you can omit your optional extension.
363
364The preferred way to do this, of course, would be for you to tell the
365Distutils which optional features (C libraries, system calls, external
366utilities, etc.) you're looking for, and it would inspect the user's
367system and try to find them. This functionality may appear in a future
368version of the Distutils, but it isn't there now. So, for the time
369being, we rely on the user building and installing your software to
370provide the necessary information. The vehicle for doing so is the
371setup configuration file, \file{setup.cfg}.
372
373\XXX{need more here!}
374
375
376\section{Creating a Source Distribution}
Greg Warde78298a2000-04-28 17:12:24 +0000377\label{source-dist}
Greg Ward16aafcd2000-04-09 04:06:44 +0000378
Greg Warde78298a2000-04-28 17:12:24 +0000379As shown in section~\ref{simple-example}, you use the
Greg Ward16aafcd2000-04-09 04:06:44 +0000380\command{sdist} command to create a source distribution. In the
381simplest case,
382\begin{verbatim}
383python setup.py sdist
384\end{verbatim}
Greg Ward19c67f82000-06-24 01:33:16 +0000385(assuming you haven't specified any \command{sdist} options in the setup
386script or config file), \command{sdist} creates the archive of the
Greg Ward16aafcd2000-04-09 04:06:44 +0000387default format for the current platform. The default formats are:
388\begin{tableii}{ll}{textrm}%
389 {Platform}{Default archive format for source distributions}
390 \lineii{Unix}{gzipped tar file (\file{.tar.gz})}
391 \lineii{Windows}{zip file}
392\end{tableii}
Greg Wardd5767a52000-04-19 22:48:09 +0000393You can specify as many formats as you like using the
394\longprogramopt{formats} option, for example:
Greg Ward16aafcd2000-04-09 04:06:44 +0000395\begin{verbatim}
396python setup.py sdist --formats=gztar,zip
397\end{verbatim}
398to create a gzipped tarball and a zip file. The available formats are:
Greg Ward46b98e32000-04-14 01:53:36 +0000399\begin{tableiii}{l|l|c}{code}%
400 {Format}{Description}{Notes}
401 \lineiii{zip}{zip file (\file{.zip})}{(1)}
402 \lineiii{gztar}{gzipped tar file (\file{.tar.gz})}{(2)}
403 \lineiii{ztar}{compressed tar file (\file{.tar.Z})}{}
404 \lineiii{tar}{tar file (\file{.tar})}{}
405\end{tableiii}
406
407\noindent Notes:
408\begin{description}
409\item[(1)] default on Windows
410\item[(2)] default on Unix
411\end{description}
Greg Ward16aafcd2000-04-09 04:06:44 +0000412
413
414\subsection{The manifest and manifest template}
Greg Warde78298a2000-04-28 17:12:24 +0000415\label{manifest}
Greg Ward16aafcd2000-04-09 04:06:44 +0000416
417Without any additional information, the \command{sdist} command puts a
418minimal set of files into the source distribution:
419\begin{itemize}
Greg Wardfacb8db2000-04-09 04:32:40 +0000420\item all Python source files implied by the \option{py\_modules} and
Greg Ward16aafcd2000-04-09 04:06:44 +0000421 \option{packages} options
Greg Wardfacb8db2000-04-09 04:32:40 +0000422\item all C source files mentioned in the \option{ext\_modules} or
Greg Ward16aafcd2000-04-09 04:06:44 +0000423 \option{libraries} options (\XXX{getting C library sources currently
Greg Wardfacb8db2000-04-09 04:32:40 +0000424 broken -- no get\_source\_files() method in build\_clib.py!})
Greg Ward16aafcd2000-04-09 04:06:44 +0000425\item anything that looks like a test script: \file{test/test*.py}
426 (currently, the Distutils don't do anything with test scripts except
427 include them in source distributions, but in the future there will be
428 a standard for testing Python module distributions)
429\item \file{README.txt} (or \file{README}) and \file{setup.py}
430\end{itemize}
431Sometimes this is enough, but usually you will want to specify
432additional files to distribute. The typical way to do this is to write
433a \emph{manifest template}, called \file{MANIFEST.in} by default. The
434\command{sdist} command processes this template and generates a manifest
435file, \file{MANIFEST}. (If you prefer, you can skip the manifest
436template and generate the manifest yourself: it just lists one file per
437line.)
438
439The manifest template has one command per line, where each command
440specifies a set of files to include or exclude from the source
441distribution. For an example, again we turn to the Distutils' own
442manifest template:
443\begin{verbatim}
444include *.txt
Greg Ward87da1ea2000-04-21 04:35:25 +0000445recursive-include examples *.txt *.py
Greg Ward16aafcd2000-04-09 04:06:44 +0000446prune examples/sample?/build
447\end{verbatim}
448The meanings should be fairly clear: include all files in the
449distribution root matching \code{*.txt}, all files anywhere under the
450\file{examples} directory matching \code{*.txt} or \code{*.py}, and
451exclude all directories matching \code{examples/sample?/build}. There
452are several other commands available in the manifest template
Greg Warde78298a2000-04-28 17:12:24 +0000453mini-language; see section~\ref{sdist-cmd}.
Greg Ward16aafcd2000-04-09 04:06:44 +0000454
455The order of commands in the manifest template very much matters:
456initially, we have the list of default files as described above, and
457each command in the template adds to or removes from that list of files.
458When we have fully processed the manifest template, we have our complete
459list of files. This list is written to the manifest for future
460reference, and then used to build the source distribution archive(s).
461
Greg Ward46b98e32000-04-14 01:53:36 +0000462Following the Distutils' own manifest template, let's trace how the
463\command{sdist} command will build the list of files to include in the
464Distutils source distribution:
Greg Ward16aafcd2000-04-09 04:06:44 +0000465\begin{enumerate}
466\item include all Python source files in the \file{distutils} and
467 \file{distutils/command} subdirectories (because packages
468 corresponding to those two directories were mentioned in the
469 \option{packages} option in the setup script)
470\item include \file{test/test*.py} (always included)
471\item include \file{README.txt} and \file{setup.py} (always included)
472\item include \file{*.txt} in the distribution root (this will find
473 \file{README.txt} a second time, but such redundancies are weeded out
474 later)
475\item in the sub-tree under \file{examples}, include anything matching
476 \file{*.txt}
477\item in the sub-tree under \file{examples}, include anything matching
478 \file{*.py}
479\item remove all files in the sub-trees starting at directories matching
480 \file{examples/sample?/build}---this may exclude files included by the
481 previous two steps, so it's important that the \code{prune} command in
482 the manifest template comes after the two \code{recursive-include}
483 commands
Greg Wardfacb8db2000-04-09 04:32:40 +0000484\end{enumerate}
Greg Ward16aafcd2000-04-09 04:06:44 +0000485
Greg Ward46b98e32000-04-14 01:53:36 +0000486Just like in the setup script, file and directory names in the manifest
487template should always be slash-separated; the Distutils will take care
488of converting them to the standard representation on your platform.
489That way, the manifest template is portable across operating systems.
490
Greg Ward16aafcd2000-04-09 04:06:44 +0000491
492\subsection{Manifest-related options}
Greg Warde78298a2000-04-28 17:12:24 +0000493\label{manifest-options}
Greg Ward16aafcd2000-04-09 04:06:44 +0000494
495The normal course of operations for the \command{sdist} command is as
496follows:
497\begin{itemize}
Greg Ward46b98e32000-04-14 01:53:36 +0000498\item if the manifest file, \file{MANIFEST} doesn't exist, read
499 \file{MANIFEST.in} and create the manifest
Greg Ward1d8f57a2000-08-05 00:43:11 +0000500\item if either \file{MANIFEST.in} or the setup script (\file{setup.py})
501 are more recent than \file{MANIFEST}, recreate \file{MANIFEST} by
502 reading \file{MANIFEST.in}
Greg Ward16aafcd2000-04-09 04:06:44 +0000503\item use the list of files now in \file{MANIFEST} (either just
504 generated or read in) to create the source distribution archive(s)
505\end{itemize}
506There are a couple of options that modify this behaviour.
507
508First, you might want to force the manifest to be regenerated---for
509example, if you have added or removed files or directories that match an
510existing pattern in the manifest template, you should regenerate the
511manifest:
512\begin{verbatim}
513python setup.py sdist --force-manifest
514\end{verbatim}
Greg Ward16aafcd2000-04-09 04:06:44 +0000515
516Or, you might just want to (re)generate the manifest, but not create a
517source distribution:
518\begin{verbatim}
519python setup.py sdist --manifest-only
520\end{verbatim}
Greg Warda021aca2000-04-19 22:34:11 +0000521(\longprogramopt{manifest-only} implies \longprogramopt{force-manifest}.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000522
523If you don't want to use the default file set, you can supply the
Greg Wardd5767a52000-04-19 22:48:09 +0000524\longprogramopt{no-defaults} option. If you use
525\longprogramopt{no-defaults} and don't supply a manifest template (or
526it's empty, or nothing matches the patterns in it), then your source
527distribution will be empty.
Greg Ward16aafcd2000-04-09 04:06:44 +0000528
529
530\section{Creating Built Distributions}
Greg Warde78298a2000-04-28 17:12:24 +0000531\label{built-dist}
Greg Ward16aafcd2000-04-09 04:06:44 +0000532
Greg Ward46b98e32000-04-14 01:53:36 +0000533A ``built distribution'' is what you're probably used to thinking of
534either as a ``binary package'' or an ``installer'' (depending on your
535background). It's not necessarily binary, though, because it might
536contain only Python source code and/or byte-code; and we don't call it a
537package, because that word is already spoken for in Python. (And
538``installer'' is a term specific to the Windows world. \XXX{do Mac
539 people use it?})
Greg Ward16aafcd2000-04-09 04:06:44 +0000540
Greg Ward46b98e32000-04-14 01:53:36 +0000541A built distribution is how you make life as easy as possible for
542installers of your module distribution: for users of RPM-based Linux
543systems, it's a binary RPM; for Windows users, it's an executable
544installer; for Debian-based Linux users, it's a Debian package; and so
545forth. Obviously, no one person will be able to create built
546distributions for every platform under the sun, so the Distutils is
547designed to enable module developers to concentrate on their
548specialty---writing code and creating source distributions---while an
549intermediary species of \emph{packager} springs up to turn source
Greg Ward19c67f82000-06-24 01:33:16 +0000550distributions into built distributions for as many platforms as there
Greg Ward46b98e32000-04-14 01:53:36 +0000551are packagers.
552
553Of course, the module developer could be his own packager; or the
554packager could be a volunteer ``out there'' somewhere who has access to
555a platform which the original developer does not; or it could be
556software periodically grabbing new source distributions and turning them
557into built distributions for as many platforms as the software has
558access to. Regardless of the nature of the beast, a packager uses the
559setup script and the \command{bdist} command family to generate built
560distributions.
561
562As a simple example, if I run the following command in the Distutils
563source tree:
564\begin{verbatim}
565python setup.py bdist
566\end{verbatim}
567then the Distutils builds my module distribution (the Distutils itself
568in this case), does a ``fake'' installation (also in the \file{build}
569directory), and creates the default type of built distribution for my
Greg Ward1d8f57a2000-08-05 00:43:11 +0000570platform. Currently, the default format for built distributions is a
571``dumb'' archive---tarball on Unix, ZIP file on Windows. (These are
572called ``dumb'' built distributions, because they must be unpacked in a
573specific location to work.)
574
575Thus, the above command on a Unix system creates
576\file{Distutils-0.9.1.\filevar{plat}.tar.gz}; unpacking this tarball
577from the root of the filesystemq installs the Distutils just as though
578you had downloaded the source distribution and run \code{python setup.py
579 install}. (Assuming that the target system has their Python
580installation laid out the same as you do---another reason these are
581called ``dumb'' distributions.) Obviously, for pure Python
582distributions, this isn't a huge win---but for non-pure distributions,
583which include extensions that would need to be compiled, it can mean the
584difference between someone being able to use your extensions or not.
Greg Ward46b98e32000-04-14 01:53:36 +0000585
586\XXX{filenames are inaccurate here!}
587
Greg Wardd5767a52000-04-19 22:48:09 +0000588The \command{bdist} command has a \longprogramopt{format} option,
Greg Ward1d8f57a2000-08-05 00:43:11 +0000589similar to the \command{sdist} command, which you can use to select the
590types of built distribution to generate: for example,
Greg Ward46b98e32000-04-14 01:53:36 +0000591\begin{verbatim}
592python setup.py bdist --format=zip
593\end{verbatim}
594would, when run on a Unix system, create
Greg Ward1d8f57a2000-08-05 00:43:11 +0000595\file{Distutils-0.8.\filevar{plat}.zip}---again, this archive would be
596unpacked from the root directory to install the Distutils.
Greg Ward46b98e32000-04-14 01:53:36 +0000597
598The available formats for built distributions are:
599\begin{tableiii}{l|l|c}{code}%
600 {Format}{Description}{Notes}
Greg Ward1d8f57a2000-08-05 00:43:11 +0000601 \lineiii{zip}{zip file (\file{.zip})}{}
602 \lineiii{gztar}{gzipped tar file (\file{.tar.gz})}{(1)}
Greg Ward46b98e32000-04-14 01:53:36 +0000603 \lineiii{ztar}{compressed tar file (\file{.tar.Z})}{}
604 \lineiii{tar}{tar file (\file{.tar})}{}
Greg Ward1d8f57a2000-08-05 00:43:11 +0000605 \lineiii{rpm}{RPM}{}
606 \lineiii{srpm}{source RPM}{\XXX{to do!}}
607 \lineiii{wininst}{self-extracting ZIP file for Windows}{(2)}
608 %\lineiii{wise}{Wise installer for Windows}{(3)}
Greg Ward46b98e32000-04-14 01:53:36 +0000609\end{tableiii}
610
611\noindent Notes:
612\begin{description}
Greg Ward1d8f57a2000-08-05 00:43:11 +0000613\item[(1)] default on Unix
614\item[(2)] default on Windows \XXX{to-do!}
615%\item[(3)] not implemented yet
Greg Ward46b98e32000-04-14 01:53:36 +0000616\end{description}
617
618You don't have to use the \command{bdist} command with the
Greg Wardd5767a52000-04-19 22:48:09 +0000619\longprogramopt{formats} option; you can also use the command that
Greg Ward1d8f57a2000-08-05 00:43:11 +0000620directly implements the format you're interested in. Some of these
Greg Ward46b98e32000-04-14 01:53:36 +0000621\command{bdist} ``sub-commands'' actually generate several similar
622formats; for instance, the \command{bdist\_dumb} command generates all
623the ``dumb'' archive formats (\code{tar}, \code{ztar}, \code{gztar}, and
624\code{zip}), and \command{bdist\_rpm} generates both binary and source
625RPMs. The \command{bdist} sub-commands, and the formats generated by
626each, are:
627\begin{tableii}{l|l}{command}%
628 {Command}{Formats}
629 \lineii{bdist\_dumb}{tar, ztar, gztar, zip}
630 \lineii{bdist\_rpm}{rpm, srpm}
Greg Ward1d8f57a2000-08-05 00:43:11 +0000631 \lineii{bdist\_wininst}{wininst}
632 %\lineii{bdist\_wise}{wise}
Greg Ward46b98e32000-04-14 01:53:36 +0000633\end{tableii}
Greg Ward16aafcd2000-04-09 04:06:44 +0000634
635\section{Examples}
Greg Warde78298a2000-04-28 17:12:24 +0000636\label{examples}
Greg Ward16aafcd2000-04-09 04:06:44 +0000637
638
639\subsection{Pure Python distribution (by module)}
Greg Warde78298a2000-04-28 17:12:24 +0000640\label{pure-mod}
Greg Ward16aafcd2000-04-09 04:06:44 +0000641
642
643\subsection{Pure Python distribution (by package)}
Greg Warde78298a2000-04-28 17:12:24 +0000644\label{pure-pkg}
Greg Ward16aafcd2000-04-09 04:06:44 +0000645
646
647\subsection{Single extension module}
Greg Warde78298a2000-04-28 17:12:24 +0000648\label{single-ext}
Greg Ward16aafcd2000-04-09 04:06:44 +0000649
650
651\subsection{Multiple extension modules}
Greg Warde78298a2000-04-28 17:12:24 +0000652\label{multiple-ext}
Greg Ward16aafcd2000-04-09 04:06:44 +0000653
654
655\subsection{Putting it all together}
656
657
Greg Ward4a9e7222000-04-25 02:57:36 +0000658
659\section{Extending the Distutils}
Greg Warde78298a2000-04-28 17:12:24 +0000660\label{extending}
Greg Ward4a9e7222000-04-25 02:57:36 +0000661
662
663\subsection{Extending existing commands}
Greg Warde78298a2000-04-28 17:12:24 +0000664\label{extend-existing}
Greg Ward4a9e7222000-04-25 02:57:36 +0000665
666
667\subsection{Writing new commands}
Greg Warde78298a2000-04-28 17:12:24 +0000668\label{new-commands}
Greg Ward4a9e7222000-04-25 02:57:36 +0000669
670
671
Greg Ward16aafcd2000-04-09 04:06:44 +0000672\section{Reference}
Greg Warde78298a2000-04-28 17:12:24 +0000673\label{ref}
Greg Ward16aafcd2000-04-09 04:06:44 +0000674
675
Greg Wardfacb8db2000-04-09 04:32:40 +0000676\subsection{Building modules: the \protect\command{build} command family}
Greg Warde78298a2000-04-28 17:12:24 +0000677\label{build-cmds}
Greg Ward16aafcd2000-04-09 04:06:44 +0000678
Greg Wardfacb8db2000-04-09 04:32:40 +0000679\subsubsection{\protect\command{build}}
Greg Warde78298a2000-04-28 17:12:24 +0000680\label{build-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000681
Greg Wardfacb8db2000-04-09 04:32:40 +0000682\subsubsection{\protect\command{build\_py}}
Greg Warde78298a2000-04-28 17:12:24 +0000683\label{build-py-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000684
Greg Wardfacb8db2000-04-09 04:32:40 +0000685\subsubsection{\protect\command{build\_ext}}
Greg Warde78298a2000-04-28 17:12:24 +0000686\label{build-ext-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000687
Greg Wardfacb8db2000-04-09 04:32:40 +0000688\subsubsection{\protect\command{build\_clib}}
Greg Warde78298a2000-04-28 17:12:24 +0000689\label{build-clib-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000690
691
Greg Wardfacb8db2000-04-09 04:32:40 +0000692\subsection{Installing modules: the \protect\command{install} command family}
Greg Warde78298a2000-04-28 17:12:24 +0000693\label{install-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000694
Gregory P. Smith147e5f32000-05-12 00:58:18 +0000695The install command ensures that the build commands have been run and then
696runs the subcommands \command{install\_lib},
697\command{install\_data} and
698\command{install\_scripts}.
699
700\subsubsection{\protect\command{install\_lib}}
701\label{sec:install-lib-cmd}
702
703\subsubsection{\protect\command{install\_data}}
704\label{sec:install-data-cmd}
705This command installs all data files provided with the distribution.
706
707\subsubsection{\protect\command{install\_scripts}}
708\label{sec:install-scripts-cmd}
709This command installs all (Python) scripts in the distribution.
710
Greg Ward16aafcd2000-04-09 04:06:44 +0000711
Greg Wardfacb8db2000-04-09 04:32:40 +0000712\subsection{Cleaning up: the \protect\command{clean} command}
Greg Warde78298a2000-04-28 17:12:24 +0000713\label{clean-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000714
715
Greg Wardfacb8db2000-04-09 04:32:40 +0000716\subsection{Creating a source distribution: the \protect\command{sdist} command}
Greg Warde78298a2000-04-28 17:12:24 +0000717\label{sdist-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000718
719
720\XXX{fragment moved down from above: needs context!}
721The manifest template commands are:
722\begin{tableii}{ll}{command}{Command}{Description}
Greg Ward87da1ea2000-04-21 04:35:25 +0000723 \lineii{include \var{pat1} \var{pat2} ... }
724 {include all files matching any of the listed patterns}
725 \lineii{exclude \var{pat1} \var{pat2} ... }
726 {exclude all files matching any of the listed patterns}
727 \lineii{recursive-include \var{dir} \var{pat1} \var{pat2} ... }
728 {include all files under \var{dir} matching any of the listed patterns}
729 \lineii{recursive-exclude \var{dir} \var{pat1} \var{pat2} ...}
730 {exclude all files under \var{dir} matching any of the listed patterns}
731 \lineii{global-include \var{pat1} \var{pat2} ...}
Greg Ward1bbe3292000-06-25 03:14:13 +0000732 {include all files anywhere in the source tree matching\\&
Greg Ward87da1ea2000-04-21 04:35:25 +0000733 any of the listed patterns}
734 \lineii{global-exclude \var{pat1} \var{pat2} ...}
Greg Ward1bbe3292000-06-25 03:14:13 +0000735 {exclude all files anywhere in the source tree matching\\&
Greg Ward87da1ea2000-04-21 04:35:25 +0000736 any of the listed patterns}
Greg Ward16aafcd2000-04-09 04:06:44 +0000737 \lineii{prune \var{dir}}{exclude all files under \var{dir}}
738 \lineii{graft \var{dir}}{include all files under \var{dir}}
739\end{tableii}
740The patterns here are Unix-style ``glob'' patterns: \code{*} matches any
741sequence of regular filename characters, \code{?} matches any single
742regular filename character, and \code{[\var{range}]} matches any of the
743characters in \var{range} (e.g., \code{a-z}, \code{a-zA-Z},
Greg Wardfacb8db2000-04-09 04:32:40 +0000744\code{a-f0-9\_.}). The definition of ``regular filename character'' is
Greg Ward16aafcd2000-04-09 04:06:44 +0000745platform-specific: on Unix it is anything except slash; on Windows
746anything except backslash or colon; on Mac OS anything except colon.
747\XXX{Windows and Mac OS support not there yet}
748
749
Greg Wardd5767a52000-04-19 22:48:09 +0000750\subsection{Creating a ``built'' distribution: the
751 \protect\command{bdist} command family}
Greg Warde78298a2000-04-28 17:12:24 +0000752\label{bdist-cmds}
Greg Ward16aafcd2000-04-09 04:06:44 +0000753
754
Greg Wardfacb8db2000-04-09 04:32:40 +0000755\subsubsection{\protect\command{blib}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000756
Greg Wardfacb8db2000-04-09 04:32:40 +0000757\subsubsection{\protect\command{blib\_dumb}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000758
Greg Wardfacb8db2000-04-09 04:32:40 +0000759\subsubsection{\protect\command{blib\_rpm}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000760
Greg Wardfacb8db2000-04-09 04:32:40 +0000761\subsubsection{\protect\command{blib\_wise}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000762
763
764
765
766
767
768
769
Greg Wardabc52162000-02-26 00:52:48 +0000770\end{document}