blob: 8f171151939bb38e1b63d81da472f336afdb2ece [file] [log] [blame]
Greg Wardabc52162000-02-26 00:52:48 +00001\documentclass{howto}
2\usepackage{ltxmarkup}
Greg Wardfacb8db2000-04-09 04:32:40 +00003\usepackage{times}
Greg Ward16aafcd2000-04-09 04:06:44 +00004\usepackage{distutils}
Greg Wardabc52162000-02-26 00:52:48 +00005
Greg Ward16aafcd2000-04-09 04:06:44 +00006\title{Distributing Python Modules}
Greg Wardabc52162000-02-26 00:52:48 +00007
Greg Wardabc52162000-02-26 00:52:48 +00008\author{Greg Ward}
9\authoraddress{E-mail: \email{gward@python.net}}
10
Greg Ward16aafcd2000-04-09 04:06:44 +000011
Greg Wardabc52162000-02-26 00:52:48 +000012\begin{document}
13
Greg Wardfacb8db2000-04-09 04:32:40 +000014\maketitle
15\tableofcontents
Greg Ward16aafcd2000-04-09 04:06:44 +000016
17\section{Introduction}
Greg Warde78298a2000-04-28 17:12:24 +000018\label{intro}
Greg Ward16aafcd2000-04-09 04:06:44 +000019
20In the past, Python module developers have not had much infrastructure
21support for distributing modules, nor have Python users had much support
22for installing and maintaining third-party modules. With the
23introduction of the Python Distribution Utilities (Distutils for short)
Fred Drake01df4532000-06-30 03:36:41 +000024in Python 2.0, this situation should start to improve.
Greg Ward16aafcd2000-04-09 04:06:44 +000025
26This document only covers using the Distutils to distribute your Python
Fred Drake01df4532000-06-30 03:36:41 +000027modules. Using the Distutils does not tie you to Python 2.0, though:
Greg Ward16aafcd2000-04-09 04:06:44 +000028the Distutils work just fine with Python 1.5, and it is reasonable (and
29expected to become commonplace) to expect users of Python 1.5 to
30download and install the Distutils separately before they can install
Fred Drake01df4532000-06-30 03:36:41 +000031your modules. Python 2.0 users, of course, won't have to add anything
Greg Ward16aafcd2000-04-09 04:06:44 +000032to their Python installation in order to use the Distutils to install
33third-party modules.
34
35This document concentrates on the role of developer/distributor: if
Fred Drake01df4532000-06-30 03:36:41 +000036you're looking for information on installing Python modules, you
37should refer to the \citetitle[../inst/inst.html]{Installing Python
38Modules} manual.
Greg Ward16aafcd2000-04-09 04:06:44 +000039
40
Greg Wardfacb8db2000-04-09 04:32:40 +000041\section{Concepts \& Terminology}
Greg Warde78298a2000-04-28 17:12:24 +000042\label{concepts}
Greg Ward16aafcd2000-04-09 04:06:44 +000043
44Using the Distutils is quite simple, both for module developers and for
45users/administrators installing third-party modules. As a developer,
46your responsibilites (apart from writing solid, well-documented and
47well-tested code, of course!) are:
48\begin{itemize}
49\item write a setup script (\file{setup.py} by convention)
50\item (optional) write a setup configuration file
51\item create a source distribution
52\item (optional) create one or more built (binary) distributions
53\end{itemize}
54Each of these tasks is covered in this document.
55
56Not all module developers have access to a multitude of platforms, so
57it's not always feasible to expect them to create a multitude of built
58distributions. It is hoped that a class of intermediaries, called
Greg Ward19c67f82000-06-24 01:33:16 +000059\emph{packagers}, will arise to address this need. Packagers will take
60source distributions released by module developers, build them on one or
61more platforms, and release the resulting built distributions. Thus,
62users on the most popular platforms will be able to install most popular
63Python module distributions in the most natural way for their platform,
64without having to run a single setup script or compile a line of code.
Greg Ward16aafcd2000-04-09 04:06:44 +000065
66
67\subsection{A simple example}
Greg Warde78298a2000-04-28 17:12:24 +000068\label{simple-example}
Greg Ward16aafcd2000-04-09 04:06:44 +000069
70The setup script is usually quite simple, although since it's written in
71Python, there are no arbitrary limits to what you can do. If all you
72want to do is distribute a module called \module{foo}, contained in a
Greg Ward370248d2000-06-24 01:45:47 +000073file \file{foo.py}, then your setup script can be as little as this:
Greg Ward16aafcd2000-04-09 04:06:44 +000074\begin{verbatim}
75from distutils.core import setup
76setup (name = "foo",
77 version = "1.0",
78 py_modules = ["foo"])
79\end{verbatim}
Greg Ward370248d2000-06-24 01:45:47 +000080
Greg Ward16aafcd2000-04-09 04:06:44 +000081Some observations:
82\begin{itemize}
Greg Ward370248d2000-06-24 01:45:47 +000083\item most information that you supply to the Distutils is supplied as
Greg Wardfacb8db2000-04-09 04:32:40 +000084 keyword arguments to the \function{setup()} function
Greg Ward16aafcd2000-04-09 04:06:44 +000085\item those keyword arguments fall into two categories: package
86 meta-data (name, version number) and information about what's in the
Greg Ward370248d2000-06-24 01:45:47 +000087 package (a list of pure Python modules, in this case)
Greg Ward16aafcd2000-04-09 04:06:44 +000088\item modules are specified by module name, not filename (the same will
89 hold true for packages and extensions)
90\item it's recommended that you supply a little more meta-data, in
91 particular your name, email address and a URL for the project
92\end{itemize}
93
Greg Ward370248d2000-06-24 01:45:47 +000094To create a source distribution for this module, you would create a
95setup script, \file{setup.py}, containing the above code, and run:
Greg Ward16aafcd2000-04-09 04:06:44 +000096\begin{verbatim}
97python setup.py sdist
98\end{verbatim}
99which will create an archive file (e.g., tarball on Unix, zip file on
100Windows) containing your setup script, \file{setup.py}, and your module,
101\file{foo.py}. The archive file will be named \file{Foo-1.0.tar.gz} (or
102\file{.zip}), and will unpack into a directory \file{Foo-1.0}.
103
104If an end-user wishes to install your \module{foo} module, all she has
Greg Ward59d382e2000-05-26 01:04:47 +0000105to do is download \file{Foo-1.0.tar.gz} (or \file{.zip}), unpack it,
Greg Ward16aafcd2000-04-09 04:06:44 +0000106and---from the \file{Foo-1.0} directory---run
107\begin{verbatim}
108python setup.py install
109\end{verbatim}
110which will ultimately copy \file{foo.py} to the appropriate directory
111for third-party modules in their Python installation.
112
113This simple example demonstrates some fundamental concepts of the
114Distutils: first, both developers and installers have the same basic
115user interface, i.e. the setup script. The difference is which
116Distutils \emph{commands} they use: the \command{sdist} command is
117almost exclusively for module developers, while \command{install} is
118more often for installers (although most developers will want to install
119their own code occasionally).
120
121\XXX{only partially implemented}%
122If you want to make things really easy for your users, you can create
123one or more built distributions for them. For instance, if you are
124running on a Windows machine, and want to make things easy for other
125Windows users, you can create an executable installer (the most
126appropriate type of built distribution for this platform) with the
Greg Ward59d382e2000-05-26 01:04:47 +0000127\command{bdist\_wininst} command. For example:
Greg Ward16aafcd2000-04-09 04:06:44 +0000128\begin{verbatim}
Greg Ward59d382e2000-05-26 01:04:47 +0000129python setup.py bdist_wininst
Greg Ward16aafcd2000-04-09 04:06:44 +0000130\end{verbatim}
Greg Wardfacb8db2000-04-09 04:32:40 +0000131will create an executable installer, \file{Foo-1\_0.exe}, in the current
Greg Ward16aafcd2000-04-09 04:06:44 +0000132directory.
133
Greg Ward59d382e2000-05-26 01:04:47 +0000134(Another way to create executable installers for Windows is with the
135\command{bdist\_wise} command, which uses Wise---the commercial
136installer-generator used to create Python's own installer---to create
137the installer. Wise-based installers are more appropriate for large,
138industrial-strength applications that need the full capabilities of a
139``real'' installer. \command{bdist\_wininst} creates a self-extracting
140zip file with a minimal user interface, which is enough for small- to
141medium-sized module collections. You'll need to have version XXX of
Greg Ward370248d2000-06-24 01:45:47 +0000142Wise installed on your system for the \command{bdist\_wise} command to
143work; it's available from \url{http://foo/bar/baz}.)
Greg Ward59d382e2000-05-26 01:04:47 +0000144
145Other \command{bdist} commands exist for other platforms: for example,
146\command{bdist\_rpm} for RPM-based Linux systems, (\command{bdist\_deb})
147for Debian-based Linux systems, and so forth. See
148section~\ref{bdist-cmds} for details on all the \command{bdist}
149commands.
Greg Ward16aafcd2000-04-09 04:06:44 +0000150
151
152\subsection{General Python terminology}
Greg Warde78298a2000-04-28 17:12:24 +0000153\label{python-terms}
Greg Ward16aafcd2000-04-09 04:06:44 +0000154
155If you're reading this document, you probably have a good idea of what
156modules, extensions, and so forth are. Nevertheless, just to be sure
157that everyone is operating from a common starting point, we offer the
158following glossary of common Python terms:
159\begin{description}
160\item[module] the basic unit of code reusability in Python: a block of
161 code imported by some other code. There are three types of modules
162 that concern us here: pure Python modules, extension modules, and
163 packages.
164\item[pure Python module] a module written in Python and contained in a
165 single \file{.py} file (and possibly associated \file{.pyc} and/or
166 \file{.pyo} files). Sometimes referred to as a ``pure module.''
167\item[extension module] a module written in the low-level language of
168 the Python implemention: C/C++ for CPython, Java for JPython.
169 Typically contained in a single dynamically loadable pre-compiled
170 file, e.g. a shared object (\file{.so}) file for CPython extensions on
171 Unix, a DLL (given the \file{.pyd} extension) for CPython extensions
Greg Ward1bbe3292000-06-25 03:14:13 +0000172 on Windows, or a Java class file for JPython extensions. (Note that
173 currently, the Distutils only handles C/C++ extensions for CPython.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000174\item[package] a module that contains other modules; typically contained
175 in a directory in the filesystem and distinguished from other
176 directories by the presence of a file \file{\_\_init\_\_.py}.
Greg Ward6153fa12000-05-26 02:24:28 +0000177\item[root package] the root of the hierarchy of packages. (This isn't
178 really a package, since it doesn't have an \file{\_\_init\_\_.py}
179 file. But we have to call it something.) The vast majority of the
180 standard library is in the root package, as are many small, standalone
181 third-party modules that don't belong to a larger module collection.
182 Unlike regular packages, modules in the root package can be found in
183 many directories: in fact, every directory listed in \code{sys.path}
184 can contribute modules to the root package.
Greg Ward16aafcd2000-04-09 04:06:44 +0000185\end{description}
186
187
188\subsection{Distutils-specific terminology}
Greg Warde78298a2000-04-28 17:12:24 +0000189\label{distutils-term}
Greg Ward16aafcd2000-04-09 04:06:44 +0000190
191The following terms apply more specifically to the domain of
192distributing Python modules using the Distutils:
193\begin{description}
194\item[module distribution] a collection of Python modules distributed
195 together as a single downloadable resource and meant to be installed
196 \emph{en masse}. Examples of some well-known module distributions are
197 Numeric Python, PyXML, PIL (the Python Imaging Library), or
198 mxDateTime. (This would be called a \emph{package}, except that term
Greg Ward59d382e2000-05-26 01:04:47 +0000199 is already taken in the Python context: a single module distribution
200 may contain zero, one, or many Python packages.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000201\item[pure module distribution] a module distribution that contains only
202 pure Python modules and packages. Sometimes referred to as a ``pure
203 distribution.''
204\item[non-pure module distribution] a module distribution that contains
205 at least one extension module. Sometimes referred to as a ``non-pure
206 distribution.''
207\item[distribution root] the top-level directory of your source tree (or
208 source distribution); the directory where \file{setup.py} exists and
209 is run from
210\end{description}
211
212
213\section{Writing the Setup Script}
Greg Warde78298a2000-04-28 17:12:24 +0000214\label{setup-script}
Greg Ward16aafcd2000-04-09 04:06:44 +0000215
216The setup script is the centre of all activity in building,
217distributing, and installing modules using the Distutils. The main
218purpose of the setup script is to describe your module distribution to
Greg Wardd5767a52000-04-19 22:48:09 +0000219the Distutils, so that the various commands that operate on your modules
Greg Ward59d382e2000-05-26 01:04:47 +0000220do the right thing. As we saw in section~\ref{simple-example} above,
221the setup script consists mainly of a call to \function{setup()}, and
Greg Ward1bbe3292000-06-25 03:14:13 +0000222most information supplied to the Distutils by the module developer is
223supplied as keyword arguments to \function{setup()}.
Greg Ward16aafcd2000-04-09 04:06:44 +0000224
225Here's a slightly more involved example, which we'll follow for the next
226couple of sections: the Distutils' own setup script. (Keep in mind that
Fred Drake01df4532000-06-30 03:36:41 +0000227although the Distutils are included with Python 2.0, they also have an
Greg Ward16aafcd2000-04-09 04:06:44 +0000228independent existence so that Python 1.5 users can use them to install
Greg Ward59d382e2000-05-26 01:04:47 +0000229other module distributions. The Distutils' own setup script is used to
230install the package into Python 1.5.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000231
232\begin{verbatim}
233#!/usr/bin/env python
234
235from distutils.core import setup
236
237setup (name = "Distutils",
238 version = "1.0",
239 description = "Python Module Distribution Utilities",
240 author = "Greg Ward",
241 author_email = "gward@python.net",
242 url = "http://www.python.org/sigs/distutils-sig/",
243
244 packages = ['distutils', 'distutils.command'],
245 )
246\end{verbatim}
247There are only two differences between this and the trivial one-file
Greg Warde78298a2000-04-28 17:12:24 +0000248distribution presented in section~\ref{simple-example}: more
Greg Ward16aafcd2000-04-09 04:06:44 +0000249meta-data, and the specification of pure Python modules by package,
250rather than by module. This is important since the Distutils consist of
251a couple of dozen modules split into (so far) two packages; an explicit
252list of every module would be tedious to generate and difficult to
253maintain.
254
Greg Ward46b98e32000-04-14 01:53:36 +0000255Note that any pathnames (files or directories) supplied in the setup
256script should be written using the Unix convention, i.e.
257slash-separated. The Distutils will take care of converting this
Greg Ward59d382e2000-05-26 01:04:47 +0000258platform-neutral representation into whatever is appropriate on your
Greg Ward46b98e32000-04-14 01:53:36 +0000259current platform before actually using the pathname. This makes your
260setup script portable across operating systems, which of course is one
261of the major goals of the Distutils. In this spirit, all pathnames in
Greg Ward59d382e2000-05-26 01:04:47 +0000262this document are slash-separated (Mac OS programmers should keep in
263mind that the \emph{absence} of a leading slash indicates a relative
264path, the opposite of the Mac OS convention with colons).
Greg Ward46b98e32000-04-14 01:53:36 +0000265
Greg Ward16aafcd2000-04-09 04:06:44 +0000266
267\subsection{Package directories}
Greg Warde78298a2000-04-28 17:12:24 +0000268\label{package-dirs}
Greg Ward16aafcd2000-04-09 04:06:44 +0000269
270The \option{packages} option tells the Distutils to process (build,
271distribute, install, etc.) all pure Python modules found in each package
272mentioned in the \option{packages} list. In order to do this, of
273course, there has to be a correspondence between package names and
274directories in the filesystem. The default correspondence is the most
Greg Ward1ecc2512000-04-19 22:36:24 +0000275obvious one, i.e. package \module{distutils} is found in the directory
Greg Ward16aafcd2000-04-09 04:06:44 +0000276\file{distutils} relative to the distribution root. Thus, when you say
277\code{packages = ['foo']} in your setup script, you are promising that
278the Distutils will find a file \file{foo/\_\_init\_\_.py} (which might
279be spelled differently on your system, but you get the idea) relative to
280the directory where your setup script lives. (If you break this
281promise, the Distutils will issue a warning but process the broken
282package anyways.)
283
284If you use a different convention to lay out your source directory,
285that's no problem: you just have to supply the \option{package\_dir}
286option to tell the Distutils about your convention. For example, say
287you keep all Python source under \file{lib}, so that modules not in any
Greg Ward1ecc2512000-04-19 22:36:24 +0000288package are right in \file{lib}, modules in the \module{foo} package
Greg Ward16aafcd2000-04-09 04:06:44 +0000289are in \file{lib/foo}, and so forth. Then you would put
290\begin{verbatim}
291package_dir = {'': 'lib'}
292\end{verbatim}
293in your setup script. (The keys to this dictionary are package names,
294and an empty package name stands for the ``root package,'' i.e. no
295package at all. The values are directory names relative to your
296distribution root.) In this case, when you say
297\code{packages = ['foo']}, you are promising that the file
298\file{lib/foo/\_\_init\_\_.py} exists.
299
Greg Ward1ecc2512000-04-19 22:36:24 +0000300Another possible convention is to put the \module{foo} package right in
301\file{lib}, the \module{foo.bar} package in \file{lib/bar}, etc. This
Greg Ward16aafcd2000-04-09 04:06:44 +0000302would be written in the setup script as
303\begin{verbatim}
304package_dir = {'foo': 'lib'}
305\end{verbatim}
Greg Ward59d382e2000-05-26 01:04:47 +0000306A \code{\var{package}: \var{dir}} entry in the \option{package\_dir}
307dictionary implicitly applies to all packages below \var{package}, so
308the \module{foo.bar} case is automatically handled here. In this
309example, having \code{packages = ['foo', 'foo.bar']} tells the Distutils
310to look for \file{lib/\_\_init\_\_.py} and
311\file{lib/bar/\_\_init\_\_.py}. (Keep in mind that although
312\option{package\_dir} applies recursively, you must explicitly list all
313packages in \option{packages}: the Distutils will \emph{not} recursively
314scan your source tree looking for any directory with an
315\file{\_\_init\_\_.py} file.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000316
317
318\subsection{Listing individual modules}
Greg Warde78298a2000-04-28 17:12:24 +0000319\label{listing-modules}
Greg Ward16aafcd2000-04-09 04:06:44 +0000320
321For a small module distribution, you might prefer to list all modules
322rather than listing packages---especially the case of a single module
323that goes in the ``root package'' (i.e., no package at all). This
Greg Warde78298a2000-04-28 17:12:24 +0000324simplest case was shown in section~\ref{simple-example}; here is a
Greg Ward16aafcd2000-04-09 04:06:44 +0000325slightly more involved example:
326\begin{verbatim}
327py_modules = ['mod1', 'pkg.mod2']
328\end{verbatim}
329This describes two modules, one of them in the ``root'' package, the
Greg Wardd5767a52000-04-19 22:48:09 +0000330other in the \module{pkg} package. Again, the default package/directory
331layout implies that these two modules can be found in \file{mod1.py} and
332\file{pkg/mod2.py}, and that \file{pkg/\_\_init\_\_.py} exists as well.
333And again, you can override the package/directory layout using the
Greg Ward59d382e2000-05-26 01:04:47 +0000334\option{package\_dir} option.
335
336
337\subsection{Describing extension modules}
338\label{sec:describing-extensions}
339
340\XXX{be sure to describe the whole \code{build\_info} dict, including
341 \code{extra\_compile\_args} and \code{extra\_link\_args}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000342
343
344\section{Writing the Setup Configuration File}
Greg Warde78298a2000-04-28 17:12:24 +0000345\label{setup-config}
Greg Ward16aafcd2000-04-09 04:06:44 +0000346
347\XXX{not implemented yet!}
348
349Often, it's not possible to write down everything needed to build a
350distribution \emph{a priori}. You need to get some information from the
351user, or from the user's system, in order to proceed. For example, you
352might include an optional extension module that provides an interface to
353a particular C library. If that library is installed on the user's
354system, then you can build your optional extension---but you need to
355know where to find the header and library file. If it's not installed,
356you need to know this so you can omit your optional extension.
357
358The preferred way to do this, of course, would be for you to tell the
359Distutils which optional features (C libraries, system calls, external
360utilities, etc.) you're looking for, and it would inspect the user's
361system and try to find them. This functionality may appear in a future
362version of the Distutils, but it isn't there now. So, for the time
363being, we rely on the user building and installing your software to
364provide the necessary information. The vehicle for doing so is the
365setup configuration file, \file{setup.cfg}.
366
367\XXX{need more here!}
368
369
370\section{Creating a Source Distribution}
Greg Warde78298a2000-04-28 17:12:24 +0000371\label{source-dist}
Greg Ward16aafcd2000-04-09 04:06:44 +0000372
Greg Warde78298a2000-04-28 17:12:24 +0000373As shown in section~\ref{simple-example}, you use the
Greg Ward16aafcd2000-04-09 04:06:44 +0000374\command{sdist} command to create a source distribution. In the
375simplest case,
376\begin{verbatim}
377python setup.py sdist
378\end{verbatim}
Greg Ward19c67f82000-06-24 01:33:16 +0000379(assuming you haven't specified any \command{sdist} options in the setup
380script or config file), \command{sdist} creates the archive of the
Greg Ward16aafcd2000-04-09 04:06:44 +0000381default format for the current platform. The default formats are:
382\begin{tableii}{ll}{textrm}%
383 {Platform}{Default archive format for source distributions}
384 \lineii{Unix}{gzipped tar file (\file{.tar.gz})}
385 \lineii{Windows}{zip file}
386\end{tableii}
Greg Wardd5767a52000-04-19 22:48:09 +0000387You can specify as many formats as you like using the
388\longprogramopt{formats} option, for example:
Greg Ward16aafcd2000-04-09 04:06:44 +0000389\begin{verbatim}
390python setup.py sdist --formats=gztar,zip
391\end{verbatim}
392to create a gzipped tarball and a zip file. The available formats are:
Greg Ward46b98e32000-04-14 01:53:36 +0000393\begin{tableiii}{l|l|c}{code}%
394 {Format}{Description}{Notes}
395 \lineiii{zip}{zip file (\file{.zip})}{(1)}
396 \lineiii{gztar}{gzipped tar file (\file{.tar.gz})}{(2)}
397 \lineiii{ztar}{compressed tar file (\file{.tar.Z})}{}
398 \lineiii{tar}{tar file (\file{.tar})}{}
399\end{tableiii}
400
401\noindent Notes:
402\begin{description}
403\item[(1)] default on Windows
404\item[(2)] default on Unix
405\end{description}
Greg Ward16aafcd2000-04-09 04:06:44 +0000406
407
408\subsection{The manifest and manifest template}
Greg Warde78298a2000-04-28 17:12:24 +0000409\label{manifest}
Greg Ward16aafcd2000-04-09 04:06:44 +0000410
411Without any additional information, the \command{sdist} command puts a
412minimal set of files into the source distribution:
413\begin{itemize}
Greg Wardfacb8db2000-04-09 04:32:40 +0000414\item all Python source files implied by the \option{py\_modules} and
Greg Ward16aafcd2000-04-09 04:06:44 +0000415 \option{packages} options
Greg Wardfacb8db2000-04-09 04:32:40 +0000416\item all C source files mentioned in the \option{ext\_modules} or
Greg Ward16aafcd2000-04-09 04:06:44 +0000417 \option{libraries} options (\XXX{getting C library sources currently
Greg Wardfacb8db2000-04-09 04:32:40 +0000418 broken -- no get\_source\_files() method in build\_clib.py!})
Greg Ward16aafcd2000-04-09 04:06:44 +0000419\item anything that looks like a test script: \file{test/test*.py}
420 (currently, the Distutils don't do anything with test scripts except
421 include them in source distributions, but in the future there will be
422 a standard for testing Python module distributions)
423\item \file{README.txt} (or \file{README}) and \file{setup.py}
424\end{itemize}
425Sometimes this is enough, but usually you will want to specify
426additional files to distribute. The typical way to do this is to write
427a \emph{manifest template}, called \file{MANIFEST.in} by default. The
428\command{sdist} command processes this template and generates a manifest
429file, \file{MANIFEST}. (If you prefer, you can skip the manifest
430template and generate the manifest yourself: it just lists one file per
431line.)
432
433The manifest template has one command per line, where each command
434specifies a set of files to include or exclude from the source
435distribution. For an example, again we turn to the Distutils' own
436manifest template:
437\begin{verbatim}
438include *.txt
Greg Ward87da1ea2000-04-21 04:35:25 +0000439recursive-include examples *.txt *.py
Greg Ward16aafcd2000-04-09 04:06:44 +0000440prune examples/sample?/build
441\end{verbatim}
442The meanings should be fairly clear: include all files in the
443distribution root matching \code{*.txt}, all files anywhere under the
444\file{examples} directory matching \code{*.txt} or \code{*.py}, and
445exclude all directories matching \code{examples/sample?/build}. There
446are several other commands available in the manifest template
Greg Warde78298a2000-04-28 17:12:24 +0000447mini-language; see section~\ref{sdist-cmd}.
Greg Ward16aafcd2000-04-09 04:06:44 +0000448
449The order of commands in the manifest template very much matters:
450initially, we have the list of default files as described above, and
451each command in the template adds to or removes from that list of files.
452When we have fully processed the manifest template, we have our complete
453list of files. This list is written to the manifest for future
454reference, and then used to build the source distribution archive(s).
455
Greg Ward46b98e32000-04-14 01:53:36 +0000456Following the Distutils' own manifest template, let's trace how the
457\command{sdist} command will build the list of files to include in the
458Distutils source distribution:
Greg Ward16aafcd2000-04-09 04:06:44 +0000459\begin{enumerate}
460\item include all Python source files in the \file{distutils} and
461 \file{distutils/command} subdirectories (because packages
462 corresponding to those two directories were mentioned in the
463 \option{packages} option in the setup script)
464\item include \file{test/test*.py} (always included)
465\item include \file{README.txt} and \file{setup.py} (always included)
466\item include \file{*.txt} in the distribution root (this will find
467 \file{README.txt} a second time, but such redundancies are weeded out
468 later)
469\item in the sub-tree under \file{examples}, include anything matching
470 \file{*.txt}
471\item in the sub-tree under \file{examples}, include anything matching
472 \file{*.py}
473\item remove all files in the sub-trees starting at directories matching
474 \file{examples/sample?/build}---this may exclude files included by the
475 previous two steps, so it's important that the \code{prune} command in
476 the manifest template comes after the two \code{recursive-include}
477 commands
Greg Wardfacb8db2000-04-09 04:32:40 +0000478\end{enumerate}
Greg Ward16aafcd2000-04-09 04:06:44 +0000479
Greg Ward46b98e32000-04-14 01:53:36 +0000480Just like in the setup script, file and directory names in the manifest
481template should always be slash-separated; the Distutils will take care
482of converting them to the standard representation on your platform.
483That way, the manifest template is portable across operating systems.
484
Greg Ward16aafcd2000-04-09 04:06:44 +0000485
486\subsection{Manifest-related options}
Greg Warde78298a2000-04-28 17:12:24 +0000487\label{manifest-options}
Greg Ward16aafcd2000-04-09 04:06:44 +0000488
489The normal course of operations for the \command{sdist} command is as
490follows:
491\begin{itemize}
Greg Ward46b98e32000-04-14 01:53:36 +0000492\item if the manifest file, \file{MANIFEST} doesn't exist, read
493 \file{MANIFEST.in} and create the manifest
494\item if \file{MANIFEST.in} is more recent than \file{MANIFEST},
495 recreate \file{MANIFEST} by reading \file{MANIFEST.in}
Greg Ward16aafcd2000-04-09 04:06:44 +0000496\item use the list of files now in \file{MANIFEST} (either just
497 generated or read in) to create the source distribution archive(s)
498\end{itemize}
499There are a couple of options that modify this behaviour.
500
501First, you might want to force the manifest to be regenerated---for
502example, if you have added or removed files or directories that match an
503existing pattern in the manifest template, you should regenerate the
504manifest:
505\begin{verbatim}
506python setup.py sdist --force-manifest
507\end{verbatim}
508\XXX{this is stupid, but is there a better way to do it without
509 reprocessing MANIFEST.in every single bloody time?}
510
511Or, you might just want to (re)generate the manifest, but not create a
512source distribution:
513\begin{verbatim}
514python setup.py sdist --manifest-only
515\end{verbatim}
Greg Warda021aca2000-04-19 22:34:11 +0000516(\longprogramopt{manifest-only} implies \longprogramopt{force-manifest}.)
Greg Ward16aafcd2000-04-09 04:06:44 +0000517
518If you don't want to use the default file set, you can supply the
Greg Wardd5767a52000-04-19 22:48:09 +0000519\longprogramopt{no-defaults} option. If you use
520\longprogramopt{no-defaults} and don't supply a manifest template (or
521it's empty, or nothing matches the patterns in it), then your source
522distribution will be empty.
Greg Ward16aafcd2000-04-09 04:06:44 +0000523
524
525\section{Creating Built Distributions}
Greg Warde78298a2000-04-28 17:12:24 +0000526\label{built-dist}
Greg Ward16aafcd2000-04-09 04:06:44 +0000527
Greg Ward46b98e32000-04-14 01:53:36 +0000528A ``built distribution'' is what you're probably used to thinking of
529either as a ``binary package'' or an ``installer'' (depending on your
530background). It's not necessarily binary, though, because it might
531contain only Python source code and/or byte-code; and we don't call it a
532package, because that word is already spoken for in Python. (And
533``installer'' is a term specific to the Windows world. \XXX{do Mac
534 people use it?})
Greg Ward16aafcd2000-04-09 04:06:44 +0000535
Greg Ward46b98e32000-04-14 01:53:36 +0000536A built distribution is how you make life as easy as possible for
537installers of your module distribution: for users of RPM-based Linux
538systems, it's a binary RPM; for Windows users, it's an executable
539installer; for Debian-based Linux users, it's a Debian package; and so
540forth. Obviously, no one person will be able to create built
541distributions for every platform under the sun, so the Distutils is
542designed to enable module developers to concentrate on their
543specialty---writing code and creating source distributions---while an
544intermediary species of \emph{packager} springs up to turn source
Greg Ward19c67f82000-06-24 01:33:16 +0000545distributions into built distributions for as many platforms as there
Greg Ward46b98e32000-04-14 01:53:36 +0000546are packagers.
547
548Of course, the module developer could be his own packager; or the
549packager could be a volunteer ``out there'' somewhere who has access to
550a platform which the original developer does not; or it could be
551software periodically grabbing new source distributions and turning them
552into built distributions for as many platforms as the software has
553access to. Regardless of the nature of the beast, a packager uses the
554setup script and the \command{bdist} command family to generate built
555distributions.
556
557As a simple example, if I run the following command in the Distutils
558source tree:
559\begin{verbatim}
560python setup.py bdist
561\end{verbatim}
562then the Distutils builds my module distribution (the Distutils itself
563in this case), does a ``fake'' installation (also in the \file{build}
564directory), and creates the default type of built distribution for my
565platform. In Distutils 0.8, only two types of built distribution are
566supported: \code{gztar} (default on non-Linux Unix) and \code{zip}
567(default on Windows). Thus, the above command on a Unix system creates
568\file{Distutils-0.8.built-posix.tar.gz}; unpacking this tarball from
569Python's \filevar{prefix} directory installs the Distutils just as
570though you had downloaded the source distribution and run \code{python
571 setup.py install}. Obviously, for pure Python distributions, this
572isn't a huge win---but for non-pure distributions, which include
573extensions that would need to be compiled, it can mean the difference
574between someone being able to use your extensions or not.
575
576\XXX{filenames are inaccurate here!}
577
Greg Wardd5767a52000-04-19 22:48:09 +0000578The \command{bdist} command has a \longprogramopt{format} option,
579similar to the \command{sdist} command, that you can use to select which
580formats to generate: for example,
Greg Ward46b98e32000-04-14 01:53:36 +0000581\begin{verbatim}
582python setup.py bdist --format=zip
583\end{verbatim}
584would, when run on a Unix system, create
585\file{Distutils-0.8.built-posix.tar.gz}---again, this archive would be
586unpacked from Python's \filevar{prefix} directory to install the
587Distutils.
588
589The available formats for built distributions are:
590\begin{tableiii}{l|l|c}{code}%
591 {Format}{Description}{Notes}
592 \lineiii{zip}{zip file (\file{.zip})}{(1)}
593 \lineiii{gztar}{gzipped tar file (\file{.tar.gz})}{(2)}
594 \lineiii{ztar}{compressed tar file (\file{.tar.Z})}{}
595 \lineiii{tar}{tar file (\file{.tar})}{}
596 \lineiii{rpm}{RPM}{(3)}
597 \lineiii{srpm}{source RPM}{}
598 \lineiii{wise}{Wise installer for Windows}{}
599\end{tableiii}
600
601\noindent Notes:
602\begin{description}
603\item[(1)] default on Windows
604\item[(2)] default on Unix
605\item[(3)] not implemented yet; will be default on RPM-based Linux
606 systems
607\item[(5)] not implemented yet; will be default on Windows
608\end{description}
609
610You don't have to use the \command{bdist} command with the
Greg Wardd5767a52000-04-19 22:48:09 +0000611\longprogramopt{formats} option; you can also use the command that
612directly implements the format you're interested in. Many of these
Greg Ward46b98e32000-04-14 01:53:36 +0000613\command{bdist} ``sub-commands'' actually generate several similar
614formats; for instance, the \command{bdist\_dumb} command generates all
615the ``dumb'' archive formats (\code{tar}, \code{ztar}, \code{gztar}, and
616\code{zip}), and \command{bdist\_rpm} generates both binary and source
617RPMs. The \command{bdist} sub-commands, and the formats generated by
618each, are:
619\begin{tableii}{l|l}{command}%
620 {Command}{Formats}
621 \lineii{bdist\_dumb}{tar, ztar, gztar, zip}
622 \lineii{bdist\_rpm}{rpm, srpm}
623 \lineii{bdist\_wise}{wise}
624\end{tableii}
Greg Ward16aafcd2000-04-09 04:06:44 +0000625
626\section{Examples}
Greg Warde78298a2000-04-28 17:12:24 +0000627\label{examples}
Greg Ward16aafcd2000-04-09 04:06:44 +0000628
629
630\subsection{Pure Python distribution (by module)}
Greg Warde78298a2000-04-28 17:12:24 +0000631\label{pure-mod}
Greg Ward16aafcd2000-04-09 04:06:44 +0000632
633
634\subsection{Pure Python distribution (by package)}
Greg Warde78298a2000-04-28 17:12:24 +0000635\label{pure-pkg}
Greg Ward16aafcd2000-04-09 04:06:44 +0000636
637
638\subsection{Single extension module}
Greg Warde78298a2000-04-28 17:12:24 +0000639\label{single-ext}
Greg Ward16aafcd2000-04-09 04:06:44 +0000640
641
642\subsection{Multiple extension modules}
Greg Warde78298a2000-04-28 17:12:24 +0000643\label{multiple-ext}
Greg Ward16aafcd2000-04-09 04:06:44 +0000644
645
646\subsection{Putting it all together}
647
648
Greg Ward4a9e7222000-04-25 02:57:36 +0000649
650\section{Extending the Distutils}
Greg Warde78298a2000-04-28 17:12:24 +0000651\label{extending}
Greg Ward4a9e7222000-04-25 02:57:36 +0000652
653
654\subsection{Extending existing commands}
Greg Warde78298a2000-04-28 17:12:24 +0000655\label{extend-existing}
Greg Ward4a9e7222000-04-25 02:57:36 +0000656
657
658\subsection{Writing new commands}
Greg Warde78298a2000-04-28 17:12:24 +0000659\label{new-commands}
Greg Ward4a9e7222000-04-25 02:57:36 +0000660
661
662
Greg Ward16aafcd2000-04-09 04:06:44 +0000663\section{Reference}
Greg Warde78298a2000-04-28 17:12:24 +0000664\label{ref}
Greg Ward16aafcd2000-04-09 04:06:44 +0000665
666
Greg Wardfacb8db2000-04-09 04:32:40 +0000667\subsection{Building modules: the \protect\command{build} command family}
Greg Warde78298a2000-04-28 17:12:24 +0000668\label{build-cmds}
Greg Ward16aafcd2000-04-09 04:06:44 +0000669
Greg Wardfacb8db2000-04-09 04:32:40 +0000670\subsubsection{\protect\command{build}}
Greg Warde78298a2000-04-28 17:12:24 +0000671\label{build-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000672
Greg Wardfacb8db2000-04-09 04:32:40 +0000673\subsubsection{\protect\command{build\_py}}
Greg Warde78298a2000-04-28 17:12:24 +0000674\label{build-py-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000675
Greg Wardfacb8db2000-04-09 04:32:40 +0000676\subsubsection{\protect\command{build\_ext}}
Greg Warde78298a2000-04-28 17:12:24 +0000677\label{build-ext-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000678
Greg Wardfacb8db2000-04-09 04:32:40 +0000679\subsubsection{\protect\command{build\_clib}}
Greg Warde78298a2000-04-28 17:12:24 +0000680\label{build-clib-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000681
682
Greg Wardfacb8db2000-04-09 04:32:40 +0000683\subsection{Installing modules: the \protect\command{install} command family}
Greg Warde78298a2000-04-28 17:12:24 +0000684\label{install-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000685
Gregory P. Smith147e5f32000-05-12 00:58:18 +0000686The install command ensures that the build commands have been run and then
687runs the subcommands \command{install\_lib},
688\command{install\_data} and
689\command{install\_scripts}.
690
691\subsubsection{\protect\command{install\_lib}}
692\label{sec:install-lib-cmd}
693
694\subsubsection{\protect\command{install\_data}}
695\label{sec:install-data-cmd}
696This command installs all data files provided with the distribution.
697
698\subsubsection{\protect\command{install\_scripts}}
699\label{sec:install-scripts-cmd}
700This command installs all (Python) scripts in the distribution.
701
Greg Ward16aafcd2000-04-09 04:06:44 +0000702
Greg Wardfacb8db2000-04-09 04:32:40 +0000703\subsection{Cleaning up: the \protect\command{clean} command}
Greg Warde78298a2000-04-28 17:12:24 +0000704\label{clean-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000705
706
Greg Wardfacb8db2000-04-09 04:32:40 +0000707\subsection{Creating a source distribution: the \protect\command{sdist} command}
Greg Warde78298a2000-04-28 17:12:24 +0000708\label{sdist-cmd}
Greg Ward16aafcd2000-04-09 04:06:44 +0000709
710
711\XXX{fragment moved down from above: needs context!}
712The manifest template commands are:
713\begin{tableii}{ll}{command}{Command}{Description}
Greg Ward87da1ea2000-04-21 04:35:25 +0000714 \lineii{include \var{pat1} \var{pat2} ... }
715 {include all files matching any of the listed patterns}
716 \lineii{exclude \var{pat1} \var{pat2} ... }
717 {exclude all files matching any of the listed patterns}
718 \lineii{recursive-include \var{dir} \var{pat1} \var{pat2} ... }
719 {include all files under \var{dir} matching any of the listed patterns}
720 \lineii{recursive-exclude \var{dir} \var{pat1} \var{pat2} ...}
721 {exclude all files under \var{dir} matching any of the listed patterns}
722 \lineii{global-include \var{pat1} \var{pat2} ...}
Greg Ward1bbe3292000-06-25 03:14:13 +0000723 {include all files anywhere in the source tree matching\\&
Greg Ward87da1ea2000-04-21 04:35:25 +0000724 any of the listed patterns}
725 \lineii{global-exclude \var{pat1} \var{pat2} ...}
Greg Ward1bbe3292000-06-25 03:14:13 +0000726 {exclude all files anywhere in the source tree matching\\&
Greg Ward87da1ea2000-04-21 04:35:25 +0000727 any of the listed patterns}
Greg Ward16aafcd2000-04-09 04:06:44 +0000728 \lineii{prune \var{dir}}{exclude all files under \var{dir}}
729 \lineii{graft \var{dir}}{include all files under \var{dir}}
730\end{tableii}
731The patterns here are Unix-style ``glob'' patterns: \code{*} matches any
732sequence of regular filename characters, \code{?} matches any single
733regular filename character, and \code{[\var{range}]} matches any of the
734characters in \var{range} (e.g., \code{a-z}, \code{a-zA-Z},
Greg Wardfacb8db2000-04-09 04:32:40 +0000735\code{a-f0-9\_.}). The definition of ``regular filename character'' is
Greg Ward16aafcd2000-04-09 04:06:44 +0000736platform-specific: on Unix it is anything except slash; on Windows
737anything except backslash or colon; on Mac OS anything except colon.
738\XXX{Windows and Mac OS support not there yet}
739
740
Greg Wardd5767a52000-04-19 22:48:09 +0000741\subsection{Creating a ``built'' distribution: the
742 \protect\command{bdist} command family}
Greg Warde78298a2000-04-28 17:12:24 +0000743\label{bdist-cmds}
Greg Ward16aafcd2000-04-09 04:06:44 +0000744
745
Greg Wardfacb8db2000-04-09 04:32:40 +0000746\subsubsection{\protect\command{blib}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000747
Greg Wardfacb8db2000-04-09 04:32:40 +0000748\subsubsection{\protect\command{blib\_dumb}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000749
Greg Wardfacb8db2000-04-09 04:32:40 +0000750\subsubsection{\protect\command{blib\_rpm}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000751
Greg Wardfacb8db2000-04-09 04:32:40 +0000752\subsubsection{\protect\command{blib\_wise}}
Greg Ward16aafcd2000-04-09 04:06:44 +0000753
754
755
756
757
758
759
760
Greg Wardabc52162000-02-26 00:52:48 +0000761\end{document}