| \documentclass{howto} |
| \usepackage{ltxmarkup} |
| \usepackage{times} |
| \usepackage{distutils} |
| |
| \title{Distributing Python Modules} |
| |
| \author{Greg Ward} |
| \authoraddress{E-mail: \email{gward@python.net}} |
| |
| \makeindex |
| |
| \begin{document} |
| |
| \maketitle |
| \begin{abstract} |
| \noindent |
| This document describes the Python Distribution Utilities |
| (``Distutils'') from the module developer's point-of-view, describing |
| how to use the Distutils to make Python modules and extensions easily |
| available to a wider audience with very little overhead for |
| build/release/install mechanics. |
| \end{abstract} |
| |
| \tableofcontents |
| |
| \section{Introduction} |
| \label{intro} |
| |
| In the past, Python module developers have not had much infrastructure |
| support for distributing modules, nor have Python users had much support |
| for installing and maintaining third-party modules. With the |
| introduction of the Python Distribution Utilities (Distutils for short) |
| in Python 1.6, this situation should start to improve. |
| |
| This document only covers using the Distutils to distribute your Python |
| modules. Using the Distutils does not tie you to Python 1.6, though: |
| the Distutils work just fine with Python 1.5.2, and it is reasonable |
| (and expected to become commonplace) to expect users of Python 1.5.2 to |
| download and install the Distutils separately before they can install |
| your modules. Python 1.6 (or later) users, of course, won't have to add |
| anything to their Python installation in order to use the Distutils to |
| install third-party modules. |
| |
| This document concentrates on the role of developer/distributor: if |
| you're looking for information on installing Python modules, you |
| should refer to the \citetitle[../inst/inst.html]{Installing Python |
| Modules} manual. |
| |
| |
| \section{Concepts \& Terminology} |
| \label{concepts} |
| |
| Using the Distutils is quite simple, both for module developers and for |
| users/administrators installing third-party modules. As a developer, |
| your responsibilites (apart from writing solid, well-documented and |
| well-tested code, of course!) are: |
| \begin{itemize} |
| \item write a setup script (\file{setup.py} by convention) |
| \item (optional) write a setup configuration file |
| \item create a source distribution |
| \item (optional) create one or more built (binary) distributions |
| \end{itemize} |
| Each of these tasks is covered in this document. |
| |
| Not all module developers have access to a multitude of platforms, so |
| it's not always feasible to expect them to create a multitude of built |
| distributions. It is hoped that a class of intermediaries, called |
| \emph{packagers}, will arise to address this need. Packagers will take |
| source distributions released by module developers, build them on one or |
| more platforms, and release the resulting built distributions. Thus, |
| users on the most popular platforms will be able to install most popular |
| Python module distributions in the most natural way for their platform, |
| without having to run a single setup script or compile a line of code. |
| |
| |
| \subsection{A simple example} |
| \label{simple-example} |
| |
| The setup script is usually quite simple, although since it's written in |
| Python, there are no arbitrary limits to what you can do with |
| it.\footnote{But be careful about putting arbitrarily expensive |
| operations in your setup script; unlike, say, Autoconf-style configure |
| scripts, the setup script may be run multiple times in the course of |
| building and installing your module distribution. If you need to |
| insert potentially expensive processing steps into the Distutils |
| chain, see section~\ref{extending} on extending the Distutils.} If |
| all you want to do is distribute a module called \module{foo}, contained |
| in a file \file{foo.py}, then your setup script can be as little as |
| this: |
| \begin{verbatim} |
| from distutils.core import setup |
| setup (name = "foo", |
| version = "1.0", |
| py_modules = ["foo"]) |
| \end{verbatim} |
| |
| Some observations: |
| \begin{itemize} |
| \item most information that you supply to the Distutils is supplied as |
| keyword arguments to the \function{setup()} function |
| \item those keyword arguments fall into two categories: package |
| meta-data (name, version number) and information about what's in the |
| package (a list of pure Python modules, in this case) |
| \item modules are specified by module name, not filename (the same will |
| hold true for packages and extensions) |
| \item it's recommended that you supply a little more meta-data, in |
| particular your name, email address and a URL for the project |
| (see section~\ref{setup-script} for an example) |
| \end{itemize} |
| |
| To create a source distribution for this module, you would create a |
| setup script, \file{setup.py}, containing the above code, and run: |
| \begin{verbatim} |
| python setup.py sdist |
| \end{verbatim} |
| which will create an archive file (e.g., tarball on Unix, zip file on |
| Windows) containing your setup script, \file{setup.py}, and your module, |
| \file{foo.py}. The archive file will be named \file{Foo-1.0.tar.gz} (or |
| \file{.zip}), and will unpack into a directory \file{Foo-1.0}. |
| |
| If an end-user wishes to install your \module{foo} module, all she has |
| to do is download \file{Foo-1.0.tar.gz} (or \file{.zip}), unpack it, |
| and---from the \file{Foo-1.0} directory---run |
| \begin{verbatim} |
| python setup.py install |
| \end{verbatim} |
| which will ultimately copy \file{foo.py} to the appropriate directory |
| for third-party modules in their Python installation. |
| |
| This simple example demonstrates some fundamental concepts of the |
| Distutils: first, both developers and installers have the same basic |
| user interface, i.e. the setup script. The difference is which |
| Distutils \emph{commands} they use: the \command{sdist} command is |
| almost exclusively for module developers, while \command{install} is |
| more often for installers (although most developers will want to install |
| their own code occasionally). |
| |
| If you want to make things really easy for your users, you can create |
| one or more built distributions for them. For instance, if you are |
| running on a Windows machine, and want to make things easy for other |
| Windows users, you can create an executable installer (the most |
| appropriate type of built distribution for this platform) with the |
| \command{bdist\_wininst} command. For example: |
| \begin{verbatim} |
| python setup.py bdist_wininst |
| \end{verbatim} |
| will create an executable installer, \file{Foo-1.0.win32.exe}, in the |
| current directory. |
| |
| \XXX{not implemented yet} |
| (Another way to create executable installers for Windows is with the |
| \command{bdist\_wise} command, which uses Wise---the commercial |
| installer-generator used to create Python's own installer---to create |
| the installer. Wise-based installers are more appropriate for large, |
| industrial-strength applications that need the full capabilities of a |
| ``real'' installer. \command{bdist\_wininst} creates a self-extracting |
| zip file with a minimal user interface, which is enough for small- to |
| medium-sized module collections. You'll need to have version XXX of |
| Wise installed on your system for the \command{bdist\_wise} command to |
| work; it's available from \url{http://foo/bar/baz}.) |
| |
| Currently (Distutils 0.9.2), the are only other useful built |
| distribution format is RPM, implemented by the \command{bdist\_rpm} |
| command. For example, the following command will create an RPM file |
| called \file{Foo-1.0.noarch.rpm}: |
| \begin{verbatim} |
| python setup.py bdist_rpm |
| \end{verbatim} |
| (This uses the \command{rpm} command, so has to be run on an RPM-based |
| system such as Red Hat Linux, SuSE Linux, or Mandrake Linux.) |
| |
| You can find out what distribution formats are available at any time by |
| running |
| \begin{verbatim} |
| python setup.py bdist --help-formats |
| \end{verbatim} |
| |
| |
| \subsection{General Python terminology} |
| \label{python-terms} |
| |
| If you're reading this document, you probably have a good idea of what |
| modules, extensions, and so forth are. Nevertheless, just to be sure |
| that everyone is operating from a common starting point, we offer the |
| following glossary of common Python terms: |
| \begin{description} |
| \item[module] the basic unit of code reusability in Python: a block of |
| code imported by some other code. Three types of modules concern us |
| here: pure Python modules, extension modules, and packages. |
| \item[pure Python module] a module written in Python and contained in a |
| single \file{.py} file (and possibly associated \file{.pyc} and/or |
| \file{.pyo} files). Sometimes referred to as a ``pure module.'' |
| \item[extension module] a module written in the low-level language of |
| the Python implemention: C/C++ for CPython, Java for JPython. |
| Typically contained in a single dynamically loadable pre-compiled |
| file, e.g. a shared object (\file{.so}) file for CPython extensions on |
| Unix, a DLL (given the \file{.pyd} extension) for CPython extensions |
| on Windows, or a Java class file for JPython extensions. (Note that |
| currently, the Distutils only handles C/C++ extensions for CPython.) |
| \item[package] a module that contains other modules; typically contained |
| in a directory in the filesystem and distinguished from other |
| directories by the presence of a file \file{\_\_init\_\_.py}. |
| \item[root package] the root of the hierarchy of packages. (This isn't |
| really a package, since it doesn't have an \file{\_\_init\_\_.py} |
| file. But we have to call it something.) The vast majority of the |
| standard library is in the root package, as are many small, standalone |
| third-party modules that don't belong to a larger module collection. |
| Unlike regular packages, modules in the root package can be found in |
| many directories: in fact, every directory listed in \code{sys.path} |
| can contribute modules to the root package. |
| \end{description} |
| |
| |
| \subsection{Distutils-specific terminology} |
| \label{distutils-term} |
| |
| The following terms apply more specifically to the domain of |
| distributing Python modules using the Distutils: |
| \begin{description} |
| \item[module distribution] a collection of Python modules distributed |
| together as a single downloadable resource and meant to be installed |
| \emph{en masse}. Examples of some well-known module distributions are |
| Numeric Python, PyXML, PIL (the Python Imaging Library), or |
| mxDateTime. (This would be called a \emph{package}, except that term |
| is already taken in the Python context: a single module distribution |
| may contain zero, one, or many Python packages.) |
| \item[pure module distribution] a module distribution that contains only |
| pure Python modules and packages. Sometimes referred to as a ``pure |
| distribution.'' |
| \item[non-pure module distribution] a module distribution that contains |
| at least one extension module. Sometimes referred to as a ``non-pure |
| distribution.'' |
| \item[distribution root] the top-level directory of your source tree (or |
| source distribution); the directory where \file{setup.py} exists and |
| is run from |
| \end{description} |
| |
| |
| \section{Writing the Setup Script} |
| \label{setup-script} |
| |
| The setup script is the centre of all activity in building, |
| distributing, and installing modules using the Distutils. The main |
| purpose of the setup script is to describe your module distribution to |
| the Distutils, so that the various commands that operate on your modules |
| do the right thing. As we saw in section~\ref{simple-example} above, |
| the setup script consists mainly of a call to \function{setup()}, and |
| most information supplied to the Distutils by the module developer is |
| supplied as keyword arguments to \function{setup()}. |
| |
| Here's a slightly more involved example, which we'll follow for the next |
| couple of sections: the Distutils' own setup script. (Keep in mind that |
| although the Distutils are included with Python 1.6 and later, they also |
| have an independent existence so that Python 1.5.2 users can use them to |
| install other module distributions. The Distutils' own setup script, |
| shown here, is used to install the package into Python 1.5.2.) |
| |
| \begin{verbatim} |
| #!/usr/bin/env python |
| |
| from distutils.core import setup |
| |
| setup (name = "Distutils", |
| version = "1.0", |
| description = "Python Distribution Utilities", |
| author = "Greg Ward", |
| author_email = "gward@python.net", |
| url = "http://www.python.org/sigs/distutils-sig/", |
| |
| packages = ['distutils', 'distutils.command'], |
| ) |
| \end{verbatim} |
| There are only two differences between this and the trivial one-file |
| distribution presented in section~\ref{simple-example}: more |
| meta-data, and the specification of pure Python modules by package, |
| rather than by module. This is important since the Distutils consist of |
| a couple of dozen modules split into (so far) two packages; an explicit |
| list of every module would be tedious to generate and difficult to |
| maintain. |
| |
| Note that any pathnames (files or directories) supplied in the setup |
| script should be written using the Unix convention, i.e. |
| slash-separated. The Distutils will take care of converting this |
| platform-neutral representation into whatever is appropriate on your |
| current platform before actually using the pathname. This makes your |
| setup script portable across operating systems, which of course is one |
| of the major goals of the Distutils. In this spirit, all pathnames in |
| this document are slash-separated (Mac OS programmers should keep in |
| mind that the \emph{absence} of a leading slash indicates a relative |
| path, the opposite of the Mac OS convention with colons). |
| |
| |
| \subsection{Listing whole packages} |
| \label{listing-packages} |
| |
| The \option{packages} option tells the Distutils to process (build, |
| distribute, install, etc.) all pure Python modules found in each package |
| mentioned in the \option{packages} list. In order to do this, of |
| course, there has to be a correspondence between package names and |
| directories in the filesystem. The default correspondence is the most |
| obvious one, i.e. package \module{distutils} is found in the directory |
| \file{distutils} relative to the distribution root. Thus, when you say |
| \code{packages = ['foo']} in your setup script, you are promising that |
| the Distutils will find a file \file{foo/\_\_init\_\_.py} (which might |
| be spelled differently on your system, but you get the idea) relative to |
| the directory where your setup script lives. (If you break this |
| promise, the Distutils will issue a warning but process the broken |
| package anyways.) |
| |
| If you use a different convention to lay out your source directory, |
| that's no problem: you just have to supply the \option{package\_dir} |
| option to tell the Distutils about your convention. For example, say |
| you keep all Python source under \file{lib}, so that modules in the |
| ``root package'' (i.e., not in any package at all) are right in |
| \file{lib}, modules in the \module{foo} package are in \file{lib/foo}, |
| and so forth. Then you would put |
| \begin{verbatim} |
| package_dir = {'': 'lib'} |
| \end{verbatim} |
| in your setup script. (The keys to this dictionary are package names, |
| and an empty package name stands for the root package. The values are |
| directory names relative to your distribution root.) In this case, when |
| you say \code{packages = ['foo']}, you are promising that the file |
| \file{lib/foo/\_\_init\_\_.py} exists. |
| |
| Another possible convention is to put the \module{foo} package right in |
| \file{lib}, the \module{foo.bar} package in \file{lib/bar}, etc. This |
| would be written in the setup script as |
| \begin{verbatim} |
| package_dir = {'foo': 'lib'} |
| \end{verbatim} |
| A \code{\var{package}: \var{dir}} entry in the \option{package\_dir} |
| dictionary implicitly applies to all packages below \var{package}, so |
| the \module{foo.bar} case is automatically handled here. In this |
| example, having \code{packages = ['foo', 'foo.bar']} tells the Distutils |
| to look for \file{lib/\_\_init\_\_.py} and |
| \file{lib/bar/\_\_init\_\_.py}. (Keep in mind that although |
| \option{package\_dir} applies recursively, you must explicitly list all |
| packages in \option{packages}: the Distutils will \emph{not} recursively |
| scan your source tree looking for any directory with an |
| \file{\_\_init\_\_.py} file.) |
| |
| |
| \subsection{Listing individual modules} |
| \label{listing-modules} |
| |
| For a small module distribution, you might prefer to list all modules |
| rather than listing packages---especially the case of a single module |
| that goes in the ``root package'' (i.e., no package at all). This |
| simplest case was shown in section~\ref{simple-example}; here is a |
| slightly more involved example: |
| \begin{verbatim} |
| py_modules = ['mod1', 'pkg.mod2'] |
| \end{verbatim} |
| This describes two modules, one of them in the ``root'' package, the |
| other in the \module{pkg} package. Again, the default package/directory |
| layout implies that these two modules can be found in \file{mod1.py} and |
| \file{pkg/mod2.py}, and that \file{pkg/\_\_init\_\_.py} exists as well. |
| And again, you can override the package/directory correspondence using |
| the \option{package\_dir} option. |
| |
| |
| \subsection{Describing extension modules} |
| \label{describing-extensions} |
| |
| Just as writing Python extension modules is a bit more complicated than |
| writing pure Python modules, describing them to the Distutils is a bit |
| more complicated. Unlike pure modules, it's not enough just to list |
| modules or packages and expect the Distutils to go out and find the |
| right files; you have to specify the extension name, source file(s), and |
| any compile/link requirements (include directories, libraries to link |
| with, etc.). |
| |
| All of this is done through another keyword argument to |
| \function{setup()}, the \option{extensions} option. \option{extensions} |
| is just a list of \class{Extension} instances, each of which describes a |
| single extension module. Suppose your distribution includes a single |
| extension, called \module{foo} and implemented by \file{foo.c}. If no |
| additional instructions to the compiler/linker are needed, describing |
| this extension is quite simple: |
| \begin{verbatim} |
| Extension("foo", ["foo.c"]) |
| \end{verbatim} |
| The \class{Extension} class can be imported from |
| \module{distutils.core}, along with \function{setup()}. Thus, the setup |
| script for a module distribution that contains only this one extension |
| and nothing else might be: |
| \begin{verbatim} |
| from distutils.core import setup, Extension |
| setup(name = "foo", version = "1.0", |
| extensions = [Extension("foo", ["foo.c"])]) |
| \end{verbatim} |
| |
| The \class{Extension} class (actually, the underlying extension-building |
| machinery implemented by the \command{built\_ext} command) supports a |
| great deal of flexibility in describing Python extensions, which is |
| explained in the following sections. |
| |
| |
| \subsubsection{Extension names and packages} |
| |
| The first argument to the \class{Extension} constructor is always the |
| name of the extension, including any package names. For example, |
| \begin{verbatim} |
| Extension("foo", ["src/foo1.c", "src/foo2.c"]) |
| \end{verbatim} |
| describes an extension that lives in the root package, while |
| \begin{verbatim} |
| Extension("pkg.foo", ["src/foo1.c", "src/foo2.c"]) |
| \end{verbatim} |
| describes the same extension in the \module{pkg} package. The source |
| files and resulting object code are identical in both cases; the only |
| difference is where in the filesystem (and therefore where in Python's |
| namespace hierarchy) the resulting extension lives. |
| |
| If you have a number of extensions all in the same package (or all under |
| the same base package), use the \option{ext\_package} keyword argument |
| to \function{setup()}. For example, |
| \begin{verbatim} |
| setup(... |
| ext_package = "pkg", |
| extensions = [Extension("foo", ["foo.c"]), |
| Extension("subpkg.bar", ["bar.c"])] |
| ) |
| \end{verbatim} |
| will compile \file{foo.c} to the extension \module{pkg.foo}, and |
| \file{bar.c} to \module{pkg.subpkg.bar}. |
| |
| |
| \subsubsection{Extension source files} |
| |
| The second argument to the \class{Extension} constructor is a list of |
| source files. Since the Distutils currently only support C/C++ |
| extensions, these are normally C/C++ source files. (Be sure to use |
| appropriate extensions to distinguish C++ source files: \file{.cc} and |
| \file{.cpp} seem to be recognized by both Unix and Windows compilers.) |
| |
| However, you can also include SWIG interface (\file{.i}) files in the |
| list; the \command{build\_ext} command knows how to deal with SWIG |
| extensions: it will run SWIG on the interface file and compile the |
| resulting C/C++ file into your extension. |
| |
| \XXX{SWIG support is rough around the edges and largely untested; |
| especially SWIG support of C++ extensions! Explain in more detail |
| here when the interface firms up.} |
| |
| On some platforms, you can include non-source files that are processed |
| by the compiler and included in your extension. Currently, this just |
| means Windows resource files for Visual C++. \XXX{get more detail on |
| this feature from Thomas Heller!} |
| |
| |
| \subsubsection{Preprocessor options} |
| |
| Three optional arguments to \class{Extension} will help if you need to |
| specify include directories to search or preprocessor macros to |
| define/undefine: \code{include\_dirs}, \code{define\_macros}, and |
| \code{undef\_macros}. |
| |
| For example, if your extension requires header files in the |
| \file{include} directory under your distribution root, use the |
| \code{include\_dirs} option: |
| \begin{verbatim} |
| Extension("foo", ["foo.c"], include_dirs=["include"]) |
| \end{verbatim} |
| |
| You can specify absolute directories there; if you know that your |
| extension will only be built on Unix systems with X11R6 installed to |
| \file{/usr}, you can get away with |
| \begin{verbatim} |
| Extension("foo", ["foo.c"], include_dirs=["/usr/include/X11"]) |
| \end{verbatim} |
| You should avoid this sort of non-portable usage if you plan to |
| distribute your code: it's probably better to write your code to include |
| (e.g.) \code{<X11/Xlib.h>}. |
| |
| If you need to include header files from some other Python extension, |
| you can take advantage of the fact that the Distutils install extension |
| header files in a consistent way. For example, the Numerical Python |
| header files are installed (on a standard Unix installation) to |
| \file{/usr/local/include/python1.5/Numerical}. (The exact location will |
| differ according to your platform and Python installation.) Since the |
| Python include directory---\file{/usr/local/include/python1.5} in this |
| case---is always included in the search path when building Python |
| extensions, the best approach is to include (e.g.) |
| \code{<Numerical/arrayobject.h>}. If you insist on putting the |
| \file{Numerical} include directory right into your header search path, |
| though, you can find that directory using the Distutils |
| \module{sysconfig} module: |
| \begin{verbatim} |
| from distutils.sysconfig import get_python_inc |
| incdir = os.path.join(get_python_inc(plat_specific=1), "Numerical") |
| setup(..., |
| Extension(..., include_dirs=[incdir])) |
| \end{verbatim} |
| Even though this is quite portable---it will work on any Python |
| installation, regardless of platform---it's probably easier to just |
| write your C code in the sensible way. |
| |
| You can define and undefine pre-processor macros with the |
| \code{define\_macros} and \code{undef\_macros} options. |
| \code{define\_macros} takes a list of \code{(name, value)} tuples, where |
| \code{name} is the name of the macro to define (a string) and |
| \code{value} is its value: either a string or \code{None}. (Defining a |
| macro \code{FOO} to \code{None} is the equivalent of a bare |
| \code{\#define FOO} in your C source: with most compilers, this sets |
| \code{FOO} to the string \code{1}.) \code{undef\_macros} is just |
| a list of macros to undefine. |
| |
| For example: |
| \begin{verbatim} |
| Extension(..., |
| define_macros=[('NDEBUG', '1')], |
| ('HAVE_STRFTIME', None), |
| undef_macros=['HAVE_FOO', 'HAVE_BAR']) |
| \end{verbatim} |
| is the equivalent of having this at the top of every C source file: |
| \begin{verbatim} |
| #define NDEBUG 1 |
| #define HAVE_STRFTIME |
| #undef HAVE_FOO |
| #undef HAVE_BAR |
| \end{verbatim} |
| |
| |
| \subsubsection{Library options} |
| |
| You can also specify the libraries to link against when building your |
| extension, and the directories to search for those libraries. The |
| \code{libraries} option is a list of libraries to link against, |
| \code{library\_dirs} is a list of directories to search for libraries at |
| link-time, and \code{runtime\_library\_dirs} is a list of directories to |
| search for shared (dynamically loaded) libraries at run-time. |
| |
| For example, if you need to link against libraries known to be in the |
| standard library search path on target systems |
| \begin{verbatim} |
| Extension(..., |
| libraries=["gdbm", "readline"]) |
| \end{verbatim} |
| |
| If you need to link with libraries in a non-standard location, you'll |
| have to include the location in \code{library\_dirs}: |
| \begin{verbatim} |
| Extension(..., |
| library_dirs=["/usr/X11R6/lib"], |
| libraries=["X11", "Xt"]) |
| \end{verbatim} |
| (Again, this sort of non-portable construct should be avoided if you |
| intend to distribute your code.) |
| |
| \XXX{still undocumented: extra\_objects, extra\_compile\_args, |
| extra\_link\_args, export\_symbols---none of which are frequently |
| needed, some of which might be completely unnecessary!} |
| |
| |
| \section{Writing the Setup Configuration File} |
| \label{setup-config} |
| |
| Often, it's not possible to write down everything needed to build a |
| distribution \emph{a priori}: you may need to get some information from |
| the user, or from the user's system, in order to proceed. As long as |
| that information is fairly simple---a list of directories to search for |
| C header files or libraries, for example---then providing a |
| configuration file, \file{setup.cfg}, for users to edit is a cheap and |
| easy way to solicit it. Configuration files also let you provide |
| default values for any command option, which the installer can then |
| override either on the command-line or by editing the config file. |
| |
| (If you have more advanced needs, such as determining which extensions |
| to build based on what capabilities are present on the target system, |
| then you need the Distutils ``auto-configuration'' facility. This |
| started to appear in Distutils 0.9 but, as of this writing, isn't mature |
| or stable enough yet for real-world use.) |
| |
| \XXX{should reference description of distutils config files in |
| ``Installing'' manual here} |
| |
| The setup configuration file is a useful middle-ground between the setup |
| script---which, ideally, would be opaque to installers\footnote{This |
| ideal probably won't be achieved until auto-configuration is fully |
| supported by the Distutils.}---and the command-line to the setup |
| script, which is outside of your control and entirely up to the |
| installer. In fact, \file{setup.cfg} (and any other Distutils |
| configuration files present on the target system) are processed after |
| the contents of the setup script, but before the command-line. This has |
| several useful consequences: |
| \begin{itemize} |
| \item installers can override some of what you put in \file{setup.py} by |
| editing \file{setup.cfg} |
| \item you can provide non-standard defaults for options that are not |
| easily set in \file{setup.py} |
| \item installers can override anything in \file{setup.cfg} using the |
| command-line options to \file{setup.py} |
| \end{itemize} |
| |
| The basic syntax of the configuration file is simple: |
| \begin{verbatim} |
| [command] |
| option=value |
| ... |
| \end{verbatim} |
| where \var{command} is one of the Distutils commands (e.g. |
| \command{build\_py}, \command{install}), and \var{option} is one of the |
| options that command supports. Any number of options can be supplied |
| for each command, and any number of command sections can be included in |
| the file. Blank lines are ignored, as are comments (from a \verb+#+ |
| character to end-of-line). Long option values can be split across |
| multiple lines simply by indenting the continuation lines. |
| |
| You can find out the list of options supported by a particular command |
| with the universal \longprogramopt{help} option, e.g. |
| \begin{verbatim} |
| > python setup.py --help build_ext |
| [...] |
| Options for 'build_ext' command: |
| --build-lib (-b) directory for compiled extension modules |
| --build-temp (-t) directory for temporary files (build by-products) |
| --inplace (-i) ignore build-lib and put compiled extensions into the |
| source directory alongside your pure Python modules |
| --include-dirs (-I) list of directories to search for header files |
| --define (-D) C preprocessor macros to define |
| --undef (-U) C preprocessor macros to undefine |
| [...] |
| \end{verbatim} |
| Or consult section \ref{reference} of this document (the command |
| reference). |
| |
| Note that an option spelled \longprogramopt{foo-bar} on the command-line |
| is spelled \option{foo\_bar} in configuration files. |
| |
| For example, say you want your extensions to be built |
| ``in-place''---that is, you have an extension \module{pkg.ext}, and you |
| want the compiled extension file (\file{ext.so} on Unix, say) to be put |
| in the same source directory as your pure Python modules |
| \module{pkg.mod1} and \module{pkg.mod2}. You can always use the |
| \longprogramopt{inplace} option on the command-line to ensure this: |
| \begin{verbatim} |
| python setup.py build_ext --inplace |
| \end{verbatim} |
| But this requires that you always specify the \command{build\_ext} |
| command explicitly, and remember to provide \longprogramopt{inplace}. |
| An easier way is to ``set and forget'' this option, by encoding it in |
| \file{setup.cfg}, the configuration file for this distribution: |
| \begin{verbatim} |
| [build_ext] |
| inplace=1 |
| \end{verbatim} |
| This will affect all builds of this module distribution, whether or not |
| you explcitly specify \command{build\_ext}. If you include |
| \file{setup.cfg} in your source distribution, it will also affect |
| end-user builds---which is probably a bad idea for this option, since |
| always building extensions in-place would break installation of the |
| module distribution. In certain peculiar cases, though, modules are |
| built right in their installation directory, so this is conceivably a |
| useful ability. (Distributing extensions that expect to be built in |
| their installation directory is almost always a bad idea, though.) |
| |
| Another example: certain commands take a lot of options that don't |
| change from run-to-run; for example, \command{bdist\_rpm} needs to know |
| everything required to generate a ``spec'' file for creating an RPM |
| distribution. Some of this information comes from the setup script, and |
| some is automatically generated by the Distutils (such as the list of |
| files installed). But some of it has to be supplied as options to |
| \command{bdist\_rpm}, which would be very tedious to do on the |
| command-line for every run. Hence, here is a snippet from the |
| Distutils' own \file{setup.cfg}: |
| \begin{verbatim} |
| [bdist_rpm] |
| release = 1 |
| packager = Greg Ward <gward@python.net> |
| doc_files = CHANGES.txt |
| README.txt |
| USAGE.txt |
| doc/ |
| examples/ |
| \end{verbatim} |
| Note that the \option{doc\_files} option is simply a |
| whitespace-separated string split across multiple lines for readability. |
| |
| |
| \section{Creating a Source Distribution} |
| \label{source-dist} |
| |
| As shown in section~\ref{simple-example}, you use the |
| \command{sdist} command to create a source distribution. In the |
| simplest case, |
| \begin{verbatim} |
| python setup.py sdist |
| \end{verbatim} |
| (assuming you haven't specified any \command{sdist} options in the setup |
| script or config file), \command{sdist} creates the archive of the |
| default format for the current platform. The default format is gzip'ed |
| tar file (\file{.tar.gz}) on Unix, and ZIP file on Windows. \XXX{no Mac |
| OS support here} |
| |
| You can specify as many formats as you like using the |
| \longprogramopt{formats} option, for example: |
| \begin{verbatim} |
| python setup.py sdist --formats=gztar,zip |
| \end{verbatim} |
| to create a gzipped tarball and a zip file. The available formats are: |
| \begin{tableiii}{l|l|c}{code}% |
| {Format}{Description}{Notes} |
| \lineiii{zip}{zip file (\file{.zip})}{(1),(3)} |
| \lineiii{gztar}{gzip'ed tar file (\file{.tar.gz})}{(2),(4)} |
| \lineiii{bztar}{bzip2'ed tar file (\file{.tar.gz})}{(4)} |
| \lineiii{ztar}{compressed tar file (\file{.tar.Z})}{(4)} |
| \lineiii{tar}{tar file (\file{.tar})}{(4)} |
| \end{tableiii} |
| |
| \noindent Notes: |
| \begin{description} |
| \item[(1)] default on Windows |
| \item[(2)] default on Unix |
| \item[(3)] under both Unix and Windows, requires either external |
| Info-ZIP utility \emph{or} the \module{zipfile} module |
| \item[(4)] requires external utilities: \program{tar} and possibly one |
| of \program{gzip}, \program{bzip2}, or \program{compress} |
| \end{description} |
| |
| |
| |
| \subsection{Specifying the files to distribute} |
| \label{manifest} |
| |
| If you don't supply an explicit list of files (or instructions on how to |
| generate one), the \command{sdist} command puts a minimal default set |
| into the source distribution: |
| \begin{itemize} |
| \item all Python source files implied by the \option{py\_modules} and |
| \option{packages} options |
| \item all C source files mentioned in the \option{ext\_modules} or |
| \option{libraries} options (\XXX{getting C library sources currently |
| broken -- no get\_source\_files() method in build\_clib.py!}) |
| \item anything that looks like a test script: \file{test/test*.py} |
| (currently, the Distutils don't do anything with test scripts except |
| include them in source distributions, but in the future there will be |
| a standard for testing Python module distributions) |
| \item \file{README.txt} (or \file{README}), \file{setup.py} (or whatever |
| you called your setup script), and \file{setup.cfg} |
| \end{itemize} |
| Sometimes this is enough, but usually you will want to specify |
| additional files to distribute. The typical way to do this is to write |
| a \emph{manifest template}, called \file{MANIFEST.in} by default. The |
| manifest template is just a list of instructions for how to generate |
| your manifest file, \file{MANIFEST}, which is the exact list of files to |
| include in your source distribution. The \command{sdist} command |
| processes this template and generates a manifest based on its |
| instructions and what it finds in the filesystem. |
| |
| If you prefer to roll your own manifest file, the format is simple: one |
| filename per line, regular files (or symlinks to them) only. If you do |
| supply your own \file{MANIFEST}, you must specify everything: the |
| default set of files described above does not apply in this case. |
| |
| The manifest template has one command per line, where each command |
| specifies a set of files to include or exclude from the source |
| distribution. For an example, again we turn to the Distutils' own |
| manifest template: |
| \begin{verbatim} |
| include *.txt |
| recursive-include examples *.txt *.py |
| prune examples/sample?/build |
| \end{verbatim} |
| The meanings should be fairly clear: include all files in the |
| distribution root matching \code{*.txt}, all files anywhere under the |
| \file{examples} directory matching \code{*.txt} or \code{*.py}, and |
| exclude all directories matching \code{examples/sample?/build}. All of |
| this is done \emph{after} the standard include set, so you can exclude |
| files from the standard set with explicit instructions in the manifest |
| template. (Or, you can use the \longprogramopt{no-defaults} option to |
| disable the standard set entirely.) There are several other commands |
| available in the manifest template mini-language; see |
| section~\ref{sdist-cmd}. |
| |
| The order of commands in the manifest template matters: initially, we |
| have the list of default files as described above, and each command in |
| the template adds to or removes from that list of files. Once we have |
| fully processed the manifest template, we remove files that should not |
| be included in the source distribution: |
| \begin{itemize} |
| \item all files in the Distutils ``build'' tree (default \file{build/}) |
| \item all files in directories named \file{RCS} or \file{CVS} |
| \end{itemize} |
| Now we have our complete list of files, which is written to the manifest |
| for future reference, and then used to build the source distribution |
| archive(s). |
| |
| You can disable the default set of included files with the |
| \longprogramopt{no-defaults} option, and you can disable the standard |
| exclude set with \longprogramopt{no-prune}. |
| |
| Following the Distutils' own manifest template, let's trace how the |
| \command{sdist} command builds the list of files to include in the |
| Distutils source distribution: |
| \begin{enumerate} |
| \item include all Python source files in the \file{distutils} and |
| \file{distutils/command} subdirectories (because packages |
| corresponding to those two directories were mentioned in the |
| \option{packages} option in the setup script---see |
| section~\ref{setup-script}) |
| \item include \file{README.txt}, \file{setup.py}, and \file{setup.cfg} |
| (standard files) |
| \item include \file{test/test*.py} (standard files) |
| \item include \file{*.txt} in the distribution root (this will find |
| \file{README.txt} a second time, but such redundancies are weeded out |
| later) |
| \item include anything matching \file{*.txt} or \file{*.py} in the |
| sub-tree under \file{examples}, |
| \item exclude all files in the sub-trees starting at directories |
| matching \file{examples/sample?/build}---this may exclude files |
| included by the previous two steps, so it's important that the |
| \code{prune} command in the manifest template comes after the |
| \code{recursive-include} command |
| \item exclude the entire \file{build} tree, and any \file{RCS} or |
| \file{CVS} directories |
| \end{enumerate} |
| Just like in the setup script, file and directory names in the manifest |
| template should always be slash-separated; the Distutils will take care |
| of converting them to the standard representation on your platform. |
| That way, the manifest template is portable across operating systems. |
| |
| |
| \subsection{Manifest-related options} |
| \label{manifest-options} |
| |
| The normal course of operations for the \command{sdist} command is as |
| follows: |
| \begin{itemize} |
| \item if the manifest file, \file{MANIFEST} doesn't exist, read |
| \file{MANIFEST.in} and create the manifest |
| \item if neither \file{MANIFEST} nor \file{MANIFEST.in} exist, create a |
| manifest with just the default file set\footnote{In versions of the |
| Distutils up to and including 0.9.2 (Python 2.0b1), this feature was |
| broken; use the \programopt{-f} (\longprogramopt{force-manifest}) |
| option to work around the bug.} |
| \item if either \file{MANIFEST.in} or the setup script (\file{setup.py}) |
| are more recent than \file{MANIFEST}, recreate \file{MANIFEST} by |
| reading \file{MANIFEST.in} |
| \item use the list of files now in \file{MANIFEST} (either just |
| generated or read in) to create the source distribution archive(s) |
| \end{itemize} |
| There are a couple of options that modify this behaviour. First, use |
| the \longprogramopt{no-defaults} and \longprogramopt{no-prune} to |
| disable the standard ``include'' and ``exclude'' sets.\footnote{Note |
| that if you have no manifest template, no manifest, and use the |
| \longprogramopt{no-defaults}, you will get an empty manifest. Another |
| bug in Distutils 0.9.2 and earlier causes an uncaught exception in |
| this case. The workaround is: Don't Do That.} |
| |
| Second, you might want to force the manifest to be regenerated---for |
| example, if you have added or removed files or directories that match an |
| existing pattern in the manifest template, you should regenerate the |
| manifest: |
| \begin{verbatim} |
| python setup.py sdist --force-manifest |
| \end{verbatim} |
| |
| Or, you might just want to (re)generate the manifest, but not create a |
| source distribution: |
| \begin{verbatim} |
| python setup.py sdist --manifest-only |
| \end{verbatim} |
| \longprogramopt{manifest-only} implies \longprogramopt{force-manifest}. |
| \programopt{-o} is a shortcut for \longprogramopt{manifest-only}, and |
| \programopt{-f} for \longprogramopt{force-manifest}. |
| |
| |
| \section{Creating Built Distributions} |
| \label{built-dist} |
| |
| A ``built distribution'' is what you're probably used to thinking of |
| either as a ``binary package'' or an ``installer'' (depending on your |
| background). It's not necessarily binary, though, because it might |
| contain only Python source code and/or byte-code; and we don't call it a |
| package, because that word is already spoken for in Python. (And |
| ``installer'' is a term specific to the Windows world. \XXX{do Mac |
| people use it?}) |
| |
| A built distribution is how you make life as easy as possible for |
| installers of your module distribution: for users of RPM-based Linux |
| systems, it's a binary RPM; for Windows users, it's an executable |
| installer; for Debian-based Linux users, it's a Debian package; and so |
| forth. Obviously, no one person will be able to create built |
| distributions for every platform under the sun, so the Distutils is |
| designed to enable module developers to concentrate on their |
| specialty---writing code and creating source distributions---while an |
| intermediary species of \emph{packager} springs up to turn source |
| distributions into built distributions for as many platforms as there |
| are packagers. |
| |
| Of course, the module developer could be his own packager; or the |
| packager could be a volunteer ``out there'' somewhere who has access to |
| a platform which the original developer does not; or it could be |
| software periodically grabbing new source distributions and turning them |
| into built distributions for as many platforms as the software has |
| access to. Regardless of the nature of the beast, a packager uses the |
| setup script and the \command{bdist} command family to generate built |
| distributions. |
| |
| As a simple example, if I run the following command in the Distutils |
| source tree: |
| \begin{verbatim} |
| python setup.py bdist |
| \end{verbatim} |
| then the Distutils builds my module distribution (the Distutils itself |
| in this case), does a ``fake'' installation (also in the \file{build} |
| directory), and creates the default type of built distribution for my |
| platform. Currently, the default format for built distributions is a |
| ``dumb'' archive---tarball on Unix, ZIP file on Windows. (These are |
| called ``dumb'' built distributions, because they must be unpacked in a |
| specific location to work.) |
| |
| Thus, the above command on a Unix system creates |
| \file{Distutils-0.9.1.\filevar{plat}.tar.gz}; unpacking this tarball |
| from the root of the filesystemq installs the Distutils just as though |
| you had downloaded the source distribution and run \code{python setup.py |
| install}. (Assuming that the target system has their Python |
| installation laid out the same as you do---another reason these are |
| called ``dumb'' distributions.) Obviously, for pure Python |
| distributions, this isn't a huge win---but for non-pure distributions, |
| which include extensions that would need to be compiled, it can mean the |
| difference between someone being able to use your extensions or not. |
| |
| \XXX{filenames are inaccurate here!} |
| |
| The \command{bdist} command has a \longprogramopt{format} option, |
| similar to the \command{sdist} command, which you can use to select the |
| types of built distribution to generate: for example, |
| \begin{verbatim} |
| python setup.py bdist --format=zip |
| \end{verbatim} |
| would, when run on a Unix system, create |
| \file{Distutils-0.8.\filevar{plat}.zip}---again, this archive would be |
| unpacked from the root directory to install the Distutils. |
| |
| The available formats for built distributions are: |
| \begin{tableiii}{l|l|c}{code}% |
| {Format}{Description}{Notes} |
| \lineiii{zip}{zip file (\file{.zip})}{} |
| \lineiii{gztar}{gzipped tar file (\file{.tar.gz})}{(1)} |
| \lineiii{ztar}{compressed tar file (\file{.tar.Z})}{} |
| \lineiii{tar}{tar file (\file{.tar})}{} |
| \lineiii{rpm}{RPM}{} |
| \lineiii{srpm}{source RPM}{\XXX{to do!}} |
| \lineiii{wininst}{self-extracting ZIP file for Windows}{(2)} |
| %\lineiii{wise}{Wise installer for Windows}{(3)} |
| \end{tableiii} |
| |
| \noindent Notes: |
| \begin{description} |
| \item[(1)] default on Unix |
| \item[(2)] default on Windows \XXX{to-do!} |
| %\item[(3)] not implemented yet |
| \end{description} |
| |
| You don't have to use the \command{bdist} command with the |
| \longprogramopt{formats} option; you can also use the command that |
| directly implements the format you're interested in. Some of these |
| \command{bdist} ``sub-commands'' actually generate several similar |
| formats; for instance, the \command{bdist\_dumb} command generates all |
| the ``dumb'' archive formats (\code{tar}, \code{ztar}, \code{gztar}, and |
| \code{zip}), and \command{bdist\_rpm} generates both binary and source |
| RPMs. The \command{bdist} sub-commands, and the formats generated by |
| each, are: |
| \begin{tableii}{l|l}{command}% |
| {Command}{Formats} |
| \lineii{bdist\_dumb}{tar, ztar, gztar, zip} |
| \lineii{bdist\_rpm}{rpm, srpm} |
| \lineii{bdist\_wininst}{wininst} |
| %\lineii{bdist\_wise}{wise} |
| \end{tableii} |
| |
| \section{Examples} |
| \label{examples} |
| |
| |
| \subsection{Pure Python distribution (by module)} |
| \label{pure-mod} |
| |
| |
| \subsection{Pure Python distribution (by package)} |
| \label{pure-pkg} |
| |
| |
| \subsection{Single extension module} |
| \label{single-ext} |
| |
| |
| \subsection{Multiple extension modules} |
| \label{multiple-ext} |
| |
| |
| \subsection{Putting it all together} |
| |
| |
| |
| \section{Extending the Distutils} |
| \label{extending} |
| |
| |
| \subsection{Extending existing commands} |
| \label{extend-existing} |
| |
| |
| \subsection{Writing new commands} |
| \label{new-commands} |
| |
| |
| |
| \section{Reference} |
| \label{reference} |
| |
| |
| \subsection{Building modules: the \protect\command{build} command family} |
| \label{build-cmds} |
| |
| \subsubsection{\protect\command{build}} |
| \label{build-cmd} |
| |
| \subsubsection{\protect\command{build\_py}} |
| \label{build-py-cmd} |
| |
| \subsubsection{\protect\command{build\_ext}} |
| \label{build-ext-cmd} |
| |
| \subsubsection{\protect\command{build\_clib}} |
| \label{build-clib-cmd} |
| |
| |
| \subsection{Installing modules: the \protect\command{install} command family} |
| \label{install-cmd} |
| |
| The install command ensures that the build commands have been run and then |
| runs the subcommands \command{install\_lib}, |
| \command{install\_data} and |
| \command{install\_scripts}. |
| |
| \subsubsection{\protect\command{install\_lib}} |
| \label{install-lib-cmd} |
| |
| \subsubsection{\protect\command{install\_data}} |
| \label{install-data-cmd} |
| This command installs all data files provided with the distribution. |
| |
| \subsubsection{\protect\command{install\_scripts}} |
| \label{install-scripts-cmd} |
| This command installs all (Python) scripts in the distribution. |
| |
| |
| \subsection{Cleaning up: the \protect\command{clean} command} |
| \label{clean-cmd} |
| |
| |
| \subsection{Creating a source distribution: the \protect\command{sdist} command} |
| \label{sdist-cmd} |
| |
| |
| \XXX{fragment moved down from above: needs context!} |
| The manifest template commands are: |
| \begin{tableii}{ll}{command}{Command}{Description} |
| \lineii{include \var{pat1} \var{pat2} ... } |
| {include all files matching any of the listed patterns} |
| \lineii{exclude \var{pat1} \var{pat2} ... } |
| {exclude all files matching any of the listed patterns} |
| \lineii{recursive-include \var{dir} \var{pat1} \var{pat2} ... } |
| {include all files under \var{dir} matching any of the listed patterns} |
| \lineii{recursive-exclude \var{dir} \var{pat1} \var{pat2} ...} |
| {exclude all files under \var{dir} matching any of the listed patterns} |
| \lineii{global-include \var{pat1} \var{pat2} ...} |
| {include all files anywhere in the source tree matching\\& |
| any of the listed patterns} |
| \lineii{global-exclude \var{pat1} \var{pat2} ...} |
| {exclude all files anywhere in the source tree matching\\& |
| any of the listed patterns} |
| \lineii{prune \var{dir}}{exclude all files under \var{dir}} |
| \lineii{graft \var{dir}}{include all files under \var{dir}} |
| \end{tableii} |
| The patterns here are Unix-style ``glob'' patterns: \code{*} matches any |
| sequence of regular filename characters, \code{?} matches any single |
| regular filename character, and \code{[\var{range}]} matches any of the |
| characters in \var{range} (e.g., \code{a-z}, \code{a-zA-Z}, |
| \code{a-f0-9\_.}). The definition of ``regular filename character'' is |
| platform-specific: on Unix it is anything except slash; on Windows |
| anything except backslash or colon; on Mac OS anything except colon. |
| \XXX{Windows and Mac OS support not there yet} |
| |
| |
| \subsection{Creating a ``built'' distribution: the |
| \protect\command{bdist} command family} |
| \label{bdist-cmds} |
| |
| |
| \subsubsection{\protect\command{blib}} |
| |
| \subsubsection{\protect\command{blib\_dumb}} |
| |
| \subsubsection{\protect\command{blib\_rpm}} |
| |
| \subsubsection{\protect\command{blib\_wise}} |
| |
| |
| |
| |
| |
| |
| |
| |
| \end{document} |