Doc/inst/inst.tex

\documentclass{howto}
\usepackage{ltxmarkup}
\usepackage{times}

\title{Installing Python Modules}

% The audience for this document includes people who don't know anything 
% about Python and aren't about to learn the language just in order to
% install and maintain it for their users, i.e. system administrators.
% Thus, I have to be sure to explain the basics at some point:
% sys.path and PYTHONPATH at least.  Should probably give pointers to
% other docs on "import site", PYTHONSTARTUP, PYTHONHOME, etc.
% 
% Also, I need to take into account that most modules out there don't
% (yet) use Distutils: briefly explain the old Makefile.pre.in
% convention (maybe move material from the E&E manual to here?), and
% explain where to copy .py and .so files manually if the distribution
% doesn't provide a mechanism for doing so.
%
% Finally, it might be useful to include all the material from my "Care
% and Feeding of a Python Installation" talk in here somewhere.  Yow!

% Hey wow, Guido didn't write this one either!
\author{Greg Ward}
\authoraddress{E-mail: \email{gward@python.net}}

% Should these be added to the standard Python doc tools?  (They'll be
% needed for my "Distributing Python Modules" guide, too.)
\newcommand{\command}[1]{\code{#1}}
\newcommand{\option}[1]{\code{#1}}
\newcommand{\filevar}[1]{{\textsl{\filenq{#1}}}}
\newcommand{\comingsoon}{\emph{Coming soon$\ \ldots$}}

% And how about these?  Very handy for writing pathnames (tilde for
% Unix, backslash for DOS/Windows).
\renewcommand{\tilde}{\raisebox{-0.5ex}{\symbol{126}}}
\newcommand{\bslash}{\symbol{92}}


\begin{document}

\maketitle

%\begin{abstract}
%\noindent
%Abstract this!
%\end{abstract}

\tableofcontents

\section{Introduction}
\label{sec:intro}

\comingsoon

\subsection{The new way: Distutils}
\label{sec:new-way}


\subsection{The old way (pure Python): whatever you feel like}
\label{sec:old-way-pure}


\subsection{The old way (extensions, \UNIX{} only): Makefile.pre.in}
\label{sec:old-way-ext}





\section{Normal Build and Install}
\label{sec:normal-install}

% This will cover:
%   * setup.py install        (the usual thing)
%   * setup.py build          (if you like doing things one-at-a-time)
%   * setup.py build install  (not necessary unless you need to supply
%                              build options--ref. next section)
%   * where things are installed, on Unix and Windows (Mac...?)
%   * simple custom install: "install --prefix=$HOME"
\comingsoon


\section{Custom Extension Building}
\label{sec:custom-ext}

% This will cover:
%   * normal extension build -- stress that it doesn't matter, you
%     do the same thing whether there are extensions or not
%   * what you might want to customize: compiler and compiler
%     flags (warn of the dangers); per-file compiler flags
%     (not handled yet!)
%   * when things go wrong: I don't know! (and I don't know what
%     to do, either!)
\comingsoon


\section{Custom Installation (\UNIX)}
\label{sec:custom-install-unix}

% XXX probably should banish mentions of Windows here to the
% separate "Non-standard installation (Windows)" section.

A \dfn{custom installation} is where you install modules to a location
that's not in Python's default module search path.  There are a couple
of reasons you might want to do this; the most typical is simply that
you don't have permission to write to the standard Python library
directory.  Or, even if you do have write permission to the standard
library, you might wish to install a module distribution into a
non-standard place for testing or experimentation.  (This is especially
useful when upgrading an existing module distribution: you might want to
make sure that your existing scripts continue to work as before, and
only then install the upgrade ``for real.'')

(XXX terminology: I keep saying ``standard Python library directory''
when I really mean ``the site-packages directory under the standard
Python library directory''.  Is there a better way?)

In any event, you can easily install to non-standard locations with a
couple of options to the \command{install} command:

\begin{tableii}{ll}{option}{Option}{Description}
  \lineii{prefix}{base dir for pure Python distributions
    (overrides \code{sys.prefix})}
  \lineii{exec-prefix}{base dir for distributions with extensions
    (overrides \code{sys.exec_prefix})}
  \lineii{install-lib}{install dir for top-level modules from pure
    Python distributions}
  \lineii{install-platlib}{install dir for top-level modules from
    distributions with extensions}
  \lineii{install-path}{extra path under \option{install-lib} or
    \option{install-platlib} to install to}
\end{tableii}


\subsection{Prefix options}
\label{sec:prefix-options}

There are a lot of picky little rules that govern the interactions of
these five options.  As usual, it's easier to explain things with
examples, so we'll save all the picky rules for later, after you've seen
a bunch of examples.  However, we really have to establish some ground
rules before we can dive into the examples:
\begin{itemize}
\item in a normal \UNIX{} installation, \code{sys.prefix} and
  \code{sys.exec\_prefix} are both \file{/usr/local}.
\item in a multi-platform \UNIX{} installation, \code{sys.prefix} and
  \code{sys.exec\_prefix} are different, and are selected when you
  configure and build Python itself.  Our canonical example of a
  multi-platform installation will have a \code{sys.prefix} of
  \file{/usr/local} and a \code{sys.exec\_prefix} of
  \file{/usr/local.\filevar{plat}} (for whatever value of \filevar{plat} 
  is appropriate).
\item the canonical place to install third-party modules is
  either \file{\filevar{prefix}/lib/python1.\filevar{X}/site-packages}
  or \file{\filevar{exec\_prefix}/lib/python1.\filevar{X}/site-packages}.
  These will be referred to as ``the site-packages directories.''
\end{itemize}


\subsubsection{Pure Python module distribution}

To demonstrate, consider a hypothetical module distribution that
contains one top-level module and a package with two modules:
\begin{tableii}{ll}{module}{Module}{Filename}
  \lineii{mymod}{\filenq{mymod.py}}
  \lineii{mypkg.mod1}{\filenq{mypkg/mod1.py}}
  \lineii{mypkg.mod2}{\filenq{mypkg/mod2.py}}
\end{tableii}
where the filenames are relative to \file{build/lib} after building, or
to some directory in \code{sys.path} after installation.

The goal of installation is to copy these files into a directory in
\code{sys.path} without interfering with the standard Python library.
The canonical, preferred, and most obvious thing to do is to put them in
the ``site-packages'' directory, which is exactly what the
\command{install} comand does by default: under a normal \UNIX{} Python
installation,
\begin{verbatim}
    python setup.py install
\end{verbatim}
installs \file{/usr/local/lib/python1.\filevar{X}/lib/site-packages/mymod.py},
with the \module{mypkg} package in a \file{mypkg} directory under
\file{site-packages}.

However, if you were interested in a standard installation, you wouldn't
be reading this section.  The next-most-standard thing to do is to
specify a custom prefix to override \code{sys.prefix}.  For example:
\begin{verbatim}
    python setup.py install --prefix=/home/greg
\end{verbatim}
is a sensible way to install Python modules to your home directory: this
results in the installation of \file{/home/greg/lib/python/mymod.py},
with the \module{mypkg} modules in \file{/home/greg/lib/python/mypkg/}.
An important point here is that in both this example and the ``plain
vanilla'' example above, the actual installation directory is derived
from the \option{prefix} option.  However, when \option{prefix} differs
from \code{sys.prefix}, the installation directory is derived
differently: the Python version and \file{site-packages} are omitted.
(The version number is part of the standard library directory name to
describe the version of the standard library, so it doesn't make sense
to include it in the name of a non-standard-library directory; likewise,
\file{site-packages} is meant to denote non-standard modules living in
the same area as the standard library, so it doesn't make sense to
include it when installing to a non-standard library.  [XXX check with
Guido that this reasoning is valid and correct; Fred disagrees!])


\subsubsection{Module distribution with extensions}

Now let's consider a different hypothetical module distribution, which
consists of a single package, \module{foo}, containing one pure Python
module and one extension module:
\begin{tableii}{ll}{module}{Module}{Filename}
  \lineii{foo.pure}{\filenq{foo/pure.py}}
  \lineii{foo.ext}{\filenq{foo/ext.so} (or \file{foo/extmodule.so})}
\end{tableii}
In this case, the two modules will be in different locations in the
build tree: \file{build/lib/foo/pure.py} and
\file{build/platlib/foo/ext.so}.  (The \file{.so} (``shared object'')
extension isn't universal, but it's the norm on \UNIX-like systems;
under Windows, the extension module will be in \file{foo/ext.pyd} or
\file{foo/extmodule.pyd}.)

Consider again a standard, plain-vanilla installation:
\begin{verbatim}
    python setup.py install
\end{verbatim}
In this case, \emph{both} modules will be installed to the site-packages
directory under \code{sys.exec\_prefix}, e.g. to
\file{/usr/local.\filevar{plat}/lib/python1.\filevar{X}/site-packages}
on a \UNIX{} system where Python was configured with
\samp{--exec-prefix=/usr/local.plat}.  (On Windows, again, there is no
site-packages directory and \code{sys.prefix} and
\code{sys.exec\_prefix} are the same---so both modules will just be
installed to \code{sys.prefix}.)

Of course, we've already established that you're not interested in
standard installations.  If you just want to install these modules to
your home directory, and you don't maintain a multi-platform home
directory, no problem---just set the prefix as before:
\begin{verbatim}
python setup.py install --prefix=/home/greg
\end{verbatim}
and both modules will be installed to \file{/home/greg/lib/python}.

Now let's say your Python installation is in \file{/usr}---as is the
case in many Linux distributions---but your local policy is to install
third-party software to a network-wide \file{/usr/local} and
\file{/usr/local.\filevar{plat}}.  That is, \code{sys.prefix} and
\code{sys.exec\_prefix} are both \file{/usr}, and you want Python
modules to be installed to either \file{/usr/local/lib/python} or
\file{/usr/local.\filevar{plat}/lib/python}.  This is one case where you
want to specify both \option{prefix} and \option{exec-prefix}:
\begin{verbatim}
python setup.py install --prefix=/usr/local \
                        --exec-prefix=/usr/local.plat
\end{verbatim}
An oddity of this situation is that for any given module distribution,
you only have to supply \emph{one} of \option{prefix} and
\option{exec-prefix}, because pure Python distributions are always
installed under \option{prefix}, and extension-containing distributions
are always installed under \option{exec-prefix}.  For consistency's
sake, though, it's best always to supply both---and the best way to do
that is by using a system-wide configuration file (see
Section~\ref{sec:config-files}).

You could use a similar scheme to maintain a multi-platform personal
Python library.  For example, if you install lots of stuff to your home
directory (not just Python modules), you might have a complete
\file{\tilde/usr} with \file{include}, \file{man}, \file{lib}, and so
forth.  (The advantage of this scheme is that it keeps those mock system
directories out of your home directory and makes it easier to support a
multi-platform personal \file{usr} tree.)  If you don't care about a
multi-platform installation, you can just install with
\begin{verbatim}
python setup.py install --prefix=$HOME/usr
\end{verbatim}
But if you want to keep separate \file{usr} trees for each architecture
that you use, you could say
\begin{verbatim}
python setup.py install --prefix=$HOME/usr \
                        --exec-prefix=$HOME/usr.plat
\end{verbatim}
for various values of \file{plat}.

% this paragraph is for Michel Sanner ;-)
(Perceptive readers will note that on a multi-platform Python
installation, multiple identical copies of \file{foo/pure.py} will be
installed, one for each platform.  This is deliberate.  First, it makes
Python's module search algorithm simpler (XXX check this): when you say
\samp{import foo.pure}, Python searches \code{sys.path} until it finds a
directory containing \file{foo/__init__.py}.  When it finds one, that
directory is deemed to be the directory containing the \module{foo}
package for this import.  Even if the search algorithm were changed
(necessitating a trip back in time to ``fix'' Python 1.5), the only way
to make multiple candidate \module{foo} directories (one for pure
Python, one for extension modules) would be to make copies of
\file{__init__.py}---in which case, why not make copies of all the pure
Python modules?  Second, if you kept pure Python modules related to
extension modules in a platform-shared directory, what happens while you
are upgrading your favourite extension from version 1.0 to 1.1 on
platforms X and Y?  After you install 1.1 for platform X, the 1.1
\file{.py} files will be in the platform-shared directory---but the 1.0
extensions will still be in the platform Y directory.  If the interval
between installing 1.1 for platform X and for platform Y is long---e.g.,
there are portability problems with platform Y---then there's a good
probability of a version mismatch between the 1.1 Python modules and the
1.0 extensions on platform Y.  The solution to both problems is to
install separate copies of the pure Python modules for every platform.
In this day and age, unnecessary disk use is no argument.)

Other ways to support a multi-platform personal Python library are
discussed below, when we cover the \option{install-lib} and
\option{install-platlib} options.


% Gory details on the prefix options (still need to work these into the
% surrounding text):
XXX need to finish these rules and give them some context!
\begin{itemize}
\item \code{sys.exec\_prefix} (and the \option{exec-prefix} option)
  only matters on a multi-platform installation.  If you don't have a
  multi-platform installation (or even know what that is), then you
  don't care about \option{exec-prefix}.
\item in a normal Windows installation, \code{sys.prefix} and
  \code{sys.exec\_prefix} are both \file{C:\bslash Program Files\bslash
    Python}; they are never different under Windows (XXX check!).
\item you may supply \emph{both} of \option{prefix} and
  \option{exec-prefix}, or \emph{neither} of them, or \emph{just}
  \option{prefix}---but you may not supply just \option{exec-prefix}.
\end{itemize}
 

\subsection{Installation directory options}
\label{sec:install-dirs}

Most of the time, it's enough to specify just \option{prefix} (and
possibly \option{exec-prefix})---your modules are installed to
\file{lib/python} under one or the other, you add the appropriate
directory to \code{sys.path}, and that's it.

However, there will inevitably be times when you want finer control over
the installation directories, and that is when the \option{install-lib},
\option{install-platlib}, and \option{install-path} options are
essential.  Normally, \option{install-lib} and \option{install-platlib}
are simply the directories where pure Python modules and extension
modules, respectively, are installed.  That is, top-level modules
(modules not in a package) are installed straight to
\option{install-lib} (or \option{install-platlib} if there are any
extensions in the module distribution).  (If \option{install-path} is
supplied, then things are a bit more complex; we'll deal with that
below.)

Normally, \option{install-lib} and \option{install-platlib} are derived
from \option{prefix} and/or \option{exec-prefix}.  For example, if you
don't supply anything, then \option{prefix} defaults to
\code{sys.prefix}, and \option{install-lib} defaults to
\file{\filevar{prefix}/lib/python1.\filevar{X}/site-packages}.  If you
supply \option{prefix} but not \option{install-lib}, then
\option{install-lib} defaults to \file{\filevar{prefix}/lib/python}
(unless you just happen to supply a prefix which equals
\code{sys.prefix}, which is treated the same as if you don't supply
\option{prefix} at all).  (The rules for \option{exec-prefix} and
\option{install-platlib} are a bit more complex; the following examples
should clarify.  Consult the Distutils source for the gory details.)

To illustrate, let's go back to our hypothetical pure-Python module
distribution containing \module{mymod}, \module{mypkg.mod1}, and
\module{mypkg.mod2}.  If you maintain a personal stash of Python modules
in your home directory, but don't like the \file{\tilde/lib/python}
convention, no problem---you can put the modules right in a
\file{\tilde/python} directory with
\begin{verbatim}
python setup.py install --install-lib=$HOME/python
\end{verbatim}
which will install \file{\$HOME/python/mymod.py},
\file{\$HOME/python/mypkg/mod1.py}, and
\file{\$HOME/python/mypkg/mod2.py}.

If you happen to install a module distribution that contains extensions,
again that's no problem---in the absence of \option{exec-prefix},
\option{install-platlib} defaults to \option{install-lib}, so the above
example will also put extension modules in \file{\$HOME/python}.
(XXX is this correct? is this the best way to describe it? should it be
implemented this way or some other way? how should it be described?)

This may not be what you want, though, if you maintain a multi-platform
stash of Python modules in your home directory.  In that case, you need
to specify \option{install-platlib}---this is the directory where module
distributions with extensions will be installed.  For example, if you
keep pure Python module distributions in \file{\tilde/python} and
extension distributions in \file{\tilde/python.plat}:
\begin{verbatim}
python setup.py install --install-lib=$HOME/python \
                        --install-platlib=$HOME/python.plat
\end{verbatim}
(Just as with \option{prefix} and \option{exec-prefix}, it's only
necessary to supply one of \option{install-lib} and
\option{install-platlib} for any given module distribution, but to
ensure consistency you should always supply them both using a
configuration file (section~\ref{sec:config-files}).)

An alternate way to maintain a multi-platform personal Python library is
in \file{\tilde/lib/python} and \file{\tilde/lib/python.plat}.  In that
case, you can get away with supplying \option{prefix} and
\option{install-platlib}:
\begin{verbatim}
python setup.py install --prefix=$HOME \
                        --install-platlib=$HOME/lib/python.plat
\end{verbatim}

Finally, the \option{install-path} option, which exists mainly to gum up
the whole works---but in a productive (and important) way.
Specifically, \option{install-path} exists to give a directory of their
own to module distributions that wouldn't otherwise have one, i.e.\ that
are not distributed as a (Python) package.

Consider a module distribution, Foo, that consists of (pure Python)
modules \module{foobar}, \module{foobaz}, and \module{fooqux}.
Obviously these are related, and if the project had started in the
Python 1.5 era (and doesn't worry about backwards compatibility with
Python 1.4), they probably would be packaged up and called
\module{foo.bar}, \module{foo.baz}, and \module{foo.qux}.
Unfortunately, they aren't, but we still want the Foo modules to go into
a directory of their own.

Normally, this will be taken care of by the module developer: he adds a
line \samp{install_path = 'Foo'} to his setup script, which has the
following consequences:
\begin{enumerate}
\item instead of \option{install-lib} the modules would be installed in
  \file{\filevar{install-lib}/Foo}
\item if \option{install-lib} is the same as the default
  \option{install-lib}---e.g., you supplied neither \option{prefix} or
  \option{install-lib}---then a \file{Foo.pth} will be created in
  \option{install-lib}, so that Python adds
  \file{\filevar{install-lib}/Foo} to \code{sys.path}
\item if \option{install-lib} is not the default, then a warning will be 
  printed, reminding you to add \file{\filevar{install-lib}/Foo} to
  \code{sys.path} yourself, such as with the \code{PYTHONPATH}
  environment variable  
\end{enumerate}

Thus, you as a module installer have to be aware of the
\option{install-path} option---especially if you maintain a personal
stash of Python modules and don't have write permission to the standard
library, so Distutils can't create \file{.pth} files for you---but you
don't often have to supply it yourself.  There are situations in which
you might want to supply it, though:
\begin{itemize}
\item a module developer forgot to include it (the distribution really
  should go in a directory of its own, but it won't unless you make it)
\item you want to override the \option{install-path} supplied by the
  developer (e.g., you'd rather have a huge jumble of files in
  \file{site-packages} than make Python wade through a bunch of
  \file{.pth} files at startup)
\end{itemize}

The first case is easy: say we're dealing with the Foo distribution
again, but the developer forgot to include \option{install-path}.  No
problem, you can supply it on the command line:
\begin{verbatim}
python setup.py install --install-path=Foo
\end{verbatim}
Note that this will work just fine if you supply \option{prefix} or
\option{install-lib}---but of course, you'll probably have to ensure
that the \file{Foo} directory is in \code{sys.path} yourself.

If you're really fanatical about keeping track of what you have
installed, you might want to supply your own \option{install-path} that
records the version as well as the name of the module distribution; this 
overrides any \option{install-path} included by the module developer in
the setup script:
\begin{verbatim}
python setup.py install --install-path=Foo-1.3
\end{verbatim}

Finally, you can disable \option{install-path} entirely:
\begin{verbatim}
python setup.py install --install-path=''
\end{verbatim}
...but the mess that will result (modules from many different
distributions in the same \option{install-lib} and
\option{install-platlib} directories) is your own problem.

% Points to make
%   * only one of prefix or exec_prefix matters
%   * don't have to specify exec_prefix unless != prefix
%   * thus, usually enough to supply prefix
%   * only have to supply install_lib if you don't like
%     "prefix/lib/python"
%   * likewise for install_platlib and exec_prefix
%   * don't have to supply install_platlib unless != install_lib (??)
%   * in the absence of install_path, top-level modules wind up in
%     install_lib or install_platlib
In case you're interested, here are the exact rules for how
\option{install-lib} and \option{install-platlib} are initialized, and
how they and \option{install-path} affect where modules (pure Python and
extensions) are installed to:
\begin{itemize}
\item If you don't supply \option{prefix} (and possibly
  \option{exec-prefix}), then \option{install-lib} and
  \option{install-platlib} will be, respectively,
  \file{\filevar{\$prefix}/lib/python1.\filevar{X}/site-packages} and
  \file{\filevar{\$exec\_prefix}/lib/python1.\filevar{X}/site-packages}.  In a
  normal \UNIX{} installation, both of these resolve to
  \file{/usr/local/lib/python1.\filevar{X}/site-packages}.
\item in the absence of an \option{install-path} option, top-level
  modules and packages from a pure Python distribution are installed to
  \option{install-lib}
\item in the absence of an \option{install-path} option, top-level
  modules and packages from a distribution that contains \emph{any}
  extension modules are installed to \option{install-platlib}.
\item \emph{there're more, but I don't remember everything offhand}
%\item \option{install-lib} is initialized from \option{prefix} (which
%  in turn is initialized from \code{sys.prefix})---so you should 
\end{itemize}


\section{Custom Installation (Windows)}
\label{sec:custom-install-windows}

\comingsoon


\section{Configuration Files}
\label{sec:config-files}

\comingsoon



\end{document}