Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 1 | :mod:`venv` --- Creation of virtual environments |
| 2 | ================================================ |
| 3 | |
| 4 | .. module:: venv |
| 5 | :synopsis: Creation of virtual environments. |
| 6 | .. moduleauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk> |
| 7 | .. sectionauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk> |
| 8 | |
| 9 | |
| 10 | .. index:: pair: Environments; virtual |
| 11 | |
| 12 | .. versionadded:: 3.3 |
| 13 | |
| 14 | **Source code:** :source:`Lib/venv.py` |
| 15 | |
| 16 | -------------- |
| 17 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 18 | The :mod:`venv` module provides support for creating lightweight "virtual |
| 19 | environments" with their own site directories, optionally isolated from system |
| 20 | site directories. Each virtual environment has its own Python binary (allowing |
| 21 | creation of environments with various Python versions) and can have its own |
| 22 | independent set of installed Python packages in its site directories. |
| 23 | |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 24 | |
| 25 | Creating virtual environments |
| 26 | ----------------------------- |
| 27 | |
Vinay Sajip | c4618e3 | 2012-07-10 08:21:07 +0100 | [diff] [blame] | 28 | .. include:: /using/venv-create.inc |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 29 | |
Vinay Sajip | a945ad1 | 2012-07-09 09:24:59 +0100 | [diff] [blame] | 30 | |
Vinay Sajip | cd9b746 | 2012-07-09 10:37:01 +0100 | [diff] [blame] | 31 | .. _venv-def: |
| 32 | |
Vinay Sajip | a945ad1 | 2012-07-09 09:24:59 +0100 | [diff] [blame] | 33 | .. note:: A virtual environment (also called a ``venv``) is a Python |
| 34 | environment such that the Python interpreter, libraries and scripts |
| 35 | installed into it are isolated from those installed in other virtual |
| 36 | environments, and (by default) any libraries installed in a "system" Python, |
| 37 | i.e. one which is installed as part of your operating system. |
| 38 | |
| 39 | A venv is a directory tree which contains Python executable files and |
| 40 | other files which indicate that it is a venv. |
| 41 | |
Vinay Sajip | c4618e3 | 2012-07-10 08:21:07 +0100 | [diff] [blame] | 42 | Common installation tools such as ``Distribute`` and ``pip`` work as |
Vinay Sajip | a945ad1 | 2012-07-09 09:24:59 +0100 | [diff] [blame] | 43 | expected with venvs - i.e. when a venv is active, they install Python |
| 44 | packages into the venv without needing to be told to do so explicitly. |
Vinay Sajip | c4618e3 | 2012-07-10 08:21:07 +0100 | [diff] [blame] | 45 | Of course, you need to install them into the venv first: this could be |
| 46 | done by running ``distribute_setup.py`` with the venv activated, |
| 47 | followed by running ``easy_install pip``. Alternatively, you could download |
| 48 | the source tarballs and run ``python setup.py install`` after unpacking, |
| 49 | with the venv activated. |
Vinay Sajip | a945ad1 | 2012-07-09 09:24:59 +0100 | [diff] [blame] | 50 | |
| 51 | When a venv is active (i.e. the venv's Python interpreter is running), the |
| 52 | attributes :attr:`sys.prefix` and :attr:`sys.exec_prefix` point to the base |
| 53 | directory of the venv, whereas :attr:`sys.base_prefix` and |
| 54 | :attr:`sys.base_exec_prefix` point to the non-venv Python installation |
| 55 | which was used to create the venv. If a venv is not active, then |
| 56 | :attr:`sys.prefix` is the same as :attr:`sys.base_prefix` and |
| 57 | :attr:`sys.exec_prefix` is the same as :attr:`sys.base_exec_prefix` (they |
| 58 | all point to a non-venv Python installation). |
| 59 | |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 60 | |
| 61 | API |
| 62 | --- |
| 63 | |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 64 | .. highlight:: python |
| 65 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 66 | The high-level method described above makes use of a simple API which provides |
| 67 | mechanisms for third-party virtual environment creators to customize environment |
| 68 | creation according to their needs, the :class:`EnvBuilder` class. |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 69 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 70 | .. class:: EnvBuilder(system_site_packages=False, clear=False, symlinks=False, upgrade=False) |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 71 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 72 | The :class:`EnvBuilder` class accepts the following keyword arguments on |
| 73 | instantiation: |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 74 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 75 | * ``system_site_packages`` -- a Boolean value indicating that the system Python |
| 76 | site-packages should be available to the environment (defaults to ``False``). |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 77 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 78 | * ``clear`` -- a Boolean value which, if True, will delete any existing target |
| 79 | directory instead of raising an exception (defaults to ``False``). |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 80 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 81 | * ``symlinks`` -- a Boolean value indicating whether to attempt to symlink the |
| 82 | Python binary (and any necessary DLLs or other binaries, |
| 83 | e.g. ``pythonw.exe``), rather than copying. Defaults to ``True`` on Linux and |
Vinay Sajip | 90db661 | 2012-07-17 17:33:46 +0100 | [diff] [blame] | 84 | Unix systems, but ``False`` on Windows. |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 85 | |
Vinay Sajip | a945ad1 | 2012-07-09 09:24:59 +0100 | [diff] [blame] | 86 | * ``upgrade`` -- a Boolean value which, if True, will upgrade an existing |
| 87 | environment with the running Python - for use when that Python has been |
| 88 | upgraded in-place (defaults to ``False``). |
| 89 | |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 90 | |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 91 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 92 | Creators of third-party virtual environment tools will be free to use the |
| 93 | provided ``EnvBuilder`` class as a base class. |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 94 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 95 | The returned env-builder is an object which has a method, ``create``: |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 96 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 97 | .. method:: create(env_dir) |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 98 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 99 | This method takes as required argument the path (absolute or relative to |
| 100 | the current directory) of the target directory which is to contain the |
| 101 | virtual environment. The ``create`` method will either create the |
| 102 | environment in the specified directory, or raise an appropriate |
| 103 | exception. |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 104 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 105 | The ``create`` method of the ``EnvBuilder`` class illustrates the hooks |
| 106 | available for subclass customization:: |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 107 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 108 | def create(self, env_dir): |
| 109 | """ |
| 110 | Create a virtualized Python environment in a directory. |
| 111 | env_dir is the target directory to create an environment in. |
| 112 | """ |
| 113 | env_dir = os.path.abspath(env_dir) |
| 114 | context = self.create_directories(env_dir) |
| 115 | self.create_configuration(context) |
| 116 | self.setup_python(context) |
| 117 | self.setup_scripts(context) |
| 118 | self.post_setup(context) |
Vinay Sajip | 7ded1f0 | 2012-05-26 03:45:29 +0100 | [diff] [blame] | 119 | |
Georg Brandl | dbab58f | 2012-06-24 16:37:59 +0200 | [diff] [blame] | 120 | Each of the methods :meth:`create_directories`, |
| 121 | :meth:`create_configuration`, :meth:`setup_python`, |
| 122 | :meth:`setup_scripts` and :meth:`post_setup` can be overridden. |
| 123 | |
| 124 | .. method:: create_directories(env_dir) |
| 125 | |
| 126 | Creates the environment directory and all necessary directories, and |
| 127 | returns a context object. This is just a holder for attributes (such as |
| 128 | paths), for use by the other methods. |
| 129 | |
| 130 | .. method:: create_configuration(context) |
| 131 | |
| 132 | Creates the ``pyvenv.cfg`` configuration file in the environment. |
| 133 | |
| 134 | .. method:: setup_python(context) |
| 135 | |
| 136 | Creates a copy of the Python executable (and, under Windows, DLLs) in |
| 137 | the environment. |
| 138 | |
| 139 | .. method:: setup_scripts(context) |
| 140 | |
| 141 | Installs activation scripts appropriate to the platform into the virtual |
| 142 | environment. |
| 143 | |
| 144 | .. method:: post_setup(context) |
| 145 | |
| 146 | A placeholder method which can be overridden in third party |
| 147 | implementations to pre-install packages in the virtual environment or |
| 148 | perform other post-creation steps. |
| 149 | |
| 150 | In addition, :class:`EnvBuilder` provides this utility method that can be |
| 151 | called from :meth:`setup_scripts` or :meth:`post_setup` in subclasses to |
| 152 | assist in installing custom scripts into the virtual environment. |
| 153 | |
| 154 | .. method:: install_scripts(context, path) |
| 155 | |
| 156 | *path* is the path to a directory that should contain subdirectories |
| 157 | "common", "posix", "nt", each containing scripts destined for the bin |
| 158 | directory in the environment. The contents of "common" and the |
| 159 | directory corresponding to :data:`os.name` are copied after some text |
| 160 | replacement of placeholders: |
| 161 | |
| 162 | * ``__VENV_DIR__`` is replaced with the absolute path of the environment |
| 163 | directory. |
| 164 | |
| 165 | * ``__VENV_NAME__`` is replaced with the environment name (final path |
| 166 | segment of environment directory). |
| 167 | |
| 168 | * ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory |
| 169 | (either ``bin`` or ``Scripts``). |
| 170 | |
| 171 | * ``__VENV_PYTHON__`` is replaced with the absolute path of the |
| 172 | environment's executable. |
| 173 | |
| 174 | |
| 175 | There is also a module-level convenience function: |
| 176 | |
| 177 | .. function:: create(env_dir, system_site_packages=False, clear=False, symlinks=False) |
| 178 | |
| 179 | Create an :class:`EnvBuilder` with the given keyword arguments, and call its |
| 180 | :meth:`~EnvBuilder.create` method with the *env_dir* argument. |
Vinay Sajip | 2b4fcfb | 2013-01-30 13:44:00 +0000 | [diff] [blame] | 181 | |
| 182 | An example of extending ``EnvBuilder`` |
| 183 | -------------------------------------- |
| 184 | |
| 185 | The following script shows how to extend :class:`EnvBuilder` by implementing a |
| 186 | subclass which installs Distribute and pip into a created venv:: |
| 187 | |
| 188 | import os |
| 189 | import os.path |
| 190 | from subprocess import Popen, PIPE |
| 191 | import sys |
| 192 | from threading import Thread |
| 193 | from urllib.parse import urlparse |
| 194 | from urllib.request import urlretrieve |
| 195 | import venv |
| 196 | |
| 197 | class DistributeEnvBuilder(venv.EnvBuilder): |
| 198 | """ |
| 199 | This builder installs Distribute and pip so that you can pip or |
| 200 | easy_install other packages into the created environment. |
| 201 | |
| 202 | :param nodist: If True, Distribute is not installed into the created |
| 203 | environment. |
| 204 | :param nopip: If True, pip is not installed into the created |
| 205 | environment. |
| 206 | :param progress: If Distribute or pip are installed, the progress of the |
| 207 | installation can be monitored by passing a progress |
| 208 | callable. If specified, it is called with two |
| 209 | arguments: a string indicating some progress, and a |
| 210 | context indicating where the string is coming from. |
| 211 | The context argument can have one of three values: |
| 212 | 'main', indicating that it is called from virtualize() |
| 213 | itself, and 'stdout' and 'stderr', which are obtained |
| 214 | by reading lines from the output streams of a subprocess |
| 215 | which is used to install the app. |
| 216 | |
| 217 | If a callable is not specified, default progress |
| 218 | information is output to sys.stderr. |
| 219 | """ |
| 220 | |
| 221 | def __init__(self, *args, **kwargs): |
| 222 | self.nodist = kwargs.pop('nodist', False) |
| 223 | self.nopip = kwargs.pop('nopip', False) |
| 224 | self.progress = kwargs.pop('progress', None) |
| 225 | self.verbose = kwargs.pop('verbose', False) |
| 226 | super().__init__(*args, **kwargs) |
| 227 | |
| 228 | def post_setup(self, context): |
| 229 | """ |
| 230 | Set up any packages which need to be pre-installed into the |
| 231 | environment being created. |
| 232 | |
| 233 | :param context: The information for the environment creation request |
| 234 | being processed. |
| 235 | """ |
| 236 | if not self.nodist: |
| 237 | self.install_distribute(context) |
| 238 | if not self.nopip: |
| 239 | self.install_pip(context) |
| 240 | |
| 241 | def reader(self, stream, context): |
| 242 | """ |
| 243 | Read lines from a subprocess' output stream and either pass to a progress |
| 244 | callable (if specified) or write progress information to sys.stderr. |
| 245 | """ |
| 246 | progress = self.progress |
| 247 | while True: |
| 248 | s = stream.readline() |
| 249 | if not s: |
| 250 | break |
| 251 | if progress is not None: |
| 252 | progress(s, context) |
| 253 | else: |
| 254 | if not self.verbose: |
| 255 | sys.stderr.write('.') |
| 256 | else: |
| 257 | sys.stderr.write(s.decode('utf-8')) |
| 258 | sys.stderr.flush() |
| 259 | |
| 260 | def install_script(self, context, name, url): |
| 261 | _, _, path, _, _, _ = urlparse(url) |
| 262 | fn = os.path.split(path)[-1] |
| 263 | binpath = context.bin_path |
| 264 | distpath = os.path.join(binpath, fn) |
| 265 | # Download script into the env's binaries folder |
| 266 | urlretrieve(url, distpath) |
| 267 | progress = self.progress |
| 268 | if progress is not None: |
| 269 | progress('Installing %s' %name, 'main') |
| 270 | else: |
| 271 | sys.stderr.write('Installing %s ' % name) |
| 272 | sys.stderr.flush() |
| 273 | # Install in the env |
| 274 | args = [context.env_exe, fn] |
| 275 | p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) |
| 276 | t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) |
| 277 | t1.start() |
| 278 | t2 = Thread(target=self.reader, args=(p.stderr, 'stderr')) |
| 279 | t2.start() |
| 280 | p.wait() |
| 281 | t1.join() |
| 282 | t2.join() |
| 283 | if progress is not None: |
| 284 | progress('done.', 'main') |
| 285 | else: |
| 286 | sys.stderr.write('done.\n') |
| 287 | # Clean up - no longer needed |
| 288 | os.unlink(distpath) |
| 289 | |
| 290 | def install_distribute(self, context): |
| 291 | """ |
| 292 | Install Distribute in the environment. |
| 293 | |
| 294 | :param context: The information for the environment creation request |
| 295 | being processed. |
| 296 | """ |
| 297 | url = 'http://python-distribute.org/distribute_setup.py' |
| 298 | self.install_script(context, 'distribute', url) |
| 299 | # clear up the distribute archive which gets downloaded |
| 300 | pred = lambda o: o.startswith('distribute-') and o.endswith('.tar.gz') |
| 301 | files = filter(pred, os.listdir(context.bin_path)) |
| 302 | for f in files: |
| 303 | f = os.path.join(context.bin_path, f) |
| 304 | os.unlink(f) |
| 305 | |
| 306 | def install_pip(self, context): |
| 307 | """ |
| 308 | Install pip in the environment. |
| 309 | |
| 310 | :param context: The information for the environment creation request |
| 311 | being processed. |
| 312 | """ |
| 313 | url = 'https://raw.github.com/pypa/pip/master/contrib/get-pip.py' |
| 314 | self.install_script(context, 'pip', url) |
| 315 | |
| 316 | def main(args=None): |
| 317 | compatible = True |
| 318 | if sys.version_info < (3, 3): |
| 319 | compatible = False |
| 320 | elif not hasattr(sys, 'base_prefix'): |
| 321 | compatible = False |
| 322 | if not compatible: |
| 323 | raise ValueError('This script is only for use with ' |
| 324 | 'Python 3.3 or later') |
| 325 | else: |
| 326 | import argparse |
| 327 | |
| 328 | parser = argparse.ArgumentParser(prog=__name__, |
| 329 | description='Creates virtual Python ' |
| 330 | 'environments in one or ' |
| 331 | 'more target ' |
| 332 | 'directories.') |
| 333 | parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', |
| 334 | help='A directory to create the environment in.') |
| 335 | parser.add_argument('--no-distribute', default=False, |
| 336 | action='store_true', dest='nodist', |
| 337 | help="Don't install Distribute in the virtual " |
| 338 | "environment.") |
| 339 | parser.add_argument('--no-pip', default=False, |
| 340 | action='store_true', dest='nopip', |
| 341 | help="Don't install pip in the virtual " |
| 342 | "environment.") |
| 343 | parser.add_argument('--system-site-packages', default=False, |
| 344 | action='store_true', dest='system_site', |
| 345 | help='Give the virtual environment access to the ' |
| 346 | 'system site-packages dir.') |
| 347 | if os.name == 'nt': |
| 348 | use_symlinks = False |
| 349 | else: |
| 350 | use_symlinks = True |
| 351 | parser.add_argument('--symlinks', default=use_symlinks, |
| 352 | action='store_true', dest='symlinks', |
| 353 | help='Try to use symlinks rather than copies, ' |
| 354 | 'when symlinks are not the default for ' |
| 355 | 'the platform.') |
| 356 | parser.add_argument('--clear', default=False, action='store_true', |
| 357 | dest='clear', help='Delete the contents of the ' |
| 358 | 'environment directory if it ' |
| 359 | 'already exists, before ' |
| 360 | 'environment creation.') |
| 361 | parser.add_argument('--upgrade', default=False, action='store_true', |
| 362 | dest='upgrade', help='Upgrade the environment ' |
| 363 | 'directory to use this version ' |
| 364 | 'of Python, assuming Python ' |
| 365 | 'has been upgraded in-place.') |
| 366 | parser.add_argument('--verbose', default=False, action='store_true', |
| 367 | dest='verbose', help='Display the output ' |
| 368 | 'from the scripts which ' |
| 369 | 'install Distribute and pip.') |
| 370 | options = parser.parse_args(args) |
| 371 | if options.upgrade and options.clear: |
| 372 | raise ValueError('you cannot supply --upgrade and --clear together.') |
| 373 | builder = DistributeEnvBuilder(system_site_packages=options.system_site, |
| 374 | clear=options.clear, |
| 375 | symlinks=options.symlinks, |
| 376 | upgrade=options.upgrade, |
| 377 | nodist=options.nodist, |
| 378 | nopip=options.nopip, |
| 379 | verbose=options.verbose) |
| 380 | for d in options.dirs: |
| 381 | builder.create(d) |
| 382 | |
| 383 | if __name__ == '__main__': |
| 384 | rc = 1 |
| 385 | try: |
| 386 | main() |
| 387 | rc = 0 |
| 388 | except Exception as e: |
| 389 | print('Error: %s' % e, file=sys.stderr) |
| 390 | sys.exit(rc) |
| 391 | |
| 392 | This script is also available for download `online |
| 393 | <https://gist.github.com/4673395>`_. |