1 :mod:`venv` --- Creation of virtual environments 2 ================================================ 3 4 .. module:: venv 5 :synopsis: Creation of virtual environments. 6 7 .. moduleauthor:: Vinay Sajip <vinay_sajip (a] yahoo.co.uk> 8 .. sectionauthor:: Vinay Sajip <vinay_sajip (a] yahoo.co.uk> 9 10 .. versionadded:: 3.3 11 12 **Source code:** :source:`Lib/venv/` 13 14 .. index:: pair: Environments; virtual 15 16 -------------- 17 18 The :mod:`venv` module provides support for creating lightweight "virtual 19 environments" with their own site directories, optionally isolated from system 20 site directories. Each virtual environment has its own Python binary (allowing 21 creation of environments with various Python versions) and can have its own 22 independent set of installed Python packages in its site directories. 23 24 See :pep:`405` for more information about Python virtual environments. 25 26 .. note:: 27 The ``pyvenv`` script has been deprecated as of Python 3.6 in favor of using 28 ``python3 -m venv`` to help prevent any potential confusion as to which 29 Python interpreter a virtual environment will be based on. 30 31 32 Creating virtual environments 33 ----------------------------- 34 35 .. include:: /using/venv-create.inc 36 37 38 .. _venv-def: 39 40 .. note:: A virtual environment is a Python environment such that the Python 41 interpreter, libraries and scripts installed into it are isolated from those 42 installed in other virtual environments, and (by default) any libraries 43 installed in a "system" Python, i.e., one which is installed as part of your 44 operating system. 45 46 A virtual environment is a directory tree which contains Python executable 47 files and other files which indicate that it is a virtual environment. 48 49 Common installation tools such as ``Setuptools`` and ``pip`` work as 50 expected with virtual environments. In other words, when a virtual 51 environment is active, they install Python packages into the virtual 52 environment without needing to be told to do so explicitly. 53 54 When a virtual environment is active (i.e., the virtual environment's Python 55 interpreter is running), the attributes :attr:`sys.prefix` and 56 :attr:`sys.exec_prefix` point to the base directory of the virtual 57 environment, whereas :attr:`sys.base_prefix` and 58 :attr:`sys.base_exec_prefix` point to the non-virtual environment Python 59 installation which was used to create the virtual environment. If a virtual 60 environment is not active, then :attr:`sys.prefix` is the same as 61 :attr:`sys.base_prefix` and :attr:`sys.exec_prefix` is the same as 62 :attr:`sys.base_exec_prefix` (they all point to a non-virtual environment 63 Python installation). 64 65 When a virtual environment is active, any options that change the 66 installation path will be ignored from all distutils configuration files to 67 prevent projects being inadvertently installed outside of the virtual 68 environment. 69 70 When working in a command shell, users can make a virtual environment active 71 by running an ``activate`` script in the virtual environment's executables 72 directory (the precise filename is shell-dependent), which prepends the 73 virtual environment's directory for executables to the ``PATH`` environment 74 variable for the running shell. There should be no need in other 75 circumstances to activate a virtual environmentscripts installed into 76 virtual environments have a "shebang" line which points to the virtual 77 environment's Python interpreter. This means that the script will run with 78 that interpreter regardless of the value of ``PATH``. On Windows, "shebang" 79 line processing is supported if you have the Python Launcher for Windows 80 installed (this was added to Python in 3.3 - see :pep:`397` for more 81 details). Thus, double-clicking an installed script in a Windows Explorer 82 window should run the script with the correct interpreter without there 83 needing to be any reference to its virtual environment in ``PATH``. 84 85 86 .. _venv-api: 87 88 API 89 --- 90 91 .. highlight:: python 92 93 The high-level method described above makes use of a simple API which provides 94 mechanisms for third-party virtual environment creators to customize environment 95 creation according to their needs, the :class:`EnvBuilder` class. 96 97 .. class:: EnvBuilder(system_site_packages=False, clear=False, \ 98 symlinks=False, upgrade=False, with_pip=False, \ 99 prompt=None) 100 101 The :class:`EnvBuilder` class accepts the following keyword arguments on 102 instantiation: 103 104 * ``system_site_packages`` -- a Boolean value indicating that the system Python 105 site-packages should be available to the environment (defaults to ``False``). 106 107 * ``clear`` -- a Boolean value which, if true, will delete the contents of 108 any existing target directory, before creating the environment. 109 110 * ``symlinks`` -- a Boolean value indicating whether to attempt to symlink the 111 Python binary (and any necessary DLLs or other binaries, 112 e.g. ``pythonw.exe``), rather than copying. Defaults to ``True`` on Linux and 113 Unix systems, but ``False`` on Windows. 114 115 * ``upgrade`` -- a Boolean value which, if true, will upgrade an existing 116 environment with the running Python - for use when that Python has been 117 upgraded in-place (defaults to ``False``). 118 119 * ``with_pip`` -- a Boolean value which, if true, ensures pip is 120 installed in the virtual environment. This uses :mod:`ensurepip` with 121 the ``--default-pip`` option. 122 123 * ``prompt`` -- a String to be used after virtual environment is activated 124 (defaults to ``None`` which means directory name of the environment would 125 be used). 126 127 .. versionchanged:: 3.4 128 Added the ``with_pip`` parameter 129 130 .. versionadded:: 3.6 131 Added the ``prompt`` parameter 132 133 134 Creators of third-party virtual environment tools will be free to use the 135 provided ``EnvBuilder`` class as a base class. 136 137 The returned env-builder is an object which has a method, ``create``: 138 139 .. method:: create(env_dir) 140 141 This method takes as required argument the path (absolute or relative to 142 the current directory) of the target directory which is to contain the 143 virtual environment. The ``create`` method will either create the 144 environment in the specified directory, or raise an appropriate 145 exception. 146 147 The ``create`` method of the ``EnvBuilder`` class illustrates the hooks 148 available for subclass customization:: 149 150 def create(self, env_dir): 151 """ 152 Create a virtualized Python environment in a directory. 153 env_dir is the target directory to create an environment in. 154 """ 155 env_dir = os.path.abspath(env_dir) 156 context = self.ensure_directories(env_dir) 157 self.create_configuration(context) 158 self.setup_python(context) 159 self.setup_scripts(context) 160 self.post_setup(context) 161 162 Each of the methods :meth:`ensure_directories`, 163 :meth:`create_configuration`, :meth:`setup_python`, 164 :meth:`setup_scripts` and :meth:`post_setup` can be overridden. 165 166 .. method:: ensure_directories(env_dir) 167 168 Creates the environment directory and all necessary directories, and 169 returns a context object. This is just a holder for attributes (such as 170 paths), for use by the other methods. The directories are allowed to 171 exist already, as long as either ``clear`` or ``upgrade`` were 172 specified to allow operating on an existing environment directory. 173 174 .. method:: create_configuration(context) 175 176 Creates the ``pyvenv.cfg`` configuration file in the environment. 177 178 .. method:: setup_python(context) 179 180 Creates a copy of the Python executable (and, under Windows, DLLs) in 181 the environment. On a POSIX system, if a specific executable 182 ``python3.x`` was used, symlinks to ``python`` and ``python3`` will be 183 created pointing to that executable, unless files with those names 184 already exist. 185 186 .. method:: setup_scripts(context) 187 188 Installs activation scripts appropriate to the platform into the virtual 189 environment. 190 191 .. method:: post_setup(context) 192 193 A placeholder method which can be overridden in third party 194 implementations to pre-install packages in the virtual environment or 195 perform other post-creation steps. 196 197 In addition, :class:`EnvBuilder` provides this utility method that can be 198 called from :meth:`setup_scripts` or :meth:`post_setup` in subclasses to 199 assist in installing custom scripts into the virtual environment. 200 201 .. method:: install_scripts(context, path) 202 203 *path* is the path to a directory that should contain subdirectories 204 "common", "posix", "nt", each containing scripts destined for the bin 205 directory in the environment. The contents of "common" and the 206 directory corresponding to :data:`os.name` are copied after some text 207 replacement of placeholders: 208 209 * ``__VENV_DIR__`` is replaced with the absolute path of the environment 210 directory. 211 212 * ``__VENV_NAME__`` is replaced with the environment name (final path 213 segment of environment directory). 214 215 * ``__VENV_PROMPT__`` is replaced with the prompt (the environment 216 name surrounded by parentheses and with a following space) 217 218 * ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory 219 (either ``bin`` or ``Scripts``). 220 221 * ``__VENV_PYTHON__`` is replaced with the absolute path of the 222 environment's executable. 223 224 The directories are allowed to exist (for when an existing environment 225 is being upgraded). 226 227 There is also a module-level convenience function: 228 229 .. function:: create(env_dir, system_site_packages=False, clear=False, \ 230 symlinks=False, with_pip=False) 231 232 Create an :class:`EnvBuilder` with the given keyword arguments, and call its 233 :meth:`~EnvBuilder.create` method with the *env_dir* argument. 234 235 .. versionchanged:: 3.4 236 Added the ``with_pip`` parameter 237 238 An example of extending ``EnvBuilder`` 239 -------------------------------------- 240 241 The following script shows how to extend :class:`EnvBuilder` by implementing a 242 subclass which installs setuptools and pip into a created virtual environment:: 243 244 import os 245 import os.path 246 from subprocess import Popen, PIPE 247 import sys 248 from threading import Thread 249 from urllib.parse import urlparse 250 from urllib.request import urlretrieve 251 import venv 252 253 class ExtendedEnvBuilder(venv.EnvBuilder): 254 """ 255 This builder installs setuptools and pip so that you can pip or 256 easy_install other packages into the created virtual environment. 257 258 :param nodist: If True, setuptools and pip are not installed into the 259 created virtual environment. 260 :param nopip: If True, pip is not installed into the created 261 virtual environment. 262 :param progress: If setuptools or pip are installed, the progress of the 263 installation can be monitored by passing a progress 264 callable. If specified, it is called with two 265 arguments: a string indicating some progress, and a 266 context indicating where the string is coming from. 267 The context argument can have one of three values: 268 'main', indicating that it is called from virtualize() 269 itself, and 'stdout' and 'stderr', which are obtained 270 by reading lines from the output streams of a subprocess 271 which is used to install the app. 272 273 If a callable is not specified, default progress 274 information is output to sys.stderr. 275 """ 276 277 def __init__(self, *args, **kwargs): 278 self.nodist = kwargs.pop('nodist', False) 279 self.nopip = kwargs.pop('nopip', False) 280 self.progress = kwargs.pop('progress', None) 281 self.verbose = kwargs.pop('verbose', False) 282 super().__init__(*args, **kwargs) 283 284 def post_setup(self, context): 285 """ 286 Set up any packages which need to be pre-installed into the 287 virtual environment being created. 288 289 :param context: The information for the virtual environment 290 creation request being processed. 291 """ 292 os.environ['VIRTUAL_ENV'] = context.env_dir 293 if not self.nodist: 294 self.install_setuptools(context) 295 # Can't install pip without setuptools 296 if not self.nopip and not self.nodist: 297 self.install_pip(context) 298 299 def reader(self, stream, context): 300 """ 301 Read lines from a subprocess' output stream and either pass to a progress 302 callable (if specified) or write progress information to sys.stderr. 303 """ 304 progress = self.progress 305 while True: 306 s = stream.readline() 307 if not s: 308 break 309 if progress is not None: 310 progress(s, context) 311 else: 312 if not self.verbose: 313 sys.stderr.write('.') 314 else: 315 sys.stderr.write(s.decode('utf-8')) 316 sys.stderr.flush() 317 stream.close() 318 319 def install_script(self, context, name, url): 320 _, _, path, _, _, _ = urlparse(url) 321 fn = os.path.split(path)[-1] 322 binpath = context.bin_path 323 distpath = os.path.join(binpath, fn) 324 # Download script into the virtual environment's binaries folder 325 urlretrieve(url, distpath) 326 progress = self.progress 327 if self.verbose: 328 term = '\n' 329 else: 330 term = '' 331 if progress is not None: 332 progress('Installing %s ...%s' % (name, term), 'main') 333 else: 334 sys.stderr.write('Installing %s ...%s' % (name, term)) 335 sys.stderr.flush() 336 # Install in the virtual environment 337 args = [context.env_exe, fn] 338 p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) 339 t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) 340 t1.start() 341 t2 = Thread(target=self.reader, args=(p.stderr, 'stderr')) 342 t2.start() 343 p.wait() 344 t1.join() 345 t2.join() 346 if progress is not None: 347 progress('done.', 'main') 348 else: 349 sys.stderr.write('done.\n') 350 # Clean up - no longer needed 351 os.unlink(distpath) 352 353 def install_setuptools(self, context): 354 """ 355 Install setuptools in the virtual environment. 356 357 :param context: The information for the virtual environment 358 creation request being processed. 359 """ 360 url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py' 361 self.install_script(context, 'setuptools', url) 362 # clear up the setuptools archive which gets downloaded 363 pred = lambda o: o.startswith('setuptools-') and o.endswith('.tar.gz') 364 files = filter(pred, os.listdir(context.bin_path)) 365 for f in files: 366 f = os.path.join(context.bin_path, f) 367 os.unlink(f) 368 369 def install_pip(self, context): 370 """ 371 Install pip in the virtual environment. 372 373 :param context: The information for the virtual environment 374 creation request being processed. 375 """ 376 url = 'https://raw.github.com/pypa/pip/master/contrib/get-pip.py' 377 self.install_script(context, 'pip', url) 378 379 def main(args=None): 380 compatible = True 381 if sys.version_info < (3, 3): 382 compatible = False 383 elif not hasattr(sys, 'base_prefix'): 384 compatible = False 385 if not compatible: 386 raise ValueError('This script is only for use with ' 387 'Python 3.3 or later') 388 else: 389 import argparse 390 391 parser = argparse.ArgumentParser(prog=__name__, 392 description='Creates virtual Python ' 393 'environments in one or ' 394 'more target ' 395 'directories.') 396 parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', 397 help='A directory in which to create the 398 'virtual environment.') 399 parser.add_argument('--no-setuptools', default=False, 400 action='store_true', dest='nodist', 401 help="Don't install setuptools or pip in the " 402 "virtual environment.") 403 parser.add_argument('--no-pip', default=False, 404 action='store_true', dest='nopip', 405 help="Don't install pip in the virtual " 406 "environment.") 407 parser.add_argument('--system-site-packages', default=False, 408 action='store_true', dest='system_site', 409 help='Give the virtual environment access to the ' 410 'system site-packages dir.') 411 if os.name == 'nt': 412 use_symlinks = False 413 else: 414 use_symlinks = True 415 parser.add_argument('--symlinks', default=use_symlinks, 416 action='store_true', dest='symlinks', 417 help='Try to use symlinks rather than copies, ' 418 'when symlinks are not the default for ' 419 'the platform.') 420 parser.add_argument('--clear', default=False, action='store_true', 421 dest='clear', help='Delete the contents of the ' 422 'virtual environment ' 423 'directory if it already ' 424 'exists, before virtual ' 425 'environment creation.') 426 parser.add_argument('--upgrade', default=False, action='store_true', 427 dest='upgrade', help='Upgrade the virtual ' 428 'environment directory to ' 429 'use this version of ' 430 'Python, assuming Python ' 431 'has been upgraded ' 432 'in-place.') 433 parser.add_argument('--verbose', default=False, action='store_true', 434 dest='verbose', help='Display the output ' 435 'from the scripts which ' 436 'install setuptools and pip.') 437 options = parser.parse_args(args) 438 if options.upgrade and options.clear: 439 raise ValueError('you cannot supply --upgrade and --clear together.') 440 builder = ExtendedEnvBuilder(system_site_packages=options.system_site, 441 clear=options.clear, 442 symlinks=options.symlinks, 443 upgrade=options.upgrade, 444 nodist=options.nodist, 445 nopip=options.nopip, 446 verbose=options.verbose) 447 for d in options.dirs: 448 builder.create(d) 449 450 if __name__ == '__main__': 451 rc = 1 452 try: 453 main() 454 rc = 0 455 except Exception as e: 456 print('Error: %s' % e, file=sys.stderr) 457 sys.exit(rc) 458 459 460 This script is also available for download `online 461 <https://gist.github.com/4673395>`_. 462
