1 :mod:`venv` --- Creation of virtual environments 2 ================================================ 3 4 .. module:: venv 5 :synopsis: Creation of virtual environments. 6 7 .. moduleauthor:: Vinay Sajip <vinay_sajip (a] yahoo.co.uk> 8 .. sectionauthor:: Vinay Sajip <vinay_sajip (a] yahoo.co.uk> 9 10 .. versionadded:: 3.3 11 12 **Source code:** :source:`Lib/venv/` 13 14 .. index:: pair: Environments; virtual 15 16 -------------- 17 18 The :mod:`venv` module provides support for creating lightweight "virtual 19 environments" with their own site directories, optionally isolated from system 20 site directories. Each virtual environment has its own Python binary (which 21 matches the version of the binary that was used to create this environment) and 22 can have its own independent set of installed Python packages in its site 23 directories. 24 25 See :pep:`405` for more information about Python virtual environments. 26 27 .. seealso:: 28 29 `Python Packaging User Guide: Creating and using virtual environments 30 <https://packaging.python.org/installing/#creating-virtual-environments>`__ 31 32 .. note:: 33 The ``pyvenv`` script has been deprecated as of Python 3.6 in favor of using 34 ``python3 -m venv`` to help prevent any potential confusion as to which 35 Python interpreter a virtual environment will be based on. 36 37 38 Creating virtual environments 39 ----------------------------- 40 41 .. include:: /using/venv-create.inc 42 43 44 .. _venv-def: 45 46 .. note:: A virtual environment is a Python environment such that the Python 47 interpreter, libraries and scripts installed into it are isolated from those 48 installed in other virtual environments, and (by default) any libraries 49 installed in a "system" Python, i.e., one which is installed as part of your 50 operating system. 51 52 A virtual environment is a directory tree which contains Python executable 53 files and other files which indicate that it is a virtual environment. 54 55 Common installation tools such as ``Setuptools`` and ``pip`` work as 56 expected with virtual environments. In other words, when a virtual 57 environment is active, they install Python packages into the virtual 58 environment without needing to be told to do so explicitly. 59 60 When a virtual environment is active (i.e., the virtual environment's Python 61 interpreter is running), the attributes :attr:`sys.prefix` and 62 :attr:`sys.exec_prefix` point to the base directory of the virtual 63 environment, whereas :attr:`sys.base_prefix` and 64 :attr:`sys.base_exec_prefix` point to the non-virtual environment Python 65 installation which was used to create the virtual environment. If a virtual 66 environment is not active, then :attr:`sys.prefix` is the same as 67 :attr:`sys.base_prefix` and :attr:`sys.exec_prefix` is the same as 68 :attr:`sys.base_exec_prefix` (they all point to a non-virtual environment 69 Python installation). 70 71 When a virtual environment is active, any options that change the 72 installation path will be ignored from all distutils configuration files to 73 prevent projects being inadvertently installed outside of the virtual 74 environment. 75 76 When working in a command shell, users can make a virtual environment active 77 by running an ``activate`` script in the virtual environment's executables 78 directory (the precise filename is shell-dependent), which prepends the 79 virtual environment's directory for executables to the ``PATH`` environment 80 variable for the running shell. There should be no need in other 81 circumstances to activate a virtual environmentscripts installed into 82 virtual environments have a "shebang" line which points to the virtual 83 environment's Python interpreter. This means that the script will run with 84 that interpreter regardless of the value of ``PATH``. On Windows, "shebang" 85 line processing is supported if you have the Python Launcher for Windows 86 installed (this was added to Python in 3.3 - see :pep:`397` for more 87 details). Thus, double-clicking an installed script in a Windows Explorer 88 window should run the script with the correct interpreter without there 89 needing to be any reference to its virtual environment in ``PATH``. 90 91 92 .. _venv-api: 93 94 API 95 --- 96 97 .. highlight:: python 98 99 The high-level method described above makes use of a simple API which provides 100 mechanisms for third-party virtual environment creators to customize environment 101 creation according to their needs, the :class:`EnvBuilder` class. 102 103 .. class:: EnvBuilder(system_site_packages=False, clear=False, \ 104 symlinks=False, upgrade=False, with_pip=False, \ 105 prompt=None) 106 107 The :class:`EnvBuilder` class accepts the following keyword arguments on 108 instantiation: 109 110 * ``system_site_packages`` -- a Boolean value indicating that the system Python 111 site-packages should be available to the environment (defaults to ``False``). 112 113 * ``clear`` -- a Boolean value which, if true, will delete the contents of 114 any existing target directory, before creating the environment. 115 116 * ``symlinks`` -- a Boolean value indicating whether to attempt to symlink the 117 Python binary rather than copying. 118 119 * ``upgrade`` -- a Boolean value which, if true, will upgrade an existing 120 environment with the running Python - for use when that Python has been 121 upgraded in-place (defaults to ``False``). 122 123 * ``with_pip`` -- a Boolean value which, if true, ensures pip is 124 installed in the virtual environment. This uses :mod:`ensurepip` with 125 the ``--default-pip`` option. 126 127 * ``prompt`` -- a String to be used after virtual environment is activated 128 (defaults to ``None`` which means directory name of the environment would 129 be used). 130 131 .. versionchanged:: 3.4 132 Added the ``with_pip`` parameter 133 134 .. versionadded:: 3.6 135 Added the ``prompt`` parameter 136 137 Creators of third-party virtual environment tools will be free to use the 138 provided ``EnvBuilder`` class as a base class. 139 140 The returned env-builder is an object which has a method, ``create``: 141 142 .. method:: create(env_dir) 143 144 This method takes as required argument the path (absolute or relative to 145 the current directory) of the target directory which is to contain the 146 virtual environment. The ``create`` method will either create the 147 environment in the specified directory, or raise an appropriate 148 exception. 149 150 The ``create`` method of the ``EnvBuilder`` class illustrates the hooks 151 available for subclass customization:: 152 153 def create(self, env_dir): 154 """ 155 Create a virtualized Python environment in a directory. 156 env_dir is the target directory to create an environment in. 157 """ 158 env_dir = os.path.abspath(env_dir) 159 context = self.ensure_directories(env_dir) 160 self.create_configuration(context) 161 self.setup_python(context) 162 self.setup_scripts(context) 163 self.post_setup(context) 164 165 Each of the methods :meth:`ensure_directories`, 166 :meth:`create_configuration`, :meth:`setup_python`, 167 :meth:`setup_scripts` and :meth:`post_setup` can be overridden. 168 169 .. method:: ensure_directories(env_dir) 170 171 Creates the environment directory and all necessary directories, and 172 returns a context object. This is just a holder for attributes (such as 173 paths), for use by the other methods. The directories are allowed to 174 exist already, as long as either ``clear`` or ``upgrade`` were 175 specified to allow operating on an existing environment directory. 176 177 .. method:: create_configuration(context) 178 179 Creates the ``pyvenv.cfg`` configuration file in the environment. 180 181 .. method:: setup_python(context) 182 183 Creates a copy or symlink to the Python executable in the environment. 184 On POSIX systems, if a specific executable ``python3.x`` was used, 185 symlinks to ``python`` and ``python3`` will be created pointing to that 186 executable, unless files with those names already exist. 187 188 .. method:: setup_scripts(context) 189 190 Installs activation scripts appropriate to the platform into the virtual 191 environment. 192 193 .. method:: post_setup(context) 194 195 A placeholder method which can be overridden in third party 196 implementations to pre-install packages in the virtual environment or 197 perform other post-creation steps. 198 199 .. versionchanged:: 3.7.2 200 Windows now uses redirector scripts for ``python[w].exe`` instead of 201 copying the actual binaries. In 3.7.2 only :meth:`setup_python` does 202 nothing unless running from a build in the source tree. 203 204 .. versionchanged:: 3.7.3 205 Windows copies the redirector scripts as part of :meth:`setup_python` 206 instead of :meth:`setup_scripts`. This was not the case in 3.7.2. 207 When using symlinks, the original executables will be linked. 208 209 In addition, :class:`EnvBuilder` provides this utility method that can be 210 called from :meth:`setup_scripts` or :meth:`post_setup` in subclasses to 211 assist in installing custom scripts into the virtual environment. 212 213 .. method:: install_scripts(context, path) 214 215 *path* is the path to a directory that should contain subdirectories 216 "common", "posix", "nt", each containing scripts destined for the bin 217 directory in the environment. The contents of "common" and the 218 directory corresponding to :data:`os.name` are copied after some text 219 replacement of placeholders: 220 221 * ``__VENV_DIR__`` is replaced with the absolute path of the environment 222 directory. 223 224 * ``__VENV_NAME__`` is replaced with the environment name (final path 225 segment of environment directory). 226 227 * ``__VENV_PROMPT__`` is replaced with the prompt (the environment 228 name surrounded by parentheses and with a following space) 229 230 * ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory 231 (either ``bin`` or ``Scripts``). 232 233 * ``__VENV_PYTHON__`` is replaced with the absolute path of the 234 environment's executable. 235 236 The directories are allowed to exist (for when an existing environment 237 is being upgraded). 238 239 There is also a module-level convenience function: 240 241 .. function:: create(env_dir, system_site_packages=False, clear=False, \ 242 symlinks=False, with_pip=False) 243 244 Create an :class:`EnvBuilder` with the given keyword arguments, and call its 245 :meth:`~EnvBuilder.create` method with the *env_dir* argument. 246 247 .. versionchanged:: 3.4 248 Added the ``with_pip`` parameter 249 250 An example of extending ``EnvBuilder`` 251 -------------------------------------- 252 253 The following script shows how to extend :class:`EnvBuilder` by implementing a 254 subclass which installs setuptools and pip into a created virtual environment:: 255 256 import os 257 import os.path 258 from subprocess import Popen, PIPE 259 import sys 260 from threading import Thread 261 from urllib.parse import urlparse 262 from urllib.request import urlretrieve 263 import venv 264 265 class ExtendedEnvBuilder(venv.EnvBuilder): 266 """ 267 This builder installs setuptools and pip so that you can pip or 268 easy_install other packages into the created virtual environment. 269 270 :param nodist: If True, setuptools and pip are not installed into the 271 created virtual environment. 272 :param nopip: If True, pip is not installed into the created 273 virtual environment. 274 :param progress: If setuptools or pip are installed, the progress of the 275 installation can be monitored by passing a progress 276 callable. If specified, it is called with two 277 arguments: a string indicating some progress, and a 278 context indicating where the string is coming from. 279 The context argument can have one of three values: 280 'main', indicating that it is called from virtualize() 281 itself, and 'stdout' and 'stderr', which are obtained 282 by reading lines from the output streams of a subprocess 283 which is used to install the app. 284 285 If a callable is not specified, default progress 286 information is output to sys.stderr. 287 """ 288 289 def __init__(self, *args, **kwargs): 290 self.nodist = kwargs.pop('nodist', False) 291 self.nopip = kwargs.pop('nopip', False) 292 self.progress = kwargs.pop('progress', None) 293 self.verbose = kwargs.pop('verbose', False) 294 super().__init__(*args, **kwargs) 295 296 def post_setup(self, context): 297 """ 298 Set up any packages which need to be pre-installed into the 299 virtual environment being created. 300 301 :param context: The information for the virtual environment 302 creation request being processed. 303 """ 304 os.environ['VIRTUAL_ENV'] = context.env_dir 305 if not self.nodist: 306 self.install_setuptools(context) 307 # Can't install pip without setuptools 308 if not self.nopip and not self.nodist: 309 self.install_pip(context) 310 311 def reader(self, stream, context): 312 """ 313 Read lines from a subprocess' output stream and either pass to a progress 314 callable (if specified) or write progress information to sys.stderr. 315 """ 316 progress = self.progress 317 while True: 318 s = stream.readline() 319 if not s: 320 break 321 if progress is not None: 322 progress(s, context) 323 else: 324 if not self.verbose: 325 sys.stderr.write('.') 326 else: 327 sys.stderr.write(s.decode('utf-8')) 328 sys.stderr.flush() 329 stream.close() 330 331 def install_script(self, context, name, url): 332 _, _, path, _, _, _ = urlparse(url) 333 fn = os.path.split(path)[-1] 334 binpath = context.bin_path 335 distpath = os.path.join(binpath, fn) 336 # Download script into the virtual environment's binaries folder 337 urlretrieve(url, distpath) 338 progress = self.progress 339 if self.verbose: 340 term = '\n' 341 else: 342 term = '' 343 if progress is not None: 344 progress('Installing %s ...%s' % (name, term), 'main') 345 else: 346 sys.stderr.write('Installing %s ...%s' % (name, term)) 347 sys.stderr.flush() 348 # Install in the virtual environment 349 args = [context.env_exe, fn] 350 p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) 351 t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) 352 t1.start() 353 t2 = Thread(target=self.reader, args=(p.stderr, 'stderr')) 354 t2.start() 355 p.wait() 356 t1.join() 357 t2.join() 358 if progress is not None: 359 progress('done.', 'main') 360 else: 361 sys.stderr.write('done.\n') 362 # Clean up - no longer needed 363 os.unlink(distpath) 364 365 def install_setuptools(self, context): 366 """ 367 Install setuptools in the virtual environment. 368 369 :param context: The information for the virtual environment 370 creation request being processed. 371 """ 372 url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py' 373 self.install_script(context, 'setuptools', url) 374 # clear up the setuptools archive which gets downloaded 375 pred = lambda o: o.startswith('setuptools-') and o.endswith('.tar.gz') 376 files = filter(pred, os.listdir(context.bin_path)) 377 for f in files: 378 f = os.path.join(context.bin_path, f) 379 os.unlink(f) 380 381 def install_pip(self, context): 382 """ 383 Install pip in the virtual environment. 384 385 :param context: The information for the virtual environment 386 creation request being processed. 387 """ 388 url = 'https://raw.github.com/pypa/pip/master/contrib/get-pip.py' 389 self.install_script(context, 'pip', url) 390 391 def main(args=None): 392 compatible = True 393 if sys.version_info < (3, 3): 394 compatible = False 395 elif not hasattr(sys, 'base_prefix'): 396 compatible = False 397 if not compatible: 398 raise ValueError('This script is only for use with ' 399 'Python 3.3 or later') 400 else: 401 import argparse 402 403 parser = argparse.ArgumentParser(prog=__name__, 404 description='Creates virtual Python ' 405 'environments in one or ' 406 'more target ' 407 'directories.') 408 parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', 409 help='A directory in which to create the 410 'virtual environment.') 411 parser.add_argument('--no-setuptools', default=False, 412 action='store_true', dest='nodist', 413 help="Don't install setuptools or pip in the " 414 "virtual environment.") 415 parser.add_argument('--no-pip', default=False, 416 action='store_true', dest='nopip', 417 help="Don't install pip in the virtual " 418 "environment.") 419 parser.add_argument('--system-site-packages', default=False, 420 action='store_true', dest='system_site', 421 help='Give the virtual environment access to the ' 422 'system site-packages dir.') 423 if os.name == 'nt': 424 use_symlinks = False 425 else: 426 use_symlinks = True 427 parser.add_argument('--symlinks', default=use_symlinks, 428 action='store_true', dest='symlinks', 429 help='Try to use symlinks rather than copies, ' 430 'when symlinks are not the default for ' 431 'the platform.') 432 parser.add_argument('--clear', default=False, action='store_true', 433 dest='clear', help='Delete the contents of the ' 434 'virtual environment ' 435 'directory if it already ' 436 'exists, before virtual ' 437 'environment creation.') 438 parser.add_argument('--upgrade', default=False, action='store_true', 439 dest='upgrade', help='Upgrade the virtual ' 440 'environment directory to ' 441 'use this version of ' 442 'Python, assuming Python ' 443 'has been upgraded ' 444 'in-place.') 445 parser.add_argument('--verbose', default=False, action='store_true', 446 dest='verbose', help='Display the output ' 447 'from the scripts which ' 448 'install setuptools and pip.') 449 options = parser.parse_args(args) 450 if options.upgrade and options.clear: 451 raise ValueError('you cannot supply --upgrade and --clear together.') 452 builder = ExtendedEnvBuilder(system_site_packages=options.system_site, 453 clear=options.clear, 454 symlinks=options.symlinks, 455 upgrade=options.upgrade, 456 nodist=options.nodist, 457 nopip=options.nopip, 458 verbose=options.verbose) 459 for d in options.dirs: 460 builder.create(d) 461 462 if __name__ == '__main__': 463 rc = 1 464 try: 465 main() 466 rc = 0 467 except Exception as e: 468 print('Error: %s' % e, file=sys.stderr) 469 sys.exit(rc) 470 471 472 This script is also available for download `online 473 <https://gist.github.com/vsajip/4673395>`_. 474
