1 Quick Build Info 2 ================ 3 4 For testing, the installer should be built with the Tools/msi/build.bat 5 script: 6 7 build.bat [-x86] [-x64] [--doc] 8 9 For an official release, the installer should be built with the 10 Tools/msi/buildrelease.bat script and environment variables: 11 12 set PYTHON=<path to Python 2.7 or 3.4> 13 set SPHINXBUILD=<path to sphinx-build.exe> 14 set PATH=<path to Mercurial (hg.exe)>; 15 <path to HTML Help Compiler (hhc.exe)>;%PATH% 16 17 buildrelease.bat [-x86] [-x64] [-D] [-B] 18 [-o <output directory>] [-c <certificate name>] 19 20 See the Building the Installer section for more information. 21 22 Overview 23 ======== 24 25 Python is distributed on Windows as an installer that will configure the 26 user's system. This allows users to have a functioning copy of Python 27 without having to build it themselves. 28 29 The main tasks of the installer are: 30 31 * copy required files into the expected layout 32 * configure system settings so the installation can be located by 33 other programs 34 * add entry points for modifying, repairing and uninstalling Python 35 * make it easy to launch Python, its documentation, and IDLE 36 37 Each of these is discussed in a later section of this document. 38 39 Structure of the Installer 40 ========================== 41 42 The installer is structured as a 'layout', which consists of a number of 43 CAB and MSI files and a single EXE. 44 45 The EXE is the main entry point into the installer. It contains the UI 46 and command-line logic, as well as the ability to locate and optionally 47 download other parts of the layout. 48 49 Each MSI contains the logic required to install a component or feature 50 of Python. These MSIs should not be launched directly by users. MSIs can 51 be embedded into the EXE or automatically downloaded as needed. 52 53 Each CAB contains the files making up a Python installation. CABs are 54 embedded into their associated MSI and are never seen by users. 55 56 MSIs are only required when the related feature or component is being 57 installed. When components are not selected for installation, the 58 associated MSI is not downloaded. This allows the installer to offer 59 options to install debugging symbols and binaries without increasing 60 the initial download size by separating them into their own MSIs. 61 62 Building the Installer 63 ====================== 64 65 Before building the installer, download extra build dependencies using 66 Tools\msi\get_externals.bat. (Note that this is in addition to the 67 similarly named file in PCBuild.) 68 69 For testing, the installer should be built with the Tools/msi/build.bat 70 script: 71 72 build.bat [-x86] [-x64] [--doc] [--test-marker] [--pack] 73 74 This script will build the required configurations of Python and 75 generate an installer layout in PCBuild/(win32|amd64)/en-us. 76 77 Specify -x86 and/or -x64 to build for each platform. If neither is 78 specified, both platforms will be built. Currently, both the debug and 79 release versions of Python are required for the installer. 80 81 Specify --doc to build the documentation (.chm) file. If the file is not 82 available, it will simply be excluded from the installer. Ensure 83 %PYTHON% and %SPHINXBUILD% are set when passing this option. You may 84 also set %HTMLHELP% to the Html Help Compiler (hhc.exe), or put HHC on 85 your PATH or in externals/. 86 87 Specify --test-marker to build an installer that works side-by-side with 88 an official Python release. All registry keys and install locations will 89 include an extra marker to avoid overwriting files. This marker is 90 currently an 'x' prefix, but may change at any time. 91 92 Specify --pack to build an installer that does not require all MSIs to 93 be available alongside. This takes longer, but is easier to share. 94 95 96 For an official release, the installer should be built with the 97 Tools/msi/buildrelease.bat script: 98 99 set PYTHON=<path to Python 2.7 or 3.4> 100 set SPHINXBUILD=<path to sphinx-build.exe> 101 set PATH=<path to Mercurial (hg.exe)>; 102 <path to HTML Help Compiler (hhc.exe)>;%PATH% 103 104 buildrelease.bat [-x86] [-x64] [-D] [-B] 105 [-o <output directory>] [-c <certificate name>] 106 107 Specify -x86 and/or -x64 to build for each platform. If neither is 108 specified, both platforms will be built. Currently, both the debug and 109 release versions of Python are required for the installer. 110 111 Specify -D to skip rebuilding the documentation. The documentation is 112 required for a release and the build will fail if it is not available. 113 114 Specify -B to skip rebuilding Python. This is useful to only rebuild the 115 installer layout after a previous call to buildrelease.bat. 116 117 Specify -o to set an output directory. The installer layouts will be 118 copied to platform-specific subdirectories of this path. 119 120 Specify -c to choose a code-signing certificate to be used for all the 121 signable binaries in Python as well as each file making up the 122 installer. Official releases of Python must be signed. 123 124 Ensure %PYTHON% and %SPHINXBUILD% are set when passing this option. You 125 may also set %HTMLHELP% to the Html Help Compiler (hhc.exe), or put HHC 126 on your PATH or in externals/. You will also need Mercurial (hg.exe) on 127 your PATH. 128 129 If WiX is not found on your system, it will be automatically downloaded 130 and extracted to the externals/ directory. 131 132 To manually build layouts of the installer, build one of the projects in 133 the bundle folder. 134 135 msbuild bundle\snapshot.wixproj 136 msbuild bundle\releaseweb.wixproj 137 msbuild bundle\releaselocal.wixproj 138 msbuild bundle\full.wixproj 139 140 snapshot.wixproj produces a test installer versioned based on the date. 141 142 releaseweb.wixproj produces a release installer that does not embed any 143 of the layout. 144 145 releaselocal.wixproj produces a release installer that embeds the files 146 required for a default installation. 147 148 full.wixproj produces a test installer that embeds the entire layout. 149 150 The following properties may be passed when building these projects. 151 152 /p:BuildForRelease=(true|false) 153 When true, adds extra verification to ensure a complete installer is 154 produced. For example, binutils is required when building for a release 155 to generate MinGW-compatible libraries, and the build will be aborted if 156 this fails. Defaults to false. 157 158 /p:ReleaseUri=(any URI) 159 Used to generate unique IDs for the installers to allow side-by-side 160 installation. Forks of Python can use the same installer infrastructure 161 by providing a unique URI for this property. It does not need to be an 162 active internet address. Defaults to $(ComputerName). 163 164 Official releases use http://www.python.org/(architecture name) 165 166 /p:DownloadUrlBase=(any URI) 167 Specifies the base of a URL where missing parts of the installer layout 168 can be downloaded from. The build version and architecture will be 169 appended to create the full address. If omitted, missing components will 170 not be automatically downloaded. 171 172 /p:DownloadUrl=(any URI) 173 Specifies the full URL where missing parts of the installer layout can 174 be downloaded from. Should normally include '{2}', which will be 175 substituted for the filename. If omitted, missing components will not be 176 automatically downloaded. If specified, this value overrides 177 DownloadUrlBase. 178 179 /p:SigningCertificate=(certificate name) 180 Specifies the certificate to sign the installer layout with. If omitted, 181 the layout will not be signed. 182 183 /p:RebuildAll=(true|false) 184 When true, rebuilds all of the MSIs making up the layout. Defaults to 185 true. 186 187 Uploading the Installer 188 ======================= 189 190 For official releases, the uploadrelease.bat script should be used. 191 192 You will require PuTTY so that plink.exe and pscp.exe can be used, and your 193 SSH key can be activated in pageant.exe. PuTTY should be either on your path 194 or in %ProgramFiles(x86)%\PuTTY. 195 196 To include signatures for each uploaded file, you will need gpg2.exe on your 197 path or have run get_externals.bat. You may also need to "gpg2.exe --import" 198 your key before running the upload script. 199 200 uploadrelease.bat --host <host> --user <username> [--dry-run] [--no-gpg] 201 202 The host is the URL to the server. This can be provided by the Release 203 Manager. You should be able to SSH to this address. 204 205 The username is your own username, which you have permission to SSH into 206 the server containing downloads. 207 208 Use --dry-run to display the generated upload commands without executing 209 them. Signatures for each file will be generated but not uploaded unless 210 --no-gpg is also passed. 211 212 Use --no-gpg to suppress signature generation and upload. 213 214 The default target directory (which appears in uploadrelease.proj) is 215 correct for official Python releases, but may be overridden with 216 --target <path> for other purposes. This path should generally not include 217 any version specifier, as that will be added automatically. 218 219 Modifying the Installer 220 ======================= 221 222 The code for the installer is divided into three main groups: packages, 223 the bundle and the bootstrap application. 224 225 Packages 226 -------- 227 228 Packages appear as subdirectories of Tools/msi (other than the bundle/ 229 directory). The project file is a .wixproj and the build output is a 230 single MSI. Packages are built with the WiX Toolset. Some project files 231 share source files and use preprocessor directives to enable particular 232 features. These are typically used to keep the sources close when the 233 files are related, but produce multiple independent packages. 234 235 A package is the smallest element that may be independently installed or 236 uninstalled (as used in this installer). For example, the test suite has 237 its own package, as users can choose to add or remove it after the 238 initial installation. 239 240 All the files installed by a single package should be related, though 241 some packages may not install any files. For example, the pip package 242 executes the ensurepip package, but does not add or remove any of its 243 own files. (It is represented as a package because of its 244 installed/uninstalled nature, as opposed to the "precompile standard 245 library" option, for example.) Dependencies between packages are handled 246 by the bundle, but packages should detect when dependencies are missing 247 and raise an error. 248 249 Packages that include a lot of files may use an InstallFiles element in 250 the .wixproj file to generate sources. See lib/lib.wixproj for an 251 example, and msi.targets and csv_to_wxs.py for the implementation. This 252 element is also responsible for generating the code for cleaning up and 253 removing __pycache__ folders in any directory containing .py files. 254 255 All packages are built with the Tools/msi/common.wxs file, and so any 256 directory or property in this file may be referenced. Of particular 257 interest: 258 259 REGISTRYKEY (property) 260 The registry key for the current installation. 261 262 InstallDirectory (directory) 263 The root install directory for the current installation. Subdirectories 264 are also specified in this file (DLLs, Lib, etc.) 265 266 MenuDir (directory) 267 The Start Menu folder for the current installation. 268 269 UpgradeTable (property) 270 Every package should reference this property to include upgrade 271 information. 272 273 OptionalFeature (Component) 274 Packages that may be enabled or disabled should reference this component 275 and have an OPTIONAL_FEATURES entry in the bootstrap application to 276 properly handle Modify and Upgrade. 277 278 The .wxl_template file is specially handled by the build system for this 279 project to perform {{substitutions}} as defined in msi.targets. They 280 should be included in projects as <WxlTemplate> items, where .wxl files 281 are normally included as <EmbeddedResource> items. 282 283 Bundle 284 ------ 285 286 The bundle is compiled to the main EXE entry point that for most users 287 will represent the Python installer. It is built from Tools/msi/bundle 288 with packages references in Tools/msi/bundle/packagegroups. 289 290 Build logic for the bundle is in bundle.targets, but should be invoked 291 through one of the .wixproj files as described in Building the 292 Installer. 293 294 The UI is separated between Default.thm (UI layout), Default.wxl 295 (strings), bundle.wxs (properties) and the bootstrap application. 296 Bundle.wxs also contains the chain, which is the list of packages to 297 install and the order they should be installed in. These refer to named 298 package groups in bundle/packagegroups. 299 300 Each package group specifies one or more packages to install. Most 301 packages require two separate entries to support both per-user and 302 all-users installations. Because these reuse the same package, it does 303 not increase the overall size of the package. 304 305 Package groups refer to payload groups, which allow better control over 306 embedding and downloading files than the default settings. Whether files 307 are embedded and where they are downloaded from depends on settings 308 created by the project files. 309 310 Package references can include install conditions that determine when to 311 install the package. When a package is a dependency for others, the 312 condition should be crafted to ensure it is installed. 313 314 MSI packages are installed or uninstalled based on their current state 315 and the install condition. This makes them most suitable for features 316 that are clearly present or absent from the user's machine. 317 318 EXE packages are executed based on a customisable condition that can be 319 omitted. This makes them suitable for pre- or post-install tasks that 320 need to run regardless of whether features have been added or removed. 321 322 Bootstrap Application 323 --------------------- 324 325 The bootstrap application is a C++ application that controls the UI and 326 installation. While it does not directly compile into the main EXE of 327 the installer, it forms the main active component. Most of the 328 installation functionality is provided by WiX, and so the bootstrap 329 application is predominantly responsible for the code behind the UI that 330 is defined in the Default.thm file. The bootstrap application code is in 331 bundle/bootstrap and is built automatically when building the bundle. 332 333 Installation Layout 334 =================== 335 336 There are two installation layouts for Python on Windows, with the only 337 differences being supporting files. A layout is selected implicitly 338 based on whether the install is for all users of the machine or just for 339 the user performing the installation. 340 341 The default installation location when installing for all users is 342 "%ProgramFiles%\Python3X" for the 64-bit interpreter and 343 "%ProgramFiles(x86)%\Python3X-32" for the 32-bit interpreter. (Note that 344 the latter path is equivalent to "%ProgramFiles%\Python3X-32" when 345 running a 32-bit version of Windows.) This location requires 346 administrative privileges to install or later modify the installation. 347 348 The default installation location when installing for the current user 349 is "%LocalAppData%\Programs\Python\Python3X" for the 64-bit interpreter 350 and "%LocalAppData%\Programs\Python\Python3X-32" for the 32-bit 351 interpreter. Only the current user can access this location. This 352 provides a suitable level of protection against malicious modification 353 of Python's files. 354 355 (Default installation locations are set in Tools\msi\bundle\bundle.wxs.) 356 357 Within this install directory is the following approximate layout: 358 359 .\python[w].exe The core executable files 360 .\DLLs Stdlib extensions (*.pyd) and dependencies 361 .\Doc Documentation (*.chm) 362 .\include Development headers (*.h) 363 .\Lib Standard library 364 .\Lib\test Test suite 365 .\libs Development libraries (*.lib) 366 .\Scripts Launcher scripts (*.exe, *.py) 367 .\tcl Tcl dependencies (*.dll, *.tcl and others) 368 .\Tools Tool scripts (*.py) 369 370 When installed for all users, the following files are installed to 371 either "%SystemRoot%\System32" or "%SystemRoot%\SysWOW64" as 372 appropriate. For the current user, they are installed in the Python 373 install directory. 374 375 .\python3x.dll The core interpreter 376 .\python3.dll The stable ABI reference 377 378 When installed for all users, the following files are installed to 379 "%SystemRoot%" (typically "C:\Windows") to ensure they are always 380 available on PATH. (See Launching Python below.) For the current user, 381 they are installed in "%LocalAppData%\Programs\Python\PyLauncher". 382 383 .\py[w].exe PEP 397 launcher 384 385 System Settings 386 =============== 387 388 On installation, registry keys are created so that other applications 389 can locate and identify installations of Python. The locations of these 390 keys vary based on the install type. 391 392 For 64-bit interpreters installed for all users, the root key is: 393 HKEY_LOCAL_MACHINE\Software\Python\PythonCore\3.X 394 395 For 32-bit interpreters installed for all users on a 64-bit operating 396 system, the root key is: 397 HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\3.X-32 398 399 For 32-bit interpreters installed for all users on a 32-bit operating 400 system, the root key is: 401 HKEY_LOCAL_MACHINE\Software\Python\PythonCore\3.X-32 402 403 For 64-bit interpreters installed for the current user: 404 HKEY_CURRENT_USER\Software\Python\PythonCore\3.X 405 406 For 32-bit interpreters installed for the current user: 407 HKEY_CURRENT_USER\Software\Python\PythonCore\3.X-32 408 409 When the core Python executables are installed, a key "InstallPath" is 410 created within the root key with its default value set to the 411 executable's install directory. A value named "ExecutablePath" is added 412 with the full path to the main Python interpreter, and a key 413 "InstallGroup" is created with its default value set to the product 414 name "Python 3.X". 415 416 When the Python standard library is installed, a key "PythonPath" is 417 created within the root key with its default value set to the full path 418 to the Lib folder followed by the path to the DLLs folder, separated by 419 a semicolon. 420 421 When the documentation is installed, a key "Help" is created within the 422 root key, with a subkey "Main Python Documentation" with its default 423 value set to the full path to the installed CHM file. 424 425 426 The py.exe launcher is installed as part of a regular Python install, 427 but using a separate mechanism that allows it to more easily span 428 versions of Python. As a result, it has different root keys for its 429 registry entries: 430 431 When installed for all users on a 64-bit operating system, the 432 launcher's root key is: 433 HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\Launcher 434 435 When installed for all users on a 32-bit operating system, the 436 launcher's root key is: 437 HKEY_LOCAL_MACHINE\Software\Python\Launcher 438 439 When installed for the current user: 440 HKEY_CURRENT_USER\Software\Python\Launcher 441 442 When the launcher is installed, a key "InstallPath" is created within 443 its root key with its default value set to the launcher's install 444 directory. File associations are also created for .py, .pyw, .pyc and 445 .pyo files. 446 447 Launching Python 448 ================ 449 450 When a feature offering user entry points in the Start Menu is 451 installed, a folder "Python 3.X" is created. Every shortcut should be 452 created within this folder, and each shortcut should include the version 453 and platform to allow users to identify the shortcut in a search results 454 page. 455 456 The core Python executables creates a shortcut "Python 3.X (32-bit)" or 457 "Python 3.X (64-bit)" depending on the interpreter. 458 459 The documentation creates a shortcut "Python 3.X 32-bit Manuals" or 460 "Python 3.X 64-bit Manuals". The documentation is identical for all 461 platforms, but the shortcuts need to be separate to avoid uninstallation 462 conflicts. 463 464 Installing IDLE creates a shortcut "IDLE (Python 3.X 32-bit)" or "IDLE 465 (Python 3.X 64-bit)" depending on the interpreter. 466 467 468 For users who often launch Python from a Command Prompt, an option is 469 provided to add the directory containing python.exe to the user or 470 system PATH variable. If the option is selected, the install directory 471 and the Scripts directory will be added at the start of the system PATH 472 for an all users install and the user PATH for a per-user install. 473 474 When the user only has one version of Python installed, this will behave 475 as expected. However, because Windows searches the system PATH before 476 the user PATH, users cannot override a system-wide installation of 477 Python on their PATH. Further, because the installer can only prepend to 478 the path, later installations of Python will take precedence over 479 earlier installations, regardless of interpreter version. 480 481 Because it is not possible to automatically create a sensible PATH 482 configuration, users are recommended to use the py.exe launcher and 483 manually modify their PATH variable to add Scripts directories in their 484 preferred order. System-wide installations of Python should consider not 485 modifying PATH, or using an alternative technology to modify their 486 users' PATH variables. 487 488 489 The py.exe launcher is recommended because it uses a consistent and 490 sensible search order for Python installations. User installations are 491 preferred over system-wide installs, and later versions are preferred 492 regardless of installation order (with the exception that py.exe 493 currently prefers 2.x versions over 3.x versions without the -3 command 494 line argument). 495 496 For both 32-bit and 64-bit interpreters, the 32-bit version of the 497 launcher is installed. This ensures that the search order is always 498 consistent (as the 64-bit launcher is subtly different from the 32-bit 499 launcher) and also avoids the need to install it multiple times. Future 500 versions of Python will upgrade the launcher in-place, using Windows 501 Installer's upgrade functionality to avoid conflicts with earlier 502 installed versions. 503 504 When installed, file associations are created for .py, .pyc and .pyo 505 files to launch with py.exe and .pyw files to launch with pyw.exe. This 506 makes Python files respect shebang lines by default and also avoids 507 conflicts between multiple Python installations. 508 509 510 Repair, Modify and Uninstall 511 ============================ 512 513 After installation, Python may be modified, repaired or uninstalled by 514 running the original EXE again or via the Programs and Features applet 515 (formerly known as Add or Remove Programs). 516 517 Modifications allow features to be added or removed. The install 518 directory and kind (all users/single user) cannot be modified. Because 519 Windows Installer caches installation packages, removing features will 520 not require internet access unless the package cache has been corrupted 521 or deleted. Adding features that were not previously installed and are 522 not embedded or otherwise available will require internet access. 523 524 Repairing will rerun the installation for all currently installed 525 features, restoring files and registry keys that have been modified or 526 removed. This operation generally will not redownload any files unless 527 the cached packages have been corrupted or deleted. 528 529 Removing Python will clean up all the files and registry keys that were 530 created by the installer, as well as __pycache__ folders that are 531 explicitly handled by the installer. Python packages installed later 532 using a tool like pip will not be removed. Some components may be 533 installed by other installers and these will not be removed if another 534 product has a dependency on them. 535 536