1 **************************** 2 What's New In Python 3.7 3 **************************** 4 5 :Editor: Elvis Pranskevichus <elvis (a] magic.io> 6 7 .. Rules for maintenance: 8 9 * Anyone can add text to this document. Do not spend very much time 10 on the wording of your changes, because your text will probably 11 get rewritten to some degree. 12 13 * The maintainer will go through Misc/NEWS periodically and add 14 changes; it's therefore more important to add your changes to 15 Misc/NEWS than to this file. 16 17 * This is not a complete list of every single change; completeness 18 is the purpose of Misc/NEWS. Some changes I consider too small 19 or esoteric to include. If such a change is added to the text, 20 I'll just remove it. (This is another reason you shouldn't spend 21 too much time on writing your addition.) 22 23 * If you want to draw your new text to the attention of the 24 maintainer, add 'XXX' to the beginning of the paragraph or 25 section. 26 27 * It's OK to just add a fragmentary note about a change. For 28 example: "XXX Describe the transmogrify() function added to the 29 socket module." The maintainer will research the change and 30 write the necessary text. 31 32 * You can comment out your additions if you like, but it's not 33 necessary (especially when a final release is some months away). 34 35 * Credit the author of a patch or bugfix. Just the name is 36 sufficient; the e-mail address isn't necessary. 37 38 * It's helpful to add the bug/patch number as a comment: 39 40 XXX Describe the transmogrify() function added to the socket 41 module. 42 (Contributed by P.Y. Developer in :issue:`12345`.) 43 44 This saves the maintainer the effort of going through the Git log 45 when researching a change. 46 47 This article explains the new features in Python 3.7, compared to 3.6. 48 Python 3.7 was released on June 27, 2018. 49 For full details, see the :ref:`changelog <changelog>`. 50 51 52 Summary -- Release Highlights 53 ============================= 54 55 .. This section singles out the most important changes in Python 3.7. 56 Brevity is key. 57 58 New syntax features: 59 60 * :ref:`PEP 563 <whatsnew37-pep563>`, postponed evaluation of type annotations. 61 62 Backwards incompatible syntax changes: 63 64 * :keyword:`async` and :keyword:`await` are now reserved keywords. 65 66 New library modules: 67 68 * :mod:`contextvars`: :ref:`PEP 567 -- Context Variables <whatsnew37-pep567>` 69 * :mod:`dataclasses`: :ref:`PEP 557 -- Data Classes <whatsnew37-pep557>` 70 * :ref:`whatsnew37_importlib_resources` 71 72 New built-in features: 73 74 * :ref:`PEP 553 <whatsnew37-pep553>`, the new :func:`breakpoint` function. 75 76 Python data model improvements: 77 78 * :ref:`PEP 562 <whatsnew37-pep562>`, customization of access to 79 module attributes. 80 81 * :ref:`PEP 560 <whatsnew37-pep560>`, core support for typing module and 82 generic types. 83 84 * the insertion-order preservation nature of :ref:`dict <typesmapping>` 85 objects `has been declared`_ to be an official 86 part of the Python language spec. 87 88 .. _has been declared: https://mail.python.org/pipermail/python-dev/2017-December/151283.html 89 90 Significant improvements in the standard library: 91 92 * The :mod:`asyncio` module has received new features, significant 93 :ref:`usability and performance improvements <whatsnew37_asyncio>`. 94 95 * The :mod:`time` module gained support for 96 :ref:`functions with nanosecond resolution <whatsnew37-pep564>`. 97 98 CPython implementation improvements: 99 100 * Avoiding the use of ASCII as a default text encoding: 101 102 * :ref:`PEP 538 <whatsnew37-pep538>`, legacy C locale coercion 103 * :ref:`PEP 540 <whatsnew37-pep540>`, forced UTF-8 runtime mode 104 * :ref:`PEP 552 <whatsnew37-pep552>`, deterministic .pycs 105 * :ref:`the new development runtime mode <whatsnew37-devmode>` 106 * :ref:`PEP 565 <whatsnew37-pep565>`, improved :exc:`DeprecationWarning` 107 handling 108 109 C API improvements: 110 111 * :ref:`PEP 539 <whatsnew37-pep539>`, new C API for thread-local storage 112 113 Documentation improvements: 114 115 * :ref:`PEP 545 <whatsnew37-pep545>`, Python documentation translations 116 * New documentation translations: `Japanese <https://docs.python.org/ja/>`_, 117 `French <https://docs.python.org/fr/>`_, and 118 `Korean <https://docs.python.org/ko/>`_. 119 120 This release features notable performance improvements in many areas. 121 The :ref:`whatsnew37-perf` section lists them in detail. 122 123 For a list of changes that may affect compatibility with previous Python 124 releases please refer to the :ref:`porting-to-python-37` section. 125 126 127 New Features 128 ============ 129 130 .. _whatsnew37-pep563: 131 132 PEP 563: Postponed Evaluation of Annotations 133 -------------------------------------------- 134 135 The advent of type hints in Python uncovered two glaring usability issues 136 with the functionality of annotations added in :pep:`3107` and refined 137 further in :pep:`526`: 138 139 * annotations could only use names which were already available in the 140 current scope, in other words they didn't support forward references 141 of any kind; and 142 143 * annotating source code had adverse effects on startup time of Python 144 programs. 145 146 Both of these issues are fixed by postponing the evaluation of 147 annotations. Instead of compiling code which executes expressions in 148 annotations at their definition time, the compiler stores the annotation 149 in a string form equivalent to the AST of the expression in question. 150 If needed, annotations can be resolved at runtime using 151 :func:`typing.get_type_hints`. In the common case where this is not 152 required, the annotations are cheaper to store (since short strings 153 are interned by the interpreter) and make startup time faster. 154 155 Usability-wise, annotations now support forward references, making the 156 following syntax valid:: 157 158 class C: 159 @classmethod 160 def from_string(cls, source: str) -> C: 161 ... 162 163 def validate_b(self, obj: B) -> bool: 164 ... 165 166 class B: 167 ... 168 169 Since this change breaks compatibility, the new behavior needs to be enabled 170 on a per-module basis in Python 3.7 using a :mod:`__future__` import:: 171 172 from __future__ import annotations 173 174 It will become the default in Python 4.0. 175 176 .. seealso:: 177 178 :pep:`563` -- Postponed evaluation of annotations 179 PEP written and implemented by ukasz Langa. 180 181 182 .. _whatsnew37-pep538: 183 184 PEP 538: Legacy C Locale Coercion 185 --------------------------------- 186 187 An ongoing challenge within the Python 3 series has been determining a sensible 188 default strategy for handling the "7-bit ASCII" text encoding assumption 189 currently implied by the use of the default C or POSIX locale on non-Windows 190 platforms. 191 192 :pep:`538` updates the default interpreter command line interface to 193 automatically coerce that locale to an available UTF-8 based locale as 194 described in the documentation of the new :envvar:`PYTHONCOERCECLOCALE` 195 environment variable. Automatically setting ``LC_CTYPE`` this way means that 196 both the core interpreter and locale-aware C extensions (such as 197 :mod:`readline`) will assume the use of UTF-8 as the default text encoding, 198 rather than ASCII. 199 200 The platform support definition in :pep:`11` has also been updated to limit 201 full text handling support to suitably configured non-ASCII based locales. 202 203 As part of this change, the default error handler for :data:`~sys.stdin` and 204 :data:`~sys.stdout` is now ``surrogateescape`` (rather than ``strict``) when 205 using any of the defined coercion target locales (currently ``C.UTF-8``, 206 ``C.utf8``, and ``UTF-8``). The default error handler for :data:`~sys.stderr` 207 continues to be ``backslashreplace``, regardless of locale. 208 209 Locale coercion is silent by default, but to assist in debugging potentially 210 locale related integration problems, explicit warnings (emitted directly on 211 :data:`~sys.stderr`) can be requested by setting ``PYTHONCOERCECLOCALE=warn``. 212 This setting will also cause the Python runtime to emit a warning if the 213 legacy C locale remains active when the core interpreter is initialized. 214 215 While :pep:`538`'s locale coercion has the benefit of also affecting extension 216 modules (such as GNU ``readline``), as well as child processes (including those 217 running non-Python applications and older versions of Python), it has the 218 downside of requiring that a suitable target locale be present on the running 219 system. To better handle the case where no suitable target locale is available 220 (as occurs on RHEL/CentOS 7, for example), Python 3.7 also implements 221 :ref:`whatsnew37-pep540`. 222 223 .. seealso:: 224 225 :pep:`538` -- Coercing the legacy C locale to a UTF-8 based locale 226 PEP written and implemented by Nick Coghlan. 227 228 229 .. _whatsnew37-pep540: 230 231 PEP 540: Forced UTF-8 Runtime Mode 232 ----------------------------------- 233 234 The new :option:`-X` ``utf8`` command line option and :envvar:`PYTHONUTF8` 235 environment variable can be used to enable the CPython *UTF-8 mode*. 236 237 When in UTF-8 mode, CPython ignores the locale settings, and uses the 238 UTF-8 encoding by default. The error handlers for :data:`sys.stdin` and 239 :data:`sys.stdout` streams are set to ``surrogateescape``. 240 241 The forced UTF-8 mode can be used to change the text handling behavior in 242 an embedded Python interpreter without changing the locale settings of 243 an embedding application. 244 245 While :pep:`540`'s UTF-8 mode has the benefit of working regardless of which 246 locales are available on the running system, it has the downside of having no 247 effect on extension modules (such as GNU ``readline``), child processes running 248 non-Python applications, and child processes running older versions of Python. 249 To reduce the risk of corrupting text data when communicating with such 250 components, Python 3.7 also implements :ref:`whatsnew37-pep540`). 251 252 The UTF-8 mode is enabled by default when the locale is ``C`` or ``POSIX``, and 253 the :pep:`538` locale coercion feature fails to change it to a UTF-8 based 254 alternative (whether that failure is due to ``PYTHONCOERCECLOCALE=0`` being set, 255 ``LC_ALL`` being set, or the lack of a suitable target locale). 256 257 .. seealso:: 258 259 :pep:`540` -- Add a new UTF-8 mode 260 PEP written and implemented by Victor Stinner 261 262 263 .. _whatsnew37-pep553: 264 265 PEP 553: Built-in ``breakpoint()`` 266 ---------------------------------- 267 268 Python 3.7 includes the new built-in :func:`breakpoint` function as 269 an easy and consistent way to enter the Python debugger. 270 271 Built-in ``breakpoint()`` calls :func:`sys.breakpointhook`. By default, the 272 latter imports :mod:`pdb` and then calls ``pdb.set_trace()``, but by binding 273 ``sys.breakpointhook()`` to the function of your choosing, ``breakpoint()`` can 274 enter any debugger. Additionally, the environment variable 275 :envvar:`PYTHONBREAKPOINT` can be set to the callable of your debugger of 276 choice. Set ``PYTHONBREAKPOINT=0`` to completely disable built-in 277 ``breakpoint()``. 278 279 .. seealso:: 280 281 :pep:`553` -- Built-in breakpoint() 282 PEP written and implemented by Barry Warsaw 283 284 285 .. _whatsnew37-pep539: 286 287 PEP 539: New C API for Thread-Local Storage 288 ------------------------------------------- 289 290 While Python provides a C API for thread-local storage support; the existing 291 :ref:`Thread Local Storage (TLS) API <thread-local-storage-api>` has used 292 :c:type:`int` to represent TLS keys across all platforms. This has not 293 generally been a problem for officially-support platforms, but that is neither 294 POSIX-compliant, nor portable in any practical sense. 295 296 :pep:`539` changes this by providing a new :ref:`Thread Specific Storage (TSS) 297 API <thread-specific-storage-api>` to CPython which supersedes use of the 298 existing TLS API within the CPython interpreter, while deprecating the existing 299 API. The TSS API uses a new type :c:type:`Py_tss_t` instead of :c:type:`int` 300 to represent TSS keys--an opaque type the definition of which may depend on 301 the underlying TLS implementation. Therefore, this will allow to build CPython 302 on platforms where the native TLS key is defined in a way that cannot be safely 303 cast to :c:type:`int`. 304 305 Note that on platforms where the native TLS key is defined in a way that cannot 306 be safely cast to :c:type:`int`, all functions of the existing TLS API will be 307 no-op and immediately return failure. This indicates clearly that the old API 308 is not supported on platforms where it cannot be used reliably, and that no 309 effort will be made to add such support. 310 311 .. seealso:: 312 313 :pep:`539` -- A New C-API for Thread-Local Storage in CPython 314 PEP written by Erik M. Bray; implementation by Masayuki Yamamoto. 315 316 317 .. _whatsnew37-pep562: 318 319 PEP 562: Customization of Access to Module Attributes 320 ----------------------------------------------------- 321 322 Python 3.7 allows defining :meth:`__getattr__` on modules and will call 323 it whenever a module attribute is otherwise not found. Defining 324 :meth:`__dir__` on modules is now also allowed. 325 326 A typical example of where this may be useful is module attribute deprecation 327 and lazy loading. 328 329 .. seealso:: 330 331 :pep:`562` -- Module ``__getattr__`` and ``__dir__`` 332 PEP written and implemented by Ivan Levkivskyi 333 334 335 .. _whatsnew37-pep564: 336 337 PEP 564: New Time Functions With Nanosecond Resolution 338 ------------------------------------------------------ 339 340 The resolution of clocks in modern systems can exceed the limited precision 341 of a floating point number returned by the :func:`time.time` function 342 and its variants. To avoid loss of precision, :pep:`564` adds six new 343 "nanosecond" variants of the existing timer functions to the :mod:`time` 344 module: 345 346 * :func:`time.clock_gettime_ns` 347 * :func:`time.clock_settime_ns` 348 * :func:`time.monotonic_ns` 349 * :func:`time.perf_counter_ns` 350 * :func:`time.process_time_ns` 351 * :func:`time.time_ns` 352 353 The new functions return the number of nanoseconds as an integer value. 354 355 `Measurements <https://www.python.org/dev/peps/pep-0564/#annex-clocks-resolution-in-python>`_ 356 show that on Linux and Windows the resolution of :func:`time.time_ns` is 357 approximately 3 times better than that of :func:`time.time`. 358 359 .. seealso:: 360 361 :pep:`564` -- Add new time functions with nanosecond resolution 362 PEP written and implemented by Victor Stinner 363 364 365 .. _whatsnew37-pep565: 366 367 PEP 565: Show DeprecationWarning in ``__main__`` 368 ------------------------------------------------ 369 370 The default handling of :exc:`DeprecationWarning` has been changed such that 371 these warnings are once more shown by default, but only when the code 372 triggering them is running directly in the :mod:`__main__` module. As a result, 373 developers of single file scripts and those using Python interactively should 374 once again start seeing deprecation warnings for the APIs they use, but 375 deprecation warnings triggered by imported application, library and framework 376 modules will continue to be hidden by default. 377 378 As a result of this change, the standard library now allows developers to choose 379 between three different deprecation warning behaviours: 380 381 * :exc:`FutureWarning`: always displayed by default, recommended for warnings 382 intended to be seen by application end users (e.g. for deprecated application 383 configuration settings). 384 * :exc:`DeprecationWarning`: displayed by default only in :mod:`__main__` and when 385 running tests, recommended for warnings intended to be seen by other Python 386 developers where a version upgrade may result in changed behaviour or an 387 error. 388 * :exc:`PendingDeprecationWarning`: displayed by default only when running 389 tests, intended for cases where a future version upgrade will change the 390 warning category to :exc:`DeprecationWarning` or :exc:`FutureWarning`. 391 392 Previously both :exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning` 393 were only visible when running tests, which meant that developers primarily 394 writing single file scripts or using Python interactively could be surprised 395 by breaking changes in the APIs they used. 396 397 .. seealso:: 398 399 :pep:`565` -- Show DeprecationWarning in ``__main__`` 400 PEP written and implemented by Nick Coghlan 401 402 403 .. _whatsnew37-pep560: 404 405 PEP 560: Core Support for ``typing`` module and Generic Types 406 ------------------------------------------------------------- 407 408 Initially :pep:`484` was designed in such way that it would not introduce *any* 409 changes to the core CPython interpreter. Now type hints and the :mod:`typing` 410 module are extensively used by the community, so this restriction is removed. 411 The PEP introduces two special methods :meth:`__class_getitem__` and 412 ``__mro_entries__``, these methods are now used by most classes and special 413 constructs in :mod:`typing`. As a result, the speed of various operations 414 with types increased up to 7 times, the generic types can be used without 415 metaclass conflicts, and several long standing bugs in :mod:`typing` module are 416 fixed. 417 418 .. seealso:: 419 420 :pep:`560` -- Core support for typing module and generic types 421 PEP written and implemented by Ivan Levkivskyi 422 423 424 .. _whatsnew37-pep552: 425 426 PEP 552: Hash-based .pyc Files 427 ------------------------------ 428 429 Python has traditionally checked the up-to-dateness of bytecode cache files 430 (i.e., ``.pyc`` files) by comparing the source metadata (last-modified timestamp 431 and size) with source metadata saved in the cache file header when it was 432 generated. While effective, this invalidation method has its drawbacks. When 433 filesystem timestamps are too coarse, Python can miss source updates, leading to 434 user confusion. Additionally, having a timestamp in the cache file is 435 problematic for `build reproducibility <https://reproducible-builds.org/>`_ and 436 content-based build systems. 437 438 :pep:`552` extends the pyc format to allow the hash of the source file to be 439 used for invalidation instead of the source timestamp. Such ``.pyc`` files are 440 called "hash-based". By default, Python still uses timestamp-based invalidation 441 and does not generate hash-based ``.pyc`` files at runtime. Hash-based ``.pyc`` 442 files may be generated with :mod:`py_compile` or :mod:`compileall`. 443 444 Hash-based ``.pyc`` files come in two variants: checked and unchecked. Python 445 validates checked hash-based ``.pyc`` files against the corresponding source 446 files at runtime but doesn't do so for unchecked hash-based pycs. Unchecked 447 hash-based ``.pyc`` files are a useful performance optimization for environments 448 where a system external to Python (e.g., the build system) is responsible for 449 keeping ``.pyc`` files up-to-date. 450 451 See :ref:`pyc-invalidation` for more information. 452 453 .. seealso:: 454 455 :pep:`552` -- Deterministic pycs 456 PEP written and implemented by Benjamin Peterson 457 458 459 .. _whatsnew37-pep545: 460 461 PEP 545: Python Documentation Translations 462 ------------------------------------------ 463 464 :pep:`545` describes the process of creating and maintaining Python 465 documentation translations. 466 467 Three new translations have been added: 468 469 - Japanese: https://docs.python.org/ja/ 470 - French: https://docs.python.org/fr/ 471 - Korean: https://docs.python.org/ko/ 472 473 .. seealso:: 474 475 :pep:`545` -- Python Documentation Translations 476 PEP written and implemented by Julien Palard, Inada Naoki, and 477 Victor Stinner. 478 479 480 .. _whatsnew37-devmode: 481 482 Development Runtime Mode: -X dev 483 -------------------------------- 484 485 The new :option:`-X` ``dev`` command line option or the new 486 :envvar:`PYTHONDEVMODE` environment variable can be used to enable 487 CPython's *development mode*. When in development mode, CPython performs 488 additional runtime checks that are too expensive to be enabled by default. 489 See :option:`-X` ``dev`` documentation for the full description of the effects 490 of this mode. 491 492 493 Other Language Changes 494 ====================== 495 496 * More than 255 arguments can now be passed to a function, and a function can 497 now have more than 255 parameters. (Contributed by Serhiy Storchaka in 498 :issue:`12844` and :issue:`18896`.) 499 500 * :meth:`bytes.fromhex` and :meth:`bytearray.fromhex` now ignore all ASCII 501 whitespace, not only spaces. (Contributed by Robert Xiao in :issue:`28927`.) 502 503 * :class:`str`, :class:`bytes`, and :class:`bytearray` gained support for 504 the new :meth:`isascii() <str.isascii>` method, which can be used to 505 test if a string or bytes contain only the ASCII characters. 506 (Contributed by INADA Naoki in :issue:`32677`.) 507 508 * :exc:`ImportError` now displays module name and module ``__file__`` path when 509 ``from ... import ...`` fails. (Contributed by Matthias Bussonnier in 510 :issue:`29546`.) 511 512 * Circular imports involving absolute imports with binding a submodule to 513 a name are now supported. 514 (Contributed by Serhiy Storchaka in :issue:`30024`.) 515 516 * ``object.__format__(x, '')`` is now equivalent to ``str(x)`` rather than 517 ``format(str(self), '')``. 518 (Contributed by Serhiy Storchaka in :issue:`28974`.) 519 520 * In order to better support dynamic creation of stack traces, 521 :class:`types.TracebackType` can now be instantiated from Python code, and 522 the ``tb_next`` attribute on :ref:`tracebacks <traceback-objects>` is now 523 writable. 524 (Contributed by Nathaniel J. Smith in :issue:`30579`.) 525 526 * When using the :option:`-m` switch, ``sys.path[0]`` is now eagerly expanded 527 to the full starting directory path, rather than being left as the empty 528 directory (which allows imports from the *current* working directory at the 529 time when an import occurs) 530 (Contributed by Nick Coghlan in :issue:`33053`.) 531 532 * The new :option:`-X` ``importtime`` option or the 533 :envvar:`PYTHONPROFILEIMPORTTIME` environment variable can be used to show 534 the timing of each module import. 535 (Contributed by Victor Stinner in :issue:`31415`.) 536 537 538 New Modules 539 =========== 540 541 .. _whatsnew37-pep567: 542 543 contextvars 544 ----------- 545 546 The new :mod:`contextvars` module and a set of 547 :ref:`new C APIs <contextvarsobjects>` introduce 548 support for *context variables*. Context variables are conceptually 549 similar to thread-local variables. Unlike TLS, context variables 550 support asynchronous code correctly. 551 552 The :mod:`asyncio` and :mod:`decimal` modules have been updated to use 553 and support context variables out of the box. Particularly the active 554 decimal context is now stored in a context variable, which allows 555 decimal operations to work with the correct context in asynchronous code. 556 557 .. seealso:: 558 559 :pep:`567` -- Context Variables 560 PEP written and implemented by Yury Selivanov 561 562 563 .. _whatsnew37-pep557: 564 565 dataclasses 566 ----------- 567 568 The new :func:`~dataclasses.dataclass` decorator provides a way to declare 569 *data classes*. A data class describes its attributes using class variable 570 annotations. Its constructor and other magic methods, such as 571 :meth:`~object.__repr__`, :meth:`~object.__eq__`, and 572 :meth:`~object.__hash__` are generated automatically. 573 574 Example:: 575 576 @dataclass 577 class Point: 578 x: float 579 y: float 580 z: float = 0.0 581 582 p = Point(1.5, 2.5) 583 print(p) # produces "Point(x=1.5, y=2.5, z=0.0)" 584 585 .. seealso:: 586 587 :pep:`557` -- Data Classes 588 PEP written and implemented by Eric V. Smith 589 590 591 .. _whatsnew37_importlib_resources: 592 593 importlib.resources 594 ------------------- 595 596 The new :mod:`importlib.resources` module provides several new APIs and one 597 new ABC for access to, opening, and reading *resources* inside packages. 598 Resources are roughly similar to files inside packages, but they needn't 599 be actual files on the physical file system. Module loaders can provide a 600 :meth:`get_resource_reader()` function which returns 601 a :class:`importlib.abc.ResourceReader` instance to support this 602 new API. Built-in file path loaders and zip file loaders both support this. 603 604 Contributed by Barry Warsaw and Brett Cannon in :issue:`32248`. 605 606 .. seealso:: 607 608 `importlib_resources <http://importlib-resources.readthedocs.io/en/latest/>`_ 609 -- a PyPI backport for earlier Python versions. 610 611 612 Improved Modules 613 ================ 614 615 616 argparse 617 -------- 618 619 The new :meth:`ArgumentParser.parse_intermixed_args() 620 <argparse.ArgumentParser.parse_intermixed_args>` 621 method allows intermixing options and positional arguments. 622 (Contributed by paul.j3 in :issue:`14191`.) 623 624 625 .. _whatsnew37_asyncio: 626 627 asyncio 628 ------- 629 630 The :mod:`asyncio` module has received many new features, usability and 631 :ref:`performance improvements <whatsnew37-asyncio-perf>`. Notable changes 632 include: 633 634 * The new :term:`provisional <provisional api>` :func:`asyncio.run` function can 635 be used to run a coroutine from synchronous code by automatically creating and 636 destroying the event loop. 637 (Contributed by Yury Selivanov in :issue:`32314`.) 638 639 * asyncio gained support for :mod:`contextvars`. 640 :meth:`loop.call_soon() <asyncio.loop.call_soon>`, 641 :meth:`loop.call_soon_threadsafe() <asyncio.loop.call_soon_threadsafe>`, 642 :meth:`loop.call_later() <asyncio.loop.call_later>`, 643 :meth:`loop.call_at() <asyncio.loop.call_at>`, and 644 :meth:`Future.add_done_callback() <asyncio.Future.add_done_callback>` 645 have a new optional keyword-only *context* parameter. 646 :class:`Tasks <asyncio.Task>` now track their context automatically. 647 See :pep:`567` for more details. 648 (Contributed by Yury Selivanov in :issue:`32436`.) 649 650 * The new :func:`asyncio.create_task` function has been added as a shortcut 651 to ``asyncio.get_event_loop().create_task()``. 652 (Contributed by Andrew Svetlov in :issue:`32311`.) 653 654 * The new :meth:`loop.start_tls() <asyncio.loop.start_tls>` 655 method can be used to upgrade an existing connection to TLS. 656 (Contributed by Yury Selivanov in :issue:`23749`.) 657 658 * The new :meth:`loop.sock_recv_into() <asyncio.loop.sock_recv_into>` 659 method allows reading data from a socket directly into a provided buffer making 660 it possible to reduce data copies. 661 (Contributed by Antoine Pitrou in :issue:`31819`.) 662 663 * The new :func:`asyncio.current_task` function returns the currently running 664 :class:`~asyncio.Task` instance, and the new :func:`asyncio.all_tasks` 665 function returns a set of all existing ``Task`` instances in a given loop. 666 The :meth:`Task.current_task() <asyncio.Task.current_task>` and 667 :meth:`Task.all_tasks() <asyncio.Task.all_tasks>` methods have been deprecated. 668 (Contributed by Andrew Svetlov in :issue:`32250`.) 669 670 * The new *provisional* :class:`~asyncio.BufferedProtocol` class allows 671 implementing streaming protocols with manual control over the receive buffer. 672 (Contributed by Yury Selivanov in :issue:`32251`.) 673 674 * The new :func:`asyncio.get_running_loop` function returns the currently 675 running loop, and raises a :exc:`RuntimeError` if no loop is running. 676 This is in contrast with :func:`asyncio.get_event_loop`, which will *create* 677 a new event loop if none is running. 678 (Contributed by Yury Selivanov in :issue:`32269`.) 679 680 * The new :meth:`StreamWriter.wait_closed() <asyncio.StreamWriter.wait_closed>` 681 coroutine method allows waiting until the stream writer is closed. The new 682 :meth:`StreamWriter.is_closing() <asyncio.StreamWriter.is_closing>` method 683 can be used to determine if the writer is closing. 684 (Contributed by Andrew Svetlov in :issue:`32391`.) 685 686 * The new :meth:`loop.sock_sendfile() <asyncio.loop.sock_sendfile>` 687 coroutine method allows sending files using :mod:`os.sendfile` when possible. 688 (Contributed by Andrew Svetlov in :issue:`32410`.) 689 690 * The new :meth:`Future.get_loop() <asyncio.Future.get_loop>` and 691 ``Task.get_loop()`` methods return the instance of the loop on which a task or 692 a future were created. 693 :meth:`Server.get_loop() <asyncio.Server.get_loop>` allows doing the same for 694 :class:`asyncio.Server` objects. 695 (Contributed by Yury Selivanov in :issue:`32415` and 696 Srinivas Reddy Thatiparthy in :issue:`32418`.) 697 698 * It is now possible to control how instances of :class:`asyncio.Server` begin 699 serving. Previously, the server would start serving immediately when created. 700 The new *start_serving* keyword argument to 701 :meth:`loop.create_server() <asyncio.loop.create_server>` and 702 :meth:`loop.create_unix_server() <asyncio.loop.create_unix_server>`, 703 as well as :meth:`Server.start_serving() <asyncio.Server.start_serving>`, and 704 :meth:`Server.serve_forever() <asyncio.Server.serve_forever>` 705 can be used to decouple server instantiation and serving. The new 706 :meth:`Server.is_serving() <asyncio.Server.is_serving>` method returns ``True`` 707 if the server is serving. :class:`~asyncio.Server` objects are now 708 asynchronous context managers:: 709 710 srv = await loop.create_server(...) 711 712 async with srv: 713 # some code 714 715 # At this point, srv is closed and no longer accepts new connections. 716 717 (Contributed by Yury Selivanov in :issue:`32662`.) 718 719 * Callback objects returned by 720 :func:`loop.call_later() <asyncio.loop.call_later>` 721 gained the new :meth:`when() <asyncio.TimerHandle.when>` method which 722 returns an absolute scheduled callback timestamp. 723 (Contributed by Andrew Svetlov in :issue:`32741`.) 724 725 * The :meth:`loop.create_datagram_endpoint() \ 726 <asyncio.loop.create_datagram_endpoint>` method 727 gained support for Unix sockets. 728 (Contributed by Quentin Dawans in :issue:`31245`.) 729 730 * The :func:`asyncio.open_connection`, :func:`asyncio.start_server` functions, 731 :meth:`loop.create_connection() <asyncio.loop.create_connection>`, 732 :meth:`loop.create_server() <asyncio.loop.create_server>`, 733 :meth:`loop.create_accepted_socket() <asyncio.loop.connect_accepted_socket>` 734 methods and their corresponding UNIX socket variants now accept the 735 *ssl_handshake_timeout* keyword argument. 736 (Contributed by Neil Aspinall in :issue:`29970`.) 737 738 * The new :meth:`Handle.cancelled() <asyncio.Handle.cancelled>` method returns 739 ``True`` if the callback was cancelled. 740 (Contributed by Marat Sharafutdinov in :issue:`31943`.) 741 742 * The asyncio source has been converted to use the 743 :keyword:`async`/:keyword:`await` syntax. 744 (Contributed by Andrew Svetlov in :issue:`32193`.) 745 746 * The new :meth:`ReadTransport.is_reading() <asyncio.ReadTransport.is_reading>` 747 method can be used to determine the reading state of the transport. 748 Additionally, calls to 749 :meth:`ReadTransport.resume_reading() <asyncio.ReadTransport.resume_reading>` 750 and :meth:`ReadTransport.pause_reading() <asyncio.ReadTransport.pause_reading>` 751 are now idempotent. 752 (Contributed by Yury Selivanov in :issue:`32356`.) 753 754 * Loop methods which accept socket paths now support passing 755 :term:`path-like objects <path-like object>`. 756 (Contributed by Yury Selivanov in :issue:`32066`.) 757 758 * In :mod:`asyncio` TCP sockets on Linux are now created with ``TCP_NODELAY`` 759 flag set by default. 760 (Contributed by Yury Selivanov and Victor Stinner in :issue:`27456`.) 761 762 * Exceptions occurring in cancelled tasks are no longer logged. 763 (Contributed by Yury Selivanov in :issue:`30508`.) 764 765 * New ``WindowsSelectorEventLoopPolicy`` and 766 ``WindowsProactorEventLoopPolicy`` classes. 767 (Contributed by Yury Selivanov in :issue:`33792`.) 768 769 Several ``asyncio`` APIs have been 770 :ref:`deprecated <whatsnew37-asyncio-deprecated>`. 771 772 773 binascii 774 -------- 775 776 The :func:`~binascii.b2a_uu` function now accepts an optional *backtick* 777 keyword argument. When it's true, zeros are represented by ``'`'`` 778 instead of spaces. (Contributed by Xiang Zhang in :issue:`30103`.) 779 780 781 calendar 782 -------- 783 784 The :class:`~calendar.HTMLCalendar` class has new class attributes which ease 785 the customization of CSS classes in the produced HTML calendar. 786 (Contributed by Oz Tiram in :issue:`30095`.) 787 788 789 collections 790 ----------- 791 792 ``collections.namedtuple()`` now supports default values. 793 (Contributed by Raymond Hettinger in :issue:`32320`.) 794 795 796 compileall 797 ---------- 798 799 :func:`compileall.compile_dir` learned the new *invalidation_mode* parameter, 800 which can be used to enable 801 :ref:`hash-based .pyc invalidation <whatsnew37-pep552>`. The invalidation 802 mode can also be specified on the command line using the new 803 ``--invalidation-mode`` argument. 804 (Contributed by Benjamin Peterson in :issue:`31650`.) 805 806 807 concurrent.futures 808 ------------------ 809 810 :class:`ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor>` and 811 :class:`ThreadPoolExecutor <concurrent.futures.ThreadPoolExecutor>` now 812 support the new *initializer* and *initargs* constructor arguments. 813 (Contributed by Antoine Pitrou in :issue:`21423`.) 814 815 The :class:`ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor>` 816 can now take the multiprocessing context via the new *mp_context* argument. 817 (Contributed by Thomas Moreau in :issue:`31540`.) 818 819 820 contextlib 821 ---------- 822 823 The new :func:`~contextlib.nullcontext` is a simpler and faster no-op 824 context manager than :class:`~contextlib.ExitStack`. 825 (Contributed by Jesse-Bakker in :issue:`10049`.) 826 827 The new :func:`~contextlib.asynccontextmanager`, 828 :class:`~contextlib.AbstractAsyncContextManager`, and 829 :class:`~contextlib.AsyncExitStack` have been added to 830 complement their synchronous counterparts. (Contributed 831 by Jelle Zijlstra in :issue:`29679` and :issue:`30241`, 832 and by Alexander Mohr and Ilya Kulakov in :issue:`29302`.) 833 834 835 cProfile 836 -------- 837 838 The :mod:`cProfile` command line now accepts ``-m module_name`` as an 839 alternative to script path. (Contributed by Sanyam Khurana in :issue:`21862`.) 840 841 842 crypt 843 ----- 844 845 The :mod:`crypt` module now supports the Blowfish hashing method. 846 (Contributed by Serhiy Storchaka in :issue:`31664`.) 847 848 The :func:`~crypt.mksalt` function now allows specifying the number of rounds 849 for hashing. (Contributed by Serhiy Storchaka in :issue:`31702`.) 850 851 852 datetime 853 -------- 854 855 The new :meth:`datetime.fromisoformat() <datetime.datetime.fromisoformat>` 856 method constructs a :class:`~datetime.datetime` object from a string 857 in one of the formats output by 858 :meth:`datetime.isoformat() <datetime.datetime.isoformat>`. 859 (Contributed by Paul Ganssle in :issue:`15873`.) 860 861 The :class:`tzinfo <datetime.tzinfo>` class now supports sub-minute offsets. 862 (Contributed by Alexander Belopolsky in :issue:`5288`.) 863 864 865 dbm 866 --- 867 868 :mod:`dbm.dumb` now supports reading read-only files and no longer writes the 869 index file when it is not changed. 870 871 872 decimal 873 ------- 874 875 The :mod:`decimal` module now uses :ref:`context variables <whatsnew37-pep567>` 876 to store the decimal context. 877 (Contributed by Yury Selivanov in :issue:`32630`.) 878 879 880 dis 881 --- 882 883 The :func:`~dis.dis` function is now able to 884 disassemble nested code objects (the code of comprehensions, generator 885 expressions and nested functions, and the code used for building nested 886 classes). The maximum depth of disassembly recursion is controlled by 887 the new *depth* parameter. 888 (Contributed by Serhiy Storchaka in :issue:`11822`.) 889 890 891 distutils 892 --------- 893 894 ``README.rst`` is now included in the list of distutils standard READMEs and 895 therefore included in source distributions. 896 (Contributed by Ryan Gonzalez in :issue:`11913`.) 897 898 899 enum 900 ---- 901 902 The :class:`Enum <enum.Enum>` learned the new ``_ignore_`` class property, 903 which allows listing the names of properties which should not become 904 enum members. 905 (Contributed by Ethan Furman in :issue:`31801`.) 906 907 In Python 3.8, attempting to check for non-Enum objects in :class:`Enum` 908 classes will raise a :exc:`TypeError` (e.g. ``1 in Color``); similarly, 909 attempting to check for non-Flag objects in a :class:`Flag` member will 910 raise :exc:`TypeError` (e.g. ``1 in Perm.RW``); currently, both operations 911 return :const:`False` instead and are deprecated. 912 (Contributed by Ethan Furman in :issue:`33217`.) 913 914 915 functools 916 --------- 917 918 :func:`functools.singledispatch` now supports registering implementations 919 using type annotations. 920 (Contributed by ukasz Langa in :issue:`32227`.) 921 922 923 gc 924 -- 925 926 The new :func:`gc.freeze` function allows freezing all objects tracked 927 by the garbage collector and excluding them from future collections. 928 This can be used before a POSIX ``fork()`` call to make the GC copy-on-write 929 friendly or to speed up collection. The new :func:`gc.unfreeze` functions 930 reverses this operation. Additionally, :func:`gc.get_freeze_count` can 931 be used to obtain the number of frozen objects. 932 (Contributed by Li Zekun in :issue:`31558`.) 933 934 935 hmac 936 ---- 937 938 The :mod:`hmac` module now has an optimized one-shot :func:`~hmac.digest` 939 function, which is up to three times faster than :func:`~hmac.HMAC`. 940 (Contributed by Christian Heimes in :issue:`32433`.) 941 942 943 http.client 944 ----------- 945 946 :class:`~http.client.HTTPConnection` and :class:`~http.client.HTTPSConnection` 947 now support the new *blocksize* argument for improved upload throughput. 948 (Contributed by Nir Soffer in :issue:`31945`.) 949 950 951 http.server 952 ----------- 953 954 :class:`~http.server.SimpleHTTPRequestHandler` now supports the HTTP 955 ``If-Modified-Since`` header. The server returns the 304 response status if 956 the target file was not modified after the time specified in the header. 957 (Contributed by Pierre Quentel in :issue:`29654`.) 958 959 :class:`~http.server.SimpleHTTPRequestHandler` accepts the new *directory* 960 argument, in addition to the new ``--directory`` command line argument. 961 With this parameter, the server serves the specified directory, by default it 962 uses the current working directory. 963 (Contributed by Stphane Wirtel and Julien Palard in :issue:`28707`.) 964 965 The new :class:`ThreadingHTTPServer <http.server.ThreadingHTTPServer>` class 966 uses threads to handle requests using :class:`~socketserver.ThreadingMixin`. 967 It is used when ``http.server`` is run with ``-m``. 968 (Contributed by Julien Palard in :issue:`31639`.) 969 970 971 idlelib and IDLE 972 ---------------- 973 974 Multiple fixes for autocompletion. (Contributed by Louie Lu in :issue:`15786`.) 975 976 Module Browser (on the File menu, formerly called Class Browser), 977 now displays nested functions and classes in addition to top-level 978 functions and classes. 979 (Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy 980 in :issue:`1612262`.) 981 982 The Settings dialog (Options, Configure IDLE) has been partly rewritten 983 to improve both appearance and function. 984 (Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.) 985 986 The font sample now includes a selection of non-Latin characters so that 987 users can better see the effect of selecting a particular font. 988 (Contributed by Terry Jan Reedy in :issue:`13802`.) 989 The sample can be edited to include other characters. 990 (Contributed by Serhiy Storchaka in :issue:`31860`.) 991 992 The IDLE features formerly implemented as extensions have been reimplemented 993 as normal features. Their settings have been moved from the Extensions tab 994 to other dialog tabs. 995 (Contributed by Charles Wohlganger and Terry Jan Reedy in :issue:`27099`.) 996 997 Editor code context option revised. Box displays all context lines up to 998 maxlines. Clicking on a context line jumps the editor to that line. Context 999 colors for custom themes is added to Highlights tab of Settings dialog. 1000 (Contributed by Cheryl Sabella and Terry Jan Reedy in :issue:`33642`, 1001 :issue:`33768`, and :issue:`33679`.) 1002 1003 On Windows, a new API call tells Windows that tk scales for DPI. On Windows 1004 8.1+ or 10, with DPI compatibility properties of the Python binary 1005 unchanged, and a monitor resolution greater than 96 DPI, this should 1006 make text and lines sharper. It should otherwise have no effect. 1007 (Contributed by Terry Jan Reedy in :issue:`33656`.) 1008 1009 New in 3.7.1: 1010 1011 Output over N lines (50 by default) is squeezed down to a button. 1012 N can be changed in the PyShell section of the General page of the 1013 Settings dialog. Fewer, but possibly extra long, lines can be squeezed by 1014 right clicking on the output. Squeezed output can be expanded in place 1015 by double-clicking the button or into the clipboard or a separate window 1016 by right-clicking the button. (Contributed by Tal Einat in :issue:`1529353`.) 1017 1018 The changes above have been backported to 3.6 maintenance releases. 1019 1020 1021 importlib 1022 --------- 1023 1024 The :class:`importlib.abc.ResourceReader` ABC was introduced to 1025 support the loading of resources from packages. See also 1026 :ref:`whatsnew37_importlib_resources`. 1027 (Contributed by Barry Warsaw, Brett Cannon in :issue:`32248`.) 1028 1029 :func:`importlib.reload` now raises :exc:`ModuleNotFoundError` if the module 1030 lacks a spec. 1031 (Contributed by Garvit Khatri in :issue:`29851`.) 1032 1033 :func:`importlib.find_spec` now raises :exc:`ModuleNotFoundError` instead of 1034 :exc:`AttributeError` if the specified parent module is not a package (i.e. 1035 lacks a ``__path__`` attribute). 1036 (Contributed by Milan Oberkirch in :issue:`30436`.) 1037 1038 The new :func:`importlib.source_hash` can be used to compute the hash of 1039 the passed source. A :ref:`hash-based .pyc file <whatsnew37-pep552>` 1040 embeds the value returned by this function. 1041 1042 1043 io 1044 -- 1045 1046 The new :meth:`TextIOWrapper.reconfigure() <io.TextIOWrapper.reconfigure>` 1047 method can be used to reconfigure the text stream with the new settings. 1048 (Contributed by Antoine Pitrou in :issue:`30526` and 1049 INADA Naoki in :issue:`15216`.) 1050 1051 1052 ipaddress 1053 --------- 1054 1055 The new ``subnet_of()`` and ``supernet_of()`` methods of 1056 :class:`ipaddress.IPv6Network` and :class:`ipaddress.IPv4Network` can 1057 be used for network containment tests. 1058 (Contributed by Michel Albert and Cheryl Sabella in :issue:`20825`.) 1059 1060 1061 itertools 1062 --------- 1063 1064 :func:`itertools.islice` now accepts 1065 :meth:`integer-like objects <object.__index__>` as start, stop, 1066 and slice arguments. 1067 (Contributed by Will Roberts in :issue:`30537`.) 1068 1069 1070 locale 1071 ------ 1072 1073 The new *monetary* argument to :func:`locale.format_string` can be used 1074 to make the conversion use monetary thousands separators and 1075 grouping strings. (Contributed by Garvit in :issue:`10379`.) 1076 1077 The :func:`locale.getpreferredencoding` function now always returns ``'UTF-8'`` 1078 on Android or when in the :ref:`forced UTF-8 mode <whatsnew37-pep540>`. 1079 1080 1081 logging 1082 ------- 1083 1084 :class:`~logging.Logger` instances can now be pickled. 1085 (Contributed by Vinay Sajip in :issue:`30520`.) 1086 1087 The new :meth:`StreamHandler.setStream() <logging.StreamHandler.setStream>` 1088 method can be used to replace the logger stream after handler creation. 1089 (Contributed by Vinay Sajip in :issue:`30522`.) 1090 1091 It is now possible to specify keyword arguments to handler constructors in 1092 configuration passed to :func:`logging.config.fileConfig`. 1093 (Contributed by Preston Landers in :issue:`31080`.) 1094 1095 1096 math 1097 ---- 1098 1099 The new :func:`math.remainder` function implements the IEEE 754-style remainder 1100 operation. (Contributed by Mark Dickinson in :issue:`29962`.) 1101 1102 1103 mimetypes 1104 --------- 1105 1106 The MIME type of .bmp has been changed from ``'image/x-ms-bmp'`` to 1107 ``'image/bmp'``. 1108 (Contributed by Nitish Chandra in :issue:`22589`.) 1109 1110 1111 msilib 1112 ------ 1113 1114 The new :meth:`Database.Close() <msilib.Database.Close>` method can be used 1115 to close the :abbr:`MSI` database. 1116 (Contributed by Berker Peksag in :issue:`20486`.) 1117 1118 1119 multiprocessing 1120 --------------- 1121 1122 The new :meth:`Process.close() <multiprocessing.Process.close>` method 1123 explicitly closes the process object and releases all resources associated 1124 with it. :exc:`ValueError` is raised if the underlying process is still 1125 running. 1126 (Contributed by Antoine Pitrou in :issue:`30596`.) 1127 1128 The new :meth:`Process.kill() <multiprocessing.Process.kill>` method can 1129 be used to terminate the process using the :data:`SIGKILL` signal on Unix. 1130 (Contributed by Vitor Pereira in :issue:`30794`.) 1131 1132 Non-daemonic threads created by :class:`~multiprocessing.Process` are now 1133 joined on process exit. 1134 (Contributed by Antoine Pitrou in :issue:`18966`.) 1135 1136 1137 os 1138 -- 1139 1140 :func:`os.fwalk` now accepts the *path* argument as :class:`bytes`. 1141 (Contributed by Serhiy Storchaka in :issue:`28682`.) 1142 1143 :func:`os.scandir` gained support for :ref:`file descriptors <path_fd>`. 1144 (Contributed by Serhiy Storchaka in :issue:`25996`.) 1145 1146 The new :func:`~os.register_at_fork` function allows registering Python 1147 callbacks to be executed at process fork. 1148 (Contributed by Antoine Pitrou in :issue:`16500`.) 1149 1150 Added :func:`os.preadv` (combine the functionality of :func:`os.readv` and 1151 :func:`os.pread`) and :func:`os.pwritev` functions (combine the functionality 1152 of :func:`os.writev` and :func:`os.pwrite`). (Contributed by Pablo Galindo in 1153 :issue:`31368`.) 1154 1155 The mode argument of :func:`os.makedirs` no longer affects the file 1156 permission bits of newly-created intermediate-level directories. 1157 (Contributed by Serhiy Storchaka in :issue:`19930`.) 1158 1159 :func:`os.dup2` now returns the new file descriptor. Previously, ``None`` 1160 was always returned. 1161 (Contributed by Benjamin Peterson in :issue:`32441`.) 1162 1163 The structure returned by :func:`os.stat` now contains the 1164 :attr:`~os.stat_result.st_fstype` attribute on Solaris and its derivatives. 1165 (Contributed by Jess Cea Avin in :issue:`32659`.) 1166 1167 1168 pathlib 1169 ------- 1170 1171 The new :meth:`Path.is_mount() <pathlib.Path.is_mount>` method is now available 1172 on POSIX systems and can be used to determine whether a path is a mount point. 1173 (Contributed by Cooper Ry Lees in :issue:`30897`.) 1174 1175 1176 pdb 1177 --- 1178 1179 :func:`pdb.set_trace` now takes an optional *header* keyword-only 1180 argument. If given, it is printed to the console just before debugging 1181 begins. (Contributed by Barry Warsaw in :issue:`31389`.) 1182 1183 :mod:`pdb` command line now accepts ``-m module_name`` as an alternative to 1184 script file. (Contributed by Mario Corchero in :issue:`32206`.) 1185 1186 1187 py_compile 1188 ---------- 1189 1190 :func:`py_compile.compile` -- and by extension, :mod:`compileall` -- now 1191 respects the :envvar:`SOURCE_DATE_EPOCH` environment variable by 1192 unconditionally creating ``.pyc`` files for hash-based validation. 1193 This allows for guaranteeing 1194 `reproducible builds <https://reproducible-builds.org/>`_ of ``.pyc`` 1195 files when they are created eagerly. (Contributed by Bernhard M. Wiedemann 1196 in :issue:`29708`.) 1197 1198 1199 pydoc 1200 ----- 1201 1202 The pydoc server can now bind to an arbitrary hostname specified by the 1203 new ``-n`` command-line argument. 1204 (Contributed by Feanil Patel in :issue:`31128`.) 1205 1206 1207 queue 1208 ----- 1209 1210 The new :class:`~queue.SimpleQueue` class is an unbounded :abbr:`FIFO` queue. 1211 (Contributed by Antoine Pitrou in :issue:`14976`.) 1212 1213 1214 re 1215 -- 1216 1217 The flags :const:`re.ASCII`, :const:`re.LOCALE` and :const:`re.UNICODE` 1218 can be set within the scope of a group. 1219 (Contributed by Serhiy Storchaka in :issue:`31690`.) 1220 1221 :func:`re.split` now supports splitting on a pattern like ``r'\b'``, 1222 ``'^$'`` or ``(?=-)`` that matches an empty string. 1223 (Contributed by Serhiy Storchaka in :issue:`25054`.) 1224 1225 Regular expressions compiled with the :const:`re.LOCALE` flag no longer 1226 depend on the locale at compile time. Locale settings are applied only 1227 when the compiled regular expression is used. 1228 (Contributed by Serhiy Storchaka in :issue:`30215`.) 1229 1230 :exc:`FutureWarning` is now emitted if a regular expression contains 1231 character set constructs that will change semantically in the future, 1232 such as nested sets and set operations. 1233 (Contributed by Serhiy Storchaka in :issue:`30349`.) 1234 1235 Compiled regular expression and match objects can now be copied 1236 using :func:`copy.copy` and :func:`copy.deepcopy`. 1237 (Contributed by Serhiy Storchaka in :issue:`10076`.) 1238 1239 1240 signal 1241 ------ 1242 1243 The new *warn_on_full_buffer* argument to the :func:`signal.set_wakeup_fd` 1244 function makes it possible to specify whether Python prints a warning on 1245 stderr when the wakeup buffer overflows. 1246 (Contributed by Nathaniel J. Smith in :issue:`30050`.) 1247 1248 1249 socket 1250 ------ 1251 1252 The new :func:`socket.getblocking() <socket.socket.getblocking>` method 1253 returns ``True`` if the socket is in blocking mode and ``False`` otherwise. 1254 (Contributed by Yury Selivanov in :issue:`32373`.) 1255 1256 The new :func:`socket.close` function closes the passed socket file descriptor. 1257 This function should be used instead of :func:`os.close` for better 1258 compatibility across platforms. 1259 (Contributed by Christian Heimes in :issue:`32454`.) 1260 1261 The :mod:`socket` module now exposes the :data:`socket.TCP_CONGESTION` 1262 (Linux 2.6.13), :data:`socket.TCP_USER_TIMEOUT` (Linux 2.6.37), and 1263 :data:`socket.TCP_NOTSENT_LOWAT` (Linux 3.12) constants. 1264 (Contributed by Omar Sandoval in :issue:`26273` and 1265 Nathaniel J. Smith in :issue:`29728`.) 1266 1267 Support for :data:`socket.AF_VSOCK` sockets has been added to allow 1268 communication between virtual machines and their hosts. 1269 (Contributed by Cathy Avery in :issue:`27584`.) 1270 1271 Sockets now auto-detect family, type and protocol from file descriptor 1272 by default. 1273 (Contributed by Christian Heimes in :issue:`28134`.) 1274 1275 1276 socketserver 1277 ------------ 1278 1279 :meth:`socketserver.ThreadingMixIn.server_close` now waits until all non-daemon 1280 threads complete. :meth:`socketserver.ForkingMixIn.server_close` now waits 1281 until all child processes complete. 1282 1283 Add a new :attr:`socketserver.ForkingMixIn.block_on_close` class attribute to 1284 :class:`socketserver.ForkingMixIn` and :class:`socketserver.ThreadingMixIn` 1285 classes. Set the class attribute to ``False`` to get the pre-3.7 behaviour. 1286 1287 1288 sqlite3 1289 ------- 1290 1291 :class:`sqlite3.Connection` now exposes the :meth:`~sqlite3.Connection.backup` 1292 method when the underlying SQLite library is at version 3.6.11 or higher. 1293 (Contributed by Lele Gaifax in :issue:`27645`.) 1294 1295 The *database* argument of :func:`sqlite3.connect` now accepts any 1296 :term:`path-like object`, instead of just a string. 1297 (Contributed by Anders Lorentsen in :issue:`31843`.) 1298 1299 1300 ssl 1301 --- 1302 1303 The :mod:`ssl` module now uses OpenSSL's builtin API instead of 1304 :func:`~ssl.match_hostname` to check a host name or an IP address. Values 1305 are validated during TLS handshake. Any certificate validation error 1306 including failing the host name check now raises 1307 :exc:`~ssl.SSLCertVerificationError` and aborts the handshake with a proper 1308 TLS Alert message. The new exception contains additional information. 1309 Host name validation can be customized with 1310 :attr:`SSLContext.host_flags <ssl.SSLContext.host_flags>`. 1311 (Contributed by Christian Heimes in :issue:`31399`.) 1312 1313 .. note:: 1314 The improved host name check requires a *libssl* implementation compatible 1315 with OpenSSL 1.0.2 or 1.1. Consequently, OpenSSL 0.9.8 and 1.0.1 are no 1316 longer supported (see :ref:`37-platform-support-removals` for more details). 1317 The ssl module is mostly compatible with LibreSSL 2.7.2 and newer. 1318 1319 The ``ssl`` module no longer sends IP addresses in SNI TLS extension. 1320 (Contributed by Christian Heimes in :issue:`32185`.) 1321 1322 :func:`~ssl.match_hostname` no longer supports partial wildcards like 1323 ``www*.example.org``. :attr:`SSLContext.host_flags <ssl.SSLContext.host_flags>` 1324 has partial wildcard matching disabled by default. 1325 (Contributed by Mandeep Singh in :issue:`23033` and Christian Heimes in 1326 :issue:`31399`.) 1327 1328 The default cipher suite selection of the ``ssl`` module now uses a blacklist 1329 approach rather than a hard-coded whitelist. Python no longer re-enables 1330 ciphers that have been blocked by OpenSSL security updates. Default cipher 1331 suite selection can be configured at compile time. 1332 (Contributed by Christian Heimes in :issue:`31429`.) 1333 1334 Validation of server certificates containing internationalized domain names 1335 (IDNs) is now supported. As part of this change, the 1336 :attr:`SSLSocket.server_hostname <ssl.SSLSocket.server_hostname>` attribute 1337 now stores the expected hostname in A-label form (``"xn--pythn-mua.org"``), 1338 rather than the U-label form (``"pythn.org"``). (Contributed by 1339 Nathaniel J. Smith and Christian Heimes in :issue:`28414`.) 1340 1341 The ``ssl`` module has preliminary and experimental support for TLS 1.3 and 1342 OpenSSL 1.1.1. At the time of Python 3.7.0 release, OpenSSL 1.1.1 is still 1343 under development and TLS 1.3 hasn't been finalized yet. The TLS 1.3 1344 handshake and protocol behaves slightly differently than TLS 1.2 and earlier, 1345 see :ref:`ssl-tlsv1_3`. 1346 (Contributed by Christian Heimes in :issue:`32947`, :issue:`20995`, 1347 :issue:`29136`, :issue:`30622` and :issue:`33618`) 1348 1349 :class:`~ssl.SSLSocket` and :class:`~ssl.SSLObject` no longer have a public 1350 constructor. Direct instantiation was never a documented and supported 1351 feature. Instances must be created with :class:`~ssl.SSLContext` methods 1352 :meth:`~ssl.SSLContext.wrap_socket` and :meth:`~ssl.SSLContext.wrap_bio`. 1353 (Contributed by Christian Heimes in :issue:`32951`) 1354 1355 OpenSSL 1.1 APIs for setting the minimum and maximum TLS protocol version are 1356 available as :attr:`SSLContext.minimum_version <ssl.SSLContext.minimum_version>` 1357 and :attr:`SSLContext.maximum_version <ssl.SSLContext.maximum_version>`. 1358 Supported protocols are indicated by several new flags, such as 1359 :data:`~ssl.HAS_TLSv1_1`. 1360 (Contributed by Christian Heimes in :issue:`32609`.) 1361 1362 Added :attr:`SSLContext.post_handshake_auth` to enable and 1363 :meth:`ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3 1364 post-handshake authentication. 1365 (Contributed by Christian Heimes in :issue:`34670`.) 1366 1367 string 1368 ------ 1369 1370 :class:`string.Template` now lets you to optionally modify the regular 1371 expression pattern for braced placeholders and non-braced placeholders 1372 separately. (Contributed by Barry Warsaw in :issue:`1198569`.) 1373 1374 1375 subprocess 1376 ---------- 1377 1378 The :func:`subprocess.run` function accepts the new *capture_output* 1379 keyword argument. When true, stdout and stderr will be captured. 1380 This is equivalent to passing :data:`subprocess.PIPE` as *stdout* and 1381 *stderr* arguments. 1382 (Contributed by Bo Bayles in :issue:`32102`.) 1383 1384 The ``subprocess.run`` function and the :class:`subprocess.Popen` constructor 1385 now accept the *text* keyword argument as an alias 1386 to *universal_newlines*. 1387 (Contributed by Andrew Clegg in :issue:`31756`.) 1388 1389 On Windows the default for *close_fds* was changed from ``False`` to 1390 ``True`` when redirecting the standard handles. It's now possible to set 1391 *close_fds* to true when redirecting the standard handles. See 1392 :class:`subprocess.Popen`. This means that *close_fds* now defaults to 1393 ``True`` on all supported platforms. 1394 (Contributed by Segev Finer in :issue:`19764`.) 1395 1396 The subprocess module is now more graceful when handling 1397 :exc:`KeyboardInterrupt` during :func:`subprocess.call`, 1398 :func:`subprocess.run`, or in a :class:`~subprocess.Popen` 1399 context manager. It now waits a short amount of time for the child 1400 to exit, before continuing the handling of the ``KeyboardInterrupt`` 1401 exception. 1402 (Contributed by Gregory P. Smith in :issue:`25942`.) 1403 1404 1405 sys 1406 --- 1407 1408 The new :func:`sys.breakpointhook` hook function is called by the 1409 built-in :func:`breakpoint`. 1410 (Contributed by Barry Warsaw in :issue:`31353`.) 1411 1412 On Android, the new :func:`sys.getandroidapilevel` returns the build-time 1413 Android API version. 1414 (Contributed by Victor Stinner in :issue:`28740`.) 1415 1416 The new :func:`sys.get_coroutine_origin_tracking_depth` function returns 1417 the current coroutine origin tracking depth, as set by 1418 the new :func:`sys.set_coroutine_origin_tracking_depth`. :mod:`asyncio` 1419 has been converted to use this new API instead of 1420 the deprecated :func:`sys.set_coroutine_wrapper`. 1421 (Contributed by Nathaniel J. Smith in :issue:`32591`.) 1422 1423 1424 time 1425 ---- 1426 1427 :pep:`564` adds six new functions with nanosecond resolution to the 1428 :mod:`time` module: 1429 1430 * :func:`time.clock_gettime_ns` 1431 * :func:`time.clock_settime_ns` 1432 * :func:`time.monotonic_ns` 1433 * :func:`time.perf_counter_ns` 1434 * :func:`time.process_time_ns` 1435 * :func:`time.time_ns` 1436 1437 New clock identifiers have been added: 1438 1439 * :data:`time.CLOCK_BOOTTIME` (Linux): Identical to 1440 :data:`time.CLOCK_MONOTONIC`, except it also includes any time that the 1441 system is suspended. 1442 * :data:`time.CLOCK_PROF` (FreeBSD, NetBSD and OpenBSD): High-resolution 1443 per-process CPU timer. 1444 * :data:`time.CLOCK_UPTIME` (FreeBSD, OpenBSD): Time whose absolute value is 1445 the time the system has been running and not suspended, providing accurate 1446 uptime measurement. 1447 1448 The new :func:`time.thread_time` and :func:`time.thread_time_ns` functions 1449 can be used to get per-thread CPU time measurements. 1450 (Contributed by Antoine Pitrou in :issue:`32025`.) 1451 1452 The new :func:`time.pthread_getcpuclockid` function returns the clock ID 1453 of the thread-specific CPU-time clock. 1454 1455 1456 tkinter 1457 ------- 1458 1459 The new :class:`tkinter.ttk.Spinbox` class is now available. 1460 (Contributed by Alan Moore in :issue:`32585`.) 1461 1462 1463 tracemalloc 1464 ----------- 1465 1466 :class:`tracemalloc.Traceback` behaves more like regular tracebacks, 1467 sorting the frames from oldest to most recent. 1468 :meth:`Traceback.format() <tracemalloc.Traceback.format>` 1469 now accepts negative *limit*, truncating the result to the 1470 ``abs(limit)`` oldest frames. To get the old behaviour, use 1471 the new *most_recent_first* argument to ``Traceback.format()``. 1472 (Contributed by Jesse Bakker in :issue:`32121`.) 1473 1474 1475 types 1476 ----- 1477 1478 The new :class:`~types.WrapperDescriptorType`, 1479 :class:`~types.MethodWrapperType`, :class:`~types.MethodDescriptorType`, 1480 and :class:`~types.ClassMethodDescriptorType` classes are now available. 1481 (Contributed by Manuel Krebber and Guido van Rossum in :issue:`29377`, 1482 and Serhiy Storchaka in :issue:`32265`.) 1483 1484 The new :func:`types.resolve_bases` function resolves MRO entries 1485 dynamically as specified by :pep:`560`. 1486 (Contributed by Ivan Levkivskyi in :issue:`32717`.) 1487 1488 1489 unicodedata 1490 ----------- 1491 1492 The internal :mod:`unicodedata` database has been upgraded to use `Unicode 11 1493 <http://www.unicode.org/versions/Unicode11.0.0/>`_. (Contributed by Benjamin 1494 Peterson.) 1495 1496 1497 unittest 1498 -------- 1499 1500 The new ``-k`` command-line option allows filtering tests by a name 1501 substring or a Unix shell-like pattern. 1502 For example, ``python -m unittest -k foo`` runs 1503 ``foo_tests.SomeTest.test_something``, ``bar_tests.SomeTest.test_foo``, 1504 but not ``bar_tests.FooTest.test_something``. 1505 (Contributed by Jonas Haag in :issue:`32071`.) 1506 1507 1508 unittest.mock 1509 ------------- 1510 1511 The :const:`~unittest.mock.sentinel` attributes now preserve their identity 1512 when they are :mod:`copied <copy>` or :mod:`pickled <pickle>`. (Contributed by 1513 Serhiy Storchaka in :issue:`20804`.) 1514 1515 The new :func:`~unittest.mock.seal` function allows sealing 1516 :class:`~unittest.mock.Mock` instances, which will disallow further creation 1517 of attribute mocks. The seal is applied recursively to all attributes that 1518 are themselves mocks. 1519 (Contributed by Mario Corchero in :issue:`30541`.) 1520 1521 1522 urllib.parse 1523 ------------ 1524 1525 :func:`urllib.parse.quote` has been updated from :rfc:`2396` to :rfc:`3986`, 1526 adding ``~`` to the set of characters that are never quoted by default. 1527 (Contributed by Christian Theune and Ratnadeep Debnath in :issue:`16285`.) 1528 1529 1530 uu 1531 -- 1532 1533 The :func:`uu.encode` function now accepts an optional *backtick* 1534 keyword argument. When it's true, zeros are represented by ``'`'`` 1535 instead of spaces. (Contributed by Xiang Zhang in :issue:`30103`.) 1536 1537 1538 uuid 1539 ---- 1540 1541 The new :attr:`UUID.is_safe <uuid.UUID.is_safe>` attribute relays information 1542 from the platform about whether generated UUIDs are generated with a 1543 multiprocessing-safe method. 1544 (Contributed by Barry Warsaw in :issue:`22807`.) 1545 1546 :func:`uuid.getnode` now prefers universally administered 1547 MAC addresses over locally administered MAC addresses. 1548 This makes a better guarantee for global uniqueness of UUIDs returned 1549 from :func:`uuid.uuid1`. If only locally administered MAC addresses are 1550 available, the first such one found is returned. 1551 (Contributed by Barry Warsaw in :issue:`32107`.) 1552 1553 1554 warnings 1555 -------- 1556 1557 The initialization of the default warnings filters has changed as follows: 1558 1559 * warnings enabled via command line options (including those for :option:`-b` 1560 and the new CPython-specific :option:`-X` ``dev`` option) are always passed 1561 to the warnings machinery via the :data:`sys.warnoptions` attribute. 1562 1563 * warnings filters enabled via the command line or the environment now have the 1564 following order of precedence: 1565 1566 * the ``BytesWarning`` filter for :option:`-b` (or ``-bb``) 1567 * any filters specified with the :option:`-W` option 1568 * any filters specified with the :envvar:`PYTHONWARNINGS` environment 1569 variable 1570 * any other CPython specific filters (e.g. the ``default`` filter added 1571 for the new ``-X dev`` mode) 1572 * any implicit filters defined directly by the warnings machinery 1573 1574 * in CPython debug builds, all warnings are now displayed by default (the 1575 implicit filter list is empty) 1576 1577 (Contributed by Nick Coghlan and Victor Stinner in :issue:`20361`, 1578 :issue:`32043`, and :issue:`32230`.) 1579 1580 Deprecation warnings are once again shown by default in single-file scripts and 1581 at the interactive prompt. See :ref:`whatsnew37-pep565` for details. 1582 (Contributed by Nick Coghlan in :issue:`31975`.) 1583 1584 1585 xml 1586 --- 1587 1588 As mitigation against DTD and external entity retrieval, the 1589 :mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process 1590 external entities by default. 1591 (Contributed by Christian Heimes in :issue:`17239`.) 1592 1593 1594 xml.etree 1595 --------- 1596 1597 :ref:`ElementPath <elementtree-xpath>` predicates in the :meth:`find` 1598 methods can now compare text of the current node with ``[. = "text"]``, 1599 not only text in children. Predicates also allow adding spaces for 1600 better readability. (Contributed by Stefan Behnel in :issue:`31648`.) 1601 1602 1603 xmlrpc.server 1604 ------------- 1605 1606 :meth:`SimpleXMLRPCDispatcher.register_function <xmlrpc.server.SimpleXMLRPCDispatcher>` 1607 can now be used as a decorator. (Contributed by Xiang Zhang in 1608 :issue:`7769`.) 1609 1610 1611 zipapp 1612 ------ 1613 1614 Function :func:`~zipapp.create_archive` now accepts an optional *filter* 1615 argument to allow the user to select which files should be included in the 1616 archive. (Contributed by Irmen de Jong in :issue:`31072`.) 1617 1618 Function :func:`~zipapp.create_archive` now accepts an optional *compressed* 1619 argument to generate a compressed archive. A command line option 1620 ``--compress`` has also been added to support compression. 1621 (Contributed by Zhiming Wang in :issue:`31638`.) 1622 1623 1624 zipfile 1625 ------- 1626 1627 :class:`~zipfile.ZipFile` now accepts the new *compresslevel* parameter to 1628 control the compression level. 1629 (Contributed by Bo Bayles in :issue:`21417`.) 1630 1631 Subdirectories in archives created by ``ZipFile`` are now stored in 1632 alphabetical order. 1633 (Contributed by Bernhard M. Wiedemann in :issue:`30693`.) 1634 1635 1636 C API Changes 1637 ============= 1638 1639 A new API for thread-local storage has been implemented. See 1640 :ref:`whatsnew37-pep539` for an overview and 1641 :ref:`thread-specific-storage-api` for a complete reference. 1642 (Contributed by Masayuki Yamamoto in :issue:`25658`.) 1643 1644 The new :ref:`context variables <whatsnew37-pep567>` functionality 1645 exposes a number of :ref:`new C APIs <contextvarsobjects>`. 1646 1647 The new :c:func:`PyImport_GetModule` function returns the previously 1648 imported module with the given name. 1649 (Contributed by Eric Snow in :issue:`28411`.) 1650 1651 The new :c:macro:`Py_RETURN_RICHCOMPARE` macro eases writing rich 1652 comparison functions. 1653 (Contributed by Petr Victorin in :issue:`23699`.) 1654 1655 The new :c:macro:`Py_UNREACHABLE` macro can be used to mark unreachable 1656 code paths. 1657 (Contributed by Barry Warsaw in :issue:`31338`.) 1658 1659 The :mod:`tracemalloc` now exposes a C API through the new 1660 :c:func:`PyTraceMalloc_Track` and :c:func:`PyTraceMalloc_Untrack` 1661 functions. 1662 (Contributed by Victor Stinner in :issue:`30054`.) 1663 1664 The new :c:func:`import__find__load__start` and 1665 :c:func:`import__find__load__done` static markers can be used to trace 1666 module imports. 1667 (Contributed by Christian Heimes in :issue:`31574`.) 1668 1669 The fields :c:member:`name` and :c:member:`doc` of structures 1670 :c:type:`PyMemberDef`, :c:type:`PyGetSetDef`, 1671 :c:type:`PyStructSequence_Field`, :c:type:`PyStructSequence_Desc`, 1672 and :c:type:`wrapperbase` are now of type ``const char *`` rather of 1673 ``char *``. (Contributed by Serhiy Storchaka in :issue:`28761`.) 1674 1675 The result of :c:func:`PyUnicode_AsUTF8AndSize` and :c:func:`PyUnicode_AsUTF8` 1676 is now of type ``const char *`` rather of ``char *``. (Contributed by Serhiy 1677 Storchaka in :issue:`28769`.) 1678 1679 The result of :c:func:`PyMapping_Keys`, :c:func:`PyMapping_Values` and 1680 :c:func:`PyMapping_Items` is now always a list, rather than a list or a 1681 tuple. (Contributed by Oren Milman in :issue:`28280`.) 1682 1683 Added functions :c:func:`PySlice_Unpack` and :c:func:`PySlice_AdjustIndices`. 1684 (Contributed by Serhiy Storchaka in :issue:`27867`.) 1685 1686 :c:func:`PyOS_AfterFork` is deprecated in favour of the new functions 1687 :c:func:`PyOS_BeforeFork`, :c:func:`PyOS_AfterFork_Parent` and 1688 :c:func:`PyOS_AfterFork_Child`. (Contributed by Antoine Pitrou in 1689 :issue:`16500`.) 1690 1691 The ``PyExc_RecursionErrorInst`` singleton that was part of the public API 1692 has been removed as its members being never cleared may cause a segfault 1693 during finalization of the interpreter. Contributed by Xavier de Gaye in 1694 :issue:`22898` and :issue:`30697`. 1695 1696 Added C API support for timezones with timezone constructors 1697 :c:func:`PyTimeZone_FromOffset` and :c:func:`PyTimeZone_FromOffsetAndName`, 1698 and access to the UTC singleton with :c:data:`PyDateTime_TimeZone_UTC`. 1699 Contributed by Paul Ganssle in :issue:`10381`. 1700 1701 The type of results of :c:func:`PyThread_start_new_thread` and 1702 :c:func:`PyThread_get_thread_ident`, and the *id* parameter of 1703 :c:func:`PyThreadState_SetAsyncExc` changed from :c:type:`long` to 1704 :c:type:`unsigned long`. 1705 (Contributed by Serhiy Storchaka in :issue:`6532`.) 1706 1707 :c:func:`PyUnicode_AsWideCharString` now raises a :exc:`ValueError` if the 1708 second argument is *NULL* and the :c:type:`wchar_t*` string contains null 1709 characters. (Contributed by Serhiy Storchaka in :issue:`30708`.) 1710 1711 Changes to the startup sequence and the management of dynamic memory 1712 allocators mean that the long documented requirement to call 1713 :c:func:`Py_Initialize` before calling most C API functions is now 1714 relied on more heavily, and failing to abide by it may lead to segfaults in 1715 embedding applications. See the :ref:`porting-to-python-37` section in this 1716 document and the :ref:`pre-init-safe` section in the C API documentation 1717 for more details. 1718 1719 The new :c:func:`PyInterpreterState_GetID` returns the unique ID for a 1720 given interpreter. 1721 (Contributed by Eric Snow in :issue:`29102`.) 1722 1723 :c:func:`Py_DecodeLocale`, :c:func:`Py_EncodeLocale` now use the UTF-8 1724 encoding when the :ref:`UTF-8 mode <whatsnew37-pep540>` is enabled. 1725 (Contributed by Victor Stinner in :issue:`29240`.) 1726 1727 :c:func:`PyUnicode_DecodeLocaleAndSize` and :c:func:`PyUnicode_EncodeLocale` 1728 now use the current locale encoding for ``surrogateescape`` error handler. 1729 (Contributed by Victor Stinner in :issue:`29240`.) 1730 1731 The *start* and *end* parameters of :c:func:`PyUnicode_FindChar` are 1732 now adjusted to behave like string slices. 1733 (Contributed by Xiang Zhang in :issue:`28822`.) 1734 1735 1736 Build Changes 1737 ============= 1738 1739 Support for building ``--without-threads`` has been removed. The 1740 :mod:`threading` module is now always available. 1741 (Contributed by Antoine Pitrou in :issue:`31370`.). 1742 1743 A full copy of libffi is no longer bundled for use when building the 1744 :mod:`_ctypes <ctypes>` module on non-OSX UNIX platforms. An installed copy 1745 of libffi is now required when building ``_ctypes`` on such platforms. 1746 (Contributed by Zachary Ware in :issue:`27979`.) 1747 1748 The Windows build process no longer depends on Subversion to pull in external 1749 sources, a Python script is used to download zipfiles from GitHub instead. 1750 If Python 3.6 is not found on the system (via ``py -3.6``), NuGet is used to 1751 download a copy of 32-bit Python for this purpose. (Contributed by Zachary 1752 Ware in :issue:`30450`.) 1753 1754 The :mod:`ssl` module requires OpenSSL 1.0.2 or 1.1 compatible libssl. 1755 OpenSSL 1.0.1 has reached end of lifetime on 2016-12-31 and is no longer 1756 supported. LibreSSL is temporarily not supported as well. LibreSSL releases 1757 up to version 2.6.4 are missing required OpenSSL 1.0.2 APIs. 1758 1759 1760 .. _whatsnew37-perf: 1761 1762 Optimizations 1763 ============= 1764 1765 The overhead of calling many methods of various standard library classes 1766 implemented in C has been significantly reduced by porting more code 1767 to use the ``METH_FASTCALL`` convention. 1768 (Contributed by Victor Stinner in :issue:`29300`, :issue:`29507`, 1769 :issue:`29452`, and :issue:`29286`.) 1770 1771 Various optimizations have reduced Python startup time by 10% on Linux and 1772 up to 30% on macOS. 1773 (Contributed by Victor Stinner, INADA Naoki in :issue:`29585`, and 1774 Ivan Levkivskyi in :issue:`31333`.) 1775 1776 Method calls are now up to 20% faster due to the bytecode changes which 1777 avoid creating bound method instances. 1778 (Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.) 1779 1780 .. _whatsnew37-asyncio-perf: 1781 1782 The :mod:`asyncio` module received a number of notable optimizations for 1783 commonly used functions: 1784 1785 * The :func:`asyncio.get_event_loop` function has been reimplemented in C to 1786 make it up to 15 times faster. 1787 (Contributed by Yury Selivanov in :issue:`32296`.) 1788 1789 * :class:`asyncio.Future` callback management has been optimized. 1790 (Contributed by Yury Selivanov in :issue:`32348`.) 1791 1792 * :func:`asyncio.gather` is now up to 15% faster. 1793 (Contributed by Yury Selivanov in :issue:`32355`.) 1794 1795 * :func:`asyncio.sleep` is now up to 2 times faster when the *delay* 1796 argument is zero or negative. 1797 (Contributed by Andrew Svetlov in :issue:`32351`.) 1798 1799 * The performance overhead of asyncio debug mode has been reduced. 1800 (Contributed by Antoine Pitrou in :issue:`31970`.) 1801 1802 As a result of :ref:`PEP 560 work <whatsnew37-pep560>`, the import time 1803 of :mod:`typing` has been reduced by a factor of 7, and many typing operations 1804 are now faster. 1805 (Contributed by Ivan Levkivskyi in :issue:`32226`.) 1806 1807 :func:`sorted` and :meth:`list.sort` have been optimized for common cases 1808 to be up to 40-75% faster. 1809 (Contributed by Elliot Gorokhovsky in :issue:`28685`.) 1810 1811 :meth:`dict.copy` is now up to 5.5 times faster. 1812 (Contributed by Yury Selivanov in :issue:`31179`.) 1813 1814 :func:`hasattr` and :func:`getattr` are now about 4 times faster when 1815 *name* is not found and *obj* does not override :meth:`object.__getattr__` 1816 or :meth:`object.__getattribute__`. 1817 (Contributed by INADA Naoki in :issue:`32544`.) 1818 1819 Searching for certain Unicode characters (like Ukrainian capital "") 1820 in a string was up to 25 times slower than searching for other characters. 1821 It is now only 3 times slower in the worst case. 1822 (Contributed by Serhiy Storchaka in :issue:`24821`.) 1823 1824 The :func:`collections.namedtuple` factory has been reimplemented to 1825 make the creation of named tuples 4 to 6 times faster. 1826 (Contributed by Jelle Zijlstra with further improvements by INADA Naoki, 1827 Serhiy Storchaka, and Raymond Hettinger in :issue:`28638`.) 1828 1829 :meth:`date.fromordinal` and :meth:`date.fromtimestamp` are now up to 1830 30% faster in the common case. 1831 (Contributed by Paul Ganssle in :issue:`32403`.) 1832 1833 The :func:`os.fwalk` function is now up to 2 times faster thanks to 1834 the use of :func:`os.scandir`. 1835 (Contributed by Serhiy Storchaka in :issue:`25996`.) 1836 1837 The speed of the :func:`shutil.rmtree` function has been improved by 1838 20--40% thanks to the use of the :func:`os.scandir` function. 1839 (Contributed by Serhiy Storchaka in :issue:`28564`.) 1840 1841 Optimized case-insensitive matching and searching of :mod:`regular 1842 expressions <re>`. Searching some patterns can now be up to 20 times faster. 1843 (Contributed by Serhiy Storchaka in :issue:`30285`.) 1844 1845 :func:`re.compile` now converts ``flags`` parameter to int object if 1846 it is ``RegexFlag``. It is now as fast as Python 3.5, and faster than 1847 Python 3.6 by about 10% depending on the pattern. 1848 (Contributed by INADA Naoki in :issue:`31671`.) 1849 1850 The :meth:`~selectors.BaseSelector.modify` methods of classes 1851 :class:`selectors.EpollSelector`, :class:`selectors.PollSelector` 1852 and :class:`selectors.DevpollSelector` may be around 10% faster under 1853 heavy loads. (Contributed by Giampaolo Rodola' in :issue:`30014`) 1854 1855 Constant folding has been moved from the peephole optimizer to the new AST 1856 optimizer, which is able perform optimizations more consistently. 1857 (Contributed by Eugene Toder and INADA Naoki in :issue:`29469` and 1858 :issue:`11549`.) 1859 1860 Most functions and methods in :mod:`abc` have been rewritten in C. 1861 This makes creation of abstract base classes, and calling :func:`isinstance` 1862 and :func:`issubclass` on them 1.5x faster. This also reduces Python 1863 start-up time by up to 10%. (Contributed by Ivan Levkivskyi and INADA Naoki 1864 in :issue:`31333`) 1865 1866 Significant speed improvements to alternate constructors for 1867 :class:`datetime.date` and :class:`datetime.datetime` by using fast-path 1868 constructors when not constructing subclasses. (Contributed by Paul Ganssle 1869 in :issue:`32403`) 1870 1871 The speed of comparison of :class:`array.array` instances has been 1872 improved considerably in certain cases. It is now from 10x to 70x faster 1873 when comparing arrays holding values of the same integer type. 1874 (Contributed by Adrian Wielgosik in :issue:`24700`.) 1875 1876 The :func:`math.erf` and :func:`math.erfc` functions now use the (faster) 1877 C library implementation on most platforms. 1878 (Contributed by Serhiy Storchaka in :issue:`26121`.) 1879 1880 1881 Other CPython Implementation Changes 1882 ==================================== 1883 1884 * Trace hooks may now opt out of receiving the ``line`` and opt into receiving 1885 the ``opcode`` events from the interpreter by setting the corresponding new 1886 ``f_trace_lines`` and ``f_trace_opcodes`` attributes on the 1887 frame being traced. (Contributed by Nick Coghlan in :issue:`31344`.) 1888 1889 * Fixed some consistency problems with namespace package module attributes. 1890 Namespace module objects now have an ``__file__`` that is set to ``None`` 1891 (previously unset), and their ``__spec__.origin`` is also set to ``None`` 1892 (previously the string ``"namespace"``). See :issue:`32305`. Also, the 1893 namespace module object's ``__spec__.loader`` is set to the same value as 1894 ``__loader__`` (previously, the former was set to ``None``). See 1895 :issue:`32303`. 1896 1897 * The :func:`locals` dictionary now displays in the lexical order that 1898 variables were defined. Previously, the order was undefined. 1899 (Contributed by Raymond Hettinger in :issue:`32690`.) 1900 1901 * The :mod:`distutils` ``upload`` command no longer tries to change CR 1902 end-of-line characters to CRLF. This fixes a corruption issue with sdists 1903 that ended with a byte equivalent to CR. 1904 (Contributed by Bo Bayles in :issue:`32304`.) 1905 1906 1907 Deprecated Python Behavior 1908 ========================== 1909 1910 Yield expressions (both ``yield`` and ``yield from`` clauses) are now deprecated 1911 in comprehensions and generator expressions (aside from the iterable expression 1912 in the leftmost :keyword:`!for` clause). This ensures that comprehensions 1913 always immediately return a container of the appropriate type (rather than 1914 potentially returning a :term:`generator iterator` object), while generator 1915 expressions won't attempt to interleave their implicit output with the output 1916 from any explicit yield expressions. In Python 3.7, such expressions emit 1917 :exc:`DeprecationWarning` when compiled, in Python 3.8 this will be a 1918 :exc:`SyntaxError`. 1919 (Contributed by Serhiy Storchaka in :issue:`10544`.) 1920 1921 Returning a subclass of :class:`complex` from :meth:`object.__complex__` is 1922 deprecated and will be an error in future Python versions. This makes 1923 ``__complex__()`` consistent with :meth:`object.__int__` and 1924 :meth:`object.__float__`. 1925 (Contributed by Serhiy Storchaka in :issue:`28894`.) 1926 1927 1928 1929 Deprecated Python modules, functions and methods 1930 ================================================ 1931 1932 aifc 1933 ---- 1934 1935 :func:`aifc.openfp` has been deprecated and will be removed in Python 3.9. 1936 Use :func:`aifc.open` instead. 1937 (Contributed by Brian Curtin in :issue:`31985`.) 1938 1939 1940 .. _whatsnew37-asyncio-deprecated: 1941 1942 asyncio 1943 ------- 1944 1945 Support for directly ``await``-ing instances of :class:`asyncio.Lock` and 1946 other asyncio synchronization primitives has been deprecated. An 1947 asynchronous context manager must be used in order to acquire and release 1948 the synchronization resource. 1949 (Contributed by Andrew Svetlov in :issue:`32253`.) 1950 1951 The :meth:`asyncio.Task.current_task` and :meth:`asyncio.Task.all_tasks` 1952 methods have been deprecated. 1953 (Contributed by Andrew Svetlov in :issue:`32250`.) 1954 1955 1956 collections 1957 ----------- 1958 1959 In Python 3.8, the abstract base classes in :mod:`collections.abc` will no 1960 longer be exposed in the regular :mod:`collections` module. This will help 1961 create a clearer distinction between the concrete classes and the abstract 1962 base classes. 1963 (Contributed by Serhiy Storchaka in :issue:`25988`.) 1964 1965 1966 dbm 1967 --- 1968 1969 :mod:`dbm.dumb` now supports reading read-only files and no longer writes the 1970 index file when it is not changed. A deprecation warning is now emitted 1971 if the index file is missing and recreated in the ``'r'`` and ``'w'`` 1972 modes (this will be an error in future Python releases). 1973 (Contributed by Serhiy Storchaka in :issue:`28847`.) 1974 1975 1976 enum 1977 ---- 1978 1979 In Python 3.8, attempting to check for non-Enum objects in :class:`Enum` 1980 classes will raise a :exc:`TypeError` (e.g. ``1 in Color``); similarly, 1981 attempting to check for non-Flag objects in a :class:`Flag` member will 1982 raise :exc:`TypeError` (e.g. ``1 in Perm.RW``); currently, both operations 1983 return :const:`False` instead. 1984 (Contributed by Ethan Furman in :issue:`33217`.) 1985 1986 1987 gettext 1988 ------- 1989 1990 Using non-integer value for selecting a plural form in :mod:`gettext` is 1991 now deprecated. It never correctly worked. (Contributed by Serhiy Storchaka 1992 in :issue:`28692`.) 1993 1994 1995 importlib 1996 --------- 1997 1998 Methods 1999 :meth:`MetaPathFinder.find_module() <importlib.abc.MetaPathFinder.find_module>` 2000 (replaced by 2001 :meth:`MetaPathFinder.find_spec() <importlib.abc.MetaPathFinder.find_spec>`) 2002 and 2003 :meth:`PathEntryFinder.find_loader() <importlib.abc.PathEntryFinder.find_loader>` 2004 (replaced by 2005 :meth:`PathEntryFinder.find_spec() <importlib.abc.PathEntryFinder.find_spec>`) 2006 both deprecated in Python 3.4 now emit :exc:`DeprecationWarning`. 2007 (Contributed by Matthias Bussonnier in :issue:`29576`) 2008 2009 The :class:`importlib.abc.ResourceLoader` ABC has been deprecated in 2010 favour of :class:`importlib.abc.ResourceReader`. 2011 2012 2013 locale 2014 ------ 2015 2016 :func:`locale.format` has been deprecated, use :meth:`locale.format_string` 2017 instead. (Contributed by Garvit in :issue:`10379`.) 2018 2019 2020 macpath 2021 ------- 2022 2023 The :mod:`macpath` is now deprecated and will be removed in Python 3.8. 2024 (Contributed by Chi Hsuan Yen in :issue:`9850`.) 2025 2026 2027 threading 2028 --------- 2029 2030 :mod:`dummy_threading` and :mod:`_dummy_thread` have been deprecated. It is 2031 no longer possible to build Python with threading disabled. 2032 Use :mod:`threading` instead. 2033 (Contributed by Antoine Pitrou in :issue:`31370`.) 2034 2035 2036 socket 2037 ------ 2038 2039 The silent argument value truncation in :func:`socket.htons` and 2040 :func:`socket.ntohs` has been deprecated. In future versions of Python, 2041 if the passed argument is larger than 16 bits, an exception will be raised. 2042 (Contributed by Oren Milman in :issue:`28332`.) 2043 2044 2045 ssl 2046 --- 2047 2048 :func:`ssl.wrap_socket` is deprecated. Use 2049 :meth:`ssl.SSLContext.wrap_socket` instead. 2050 (Contributed by Christian Heimes in :issue:`28124`.) 2051 2052 2053 sunau 2054 ----- 2055 2056 :func:`sunau.openfp` has been deprecated and will be removed in Python 3.9. 2057 Use :func:`sunau.open` instead. 2058 (Contributed by Brian Curtin in :issue:`31985`.) 2059 2060 2061 sys 2062 --- 2063 2064 Deprecated :func:`sys.set_coroutine_wrapper` and 2065 :func:`sys.get_coroutine_wrapper`. 2066 2067 The undocumented ``sys.callstats()`` function has been deprecated and 2068 will be removed in a future Python version. 2069 (Contributed by Victor Stinner in :issue:`28799`.) 2070 2071 2072 wave 2073 ---- 2074 2075 :func:`wave.openfp` has been deprecated and will be removed in Python 3.9. 2076 Use :func:`wave.open` instead. 2077 (Contributed by Brian Curtin in :issue:`31985`.) 2078 2079 2080 Deprecated functions and types of the C API 2081 =========================================== 2082 2083 Function :c:func:`PySlice_GetIndicesEx` is deprecated and replaced with 2084 a macro if ``Py_LIMITED_API`` is not set or set to a value in the range 2085 between ``0x03050400`` and ``0x03060000`` (not inclusive), or is ``0x03060100`` 2086 or higher. (Contributed by Serhiy Storchaka in :issue:`27867`.) 2087 2088 :c:func:`PyOS_AfterFork` has been deprecated. Use :c:func:`PyOS_BeforeFork`, 2089 :c:func:`PyOS_AfterFork_Parent` or :c:func:`PyOS_AfterFork_Child()` instead. 2090 (Contributed by Antoine Pitrou in :issue:`16500`.) 2091 2092 2093 .. _37-platform-support-removals: 2094 2095 Platform Support Removals 2096 ========================= 2097 2098 * FreeBSD 9 and older are no longer officially supported. 2099 * For full Unicode support, including within extension modules, \*nix platforms 2100 are now expected to provide at least one of ``C.UTF-8`` (full locale), 2101 ``C.utf8`` (full locale) or ``UTF-8`` (``LC_CTYPE``-only locale) as an 2102 alternative to the legacy ``ASCII``-based ``C`` locale. 2103 * OpenSSL 0.9.8 and 1.0.1 are no longer supported, which means building CPython 2104 3.7 with SSL/TLS support on older platforms still using these versions 2105 requires custom build options that link to a more recent version of OpenSSL. 2106 2107 Notably, this issue affects the Debian 8 (aka "jessie") and Ubuntu 14.04 2108 (aka "Trusty") LTS Linux distributions, as they still use OpenSSL 1.0.1 by 2109 default. 2110 2111 Debian 9 ("stretch") and Ubuntu 16.04 ("xenial"), as well as recent releases 2112 of other LTS Linux releases (e.g. RHEL/CentOS 7.5, SLES 12-SP3), use OpenSSL 2113 1.0.2 or later, and remain supported in the default build configuration. 2114 2115 CPython's own :source:`CI configuration file <.travis.yml>` provides an 2116 example of using the SSL 2117 :source:`compatibility testing infrastructure <Tools/ssl/multissltests.py>` in 2118 CPython's test suite to build and link against OpenSSL 1.1.0 rather than an 2119 outdated system provided OpenSSL. 2120 2121 2122 API and Feature Removals 2123 ======================== 2124 2125 The following features and APIs have been removed from Python 3.7: 2126 2127 * The ``os.stat_float_times()`` function has been removed. It was introduced in 2128 Python 2.3 for backward compatibility with Python 2.2, and was deprecated 2129 since Python 3.1. 2130 2131 * Unknown escapes consisting of ``'\'`` and an ASCII letter in replacement 2132 templates for :func:`re.sub` were deprecated in Python 3.5, and will now 2133 cause an error. 2134 2135 * Removed support of the *exclude* argument in :meth:`tarfile.TarFile.add`. 2136 It was deprecated in Python 2.7 and 3.2. Use the *filter* argument instead. 2137 2138 * The ``splitunc()`` function in the :mod:`ntpath` module was deprecated in 2139 Python 3.1, and has now been removed. Use the :func:`~os.path.splitdrive` 2140 function instead. 2141 2142 * :func:`collections.namedtuple` no longer supports the *verbose* parameter 2143 or ``_source`` attribute which showed the generated source code for the 2144 named tuple class. This was part of an optimization designed to speed-up 2145 class creation. (Contributed by Jelle Zijlstra with further improvements 2146 by INADA Naoki, Serhiy Storchaka, and Raymond Hettinger in :issue:`28638`.) 2147 2148 * Functions :func:`bool`, :func:`float`, :func:`list` and :func:`tuple` no 2149 longer take keyword arguments. The first argument of :func:`int` can now 2150 be passed only as positional argument. 2151 2152 * Removed previously deprecated in Python 2.4 classes ``Plist``, ``Dict`` and 2153 ``_InternalDict`` in the :mod:`plistlib` module. Dict values in the result 2154 of functions :func:`~plistlib.readPlist` and 2155 :func:`~plistlib.readPlistFromBytes` are now normal dicts. You no longer 2156 can use attribute access to access items of these dictionaries. 2157 2158 * The ``asyncio.windows_utils.socketpair()`` function has been 2159 removed. Use the :func:`socket.socketpair` function instead, 2160 it is available on all platforms since Python 3.5. 2161 ``asyncio.windows_utils.socketpair`` was just an alias to 2162 ``socket.socketpair`` on Python 3.5 and newer. 2163 2164 * :mod:`asyncio` no longer exports the :mod:`selectors` and 2165 :mod:`_overlapped` modules as ``asyncio.selectors`` and 2166 ``asyncio._overlapped``. Replace ``from asyncio import selectors`` with 2167 ``import selectors``. 2168 2169 * Direct instantiation of :class:`ssl.SSLSocket` and :class:`ssl.SSLObject` 2170 objects is now prohibited. The constructors were never documented, tested, 2171 or designed as public constructors. Users were supposed to use 2172 :func:`ssl.wrap_socket` or :class:`ssl.SSLContext`. 2173 (Contributed by Christian Heimes in :issue:`32951`.) 2174 2175 * The unused :mod:`distutils` ``install_misc`` command has been removed. 2176 (Contributed by Eric N. Vander Weele in :issue:`29218`.) 2177 2178 2179 Module Removals 2180 =============== 2181 2182 The ``fpectl`` module has been removed. It was never enabled by 2183 default, never worked correctly on x86-64, and it changed the Python 2184 ABI in ways that caused unexpected breakage of C extensions. 2185 (Contributed by Nathaniel J. Smith in :issue:`29137`.) 2186 2187 2188 Windows-only Changes 2189 ==================== 2190 2191 The python launcher, (py.exe), can accept 32 & 64 bit specifiers **without** 2192 having to specify a minor version as well. So ``py -3-32`` and ``py -3-64`` 2193 become valid as well as ``py -3.7-32``, also the -*m*-64 and -*m.n*-64 forms 2194 are now accepted to force 64 bit python even if 32 bit would have otherwise 2195 been used. If the specified version is not available py.exe will error exit. 2196 (Contributed by Steve Barnes in :issue:`30291`.) 2197 2198 The launcher can be run as ``py -0`` to produce a list of the installed pythons, 2199 *with default marked with an asterisk*. Running ``py -0p`` will include the paths. 2200 If py is run with a version specifier that cannot be matched it will also print 2201 the *short form* list of available specifiers. 2202 (Contributed by Steve Barnes in :issue:`30362`.) 2203 2204 2205 .. _porting-to-python-37: 2206 2207 Porting to Python 3.7 2208 ===================== 2209 2210 This section lists previously described changes and other bugfixes 2211 that may require changes to your code. 2212 2213 2214 Changes in Python Behavior 2215 -------------------------- 2216 2217 * :keyword:`async` and :keyword:`await` names are now reserved keywords. 2218 Code using these names as identifiers will now raise a :exc:`SyntaxError`. 2219 (Contributed by Jelle Zijlstra in :issue:`30406`.) 2220 2221 * :pep:`479` is enabled for all code in Python 3.7, meaning that 2222 :exc:`StopIteration` exceptions raised directly or indirectly in 2223 coroutines and generators are transformed into :exc:`RuntimeError` 2224 exceptions. 2225 (Contributed by Yury Selivanov in :issue:`32670`.) 2226 2227 * :meth:`object.__aiter__` methods can no longer be declared as 2228 asynchronous. (Contributed by Yury Selivanov in :issue:`31709`.) 2229 2230 * Due to an oversight, earlier Python versions erroneously accepted the 2231 following syntax:: 2232 2233 f(1 for x in [1],) 2234 2235 class C(1 for x in [1]): 2236 pass 2237 2238 Python 3.7 now correctly raises a :exc:`SyntaxError`, as a generator 2239 expression always needs to be directly inside a set of parentheses 2240 and cannot have a comma on either side, and the duplication of the 2241 parentheses can be omitted only on calls. 2242 (Contributed by Serhiy Storchaka in :issue:`32012` and :issue:`32023`.) 2243 2244 * When using the :option:`-m` switch, the initial working directory is now added 2245 to :data:`sys.path`, rather than an empty string (which dynamically denoted 2246 the current working directory at the time of each import). Any programs that 2247 are checking for the empty string, or otherwise relying on the previous 2248 behaviour, will need to be updated accordingly (e.g. by also checking for 2249 ``os.getcwd()`` or ``os.path.dirname(__main__.__file__)``, depending on why 2250 the code was checking for the empty string in the first place). 2251 2252 2253 Changes in the Python API 2254 ------------------------- 2255 2256 * :meth:`socketserver.ThreadingMixIn.server_close` now waits until all 2257 non-daemon threads complete. Set the new 2258 :attr:`socketserver.ThreadingMixIn.block_on_close` class attribute to 2259 ``False`` to get the pre-3.7 behaviour. 2260 (Contributed by Victor Stinner in :issue:`31233` and :issue:`33540`.) 2261 2262 * :meth:`socketserver.ForkingMixIn.server_close` now waits until all 2263 child processes complete. Set the new 2264 :attr:`socketserver.ForkingMixIn.block_on_close` class attribute to ``False`` 2265 to get the pre-3.7 behaviour. 2266 (Contributed by Victor Stinner in :issue:`31151` and :issue:`33540`.) 2267 2268 * The :func:`locale.localeconv` function now temporarily sets the ``LC_CTYPE`` 2269 locale to the value of ``LC_NUMERIC`` in some cases. 2270 (Contributed by Victor Stinner in :issue:`31900`.) 2271 2272 * :meth:`pkgutil.walk_packages` now raises a :exc:`ValueError` if *path* is 2273 a string. Previously an empty list was returned. 2274 (Contributed by Sanyam Khurana in :issue:`24744`.) 2275 2276 * A format string argument for :meth:`string.Formatter.format` 2277 is now :ref:`positional-only <positional-only_parameter>`. 2278 Passing it as a keyword argument was deprecated in Python 3.5. (Contributed 2279 by Serhiy Storchaka in :issue:`29193`.) 2280 2281 * Attributes :attr:`~http.cookies.Morsel.key`, 2282 :attr:`~http.cookies.Morsel.value` and 2283 :attr:`~http.cookies.Morsel.coded_value` of class 2284 :class:`http.cookies.Morsel` are now read-only. 2285 Assigning to them was deprecated in Python 3.5. 2286 Use the :meth:`~http.cookies.Morsel.set` method for setting them. 2287 (Contributed by Serhiy Storchaka in :issue:`29192`.) 2288 2289 * The *mode* argument of :func:`os.makedirs` no longer affects the file 2290 permission bits of newly-created intermediate-level directories. 2291 To set their file permission bits you can set the umask before invoking 2292 ``makedirs()``. 2293 (Contributed by Serhiy Storchaka in :issue:`19930`.) 2294 2295 * The :attr:`struct.Struct.format` type is now :class:`str` instead of 2296 :class:`bytes`. (Contributed by Victor Stinner in :issue:`21071`.) 2297 2298 * :func:`~cgi.parse_multipart` now accepts the *encoding* and *errors* 2299 arguments and returns the same results as 2300 :class:`~FieldStorage`: for non-file fields, the value associated to a key 2301 is a list of strings, not bytes. 2302 (Contributed by Pierre Quentel in :issue:`29979`.) 2303 2304 * Due to internal changes in :mod:`socket`, calling :func:`socket.fromshare` 2305 on a socket created by :func:`socket.share <socket.socket.share>` in older 2306 Python versions is not supported. 2307 2308 * ``repr`` for :exc:`BaseException` has changed to not include the trailing 2309 comma. Most exceptions are affected by this change. 2310 (Contributed by Serhiy Storchaka in :issue:`30399`.) 2311 2312 * ``repr`` for :class:`datetime.timedelta` has changed to include the keyword 2313 arguments in the output. (Contributed by Utkarsh Upadhyay in :issue:`30302`.) 2314 2315 * Because :func:`shutil.rmtree` is now implemented using the :func:`os.scandir` 2316 function, the user specified handler *onerror* is now called with the first 2317 argument ``os.scandir`` instead of ``os.listdir`` when listing the directory 2318 is failed. 2319 2320 * Support for nested sets and set operations in regular expressions as in 2321 `Unicode Technical Standard #18`_ might be added in the future. This would 2322 change the syntax. To facilitate this future change a :exc:`FutureWarning` 2323 will be raised in ambiguous cases for the time being. 2324 That include sets starting with a literal ``'['`` or containing literal 2325 character sequences ``'--'``, ``'&&'``, ``'~~'``, and ``'||'``. To 2326 avoid a warning, escape them with a backslash. 2327 (Contributed by Serhiy Storchaka in :issue:`30349`.) 2328 2329 .. _Unicode Technical Standard #18: https://unicode.org/reports/tr18/ 2330 2331 * The result of splitting a string on a :mod:`regular expression <re>` 2332 that could match an empty string has been changed. For example 2333 splitting on ``r'\s*'`` will now split not only on whitespaces as it 2334 did previously, but also on empty strings before all non-whitespace 2335 characters and just before the end of the string. 2336 The previous behavior can be restored by changing the pattern 2337 to ``r'\s+'``. A :exc:`FutureWarning` was emitted for such patterns since 2338 Python 3.5. 2339 2340 For patterns that match both empty and non-empty strings, the result of 2341 searching for all matches may also be changed in other cases. For example 2342 in the string ``'a\n\n'``, the pattern ``r'(?m)^\s*?$'`` will not only 2343 match empty strings at positions 2 and 3, but also the string ``'\n'`` at 2344 positions 2--3. To match only blank lines, the pattern should be rewritten 2345 as ``r'(?m)^[^\S\n]*$'``. 2346 2347 :func:`re.sub()` now replaces empty matches adjacent to a previous 2348 non-empty match. For example ``re.sub('x*', '-', 'abxd')`` returns now 2349 ``'-a-b--d-'`` instead of ``'-a-b-d-'`` (the first minus between 'b' and 2350 'd' replaces 'x', and the second minus replaces an empty string between 2351 'x' and 'd'). 2352 2353 (Contributed by Serhiy Storchaka in :issue:`25054` and :issue:`32308`.) 2354 2355 * Change :func:`re.escape` to only escape regex special characters instead 2356 of escaping all characters other than ASCII letters, numbers, and ``'_'``. 2357 (Contributed by Serhiy Storchaka in :issue:`29995`.) 2358 2359 * :class:`tracemalloc.Traceback` frames are now sorted from oldest to most 2360 recent to be more consistent with :mod:`traceback`. 2361 (Contributed by Jesse Bakker in :issue:`32121`.) 2362 2363 * On OSes that support :const:`socket.SOCK_NONBLOCK` or 2364 :const:`socket.SOCK_CLOEXEC` bit flags, the 2365 :attr:`socket.type <socket.socket.type>` no longer has them applied. 2366 Therefore, checks like ``if sock.type == socket.SOCK_STREAM`` 2367 work as expected on all platforms. 2368 (Contributed by Yury Selivanov in :issue:`32331`.) 2369 2370 * On Windows the default for the *close_fds* argument of 2371 :class:`subprocess.Popen` was changed from :const:`False` to :const:`True` 2372 when redirecting the standard handles. If you previously depended on handles 2373 being inherited when using :class:`subprocess.Popen` with standard io 2374 redirection, you will have to pass ``close_fds=False`` to preserve the 2375 previous behaviour, or use 2376 :attr:`STARTUPINFO.lpAttributeList <subprocess.STARTUPINFO.lpAttributeList>`. 2377 2378 * :meth:`importlib.machinery.PathFinder.invalidate_caches` -- which implicitly 2379 affects :func:`importlib.invalidate_caches` -- now deletes entries 2380 in :data:`sys.path_importer_cache` which are set to ``None``. 2381 (Contributed by Brett Cannon in :issue:`33169`.) 2382 2383 * In :mod:`asyncio`, 2384 :meth:`loop.sock_recv() <asyncio.loop.sock_recv>`, 2385 :meth:`loop.sock_sendall() <asyncio.loop.sock_sendall>`, 2386 :meth:`loop.sock_accept() <asyncio.loop.sock_accept>`, 2387 :meth:`loop.getaddrinfo() <asyncio.loop.getaddrinfo>`, 2388 :meth:`loop.getnameinfo() <asyncio.loop.getnameinfo>` 2389 have been changed to be proper coroutine methods to match their 2390 documentation. Previously, these methods returned :class:`asyncio.Future` 2391 instances. 2392 (Contributed by Yury Selivanov in :issue:`32327`.) 2393 2394 * :attr:`asyncio.Server.sockets` now returns a copy of the internal list 2395 of server sockets, instead of returning it directly. 2396 (Contributed by Yury Selivanov in :issue:`32662`.) 2397 2398 * :attr:`Struct.format <struct.Struct.format>` is now a :class:`str` instance 2399 instead of a :class:`bytes` instance. 2400 (Contributed by Victor Stinner in :issue:`21071`.) 2401 2402 * :meth:`ast.literal_eval()` is now stricter. Addition and subtraction of 2403 arbitrary numbers are no longer allowed. 2404 (Contributed by Serhiy Storchaka in :issue:`31778`.) 2405 2406 * :meth:`Calendar.itermonthdates <calendar.Calendar.itermonthdates>` 2407 will now consistently raise an exception when a date falls outside of the 2408 ``0001-01-01`` through ``9999-12-31`` range. To support applications that 2409 cannot tolerate such exceptions, the new 2410 :meth:`Calendar.itermonthdays3 <calendar.Calendar.itermonthdays3>` and 2411 :meth:`Calendar.itermonthdays4 <calendar.Calendar.itermonthdays4>` can be used. 2412 The new methods return tuples and are not restricted by the range supported by 2413 :class:`datetime.date`. 2414 (Contributed by Alexander Belopolsky in :issue:`28292`.) 2415 2416 * :class:`collections.ChainMap` now preserves the order of the underlying 2417 mappings. (Contributed by Raymond Hettinger in :issue:`32792`.) 2418 2419 * The ``submit()`` method of :class:`concurrent.futures.ThreadPoolExecutor` 2420 and :class:`concurrent.futures.ProcessPoolExecutor` now raises 2421 a :exc:`RuntimeError` if called during interpreter shutdown. 2422 (Contributed by Mark Nemec in :issue:`33097`.) 2423 2424 * The :class:`configparser.ConfigParser` constructor now uses ``read_dict()`` 2425 to process the default values, making its behavior consistent with the 2426 rest of the parser. Non-string keys and values in the defaults 2427 dictionary are now being implicitly converted to strings. 2428 (Contributed by James Tocknell in :issue:`23835`.) 2429 2430 * Several undocumented internal imports were removed. 2431 One example is that ``os.errno`` is no longer available; use ``import errno`` 2432 directly instead. 2433 Note that such undocumented internal imports may be removed any time without 2434 notice, even in micro version releases. 2435 2436 2437 Changes in the C API 2438 -------------------- 2439 2440 The function :c:func:`PySlice_GetIndicesEx` is considered unsafe for 2441 resizable sequences. If the slice indices are not instances of :class:`int`, 2442 but objects that implement the :meth:`!__index__` method, the sequence can be 2443 resized after passing its length to :c:func:`!PySlice_GetIndicesEx`. This 2444 can lead to returning indices out of the length of the sequence. For 2445 avoiding possible problems use new functions :c:func:`PySlice_Unpack` and 2446 :c:func:`PySlice_AdjustIndices`. 2447 (Contributed by Serhiy Storchaka in :issue:`27867`.) 2448 2449 2450 CPython bytecode changes 2451 ------------------------ 2452 2453 There are two new opcodes: :opcode:`LOAD_METHOD` and :opcode:`CALL_METHOD`. 2454 (Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.) 2455 2456 The :opcode:`STORE_ANNOTATION` opcode has been removed. 2457 (Contributed by Mark Shannon in :issue:`32550`.) 2458 2459 2460 Windows-only Changes 2461 -------------------- 2462 2463 The file used to override :data:`sys.path` is now called 2464 ``<python-executable>._pth`` instead of ``'sys.path'``. 2465 See :ref:`finding_modules` for more information. 2466 (Contributed by Steve Dower in :issue:`28137`.) 2467 2468 2469 Other CPython implementation changes 2470 ------------------------------------ 2471 2472 In preparation for potential future changes to the public CPython runtime 2473 initialization API (see :pep:`432` for an initial, but somewhat outdated, 2474 draft), CPython's internal startup 2475 and configuration management logic has been significantly refactored. While 2476 these updates are intended to be entirely transparent to both embedding 2477 applications and users of the regular CPython CLI, they're being mentioned 2478 here as the refactoring changes the internal order of various operations 2479 during interpreter startup, and hence may uncover previously latent defects, 2480 either in embedding applications, or in CPython itself. 2481 (Initially contributed by Nick Coghlan and Eric Snow as part of 2482 :issue:`22257`, and further updated by Nick, Eric, and Victor Stinner in a 2483 number of other issues). Some known details affected: 2484 2485 * :c:func:`PySys_AddWarnOptionUnicode` is not currently usable by embedding 2486 applications due to the requirement to create a Unicode object prior to 2487 calling `Py_Initialize`. Use :c:func:`PySys_AddWarnOption` instead. 2488 2489 * warnings filters added by an embedding application with 2490 :c:func:`PySys_AddWarnOption` should now more consistently take precedence 2491 over the default filters set by the interpreter 2492 2493 Due to changes in the way the default warnings filters are configured, 2494 setting :c:data:`Py_BytesWarningFlag` to a value greater than one is no longer 2495 sufficient to both emit :exc:`BytesWarning` messages and have them converted 2496 to exceptions. Instead, the flag must be set (to cause the warnings to be 2497 emitted in the first place), and an explicit ``error::BytesWarning`` 2498 warnings filter added to convert them to exceptions. 2499 2500 Due to a change in the way docstrings are handled by the compiler, the 2501 implicit ``return None`` in a function body consisting solely of a docstring 2502 is now marked as occurring on the same line as the docstring, not on the 2503 function's header line. 2504 2505 The current exception state has been moved from the frame object to the co-routine. 2506 This simplified the interpreter and fixed a couple of obscure bugs caused by 2507 having swap exception state when entering or exiting a generator. 2508 (Contributed by Mark Shannon in :issue:`25612`.) 2509 2510 Notable changes in Python 3.7.1 2511 =============================== 2512 2513 Starting in 3.7.1, :c:func:`Py_Initialize` now consistently reads and respects 2514 all of the same environment settings as :c:func:`Py_Main` (in earlier Python 2515 versions, it respected an ill-defined subset of those environment variables, 2516 while in Python 3.7.0 it didn't read any of them due to :issue:`34247`). If 2517 this behavior is unwanted, set :c:data:`Py_IgnoreEnvironmentFlag` to 1 before 2518 calling :c:func:`Py_Initialize`. 2519 2520 In 3.7.1 the C API for Context Variables 2521 :ref:`was updated <contextvarsobjects_pointertype_change>` to use 2522 :c:type:`PyObject` pointers. See also :issue:`34762`. 2523 2524 :mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process 2525 external entities by default. See also :issue:`17239`. 2526 2527 In 3.7.1 the :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token 2528 when provided with input that does not have a trailing new line. This behavior 2529 now matches what the C tokenizer does internally. 2530 (Contributed by Ammar Askar in :issue:`33899`.) 2531 2532 Notable changes in Python 3.7.2 2533 =============================== 2534 2535 In 3.7.2, :mod:`venv` on Windows no longer copies the original binaries, but 2536 creates redirector scripts named ``python.exe`` and ``pythonw.exe`` instead. 2537 This resolves a long standing issue where all virtual environments would have 2538 to be upgraded or recreated with each Python update. However, note that this 2539 release will still require recreation of virtual environments in order to get 2540 the new scripts. 2541
