1 **************************** 2 What's New In Python 3.2 3 **************************** 4 5 :Author: Raymond Hettinger 6 7 .. $Id$ 8 Rules for maintenance: 9 10 * Anyone can add text to this document. Do not spend very much time 11 on the wording of your changes, because your text will probably 12 get rewritten. (Note, during release candidate phase or just before 13 a beta release, please use the tracker instead -- this helps avoid 14 merge conflicts. If you must add a suggested entry directly, 15 please put it in an XXX comment and the maintainer will take notice). 16 17 * The maintainer will go through Misc/NEWS periodically and add 18 changes; it's therefore more important to add your changes to 19 Misc/NEWS than to this file. 20 21 * This is not a complete list of every single change; completeness 22 is the purpose of Misc/NEWS. Some changes I consider too small 23 or esoteric to include. If such a change is added to the text, 24 I'll just remove it. (This is another reason you shouldn't spend 25 too much time on writing your addition.) 26 27 * If you want to draw your new text to the attention of the 28 maintainer, add 'XXX' to the beginning of the paragraph or 29 section. 30 31 * It's OK to just add a fragmentary note about a change. For 32 example: "XXX Describe the transmogrify() function added to the 33 socket module." The maintainer will research the change and 34 write the necessary text. 35 36 * You can comment out your additions if you like, but it's not 37 necessary (especially when a final release is some months away). 38 39 * Credit the author of a patch or bugfix. Just the name is 40 sufficient; the e-mail address isn't necessary. It's helpful to 41 add the issue number: 42 43 XXX Describe the transmogrify() function added to the socket 44 module. 45 46 (Contributed by P.Y. Developer; :issue:`12345`.) 47 48 This saves the maintainer the effort of going through the SVN log 49 when researching a change. 50 51 This article explains the new features in Python 3.2 as compared to 3.1. It 52 focuses on a few highlights and gives a few examples. For full details, see the 53 `Misc/NEWS <https://hg.python.org/cpython/file/3.2/Misc/NEWS>`_ file. 54 55 .. seealso:: 56 57 :pep:`392` - Python 3.2 Release Schedule 58 59 60 PEP 384: Defining a Stable ABI 61 ============================== 62 63 In the past, extension modules built for one Python version were often 64 not usable with other Python versions. Particularly on Windows, every 65 feature release of Python required rebuilding all extension modules that 66 one wanted to use. This requirement was the result of the free access to 67 Python interpreter internals that extension modules could use. 68 69 With Python 3.2, an alternative approach becomes available: extension 70 modules which restrict themselves to a limited API (by defining 71 Py_LIMITED_API) cannot use many of the internals, but are constrained 72 to a set of API functions that are promised to be stable for several 73 releases. As a consequence, extension modules built for 3.2 in that 74 mode will also work with 3.3, 3.4, and so on. Extension modules that 75 make use of details of memory structures can still be built, but will 76 need to be recompiled for every feature release. 77 78 .. seealso:: 79 80 :pep:`384` - Defining a Stable ABI 81 PEP written by Martin von Lwis. 82 83 84 PEP 389: Argparse Command Line Parsing Module 85 ============================================= 86 87 A new module for command line parsing, :mod:`argparse`, was introduced to 88 overcome the limitations of :mod:`optparse` which did not provide support for 89 positional arguments (not just options), subcommands, required options and other 90 common patterns of specifying and validating options. 91 92 This module has already had widespread success in the community as a 93 third-party module. Being more fully featured than its predecessor, the 94 :mod:`argparse` module is now the preferred module for command-line processing. 95 The older module is still being kept available because of the substantial amount 96 of legacy code that depends on it. 97 98 Here's an annotated example parser showing features like limiting results to a 99 set of choices, specifying a *metavar* in the help screen, validating that one 100 or more positional arguments is present, and making a required option:: 101 102 import argparse 103 parser = argparse.ArgumentParser( 104 description = 'Manage servers', # main description for help 105 epilog = 'Tested on Solaris and Linux') # displayed after help 106 parser.add_argument('action', # argument name 107 choices = ['deploy', 'start', 'stop'], # three allowed values 108 help = 'action on each target') # help msg 109 parser.add_argument('targets', 110 metavar = 'HOSTNAME', # var name used in help msg 111 nargs = '+', # require one or more targets 112 help = 'url for target machines') # help msg explanation 113 parser.add_argument('-u', '--user', # -u or --user option 114 required = True, # make it a required argument 115 help = 'login as user') 116 117 Example of calling the parser on a command string:: 118 119 >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain' 120 >>> result = parser.parse_args(cmd.split()) 121 >>> result.action 122 'deploy' 123 >>> result.targets 124 ['sneezy.example.com', 'sleepy.example.com'] 125 >>> result.user 126 'skycaptain' 127 128 Example of the parser's automatically generated help:: 129 130 >>> parser.parse_args('-h'.split()) 131 132 usage: manage_cloud.py [-h] -u USER 133 {deploy,start,stop} HOSTNAME [HOSTNAME ...] 134 135 Manage servers 136 137 positional arguments: 138 {deploy,start,stop} action on each target 139 HOSTNAME url for target machines 140 141 optional arguments: 142 -h, --help show this help message and exit 143 -u USER, --user USER login as user 144 145 Tested on Solaris and Linux 146 147 An especially nice :mod:`argparse` feature is the ability to define subparsers, 148 each with their own argument patterns and help displays:: 149 150 import argparse 151 parser = argparse.ArgumentParser(prog='HELM') 152 subparsers = parser.add_subparsers() 153 154 parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup 155 parser_l.add_argument('-m', '--missiles', action='store_true') 156 parser_l.add_argument('-t', '--torpedos', action='store_true') 157 158 parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup 159 aliases=('steer', 'turn')) # equivalent names 160 parser_m.add_argument('-c', '--course', type=int, required=True) 161 parser_m.add_argument('-s', '--speed', type=int, default=0) 162 163 .. code-block:: shell-session 164 165 $ ./helm.py --help # top level help (launch and move) 166 $ ./helm.py launch --help # help for launch options 167 $ ./helm.py launch --missiles # set missiles=True and torpedos=False 168 $ ./helm.py steer --course 180 --speed 5 # set movement parameters 169 170 .. seealso:: 171 172 :pep:`389` - New Command Line Parsing Module 173 PEP written by Steven Bethard. 174 175 :ref:`upgrading-optparse-code` for details on the differences from :mod:`optparse`. 176 177 178 PEP 391: Dictionary Based Configuration for Logging 179 ==================================================== 180 181 The :mod:`logging` module provided two kinds of configuration, one style with 182 function calls for each option or another style driven by an external file saved 183 in a :mod:`ConfigParser` format. Those options did not provide the flexibility 184 to create configurations from JSON or YAML files, nor did they support 185 incremental configuration, which is needed for specifying logger options from a 186 command line. 187 188 To support a more flexible style, the module now offers 189 :func:`logging.config.dictConfig` for specifying logging configuration with 190 plain Python dictionaries. The configuration options include formatters, 191 handlers, filters, and loggers. Here's a working example of a configuration 192 dictionary:: 193 194 {"version": 1, 195 "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"}, 196 "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"} 197 }, 198 "handlers": {"console": { 199 "class": "logging.StreamHandler", 200 "formatter": "brief", 201 "level": "INFO", 202 "stream": "ext://sys.stdout"}, 203 "console_priority": { 204 "class": "logging.StreamHandler", 205 "formatter": "full", 206 "level": "ERROR", 207 "stream": "ext://sys.stderr"} 208 }, 209 "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}} 210 211 212 If that dictionary is stored in a file called :file:`conf.json`, it can be 213 loaded and called with code like this:: 214 215 >>> import json, logging.config 216 >>> with open('conf.json') as f: 217 ... conf = json.load(f) 218 ... 219 >>> logging.config.dictConfig(conf) 220 >>> logging.info("Transaction completed normally") 221 INFO : root : Transaction completed normally 222 >>> logging.critical("Abnormal termination") 223 2011-02-17 11:14:36,694 root CRITICAL Abnormal termination 224 225 .. seealso:: 226 227 :pep:`391` - Dictionary Based Configuration for Logging 228 PEP written by Vinay Sajip. 229 230 231 PEP 3148: The ``concurrent.futures`` module 232 ============================================ 233 234 Code for creating and managing concurrency is being collected in a new top-level 235 namespace, *concurrent*. Its first member is a *futures* package which provides 236 a uniform high-level interface for managing threads and processes. 237 238 The design for :mod:`concurrent.futures` was inspired by the 239 *java.util.concurrent* package. In that model, a running call and its result 240 are represented by a :class:`~concurrent.futures.Future` object that abstracts 241 features common to threads, processes, and remote procedure calls. That object 242 supports status checks (running or done), timeouts, cancellations, adding 243 callbacks, and access to results or exceptions. 244 245 The primary offering of the new module is a pair of executor classes for 246 launching and managing calls. The goal of the executors is to make it easier to 247 use existing tools for making parallel calls. They save the effort needed to 248 setup a pool of resources, launch the calls, create a results queue, add 249 time-out handling, and limit the total number of threads, processes, or remote 250 procedure calls. 251 252 Ideally, each application should share a single executor across multiple 253 components so that process and thread limits can be centrally managed. This 254 solves the design challenge that arises when each component has its own 255 competing strategy for resource management. 256 257 Both classes share a common interface with three methods: 258 :meth:`~concurrent.futures.Executor.submit` for scheduling a callable and 259 returning a :class:`~concurrent.futures.Future` object; 260 :meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls 261 at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing 262 resources. The class is a :term:`context manager` and can be used in a 263 :keyword:`with` statement to assure that resources are automatically released 264 when currently pending futures are done executing. 265 266 A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a 267 launch of four parallel threads for copying files:: 268 269 import concurrent.futures, shutil 270 with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e: 271 e.submit(shutil.copy, 'src1.txt', 'dest1.txt') 272 e.submit(shutil.copy, 'src2.txt', 'dest2.txt') 273 e.submit(shutil.copy, 'src3.txt', 'dest3.txt') 274 e.submit(shutil.copy, 'src3.txt', 'dest4.txt') 275 276 .. seealso:: 277 278 :pep:`3148` - Futures -- Execute Computations Asynchronously 279 PEP written by Brian Quinlan. 280 281 :ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an 282 example using threads to fetch multiple web pages in parallel. 283 284 :ref:`Code for computing prime numbers in 285 parallel<processpoolexecutor-example>`, an example demonstrating 286 :class:`~concurrent.futures.ProcessPoolExecutor`. 287 288 289 PEP 3147: PYC Repository Directories 290 ===================================== 291 292 Python's scheme for caching bytecode in *.pyc* files did not work well in 293 environments with multiple Python interpreters. If one interpreter encountered 294 a cached file created by another interpreter, it would recompile the source and 295 overwrite the cached file, thus losing the benefits of caching. 296 297 The issue of "pyc fights" has become more pronounced as it has become 298 commonplace for Linux distributions to ship with multiple versions of Python. 299 These conflicts also arise with CPython alternatives such as Unladen Swallow. 300 301 To solve this problem, Python's import machinery has been extended to use 302 distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and 303 Unladen Swallow each competing for a file called "mymodule.pyc", they will now 304 look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and 305 "mymodule.unladen10.pyc". And to prevent all of these new files from 306 cluttering source directories, the *pyc* files are now collected in a 307 "__pycache__" directory stored under the package directory. 308 309 Aside from the filenames and target directories, the new scheme has a few 310 aspects that are visible to the programmer: 311 312 * Imported modules now have a :attr:`__cached__` attribute which stores the name 313 of the actual file that was imported: 314 315 >>> import collections 316 >>> collections.__cached__ # doctest: +SKIP 317 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' 318 319 * The tag that is unique to each interpreter is accessible from the :mod:`imp` 320 module: 321 322 >>> import imp 323 >>> imp.get_tag() # doctest: +SKIP 324 'cpython-32' 325 326 * Scripts that try to deduce source filename from the imported file now need to 327 be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc" 328 filename. Instead, use the new functions in the :mod:`imp` module: 329 330 >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') 331 'c:/py32/lib/collections.py' 332 >>> imp.cache_from_source('c:/py32/lib/collections.py') # doctest: +SKIP 333 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' 334 335 * The :mod:`py_compile` and :mod:`compileall` modules have been updated to 336 reflect the new naming convention and target directory. The command-line 337 invocation of *compileall* has new options: ``-i`` for 338 specifying a list of files and directories to compile and ``-b`` which causes 339 bytecode files to be written to their legacy location rather than 340 *__pycache__*. 341 342 * The :mod:`importlib.abc` module has been updated with new :term:`abstract base 343 classes <abstract base class>` for loading bytecode files. The obsolete 344 ABCs, :class:`~importlib.abc.PyLoader` and 345 :class:`~importlib.abc.PyPycLoader`, have been deprecated (instructions on how 346 to stay Python 3.1 compatible are included with the documentation). 347 348 .. seealso:: 349 350 :pep:`3147` - PYC Repository Directories 351 PEP written by Barry Warsaw. 352 353 354 PEP 3149: ABI Version Tagged .so Files 355 ====================================== 356 357 The PYC repository directory allows multiple bytecode cache files to be 358 co-located. This PEP implements a similar mechanism for shared object files by 359 giving them a common directory and distinct names for each version. 360 361 The common directory is "pyshared" and the file names are made distinct by 362 identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the 363 major and minor version numbers, and optional build flags (such as "d" for 364 debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo", 365 you may see these files when the distribution package is installed:: 366 367 /usr/share/pyshared/foo.cpython-32m.so 368 /usr/share/pyshared/foo.cpython-33md.so 369 370 In Python itself, the tags are accessible from functions in the :mod:`sysconfig` 371 module:: 372 373 >>> import sysconfig 374 >>> sysconfig.get_config_var('SOABI') # find the version tag 375 'cpython-32mu' 376 >>> sysconfig.get_config_var('EXT_SUFFIX') # find the full filename extension 377 '.cpython-32mu.so' 378 379 .. seealso:: 380 381 :pep:`3149` - ABI Version Tagged .so Files 382 PEP written by Barry Warsaw. 383 384 385 PEP 3333: Python Web Server Gateway Interface v1.0.1 386 ===================================================== 387 388 This informational PEP clarifies how bytes/text issues are to be handled by the 389 WSGI protocol. The challenge is that string handling in Python 3 is most 390 conveniently handled with the :class:`str` type even though the HTTP protocol 391 is itself bytes oriented. 392 393 The PEP differentiates so-called *native strings* that are used for 394 request/response headers and metadata versus *byte strings* which are used for 395 the bodies of requests and responses. 396 397 The *native strings* are always of type :class:`str` but are restricted to code 398 points between *U+0000* through *U+00FF* which are translatable to bytes using 399 *Latin-1* encoding. These strings are used for the keys and values in the 400 environment dictionary and for response headers and statuses in the 401 :func:`start_response` function. They must follow :rfc:`2616` with respect to 402 encoding. That is, they must either be *ISO-8859-1* characters or use 403 :rfc:`2047` MIME encoding. 404 405 For developers porting WSGI applications from Python 2, here are the salient 406 points: 407 408 * If the app already used strings for headers in Python 2, no change is needed. 409 410 * If instead, the app encoded output headers or decoded input headers, then the 411 headers will need to be re-encoded to Latin-1. For example, an output header 412 encoded in utf-8 was using ``h.encode('utf-8')`` now needs to convert from 413 bytes to native strings using ``h.encode('utf-8').decode('latin-1')``. 414 415 * Values yielded by an application or sent using the :meth:`write` method 416 must be byte strings. The :func:`start_response` function and environ 417 must use native strings. The two cannot be mixed. 418 419 For server implementers writing CGI-to-WSGI pathways or other CGI-style 420 protocols, the users must to be able access the environment using native strings 421 even though the underlying platform may have a different convention. To bridge 422 this gap, the :mod:`wsgiref` module has a new function, 423 :func:`wsgiref.handlers.read_environ` for transcoding CGI variables from 424 :attr:`os.environ` into native strings and returning a new dictionary. 425 426 .. seealso:: 427 428 :pep:`3333` - Python Web Server Gateway Interface v1.0.1 429 PEP written by Phillip Eby. 430 431 432 Other Language Changes 433 ====================== 434 435 Some smaller changes made to the core Python language are: 436 437 * String formatting for :func:`format` and :meth:`str.format` gained new 438 capabilities for the format character **#**. Previously, for integers in 439 binary, octal, or hexadecimal, it caused the output to be prefixed with '0b', 440 '0o', or '0x' respectively. Now it can also handle floats, complex, and 441 Decimal, causing the output to always have a decimal point even when no digits 442 follow it. 443 444 >>> format(20, '#o') 445 '0o24' 446 >>> format(12.34, '#5.0f') 447 ' 12.' 448 449 (Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.) 450 451 * There is also a new :meth:`str.format_map` method that extends the 452 capabilities of the existing :meth:`str.format` method by accepting arbitrary 453 :term:`mapping` objects. This new method makes it possible to use string 454 formatting with any of Python's many dictionary-like objects such as 455 :class:`~collections.defaultdict`, :class:`~shelve.Shelf`, 456 :class:`~configparser.ConfigParser`, or :mod:`dbm`. It is also useful with 457 custom :class:`dict` subclasses that normalize keys before look-up or that 458 supply a :meth:`__missing__` method for unknown keys:: 459 460 >>> import shelve 461 >>> d = shelve.open('tmp.shl') 462 >>> 'The {project_name} status is {status} as of {date}'.format_map(d) 463 'The testing project status is green as of February 15, 2011' 464 465 >>> class LowerCasedDict(dict): 466 ... def __getitem__(self, key): 467 ... return dict.__getitem__(self, key.lower()) 468 >>> lcd = LowerCasedDict(part='widgets', quantity=10) 469 >>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd) 470 'There are 10 widgets in stock' 471 472 >>> class PlaceholderDict(dict): 473 ... def __missing__(self, key): 474 ... return '<{}>'.format(key) 475 >>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict()) 476 'Hello <name>, welcome to <location>' 477 478 (Suggested by Raymond Hettinger and implemented by Eric Smith in 479 :issue:`6081`.) 480 481 * The interpreter can now be started with a quiet option, ``-q``, to prevent 482 the copyright and version information from being displayed in the interactive 483 mode. The option can be introspected using the :attr:`sys.flags` attribute: 484 485 .. code-block:: shell-session 486 487 $ python -q 488 >>> sys.flags 489 sys.flags(debug=0, division_warning=0, inspect=0, interactive=0, 490 optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, 491 ignore_environment=0, verbose=0, bytes_warning=0, quiet=1) 492 493 (Contributed by Marcin Wojdyr in :issue:`1772833`). 494 495 * The :func:`hasattr` function works by calling :func:`getattr` and detecting 496 whether an exception is raised. This technique allows it to detect methods 497 created dynamically by :meth:`__getattr__` or :meth:`__getattribute__` which 498 would otherwise be absent from the class dictionary. Formerly, *hasattr* 499 would catch any exception, possibly masking genuine errors. Now, *hasattr* 500 has been tightened to only catch :exc:`AttributeError` and let other 501 exceptions pass through:: 502 503 >>> class A: 504 ... @property 505 ... def f(self): 506 ... return 1 // 0 507 ... 508 >>> a = A() 509 >>> hasattr(a, 'f') 510 Traceback (most recent call last): 511 ... 512 ZeroDivisionError: integer division or modulo by zero 513 514 (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.) 515 516 * The :func:`str` of a float or complex number is now the same as its 517 :func:`repr`. Previously, the :func:`str` form was shorter but that just 518 caused confusion and is no longer needed now that the shortest possible 519 :func:`repr` is displayed by default: 520 521 >>> import math 522 >>> repr(math.pi) 523 '3.141592653589793' 524 >>> str(math.pi) 525 '3.141592653589793' 526 527 (Proposed and implemented by Mark Dickinson; :issue:`9337`.) 528 529 * :class:`memoryview` objects now have a :meth:`~memoryview.release()` method 530 and they also now support the context management protocol. This allows timely 531 release of any resources that were acquired when requesting a buffer from the 532 original object. 533 534 >>> with memoryview(b'abcdefgh') as v: 535 ... print(v.tolist()) 536 [97, 98, 99, 100, 101, 102, 103, 104] 537 538 (Added by Antoine Pitrou; :issue:`9757`.) 539 540 * Previously it was illegal to delete a name from the local namespace if it 541 occurs as a free variable in a nested block:: 542 543 def outer(x): 544 def inner(): 545 return x 546 inner() 547 del x 548 549 This is now allowed. Remember that the target of an :keyword:`except` clause 550 is cleared, so this code which used to work with Python 2.6, raised a 551 :exc:`SyntaxError` with Python 3.1 and now works again:: 552 553 def f(): 554 def print_error(): 555 print(e) 556 try: 557 something 558 except Exception as e: 559 print_error() 560 # implicit "del e" here 561 562 (See :issue:`4617`.) 563 564 * The internal :c:type:`structsequence` tool now creates subclasses of tuple. 565 This means that C structures like those returned by :func:`os.stat`, 566 :func:`time.gmtime`, and :attr:`sys.version_info` now work like a 567 :term:`named tuple` and now work with functions and methods that 568 expect a tuple as an argument. This is a big step forward in making the C 569 structures as flexible as their pure Python counterparts: 570 571 >>> import sys 572 >>> isinstance(sys.version_info, tuple) 573 True 574 >>> 'Version %d.%d.%d %s(%d)' % sys.version_info # doctest: +SKIP 575 'Version 3.2.0 final(0)' 576 577 (Suggested by Arfrever Frehtes Taifersar Arahesis and implemented 578 by Benjamin Peterson in :issue:`8413`.) 579 580 * Warnings are now easier to control using the :envvar:`PYTHONWARNINGS` 581 environment variable as an alternative to using ``-W`` at the command line: 582 583 .. code-block:: shell-session 584 585 $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::' 586 587 (Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.) 588 589 * A new warning category, :exc:`ResourceWarning`, has been added. It is 590 emitted when potential issues with resource consumption or cleanup 591 are detected. It is silenced by default in normal release builds but 592 can be enabled through the means provided by the :mod:`warnings` 593 module, or on the command line. 594 595 A :exc:`ResourceWarning` is issued at interpreter shutdown if the 596 :data:`gc.garbage` list isn't empty, and if :attr:`gc.DEBUG_UNCOLLECTABLE` is 597 set, all uncollectable objects are printed. This is meant to make the 598 programmer aware that their code contains object finalization issues. 599 600 A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed 601 without having been explicitly closed. While the deallocator for such 602 object ensures it closes the underlying operating system resource 603 (usually, a file descriptor), the delay in deallocating the object could 604 produce various issues, especially under Windows. Here is an example 605 of enabling the warning from the command line: 606 607 .. code-block:: shell-session 608 609 $ python -q -Wdefault 610 >>> f = open("foo", "wb") 611 >>> del f 612 __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'> 613 614 (Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.) 615 616 * :class:`range` objects now support *index* and *count* methods. This is part 617 of an effort to make more objects fully implement the 618 :class:`collections.Sequence` :term:`abstract base class`. As a result, the 619 language will have a more uniform API. In addition, :class:`range` objects 620 now support slicing and negative indices, even with values larger than 621 :attr:`sys.maxsize`. This makes *range* more interoperable with lists:: 622 623 >>> range(0, 100, 2).count(10) 624 1 625 >>> range(0, 100, 2).index(10) 626 5 627 >>> range(0, 100, 2)[5] 628 10 629 >>> range(0, 100, 2)[0:5] 630 range(0, 10, 2) 631 632 (Contributed by Daniel Stutzbach in :issue:`9213`, by Alexander Belopolsky 633 in :issue:`2690`, and by Nick Coghlan in :issue:`10889`.) 634 635 * The :func:`callable` builtin function from Py2.x was resurrected. It provides 636 a concise, readable alternative to using an :term:`abstract base class` in an 637 expression like ``isinstance(x, collections.Callable)``: 638 639 >>> callable(max) 640 True 641 >>> callable(20) 642 False 643 644 (See :issue:`10518`.) 645 646 * Python's import mechanism can now load modules installed in directories with 647 non-ASCII characters in the path name. This solved an aggravating problem 648 with home directories for users with non-ASCII characters in their usernames. 649 650 (Required extensive work by Victor Stinner in :issue:`9425`.) 651 652 653 New, Improved, and Deprecated Modules 654 ===================================== 655 656 Python's standard library has undergone significant maintenance efforts and 657 quality improvements. 658 659 The biggest news for Python 3.2 is that the :mod:`email` package, :mod:`mailbox` 660 module, and :mod:`nntplib` modules now work correctly with the bytes/text model 661 in Python 3. For the first time, there is correct handling of messages with 662 mixed encodings. 663 664 Throughout the standard library, there has been more careful attention to 665 encodings and text versus bytes issues. In particular, interactions with the 666 operating system are now better able to exchange non-ASCII data using the 667 Windows MBCS encoding, locale-aware encodings, or UTF-8. 668 669 Another significant win is the addition of substantially better support for 670 *SSL* connections and security certificates. 671 672 In addition, more classes now implement a :term:`context manager` to support 673 convenient and reliable resource clean-up using a :keyword:`with` statement. 674 675 email 676 ----- 677 678 The usability of the :mod:`email` package in Python 3 has been mostly fixed by 679 the extensive efforts of R. David Murray. The problem was that emails are 680 typically read and stored in the form of :class:`bytes` rather than :class:`str` 681 text, and they may contain multiple encodings within a single email. So, the 682 email package had to be extended to parse and generate email messages in bytes 683 format. 684 685 * New functions :func:`~email.message_from_bytes` and 686 :func:`~email.message_from_binary_file`, and new classes 687 :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser` 688 allow binary message data to be parsed into model objects. 689 690 * Given bytes input to the model, :meth:`~email.message.Message.get_payload` 691 will by default decode a message body that has a 692 :mailheader:`Content-Transfer-Encoding` of *8bit* using the charset 693 specified in the MIME headers and return the resulting string. 694 695 * Given bytes input to the model, :class:`~email.generator.Generator` will 696 convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of 697 *8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`. 698 699 Headers with unencoded non-ASCII bytes are deemed to be :rfc:`2047`\ -encoded 700 using the *unknown-8bit* character set. 701 702 * A new class :class:`~email.generator.BytesGenerator` produces bytes as output, 703 preserving any unchanged non-ASCII data that was present in the input used to 704 build the model, including message bodies with a 705 :mailheader:`Content-Transfer-Encoding` of *8bit*. 706 707 * The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string 708 for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method, 709 and a new method, :meth:`~smtplib.SMTP.send_message` accepts a 710 :class:`~email.message.Message` object and can optionally obtain the 711 *from_addr* and *to_addrs* addresses directly from the object. 712 713 (Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.) 714 715 elementtree 716 ----------- 717 718 The :mod:`xml.etree.ElementTree` package and its :mod:`xml.etree.cElementTree` 719 counterpart have been updated to version 1.3. 720 721 Several new and useful functions and methods have been added: 722 723 * :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document 724 from a sequence of fragments 725 * :func:`xml.etree.ElementTree.register_namespace` for registering a global 726 namespace prefix 727 * :func:`xml.etree.ElementTree.tostringlist` for string representation 728 including all sublists 729 * :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero 730 or more elements 731 * :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and 732 subelements 733 * :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over 734 an element and its subelements 735 * :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element 736 * :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype 737 declaration 738 739 Two methods have been deprecated: 740 741 * :meth:`xml.etree.ElementTree.getchildren` use ``list(elem)`` instead. 742 * :meth:`xml.etree.ElementTree.getiterator` use ``Element.iter`` instead. 743 744 For details of the update, see `Introducing ElementTree 745 <http://effbot.org/zone/elementtree-13-intro.htm>`_ on Fredrik Lundh's website. 746 747 (Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.) 748 749 functools 750 --------- 751 752 * The :mod:`functools` module includes a new decorator for caching function 753 calls. :func:`functools.lru_cache` can save repeated queries to an external 754 resource whenever the results are expected to be the same. 755 756 For example, adding a caching decorator to a database query function can save 757 database accesses for popular searches: 758 759 >>> import functools 760 >>> @functools.lru_cache(maxsize=300) 761 ... def get_phone_number(name): 762 ... c = conn.cursor() 763 ... c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,)) 764 ... return c.fetchone()[0] 765 766 >>> for name in user_requests: # doctest: +SKIP 767 ... get_phone_number(name) # cached lookup 768 769 To help with choosing an effective cache size, the wrapped function is 770 instrumented for tracking cache statistics: 771 772 >>> get_phone_number.cache_info() # doctest: +SKIP 773 CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300) 774 775 If the phonelist table gets updated, the outdated contents of the cache can be 776 cleared with: 777 778 >>> get_phone_number.cache_clear() 779 780 (Contributed by Raymond Hettinger and incorporating design ideas from Jim 781 Baker, Miki Tebeka, and Nick Coghlan; see `recipe 498245 782 <https://code.activestate.com/recipes/498245>`_\, `recipe 577479 783 <https://code.activestate.com/recipes/577479>`_\, :issue:`10586`, and 784 :issue:`10593`.) 785 786 * The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute 787 pointing to the original callable function. This allows wrapped functions to 788 be introspected. It also copies :attr:`__annotations__` if defined. And now 789 it also gracefully skips over missing attributes such as :attr:`__doc__` which 790 might not be defined for the wrapped callable. 791 792 In the above example, the cache can be removed by recovering the original 793 function: 794 795 >>> get_phone_number = get_phone_number.__wrapped__ # uncached function 796 797 (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and 798 :issue:`8814`.) 799 800 * To help write classes with rich comparison methods, a new decorator 801 :func:`functools.total_ordering` will use existing equality and inequality 802 methods to fill in the remaining methods. 803 804 For example, supplying *__eq__* and *__lt__* will enable 805 :func:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*:: 806 807 @total_ordering 808 class Student: 809 def __eq__(self, other): 810 return ((self.lastname.lower(), self.firstname.lower()) == 811 (other.lastname.lower(), other.firstname.lower())) 812 813 def __lt__(self, other): 814 return ((self.lastname.lower(), self.firstname.lower()) < 815 (other.lastname.lower(), other.firstname.lower())) 816 817 With the *total_ordering* decorator, the remaining comparison methods 818 are filled in automatically. 819 820 (Contributed by Raymond Hettinger.) 821 822 * To aid in porting programs from Python 2, the :func:`functools.cmp_to_key` 823 function converts an old-style comparison function to 824 modern :term:`key function`: 825 826 >>> # locale-aware sort order 827 >>> sorted(iterable, key=cmp_to_key(locale.strcoll)) # doctest: +SKIP 828 829 For sorting examples and a brief sorting tutorial, see the `Sorting HowTo 830 <https://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial. 831 832 (Contributed by Raymond Hettinger.) 833 834 itertools 835 --------- 836 837 * The :mod:`itertools` module has a new :func:`~itertools.accumulate` function 838 modeled on APL's *scan* operator and Numpy's *accumulate* function: 839 840 >>> from itertools import accumulate 841 >>> list(accumulate([8, 2, 50])) 842 [8, 10, 60] 843 844 >>> prob_dist = [0.1, 0.4, 0.2, 0.3] 845 >>> list(accumulate(prob_dist)) # cumulative probability distribution 846 [0.1, 0.5, 0.7, 1.0] 847 848 For an example using :func:`~itertools.accumulate`, see the :ref:`examples for 849 the random module <random-examples>`. 850 851 (Contributed by Raymond Hettinger and incorporating design suggestions 852 from Mark Dickinson.) 853 854 collections 855 ----------- 856 857 * The :class:`collections.Counter` class now has two forms of in-place 858 subtraction, the existing *-=* operator for `saturating subtraction 859 <https://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new 860 :meth:`~collections.Counter.subtract` method for regular subtraction. The 861 former is suitable for `multisets <https://en.wikipedia.org/wiki/Multiset>`_ 862 which only have positive counts, and the latter is more suitable for use cases 863 that allow negative counts: 864 865 >>> from collections import Counter 866 >>> tally = Counter(dogs=5, cats=3) 867 >>> tally -= Counter(dogs=2, cats=8) # saturating subtraction 868 >>> tally 869 Counter({'dogs': 3}) 870 871 >>> tally = Counter(dogs=5, cats=3) 872 >>> tally.subtract(dogs=2, cats=8) # regular subtraction 873 >>> tally 874 Counter({'dogs': 3, 'cats': -5}) 875 876 (Contributed by Raymond Hettinger.) 877 878 * The :class:`collections.OrderedDict` class has a new method 879 :meth:`~collections.OrderedDict.move_to_end` which takes an existing key and 880 moves it to either the first or last position in the ordered sequence. 881 882 The default is to move an item to the last position. This is equivalent of 883 renewing an entry with ``od[k] = od.pop(k)``. 884 885 A fast move-to-end operation is useful for resequencing entries. For example, 886 an ordered dictionary can be used to track order of access by aging entries 887 from the oldest to the most recently accessed. 888 889 >>> from collections import OrderedDict 890 >>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e']) 891 >>> list(d) 892 ['a', 'b', 'X', 'd', 'e'] 893 >>> d.move_to_end('X') 894 >>> list(d) 895 ['a', 'b', 'd', 'e', 'X'] 896 897 (Contributed by Raymond Hettinger.) 898 899 * The :class:`collections.deque` class grew two new methods 900 :meth:`~collections.deque.count` and :meth:`~collections.deque.reverse` that 901 make them more substitutable for :class:`list` objects: 902 903 >>> from collections import deque 904 >>> d = deque('simsalabim') 905 >>> d.count('s') 906 2 907 >>> d.reverse() 908 >>> d 909 deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's']) 910 911 (Contributed by Raymond Hettinger.) 912 913 threading 914 --------- 915 916 The :mod:`threading` module has a new :class:`~threading.Barrier` 917 synchronization class for making multiple threads wait until all of them have 918 reached a common barrier point. Barriers are useful for making sure that a task 919 with multiple preconditions does not run until all of the predecessor tasks are 920 complete. 921 922 Barriers can work with an arbitrary number of threads. This is a generalization 923 of a `Rendezvous <https://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which 924 is defined for only two threads. 925 926 Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects 927 are suitable for use in loops. The separate *filling* and *draining* phases 928 assure that all threads get released (drained) before any one of them can loop 929 back and re-enter the barrier. The barrier fully resets after each cycle. 930 931 Example of using barriers:: 932 933 from threading import Barrier, Thread 934 935 def get_votes(site): 936 ballots = conduct_election(site) 937 all_polls_closed.wait() # do not count until all polls are closed 938 totals = summarize(ballots) 939 publish(site, totals) 940 941 all_polls_closed = Barrier(len(sites)) 942 for site in sites: 943 Thread(target=get_votes, args=(site,)).start() 944 945 In this example, the barrier enforces a rule that votes cannot be counted at any 946 polling site until all polls are closed. Notice how a solution with a barrier 947 is similar to one with :meth:`threading.Thread.join`, but the threads stay alive 948 and continue to do work (summarizing ballots) after the barrier point is 949 crossed. 950 951 If any of the predecessor tasks can hang or be delayed, a barrier can be created 952 with an optional *timeout* parameter. Then if the timeout period elapses before 953 all the predecessor tasks reach the barrier point, all waiting threads are 954 released and a :exc:`~threading.BrokenBarrierError` exception is raised:: 955 956 def get_votes(site): 957 ballots = conduct_election(site) 958 try: 959 all_polls_closed.wait(timeout=midnight - time.now()) 960 except BrokenBarrierError: 961 lockbox = seal_ballots(ballots) 962 queue.put(lockbox) 963 else: 964 totals = summarize(ballots) 965 publish(site, totals) 966 967 In this example, the barrier enforces a more robust rule. If some election 968 sites do not finish before midnight, the barrier times-out and the ballots are 969 sealed and deposited in a queue for later handling. 970 971 See `Barrier Synchronization Patterns 972 <https://parlab.eecs.berkeley.edu/wiki/_media/patterns/paraplop_g1_3.pdf>`_ for 973 more examples of how barriers can be used in parallel computing. Also, there is 974 a simple but thorough explanation of barriers in `The Little Book of Semaphores 975 <http://greenteapress.com/semaphores/downey08semaphores.pdf>`_, *section 3.6*. 976 977 (Contributed by Kristjn Valur Jnsson with an API review by Jeffrey Yasskin in 978 :issue:`8777`.) 979 980 datetime and time 981 ----------------- 982 983 * The :mod:`datetime` module has a new type :class:`~datetime.timezone` that 984 implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC 985 offset and timezone name. This makes it easier to create timezone-aware 986 datetime objects:: 987 988 >>> from datetime import datetime, timezone 989 990 >>> datetime.now(timezone.utc) 991 datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc) 992 993 >>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z") 994 datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc) 995 996 * Also, :class:`~datetime.timedelta` objects can now be multiplied by 997 :class:`float` and divided by :class:`float` and :class:`int` objects. 998 And :class:`~datetime.timedelta` objects can now divide one another. 999 1000 * The :meth:`datetime.date.strftime` method is no longer restricted to years 1001 after 1900. The new supported year range is from 1000 to 9999 inclusive. 1002 1003 * Whenever a two-digit year is used in a time tuple, the interpretation has been 1004 governed by :attr:`time.accept2dyear`. The default is ``True`` which means that 1005 for a two-digit year, the century is guessed according to the POSIX rules 1006 governing the ``%y`` strptime format. 1007 1008 Starting with Py3.2, use of the century guessing heuristic will emit a 1009 :exc:`DeprecationWarning`. Instead, it is recommended that 1010 :attr:`time.accept2dyear` be set to ``False`` so that large date ranges 1011 can be used without guesswork:: 1012 1013 >>> import time, warnings 1014 >>> warnings.resetwarnings() # remove the default warning filters 1015 1016 >>> time.accept2dyear = True # guess whether 11 means 11 or 2011 1017 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) 1018 Warning (from warnings module): 1019 ... 1020 DeprecationWarning: Century info guessed for a 2-digit year. 1021 'Fri Jan 1 12:34:56 2011' 1022 1023 >>> time.accept2dyear = False # use the full range of allowable dates 1024 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) 1025 'Fri Jan 1 12:34:56 11' 1026 1027 Several functions now have significantly expanded date ranges. When 1028 :attr:`time.accept2dyear` is false, the :func:`time.asctime` function will 1029 accept any year that fits in a C int, while the :func:`time.mktime` and 1030 :func:`time.strftime` functions will accept the full range supported by the 1031 corresponding operating system functions. 1032 1033 (Contributed by Alexander Belopolsky and Victor Stinner in :issue:`1289118`, 1034 :issue:`5094`, :issue:`6641`, :issue:`2706`, :issue:`1777412`, :issue:`8013`, 1035 and :issue:`10827`.) 1036 1037 .. XXX https://bugs.python.org/issue?%40search_text=datetime&%40sort=-activity 1038 1039 math 1040 ---- 1041 1042 The :mod:`math` module has been updated with six new functions inspired by the 1043 C99 standard. 1044 1045 The :func:`~math.isfinite` function provides a reliable and fast way to detect 1046 special values. It returns ``True`` for regular numbers and ``False`` for *Nan* or 1047 *Infinity*: 1048 1049 >>> from math import isfinite 1050 >>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))] 1051 [True, True, False, False] 1052 1053 The :func:`~math.expm1` function computes ``e**x-1`` for small values of *x* 1054 without incurring the loss of precision that usually accompanies the subtraction 1055 of nearly equal quantities: 1056 1057 >>> from math import expm1 1058 >>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x 1059 0.013765762467652909 1060 1061 The :func:`~math.erf` function computes a probability integral or `Gaussian 1062 error function <https://en.wikipedia.org/wiki/Error_function>`_. The 1063 complementary error function, :func:`~math.erfc`, is ``1 - erf(x)``: 1064 1065 .. doctest:: 1066 :options: +SKIP 1067 1068 >>> from math import erf, erfc, sqrt 1069 >>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation 1070 0.682689492137086 1071 >>> erfc(1.0/sqrt(2.0)) # portion of normal distribution outside 1 standard deviation 1072 0.31731050786291404 1073 >>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0)) 1074 1.0 1075 1076 The :func:`~math.gamma` function is a continuous extension of the factorial 1077 function. See https://en.wikipedia.org/wiki/Gamma_function for details. Because 1078 the function is related to factorials, it grows large even for small values of 1079 *x*, so there is also a :func:`~math.lgamma` function for computing the natural 1080 logarithm of the gamma function: 1081 1082 >>> from math import gamma, lgamma 1083 >>> gamma(7.0) # six factorial 1084 720.0 1085 >>> lgamma(801.0) # log(800 factorial) 1086 4551.950730698041 1087 1088 (Contributed by Mark Dickinson.) 1089 1090 abc 1091 --- 1092 1093 The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and 1094 :func:`~abc.abstractstaticmethod`. 1095 1096 These tools make it possible to define an :term:`abstract base class` that 1097 requires a particular :func:`classmethod` or :func:`staticmethod` to be 1098 implemented:: 1099 1100 class Temperature(metaclass=abc.ABCMeta): 1101 @abc.abstractclassmethod 1102 def from_fahrenheit(cls, t): 1103 ... 1104 @abc.abstractclassmethod 1105 def from_celsius(cls, t): 1106 ... 1107 1108 (Patch submitted by Daniel Urban; :issue:`5867`.) 1109 1110 io 1111 -- 1112 1113 The :class:`io.BytesIO` has a new method, :meth:`~io.BytesIO.getbuffer`, which 1114 provides functionality similar to :func:`memoryview`. It creates an editable 1115 view of the data without making a copy. The buffer's random access and support 1116 for slice notation are well-suited to in-place editing:: 1117 1118 >>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11 1119 1120 >>> def change_location(buffer, record_number, location): 1121 ... start = record_number * REC_LEN + LOC_START 1122 ... buffer[start: start+LOC_LEN] = location 1123 1124 >>> import io 1125 1126 >>> byte_stream = io.BytesIO( 1127 ... b'G3805 storeroom Main chassis ' 1128 ... b'X7899 shipping Reserve cog ' 1129 ... b'L6988 receiving Primary sprocket' 1130 ... ) 1131 >>> buffer = byte_stream.getbuffer() 1132 >>> change_location(buffer, 1, b'warehouse ') 1133 >>> change_location(buffer, 0, b'showroom ') 1134 >>> print(byte_stream.getvalue()) 1135 b'G3805 showroom Main chassis ' 1136 b'X7899 warehouse Reserve cog ' 1137 b'L6988 receiving Primary sprocket' 1138 1139 (Contributed by Antoine Pitrou in :issue:`5506`.) 1140 1141 reprlib 1142 ------- 1143 1144 When writing a :meth:`__repr__` method for a custom container, it is easy to 1145 forget to handle the case where a member refers back to the container itself. 1146 Python's builtin objects such as :class:`list` and :class:`set` handle 1147 self-reference by displaying "..." in the recursive part of the representation 1148 string. 1149 1150 To help write such :meth:`__repr__` methods, the :mod:`reprlib` module has a new 1151 decorator, :func:`~reprlib.recursive_repr`, for detecting recursive calls to 1152 :meth:`__repr__` and substituting a placeholder string instead:: 1153 1154 >>> class MyList(list): 1155 ... @recursive_repr() 1156 ... def __repr__(self): 1157 ... return '<' + '|'.join(map(repr, self)) + '>' 1158 ... 1159 >>> m = MyList('abc') 1160 >>> m.append(m) 1161 >>> m.append('x') 1162 >>> print(m) 1163 <'a'|'b'|'c'|...|'x'> 1164 1165 (Contributed by Raymond Hettinger in :issue:`9826` and :issue:`9840`.) 1166 1167 logging 1168 ------- 1169 1170 In addition to dictionary-based configuration described above, the 1171 :mod:`logging` package has many other improvements. 1172 1173 The logging documentation has been augmented by a :ref:`basic tutorial 1174 <logging-basic-tutorial>`\, an :ref:`advanced tutorial 1175 <logging-advanced-tutorial>`\, and a :ref:`cookbook <logging-cookbook>` of 1176 logging recipes. These documents are the fastest way to learn about logging. 1177 1178 The :func:`logging.basicConfig` set-up function gained a *style* argument to 1179 support three different types of string formatting. It defaults to "%" for 1180 traditional %-formatting, can be set to "{" for the new :meth:`str.format` style, or 1181 can be set to "$" for the shell-style formatting provided by 1182 :class:`string.Template`. The following three configurations are equivalent:: 1183 1184 >>> from logging import basicConfig 1185 >>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s") 1186 >>> basicConfig(style='{', format="{name} -> {levelname} {message}") 1187 >>> basicConfig(style='$', format="$name -> $levelname: $message") 1188 1189 If no configuration is set-up before a logging event occurs, there is now a 1190 default configuration using a :class:`~logging.StreamHandler` directed to 1191 :attr:`sys.stderr` for events of ``WARNING`` level or higher. Formerly, an 1192 event occurring before a configuration was set-up would either raise an 1193 exception or silently drop the event depending on the value of 1194 :attr:`logging.raiseExceptions`. The new default handler is stored in 1195 :attr:`logging.lastResort`. 1196 1197 The use of filters has been simplified. Instead of creating a 1198 :class:`~logging.Filter` object, the predicate can be any Python callable that 1199 returns ``True`` or ``False``. 1200 1201 There were a number of other improvements that add flexibility and simplify 1202 configuration. See the module documentation for a full listing of changes in 1203 Python 3.2. 1204 1205 csv 1206 --- 1207 1208 The :mod:`csv` module now supports a new dialect, :class:`~csv.unix_dialect`, 1209 which applies quoting for all fields and a traditional Unix style with ``'\n'`` as 1210 the line terminator. The registered dialect name is ``unix``. 1211 1212 The :class:`csv.DictWriter` has a new method, 1213 :meth:`~csv.DictWriter.writeheader` for writing-out an initial row to document 1214 the field names:: 1215 1216 >>> import csv, sys 1217 >>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix') 1218 >>> w.writeheader() 1219 "name","dept" 1220 >>> w.writerows([ 1221 ... {'name': 'tom', 'dept': 'accounting'}, 1222 ... {'name': 'susan', 'dept': 'Salesl'}]) 1223 "tom","accounting" 1224 "susan","sales" 1225 1226 (New dialect suggested by Jay Talbot in :issue:`5975`, and the new method 1227 suggested by Ed Abraham in :issue:`1537721`.) 1228 1229 contextlib 1230 ---------- 1231 1232 There is a new and slightly mind-blowing tool 1233 :class:`~contextlib.ContextDecorator` that is helpful for creating a 1234 :term:`context manager` that does double duty as a function decorator. 1235 1236 As a convenience, this new functionality is used by 1237 :func:`~contextlib.contextmanager` so that no extra effort is needed to support 1238 both roles. 1239 1240 The basic idea is that both context managers and function decorators can be used 1241 for pre-action and post-action wrappers. Context managers wrap a group of 1242 statements using a :keyword:`with` statement, and function decorators wrap a 1243 group of statements enclosed in a function. So, occasionally there is a need to 1244 write a pre-action or post-action wrapper that can be used in either role. 1245 1246 For example, it is sometimes useful to wrap functions or groups of statements 1247 with a logger that can track the time of entry and time of exit. Rather than 1248 writing both a function decorator and a context manager for the task, the 1249 :func:`~contextlib.contextmanager` provides both capabilities in a single 1250 definition:: 1251 1252 from contextlib import contextmanager 1253 import logging 1254 1255 logging.basicConfig(level=logging.INFO) 1256 1257 @contextmanager 1258 def track_entry_and_exit(name): 1259 logging.info('Entering: %s', name) 1260 yield 1261 logging.info('Exiting: %s', name) 1262 1263 Formerly, this would have only been usable as a context manager:: 1264 1265 with track_entry_and_exit('widget loader'): 1266 print('Some time consuming activity goes here') 1267 load_widget() 1268 1269 Now, it can be used as a decorator as well:: 1270 1271 @track_entry_and_exit('widget loader') 1272 def activity(): 1273 print('Some time consuming activity goes here') 1274 load_widget() 1275 1276 Trying to fulfill two roles at once places some limitations on the technique. 1277 Context managers normally have the flexibility to return an argument usable by 1278 a :keyword:`with` statement, but there is no parallel for function decorators. 1279 1280 In the above example, there is not a clean way for the *track_entry_and_exit* 1281 context manager to return a logging instance for use in the body of enclosed 1282 statements. 1283 1284 (Contributed by Michael Foord in :issue:`9110`.) 1285 1286 decimal and fractions 1287 --------------------- 1288 1289 Mark Dickinson crafted an elegant and efficient scheme for assuring that 1290 different numeric datatypes will have the same hash value whenever their actual 1291 values are equal (:issue:`8188`):: 1292 1293 assert hash(Fraction(3, 2)) == hash(1.5) == \ 1294 hash(Decimal("1.5")) == hash(complex(1.5, 0)) 1295 1296 Some of the hashing details are exposed through a new attribute, 1297 :attr:`sys.hash_info`, which describes the bit width of the hash value, the 1298 prime modulus, the hash values for *infinity* and *nan*, and the multiplier 1299 used for the imaginary part of a number: 1300 1301 >>> sys.hash_info # doctest: +SKIP 1302 sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003) 1303 1304 An early decision to limit the inter-operability of various numeric types has 1305 been relaxed. It is still unsupported (and ill-advised) to have implicit 1306 mixing in arithmetic expressions such as ``Decimal('1.1') + float('1.1')`` 1307 because the latter loses information in the process of constructing the binary 1308 float. However, since existing floating point value can be converted losslessly 1309 to either a decimal or rational representation, it makes sense to add them to 1310 the constructor and to support mixed-type comparisons. 1311 1312 * The :class:`decimal.Decimal` constructor now accepts :class:`float` objects 1313 directly so there in no longer a need to use the :meth:`~decimal.Decimal.from_float` 1314 method (:issue:`8257`). 1315 1316 * Mixed type comparisons are now fully supported so that 1317 :class:`~decimal.Decimal` objects can be directly compared with :class:`float` 1318 and :class:`fractions.Fraction` (:issue:`2531` and :issue:`8188`). 1319 1320 Similar changes were made to :class:`fractions.Fraction` so that the 1321 :meth:`~fractions.Fraction.from_float()` and :meth:`~fractions.Fraction.from_decimal` 1322 methods are no longer needed (:issue:`8294`): 1323 1324 >>> from decimal import Decimal 1325 >>> from fractions import Fraction 1326 >>> Decimal(1.1) 1327 Decimal('1.100000000000000088817841970012523233890533447265625') 1328 >>> Fraction(1.1) 1329 Fraction(2476979795053773, 2251799813685248) 1330 1331 Another useful change for the :mod:`decimal` module is that the 1332 :attr:`Context.clamp` attribute is now public. This is useful in creating 1333 contexts that correspond to the decimal interchange formats specified in IEEE 1334 754 (see :issue:`8540`). 1335 1336 (Contributed by Mark Dickinson and Raymond Hettinger.) 1337 1338 ftp 1339 --- 1340 1341 The :class:`ftplib.FTP` class now supports the context management protocol to 1342 unconditionally consume :exc:`socket.error` exceptions and to close the FTP 1343 connection when done:: 1344 1345 >>> from ftplib import FTP 1346 >>> with FTP("ftp1.at.proftpd.org") as ftp: 1347 ftp.login() 1348 ftp.dir() 1349 1350 '230 Anonymous login ok, restrictions apply.' 1351 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 . 1352 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .. 1353 dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS 1354 dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora 1355 1356 Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input` 1357 also grew auto-closing context managers:: 1358 1359 with fileinput.input(files=('log1.txt', 'log2.txt')) as f: 1360 for line in f: 1361 process(line) 1362 1363 (Contributed by Tarek Ziad and Giampaolo Rodol in :issue:`4972`, and 1364 by Georg Brandl in :issue:`8046` and :issue:`1286`.) 1365 1366 The :class:`~ftplib.FTP_TLS` class now accepts a *context* parameter, which is a 1367 :class:`ssl.SSLContext` object allowing bundling SSL configuration options, 1368 certificates and private keys into a single (potentially long-lived) structure. 1369 1370 (Contributed by Giampaolo Rodol; :issue:`8806`.) 1371 1372 popen 1373 ----- 1374 1375 The :func:`os.popen` and :func:`subprocess.Popen` functions now support 1376 :keyword:`with` statements for auto-closing of the file descriptors. 1377 1378 (Contributed by Antoine Pitrou and Brian Curtin in :issue:`7461` and 1379 :issue:`10554`.) 1380 1381 select 1382 ------ 1383 1384 The :mod:`select` module now exposes a new, constant attribute, 1385 :attr:`~select.PIPE_BUF`, which gives the minimum number of bytes which are 1386 guaranteed not to block when :func:`select.select` says a pipe is ready 1387 for writing. 1388 1389 >>> import select 1390 >>> select.PIPE_BUF # doctest: +SKIP 1391 512 1392 1393 (Available on Unix systems. Patch by Sbastien Sabl in :issue:`9862`) 1394 1395 gzip and zipfile 1396 ---------------- 1397 1398 :class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase` 1399 :term:`abstract base class` (except for ``truncate()``). It also has a 1400 :meth:`~gzip.GzipFile.peek` method and supports unseekable as well as 1401 zero-padded file objects. 1402 1403 The :mod:`gzip` module also gains the :func:`~gzip.compress` and 1404 :func:`~gzip.decompress` functions for easier in-memory compression and 1405 decompression. Keep in mind that text needs to be encoded as :class:`bytes` 1406 before compressing and decompressing: 1407 1408 >>> import gzip 1409 >>> s = 'Three shall be the number thou shalt count, ' 1410 >>> s += 'and the number of the counting shall be three' 1411 >>> b = s.encode() # convert to utf-8 1412 >>> len(b) 1413 89 1414 >>> c = gzip.compress(b) 1415 >>> len(c) 1416 77 1417 >>> gzip.decompress(c).decode()[:42] # decompress and convert to text 1418 'Three shall be the number thou shalt count' 1419 1420 (Contributed by Anand B. Pillai in :issue:`3488`; and by Antoine Pitrou, Nir 1421 Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and 1422 :issue:`2846`.) 1423 1424 Also, the :class:`zipfile.ZipExtFile` class was reworked internally to represent 1425 files stored inside an archive. The new implementation is significantly faster 1426 and can be wrapped in an :class:`io.BufferedReader` object for more speedups. It 1427 also solves an issue where interleaved calls to *read* and *readline* gave the 1428 wrong results. 1429 1430 (Patch submitted by Nir Aides in :issue:`7610`.) 1431 1432 tarfile 1433 ------- 1434 1435 The :class:`~tarfile.TarFile` class can now be used as a context manager. In 1436 addition, its :meth:`~tarfile.TarFile.add` method has a new option, *filter*, 1437 that controls which files are added to the archive and allows the file metadata 1438 to be edited. 1439 1440 The new *filter* option replaces the older, less flexible *exclude* parameter 1441 which is now deprecated. If specified, the optional *filter* parameter needs to 1442 be a :term:`keyword argument`. The user-supplied filter function accepts a 1443 :class:`~tarfile.TarInfo` object and returns an updated 1444 :class:`~tarfile.TarInfo` object, or if it wants the file to be excluded, the 1445 function can return ``None``:: 1446 1447 >>> import tarfile, glob 1448 1449 >>> def myfilter(tarinfo): 1450 ... if tarinfo.isfile(): # only save real files 1451 ... tarinfo.uname = 'monty' # redact the user name 1452 ... return tarinfo 1453 1454 >>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf: 1455 ... for filename in glob.glob('*.txt'): 1456 ... tf.add(filename, filter=myfilter) 1457 ... tf.list() 1458 -rw-r--r-- monty/501 902 2011-01-26 17:59:11 annotations.txt 1459 -rw-r--r-- monty/501 123 2011-01-26 17:59:11 general_questions.txt 1460 -rw-r--r-- monty/501 3514 2011-01-26 17:59:11 prion.txt 1461 -rw-r--r-- monty/501 124 2011-01-26 17:59:11 py_todo.txt 1462 -rw-r--r-- monty/501 1399 2011-01-26 17:59:11 semaphore_notes.txt 1463 1464 (Proposed by Tarek Ziad and implemented by Lars Gustbel in :issue:`6856`.) 1465 1466 hashlib 1467 ------- 1468 1469 The :mod:`hashlib` module has two new constant attributes listing the hashing 1470 algorithms guaranteed to be present in all implementations and those available 1471 on the current implementation:: 1472 1473 >>> import hashlib 1474 1475 >>> hashlib.algorithms_guaranteed 1476 {'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'} 1477 1478 >>> hashlib.algorithms_available 1479 {'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256', 1480 'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2', 1481 'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224', 1482 'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'} 1483 1484 (Suggested by Carl Chenet in :issue:`7418`.) 1485 1486 ast 1487 --- 1488 1489 The :mod:`ast` module has a wonderful a general-purpose tool for safely 1490 evaluating expression strings using the Python literal 1491 syntax. The :func:`ast.literal_eval` function serves as a secure alternative to 1492 the builtin :func:`eval` function which is easily abused. Python 3.2 adds 1493 :class:`bytes` and :class:`set` literals to the list of supported types: 1494 strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and ``None``. 1495 1496 :: 1497 1498 >>> from ast import literal_eval 1499 1500 >>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}" 1501 >>> literal_eval(request) 1502 {'args': (2, 0.5), 'req': 3, 'func': 'pow'} 1503 1504 >>> request = "os.system('do something harmful')" 1505 >>> literal_eval(request) 1506 Traceback (most recent call last): 1507 ... 1508 ValueError: malformed node or string: <_ast.Call object at 0x101739a10> 1509 1510 (Implemented by Benjamin Peterson and Georg Brandl.) 1511 1512 os 1513 -- 1514 1515 Different operating systems use various encodings for filenames and environment 1516 variables. The :mod:`os` module provides two new functions, 1517 :func:`~os.fsencode` and :func:`~os.fsdecode`, for encoding and decoding 1518 filenames: 1519 1520 >>> import os 1521 >>> filename = 'Sehenswrdigkeiten' 1522 >>> os.fsencode(filename) 1523 b'Sehensw\xc3\xbcrdigkeiten' 1524 1525 Some operating systems allow direct access to encoded bytes in the 1526 environment. If so, the :attr:`os.supports_bytes_environ` constant will be 1527 true. 1528 1529 For direct access to encoded environment variables (if available), 1530 use the new :func:`os.getenvb` function or use :data:`os.environb` 1531 which is a bytes version of :data:`os.environ`. 1532 1533 (Contributed by Victor Stinner.) 1534 1535 shutil 1536 ------ 1537 1538 The :func:`shutil.copytree` function has two new options: 1539 1540 * *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function 1541 copies a file pointed to by a symlink, not the symlink itself. This option 1542 will silence the error raised if the file doesn't exist. 1543 1544 * *copy_function*: is a callable that will be used to copy files. 1545 :func:`shutil.copy2` is used by default. 1546 1547 (Contributed by Tarek Ziad.) 1548 1549 In addition, the :mod:`shutil` module now supports :ref:`archiving operations 1550 <archiving-operations>` for zipfiles, uncompressed tarfiles, gzipped tarfiles, 1551 and bzipped tarfiles. And there are functions for registering additional 1552 archiving file formats (such as xz compressed tarfiles or custom formats). 1553 1554 The principal functions are :func:`~shutil.make_archive` and 1555 :func:`~shutil.unpack_archive`. By default, both operate on the current 1556 directory (which can be set by :func:`os.chdir`) and on any sub-directories. 1557 The archive filename needs to be specified with a full pathname. The archiving 1558 step is non-destructive (the original files are left unchanged). 1559 1560 :: 1561 1562 >>> import shutil, pprint 1563 1564 >>> os.chdir('mydata') # change to the source directory 1565 >>> f = shutil.make_archive('/var/backup/mydata', 1566 ... 'zip') # archive the current directory 1567 >>> f # show the name of archive 1568 '/var/backup/mydata.zip' 1569 >>> os.chdir('tmp') # change to an unpacking 1570 >>> shutil.unpack_archive('/var/backup/mydata.zip') # recover the data 1571 1572 >>> pprint.pprint(shutil.get_archive_formats()) # display known formats 1573 [('bztar', "bzip2'ed tar-file"), 1574 ('gztar', "gzip'ed tar-file"), 1575 ('tar', 'uncompressed tar file'), 1576 ('zip', 'ZIP file')] 1577 1578 >>> shutil.register_archive_format( # register a new archive format 1579 ... name='xz', 1580 ... function=xz.compress, # callable archiving function 1581 ... extra_args=[('level', 8)], # arguments to the function 1582 ... description='xz compression' 1583 ... ) 1584 1585 (Contributed by Tarek Ziad.) 1586 1587 sqlite3 1588 ------- 1589 1590 The :mod:`sqlite3` module was updated to pysqlite version 2.6.0. It has two new capabilities. 1591 1592 * The :attr:`sqlite3.Connection.in_transit` attribute is true if there is an 1593 active transaction for uncommitted changes. 1594 1595 * The :meth:`sqlite3.Connection.enable_load_extension` and 1596 :meth:`sqlite3.Connection.load_extension` methods allows you to load SQLite 1597 extensions from ".so" files. One well-known extension is the fulltext-search 1598 extension distributed with SQLite. 1599 1600 (Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.) 1601 1602 html 1603 ---- 1604 1605 A new :mod:`html` module was introduced with only a single function, 1606 :func:`~html.escape`, which is used for escaping reserved characters from HTML 1607 markup: 1608 1609 >>> import html 1610 >>> html.escape('x > 2 && x < 7') 1611 'x > 2 && x < 7' 1612 1613 socket 1614 ------ 1615 1616 The :mod:`socket` module has two new improvements. 1617 1618 * Socket objects now have a :meth:`~socket.socket.detach()` method which puts 1619 the socket into closed state without actually closing the underlying file 1620 descriptor. The latter can then be reused for other purposes. 1621 (Added by Antoine Pitrou; :issue:`8524`.) 1622 1623 * :func:`socket.create_connection` now supports the context management protocol 1624 to unconditionally consume :exc:`socket.error` exceptions and to close the 1625 socket when done. 1626 (Contributed by Giampaolo Rodol; :issue:`9794`.) 1627 1628 ssl 1629 --- 1630 1631 The :mod:`ssl` module added a number of features to satisfy common requirements 1632 for secure (encrypted, authenticated) internet connections: 1633 1634 * A new class, :class:`~ssl.SSLContext`, serves as a container for persistent 1635 SSL data, such as protocol settings, certificates, private keys, and various 1636 other options. It includes a :meth:`~ssl.SSLContext.wrap_socket` for creating 1637 an SSL socket from an SSL context. 1638 1639 * A new function, :func:`ssl.match_hostname`, supports server identity 1640 verification for higher-level protocols by implementing the rules of HTTPS 1641 (from :rfc:`2818`) which are also suitable for other protocols. 1642 1643 * The :func:`ssl.wrap_socket` constructor function now takes a *ciphers* 1644 argument. The *ciphers* string lists the allowed encryption algorithms using 1645 the format described in the `OpenSSL documentation 1646 <https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT>`__. 1647 1648 * When linked against recent versions of OpenSSL, the :mod:`ssl` module now 1649 supports the Server Name Indication extension to the TLS protocol, allowing 1650 multiple "virtual hosts" using different certificates on a single IP port. 1651 This extension is only supported in client mode, and is activated by passing 1652 the *server_hostname* argument to :meth:`ssl.SSLContext.wrap_socket`. 1653 1654 * Various options have been added to the :mod:`ssl` module, such as 1655 :data:`~ssl.OP_NO_SSLv2` which disables the insecure and obsolete SSLv2 1656 protocol. 1657 1658 * The extension now loads all the OpenSSL ciphers and digest algorithms. If 1659 some SSL certificates cannot be verified, they are reported as an "unknown 1660 algorithm" error. 1661 1662 * The version of OpenSSL being used is now accessible using the module 1663 attributes :data:`ssl.OPENSSL_VERSION` (a string), 1664 :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and 1665 :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). 1666 1667 (Contributed by Antoine Pitrou in :issue:`8850`, :issue:`1589`, :issue:`8322`, 1668 :issue:`5639`, :issue:`4870`, :issue:`8484`, and :issue:`8321`.) 1669 1670 nntp 1671 ---- 1672 1673 The :mod:`nntplib` module has a revamped implementation with better bytes and 1674 text semantics as well as more practical APIs. These improvements break 1675 compatibility with the nntplib version in Python 3.1, which was partly 1676 dysfunctional in itself. 1677 1678 Support for secure connections through both implicit (using 1679 :class:`nntplib.NNTP_SSL`) and explicit (using :meth:`nntplib.NNTP.starttls`) 1680 TLS has also been added. 1681 1682 (Contributed by Antoine Pitrou in :issue:`9360` and Andrew Vant in :issue:`1926`.) 1683 1684 certificates 1685 ------------ 1686 1687 :class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler` 1688 and :func:`urllib.request.urlopen` now take optional arguments to allow for 1689 server certificate checking against a set of Certificate Authorities, 1690 as recommended in public uses of HTTPS. 1691 1692 (Added by Antoine Pitrou, :issue:`9003`.) 1693 1694 imaplib 1695 ------- 1696 1697 Support for explicit TLS on standard IMAP4 connections has been added through 1698 the new :mod:`imaplib.IMAP4.starttls` method. 1699 1700 (Contributed by Lorenzo M. Catucci and Antoine Pitrou, :issue:`4471`.) 1701 1702 http.client 1703 ----------- 1704 1705 There were a number of small API improvements in the :mod:`http.client` module. 1706 The old-style HTTP 0.9 simple responses are no longer supported and the *strict* 1707 parameter is deprecated in all classes. 1708 1709 The :class:`~http.client.HTTPConnection` and 1710 :class:`~http.client.HTTPSConnection` classes now have a *source_address* 1711 parameter for a (host, port) tuple indicating where the HTTP connection is made 1712 from. 1713 1714 Support for certificate checking and HTTPS virtual hosts were added to 1715 :class:`~http.client.HTTPSConnection`. 1716 1717 The :meth:`~http.client.HTTPConnection.request` method on connection objects 1718 allowed an optional *body* argument so that a :term:`file object` could be used 1719 to supply the content of the request. Conveniently, the *body* argument now 1720 also accepts an :term:`iterable` object so long as it includes an explicit 1721 ``Content-Length`` header. This extended interface is much more flexible than 1722 before. 1723 1724 To establish an HTTPS connection through a proxy server, there is a new 1725 :meth:`~http.client.HTTPConnection.set_tunnel` method that sets the host and 1726 port for HTTP Connect tunneling. 1727 1728 To match the behavior of :mod:`http.server`, the HTTP client library now also 1729 encodes headers with ISO-8859-1 (Latin-1) encoding. It was already doing that 1730 for incoming headers, so now the behavior is consistent for both incoming and 1731 outgoing traffic. (See work by Armin Ronacher in :issue:`10980`.) 1732 1733 unittest 1734 -------- 1735 1736 The unittest module has a number of improvements supporting test discovery for 1737 packages, easier experimentation at the interactive prompt, new testcase 1738 methods, improved diagnostic messages for test failures, and better method 1739 names. 1740 1741 * The command-line call ``python -m unittest`` can now accept file paths 1742 instead of module names for running specific tests (:issue:`10620`). The new 1743 test discovery can find tests within packages, locating any test importable 1744 from the top-level directory. The top-level directory can be specified with 1745 the `-t` option, a pattern for matching files with ``-p``, and a directory to 1746 start discovery with ``-s``: 1747 1748 .. code-block:: shell-session 1749 1750 $ python -m unittest discover -s my_proj_dir -p _test.py 1751 1752 (Contributed by Michael Foord.) 1753 1754 * Experimentation at the interactive prompt is now easier because the 1755 :class:`unittest.case.TestCase` class can now be instantiated without 1756 arguments: 1757 1758 >>> from unittest import TestCase 1759 >>> TestCase().assertEqual(pow(2, 3), 8) 1760 1761 (Contributed by Michael Foord.) 1762 1763 * The :mod:`unittest` module has two new methods, 1764 :meth:`~unittest.TestCase.assertWarns` and 1765 :meth:`~unittest.TestCase.assertWarnsRegex` to verify that a given warning type 1766 is triggered by the code under test:: 1767 1768 with self.assertWarns(DeprecationWarning): 1769 legacy_function('XYZ') 1770 1771 (Contributed by Antoine Pitrou, :issue:`9754`.) 1772 1773 Another new method, :meth:`~unittest.TestCase.assertCountEqual` is used to 1774 compare two iterables to determine if their element counts are equal (whether 1775 the same elements are present with the same number of occurrences regardless 1776 of order):: 1777 1778 def test_anagram(self): 1779 self.assertCountEqual('algorithm', 'logarithm') 1780 1781 (Contributed by Raymond Hettinger.) 1782 1783 * A principal feature of the unittest module is an effort to produce meaningful 1784 diagnostics when a test fails. When possible, the failure is recorded along 1785 with a diff of the output. This is especially helpful for analyzing log files 1786 of failed test runs. However, since diffs can sometime be voluminous, there is 1787 a new :attr:`~unittest.TestCase.maxDiff` attribute that sets maximum length of 1788 diffs displayed. 1789 1790 * In addition, the method names in the module have undergone a number of clean-ups. 1791 1792 For example, :meth:`~unittest.TestCase.assertRegex` is the new name for 1793 :meth:`~unittest.TestCase.assertRegexpMatches` which was misnamed because the 1794 test uses :func:`re.search`, not :func:`re.match`. Other methods using 1795 regular expressions are now named using short form "Regex" in preference to 1796 "Regexp" -- this matches the names used in other unittest implementations, 1797 matches Python's old name for the :mod:`re` module, and it has unambiguous 1798 camel-casing. 1799 1800 (Contributed by Raymond Hettinger and implemented by Ezio Melotti.) 1801 1802 * To improve consistency, some long-standing method aliases are being 1803 deprecated in favor of the preferred names: 1804 1805 =============================== ============================== 1806 Old Name Preferred Name 1807 =============================== ============================== 1808 :meth:`assert_` :meth:`.assertTrue` 1809 :meth:`assertEquals` :meth:`.assertEqual` 1810 :meth:`assertNotEquals` :meth:`.assertNotEqual` 1811 :meth:`assertAlmostEquals` :meth:`.assertAlmostEqual` 1812 :meth:`assertNotAlmostEquals` :meth:`.assertNotAlmostEqual` 1813 =============================== ============================== 1814 1815 Likewise, the ``TestCase.fail*`` methods deprecated in Python 3.1 are expected 1816 to be removed in Python 3.3. Also see the :ref:`deprecated-aliases` section in 1817 the :mod:`unittest` documentation. 1818 1819 (Contributed by Ezio Melotti; :issue:`9424`.) 1820 1821 * The :meth:`~unittest.TestCase.assertDictContainsSubset` method was deprecated 1822 because it was misimplemented with the arguments in the wrong order. This 1823 created hard-to-debug optical illusions where tests like 1824 ``TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1})`` would fail. 1825 1826 (Contributed by Raymond Hettinger.) 1827 1828 random 1829 ------ 1830 1831 The integer methods in the :mod:`random` module now do a better job of producing 1832 uniform distributions. Previously, they computed selections with 1833 ``int(n*random())`` which had a slight bias whenever *n* was not a power of two. 1834 Now, multiple selections are made from a range up to the next power of two and a 1835 selection is kept only when it falls within the range ``0 <= x < n``. The 1836 functions and methods affected are :func:`~random.randrange`, 1837 :func:`~random.randint`, :func:`~random.choice`, :func:`~random.shuffle` and 1838 :func:`~random.sample`. 1839 1840 (Contributed by Raymond Hettinger; :issue:`9025`.) 1841 1842 poplib 1843 ------ 1844 1845 :class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a 1846 :class:`ssl.SSLContext` object allowing bundling SSL configuration options, 1847 certificates and private keys into a single (potentially long-lived) 1848 structure. 1849 1850 (Contributed by Giampaolo Rodol; :issue:`8807`.) 1851 1852 asyncore 1853 -------- 1854 1855 :class:`asyncore.dispatcher` now provides a 1856 :meth:`~asyncore.dispatcher.handle_accepted()` method 1857 returning a `(sock, addr)` pair which is called when a connection has actually 1858 been established with a new remote endpoint. This is supposed to be used as a 1859 replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids 1860 the user to call :meth:`~asyncore.dispatcher.accept()` directly. 1861 1862 (Contributed by Giampaolo Rodol; :issue:`6706`.) 1863 1864 tempfile 1865 -------- 1866 1867 The :mod:`tempfile` module has a new context manager, 1868 :class:`~tempfile.TemporaryDirectory` which provides easy deterministic 1869 cleanup of temporary directories:: 1870 1871 with tempfile.TemporaryDirectory() as tmpdirname: 1872 print('created temporary dir:', tmpdirname) 1873 1874 (Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.) 1875 1876 inspect 1877 ------- 1878 1879 * The :mod:`inspect` module has a new function 1880 :func:`~inspect.getgeneratorstate` to easily identify the current state of a 1881 generator-iterator:: 1882 1883 >>> from inspect import getgeneratorstate 1884 >>> def gen(): 1885 ... yield 'demo' 1886 >>> g = gen() 1887 >>> getgeneratorstate(g) 1888 'GEN_CREATED' 1889 >>> next(g) 1890 'demo' 1891 >>> getgeneratorstate(g) 1892 'GEN_SUSPENDED' 1893 >>> next(g, None) 1894 >>> getgeneratorstate(g) 1895 'GEN_CLOSED' 1896 1897 (Contributed by Rodolpho Eckhardt and Nick Coghlan, :issue:`10220`.) 1898 1899 * To support lookups without the possibility of activating a dynamic attribute, 1900 the :mod:`inspect` module has a new function, :func:`~inspect.getattr_static`. 1901 Unlike :func:`hasattr`, this is a true read-only search, guaranteed not to 1902 change state while it is searching:: 1903 1904 >>> class A: 1905 ... @property 1906 ... def f(self): 1907 ... print('Running') 1908 ... return 10 1909 ... 1910 >>> a = A() 1911 >>> getattr(a, 'f') 1912 Running 1913 10 1914 >>> inspect.getattr_static(a, 'f') 1915 <property object at 0x1022bd788> 1916 1917 (Contributed by Michael Foord.) 1918 1919 pydoc 1920 ----- 1921 1922 The :mod:`pydoc` module now provides a much-improved Web server interface, as 1923 well as a new command-line option ``-b`` to automatically open a browser window 1924 to display that server: 1925 1926 .. code-block:: shell-session 1927 1928 $ pydoc3.2 -b 1929 1930 (Contributed by Ron Adam; :issue:`2001`.) 1931 1932 dis 1933 --- 1934 1935 The :mod:`dis` module gained two new functions for inspecting code, 1936 :func:`~dis.code_info` and :func:`~dis.show_code`. Both provide detailed code 1937 object information for the supplied function, method, source code string or code 1938 object. The former returns a string and the latter prints it:: 1939 1940 >>> import dis, random 1941 >>> dis.show_code(random.choice) 1942 Name: choice 1943 Filename: /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py 1944 Argument count: 2 1945 Kw-only arguments: 0 1946 Number of locals: 3 1947 Stack size: 11 1948 Flags: OPTIMIZED, NEWLOCALS, NOFREE 1949 Constants: 1950 0: 'Choose a random element from a non-empty sequence.' 1951 1: 'Cannot choose from an empty sequence' 1952 Names: 1953 0: _randbelow 1954 1: len 1955 2: ValueError 1956 3: IndexError 1957 Variable names: 1958 0: self 1959 1: seq 1960 2: i 1961 1962 In addition, the :func:`~dis.dis` function now accepts string arguments 1963 so that the common idiom ``dis(compile(s, '', 'eval'))`` can be shortened 1964 to ``dis(s)``:: 1965 1966 >>> dis('3*x+1 if x%2==1 else x//2') 1967 1 0 LOAD_NAME 0 (x) 1968 3 LOAD_CONST 0 (2) 1969 6 BINARY_MODULO 1970 7 LOAD_CONST 1 (1) 1971 10 COMPARE_OP 2 (==) 1972 13 POP_JUMP_IF_FALSE 28 1973 16 LOAD_CONST 2 (3) 1974 19 LOAD_NAME 0 (x) 1975 22 BINARY_MULTIPLY 1976 23 LOAD_CONST 1 (1) 1977 26 BINARY_ADD 1978 27 RETURN_VALUE 1979 >> 28 LOAD_NAME 0 (x) 1980 31 LOAD_CONST 0 (2) 1981 34 BINARY_FLOOR_DIVIDE 1982 35 RETURN_VALUE 1983 1984 Taken together, these improvements make it easier to explore how CPython is 1985 implemented and to see for yourself what the language syntax does 1986 under-the-hood. 1987 1988 (Contributed by Nick Coghlan in :issue:`9147`.) 1989 1990 dbm 1991 --- 1992 1993 All database modules now support the :meth:`get` and :meth:`setdefault` methods. 1994 1995 (Suggested by Ray Allen in :issue:`9523`.) 1996 1997 ctypes 1998 ------ 1999 2000 A new type, :class:`ctypes.c_ssize_t` represents the C :c:type:`ssize_t` datatype. 2001 2002 site 2003 ---- 2004 2005 The :mod:`site` module has three new functions useful for reporting on the 2006 details of a given Python installation. 2007 2008 * :func:`~site.getsitepackages` lists all global site-packages directories. 2009 2010 * :func:`~site.getuserbase` reports on the user's base directory where data can 2011 be stored. 2012 2013 * :func:`~site.getusersitepackages` reveals the user-specific site-packages 2014 directory path. 2015 2016 :: 2017 2018 >>> import site 2019 >>> site.getsitepackages() 2020 ['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages', 2021 '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python', 2022 '/Library/Python/3.2/site-packages'] 2023 >>> site.getuserbase() 2024 '/Users/raymondhettinger/Library/Python/3.2' 2025 >>> site.getusersitepackages() 2026 '/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages' 2027 2028 Conveniently, some of site's functionality is accessible directly from the 2029 command-line: 2030 2031 .. code-block:: shell-session 2032 2033 $ python -m site --user-base 2034 /Users/raymondhettinger/.local 2035 $ python -m site --user-site 2036 /Users/raymondhettinger/.local/lib/python3.2/site-packages 2037 2038 (Contributed by Tarek Ziad in :issue:`6693`.) 2039 2040 sysconfig 2041 --------- 2042 2043 The new :mod:`sysconfig` module makes it straightforward to discover 2044 installation paths and configuration variables that vary across platforms and 2045 installations. 2046 2047 The module offers access simple access functions for platform and version 2048 information: 2049 2050 * :func:`~sysconfig.get_platform` returning values like *linux-i586* or 2051 *macosx-10.6-ppc*. 2052 * :func:`~sysconfig.get_python_version` returns a Python version string 2053 such as "3.2". 2054 2055 It also provides access to the paths and variables corresponding to one of 2056 seven named schemes used by :mod:`distutils`. Those include *posix_prefix*, 2057 *posix_home*, *posix_user*, *nt*, *nt_user*, *os2*, *os2_home*: 2058 2059 * :func:`~sysconfig.get_paths` makes a dictionary containing installation paths 2060 for the current installation scheme. 2061 * :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific 2062 variables. 2063 2064 There is also a convenient command-line interface: 2065 2066 .. code-block:: doscon 2067 2068 C:\Python32>python -m sysconfig 2069 Platform: "win32" 2070 Python version: "3.2" 2071 Current installation scheme: "nt" 2072 2073 Paths: 2074 data = "C:\Python32" 2075 include = "C:\Python32\Include" 2076 platinclude = "C:\Python32\Include" 2077 platlib = "C:\Python32\Lib\site-packages" 2078 platstdlib = "C:\Python32\Lib" 2079 purelib = "C:\Python32\Lib\site-packages" 2080 scripts = "C:\Python32\Scripts" 2081 stdlib = "C:\Python32\Lib" 2082 2083 Variables: 2084 BINDIR = "C:\Python32" 2085 BINLIBDEST = "C:\Python32\Lib" 2086 EXE = ".exe" 2087 INCLUDEPY = "C:\Python32\Include" 2088 LIBDEST = "C:\Python32\Lib" 2089 SO = ".pyd" 2090 VERSION = "32" 2091 abiflags = "" 2092 base = "C:\Python32" 2093 exec_prefix = "C:\Python32" 2094 platbase = "C:\Python32" 2095 prefix = "C:\Python32" 2096 projectbase = "C:\Python32" 2097 py_version = "3.2" 2098 py_version_nodot = "32" 2099 py_version_short = "3.2" 2100 srcdir = "C:\Python32" 2101 userbase = "C:\Documents and Settings\Raymond\Application Data\Python" 2102 2103 (Moved out of Distutils by Tarek Ziad.) 2104 2105 pdb 2106 --- 2107 2108 The :mod:`pdb` debugger module gained a number of usability improvements: 2109 2110 * :file:`pdb.py` now has a ``-c`` option that executes commands as given in a 2111 :file:`.pdbrc` script file. 2112 * A :file:`.pdbrc` script file can contain ``continue`` and ``next`` commands 2113 that continue debugging. 2114 * The :class:`Pdb` class constructor now accepts a *nosigint* argument. 2115 * New commands: ``l(list)``, ``ll(long list)`` and ``source`` for 2116 listing source code. 2117 * New commands: ``display`` and ``undisplay`` for showing or hiding 2118 the value of an expression if it has changed. 2119 * New command: ``interact`` for starting an interactive interpreter containing 2120 the global and local names found in the current scope. 2121 * Breakpoints can be cleared by breakpoint number. 2122 2123 (Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.) 2124 2125 configparser 2126 ------------ 2127 2128 The :mod:`configparser` module was modified to improve usability and 2129 predictability of the default parser and its supported INI syntax. The old 2130 :class:`ConfigParser` class was removed in favor of :class:`SafeConfigParser` 2131 which has in turn been renamed to :class:`~configparser.ConfigParser`. Support 2132 for inline comments is now turned off by default and section or option 2133 duplicates are not allowed in a single configuration source. 2134 2135 Config parsers gained a new API based on the mapping protocol:: 2136 2137 >>> parser = ConfigParser() 2138 >>> parser.read_string(""" 2139 ... [DEFAULT] 2140 ... location = upper left 2141 ... visible = yes 2142 ... editable = no 2143 ... color = blue 2144 ... 2145 ... [main] 2146 ... title = Main Menu 2147 ... color = green 2148 ... 2149 ... [options] 2150 ... title = Options 2151 ... """) 2152 >>> parser['main']['color'] 2153 'green' 2154 >>> parser['main']['editable'] 2155 'no' 2156 >>> section = parser['options'] 2157 >>> section['title'] 2158 'Options' 2159 >>> section['title'] = 'Options (editable: %(editable)s)' 2160 >>> section['title'] 2161 'Options (editable: no)' 2162 2163 The new API is implemented on top of the classical API, so custom parser 2164 subclasses should be able to use it without modifications. 2165 2166 The INI file structure accepted by config parsers can now be customized. Users 2167 can specify alternative option/value delimiters and comment prefixes, change the 2168 name of the *DEFAULT* section or switch the interpolation syntax. 2169 2170 There is support for pluggable interpolation including an additional interpolation 2171 handler :class:`~configparser.ExtendedInterpolation`:: 2172 2173 >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) 2174 >>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'}, 2175 ... 'custom': {'prefix': '/usr/local'}}) 2176 >>> parser.read_string(""" 2177 ... [buildout] 2178 ... parts = 2179 ... zope9 2180 ... instance 2181 ... find-links = 2182 ... ${buildout:directory}/downloads/dist 2183 ... 2184 ... [zope9] 2185 ... recipe = plone.recipe.zope9install 2186 ... location = /opt/zope 2187 ... 2188 ... [instance] 2189 ... recipe = plone.recipe.zope9instance 2190 ... zope9-location = ${zope9:location} 2191 ... zope-conf = ${custom:prefix}/etc/zope.conf 2192 ... """) 2193 >>> parser['buildout']['find-links'] 2194 '\n/home/ambv/zope9/downloads/dist' 2195 >>> parser['instance']['zope-conf'] 2196 '/usr/local/etc/zope.conf' 2197 >>> instance = parser['instance'] 2198 >>> instance['zope-conf'] 2199 '/usr/local/etc/zope.conf' 2200 >>> instance['zope9-location'] 2201 '/opt/zope' 2202 2203 A number of smaller features were also introduced, like support for specifying 2204 encoding in read operations, specifying fallback values for get-functions, or 2205 reading directly from dictionaries and strings. 2206 2207 (All changes contributed by ukasz Langa.) 2208 2209 .. XXX consider showing a difflib example 2210 2211 urllib.parse 2212 ------------ 2213 2214 A number of usability improvements were made for the :mod:`urllib.parse` module. 2215 2216 The :func:`~urllib.parse.urlparse` function now supports `IPv6 2217 <https://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`: 2218 2219 >>> import urllib.parse 2220 >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') # doctest: +NORMALIZE_WHITESPACE 2221 ParseResult(scheme='http', 2222 netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]', 2223 path='/foo/', 2224 params='', 2225 query='', 2226 fragment='') 2227 2228 The :func:`~urllib.parse.urldefrag` function now returns a :term:`named tuple`:: 2229 2230 >>> r = urllib.parse.urldefrag('http://python.org/about/#target') 2231 >>> r 2232 DefragResult(url='http://python.org/about/', fragment='target') 2233 >>> r[0] 2234 'http://python.org/about/' 2235 >>> r.fragment 2236 'target' 2237 2238 And, the :func:`~urllib.parse.urlencode` function is now much more flexible, 2239 accepting either a string or bytes type for the *query* argument. If it is a 2240 string, then the *safe*, *encoding*, and *error* parameters are sent to 2241 :func:`~urllib.parse.quote_plus` for encoding:: 2242 2243 >>> urllib.parse.urlencode([ 2244 ... ('type', 'telenovela'), 2245 ... ('name', 'Dnde Est Elisa?')], 2246 ... encoding='latin-1') 2247 'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F' 2248 2249 As detailed in :ref:`parsing-ascii-encoded-bytes`, all the :mod:`urllib.parse` 2250 functions now accept ASCII-encoded byte strings as input, so long as they are 2251 not mixed with regular strings. If ASCII-encoded byte strings are given as 2252 parameters, the return types will also be an ASCII-encoded byte strings: 2253 2254 >>> urllib.parse.urlparse(b'http://www.python.org:80/about/') # doctest: +NORMALIZE_WHITESPACE 2255 ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80', 2256 path=b'/about/', params=b'', query=b'', fragment=b'') 2257 2258 (Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in :issue:`2987`, 2259 :issue:`5468`, and :issue:`9873`.) 2260 2261 mailbox 2262 ------- 2263 2264 Thanks to a concerted effort by R. David Murray, the :mod:`mailbox` module has 2265 been fixed for Python 3.2. The challenge was that mailbox had been originally 2266 designed with a text interface, but email messages are best represented with 2267 :class:`bytes` because various parts of a message may have different encodings. 2268 2269 The solution harnessed the :mod:`email` package's binary support for parsing 2270 arbitrary email messages. In addition, the solution required a number of API 2271 changes. 2272 2273 As expected, the :meth:`~mailbox.Mailbox.add` method for 2274 :class:`mailbox.Mailbox` objects now accepts binary input. 2275 2276 :class:`~io.StringIO` and text file input are deprecated. Also, string input 2277 will fail early if non-ASCII characters are used. Previously it would fail when 2278 the email was processed in a later step. 2279 2280 There is also support for binary output. The :meth:`~mailbox.Mailbox.get_file` 2281 method now returns a file in the binary mode (where it used to incorrectly set 2282 the file to text-mode). There is also a new :meth:`~mailbox.Mailbox.get_bytes` 2283 method that returns a :class:`bytes` representation of a message corresponding 2284 to a given *key*. 2285 2286 It is still possible to get non-binary output using the old API's 2287 :meth:`~mailbox.Mailbox.get_string` method, but that approach 2288 is not very useful. Instead, it is best to extract messages from 2289 a :class:`~mailbox.Message` object or to load them from binary input. 2290 2291 (Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an 2292 initial patch by Victor Stinner in :issue:`9124`.) 2293 2294 turtledemo 2295 ---------- 2296 2297 The demonstration code for the :mod:`turtle` module was moved from the *Demo* 2298 directory to main library. It includes over a dozen sample scripts with 2299 lively displays. Being on :attr:`sys.path`, it can now be run directly 2300 from the command-line: 2301 2302 .. code-block:: shell-session 2303 2304 $ python -m turtledemo 2305 2306 (Moved from the Demo directory by Alexander Belopolsky in :issue:`10199`.) 2307 2308 Multi-threading 2309 =============== 2310 2311 * The mechanism for serializing execution of concurrently running Python threads 2312 (generally known as the :term:`GIL` or :term:`Global Interpreter Lock`) has 2313 been rewritten. Among the objectives were more predictable switching 2314 intervals and reduced overhead due to lock contention and the number of 2315 ensuing system calls. The notion of a "check interval" to allow thread 2316 switches has been abandoned and replaced by an absolute duration expressed in 2317 seconds. This parameter is tunable through :func:`sys.setswitchinterval()`. 2318 It currently defaults to 5 milliseconds. 2319 2320 Additional details about the implementation can be read from a `python-dev 2321 mailing-list message 2322 <https://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_ 2323 (however, "priority requests" as exposed in this message have not been kept 2324 for inclusion). 2325 2326 (Contributed by Antoine Pitrou.) 2327 2328 * Regular and recursive locks now accept an optional *timeout* argument to their 2329 :meth:`~threading.Lock.acquire` method. (Contributed by Antoine Pitrou; 2330 :issue:`7316`.) 2331 2332 * Similarly, :meth:`threading.Semaphore.acquire` also gained a *timeout* 2333 argument. (Contributed by Torsten Landschoff; :issue:`850728`.) 2334 2335 * Regular and recursive lock acquisitions can now be interrupted by signals on 2336 platforms using Pthreads. This means that Python programs that deadlock while 2337 acquiring locks can be successfully killed by repeatedly sending SIGINT to the 2338 process (by pressing :kbd:`Ctrl+C` in most shells). 2339 (Contributed by Reid Kleckner; :issue:`8844`.) 2340 2341 2342 Optimizations 2343 ============= 2344 2345 A number of small performance enhancements have been added: 2346 2347 * Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as 2348 being a test for membership in a set of constants. The optimizer recasts the 2349 :class:`set` as a :class:`frozenset` and stores the pre-built constant. 2350 2351 Now that the speed penalty is gone, it is practical to start writing 2352 membership tests using set-notation. This style is both semantically clear 2353 and operationally fast:: 2354 2355 extension = name.rpartition('.')[2] 2356 if extension in {'xml', 'html', 'xhtml', 'css'}: 2357 handle(name) 2358 2359 (Patch and additional tests contributed by Dave Malcolm; :issue:`6690`). 2360 2361 * Serializing and unserializing data using the :mod:`pickle` module is now 2362 several times faster. 2363 2364 (Contributed by Alexandre Vassalotti, Antoine Pitrou 2365 and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.) 2366 2367 * The `Timsort algorithm <https://en.wikipedia.org/wiki/Timsort>`_ used in 2368 :meth:`list.sort` and :func:`sorted` now runs faster and uses less memory 2369 when called with a :term:`key function`. Previously, every element of 2370 a list was wrapped with a temporary object that remembered the key value 2371 associated with each element. Now, two arrays of keys and values are 2372 sorted in parallel. This saves the memory consumed by the sort wrappers, 2373 and it saves time lost to delegating comparisons. 2374 2375 (Patch by Daniel Stutzbach in :issue:`9915`.) 2376 2377 * JSON decoding performance is improved and memory consumption is reduced 2378 whenever the same string is repeated for multiple keys. Also, JSON encoding 2379 now uses the C speedups when the ``sort_keys`` argument is true. 2380 2381 (Contributed by Antoine Pitrou in :issue:`7451` and by Raymond Hettinger and 2382 Antoine Pitrou in :issue:`10314`.) 2383 2384 * Recursive locks (created with the :func:`threading.RLock` API) now benefit 2385 from a C implementation which makes them as fast as regular locks, and between 2386 10x and 15x faster than their previous pure Python implementation. 2387 2388 (Contributed by Antoine Pitrou; :issue:`3001`.) 2389 2390 * The fast-search algorithm in stringlib is now used by the :meth:`split`, 2391 :meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on 2392 :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the 2393 algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and 2394 :meth:`rpartition`. 2395 2396 (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.) 2397 2398 2399 * Integer to string conversions now work two "digits" at a time, reducing the 2400 number of division and modulo operations. 2401 2402 (:issue:`6713` by Gawain Bolton, Mark Dickinson, and Victor Stinner.) 2403 2404 There were several other minor optimizations. Set differencing now runs faster 2405 when one operand is much larger than the other (patch by Andress Bennetts in 2406 :issue:`8685`). The :meth:`array.repeat` method has a faster implementation 2407 (:issue:`1569291` by Alexander Belopolsky). The :class:`BaseHTTPRequestHandler` 2408 has more efficient buffering (:issue:`3709` by Andrew Schaaf). The 2409 :func:`operator.attrgetter` function has been sped-up (:issue:`10160` by 2410 Christos Georgiou). And :class:`ConfigParser` loads multi-line arguments a bit 2411 faster (:issue:`7113` by ukasz Langa). 2412 2413 2414 Unicode 2415 ======= 2416 2417 Python has been updated to `Unicode 6.0.0 2418 <http://unicode.org/versions/Unicode6.0.0/>`_. The update to the standard adds 2419 over 2,000 new characters including `emoji <https://en.wikipedia.org/wiki/Emoji>`_ 2420 symbols which are important for mobile phones. 2421 2422 In addition, the updated standard has altered the character properties for two 2423 Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character 2424 (U+19DA), making the former eligible for use in identifiers while disqualifying 2425 the latter. For more information, see `Unicode Character Database Changes 2426 <http://www.unicode.org/versions/Unicode6.0.0/#Database_Changes>`_. 2427 2428 2429 Codecs 2430 ====== 2431 2432 Support was added for *cp720* Arabic DOS encoding (:issue:`1616979`). 2433 2434 MBCS encoding no longer ignores the error handler argument. In the default 2435 strict mode, it raises an :exc:`UnicodeDecodeError` when it encounters an 2436 undecodable byte sequence and an :exc:`UnicodeEncodeError` for an unencodable 2437 character. 2438 2439 The MBCS codec supports ``'strict'`` and ``'ignore'`` error handlers for 2440 decoding, and ``'strict'`` and ``'replace'`` for encoding. 2441 2442 To emulate Python3.1 MBCS encoding, select the ``'ignore'`` handler for decoding 2443 and the ``'replace'`` handler for encoding. 2444 2445 On Mac OS X, Python decodes command line arguments with ``'utf-8'`` rather than 2446 the locale encoding. 2447 2448 By default, :mod:`tarfile` uses ``'utf-8'`` encoding on Windows (instead of 2449 ``'mbcs'``) and the ``'surrogateescape'`` error handler on all operating 2450 systems. 2451 2452 2453 Documentation 2454 ============= 2455 2456 The documentation continues to be improved. 2457 2458 * A table of quick links has been added to the top of lengthy sections such as 2459 :ref:`built-in-funcs`. In the case of :mod:`itertools`, the links are 2460 accompanied by tables of cheatsheet-style summaries to provide an overview and 2461 memory jog without having to read all of the docs. 2462 2463 * In some cases, the pure Python source code can be a helpful adjunct to the 2464 documentation, so now many modules now feature quick links to the latest 2465 version of the source code. For example, the :mod:`functools` module 2466 documentation has a quick link at the top labeled: 2467 2468 **Source code** :source:`Lib/functools.py`. 2469 2470 (Contributed by Raymond Hettinger; see 2471 `rationale <https://rhettinger.wordpress.com/2011/01/28/open-your-source-more/>`_.) 2472 2473 * The docs now contain more examples and recipes. In particular, :mod:`re` 2474 module has an extensive section, :ref:`re-examples`. Likewise, the 2475 :mod:`itertools` module continues to be updated with new 2476 :ref:`itertools-recipes`. 2477 2478 * The :mod:`datetime` module now has an auxiliary implementation in pure Python. 2479 No functionality was changed. This just provides an easier-to-read alternate 2480 implementation. 2481 2482 (Contributed by Alexander Belopolsky in :issue:`9528`.) 2483 2484 * The unmaintained :file:`Demo` directory has been removed. Some demos were 2485 integrated into the documentation, some were moved to the :file:`Tools/demo` 2486 directory, and others were removed altogether. 2487 2488 (Contributed by Georg Brandl in :issue:`7962`.) 2489 2490 2491 IDLE 2492 ==== 2493 2494 * The format menu now has an option to clean source files by stripping 2495 trailing whitespace. 2496 2497 (Contributed by Raymond Hettinger; :issue:`5150`.) 2498 2499 * IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk. 2500 2501 (Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; :issue:`6075`.) 2502 2503 Code Repository 2504 =============== 2505 2506 In addition to the existing Subversion code repository at http://svn.python.org 2507 there is now a `Mercurial <https://www.mercurial-scm.org/>`_ repository at 2508 https://hg.python.org/\ . 2509 2510 After the 3.2 release, there are plans to switch to Mercurial as the primary 2511 repository. This distributed version control system should make it easier for 2512 members of the community to create and share external changesets. See 2513 :pep:`385` for details. 2514 2515 To learn to use the new version control system, see the `tutorial by Joel 2516 Spolsky <http://hginit.com>`_ or the `Guide to Mercurial Workflows 2517 <https://www.mercurial-scm.org/guide>`_. 2518 2519 2520 Build and C API Changes 2521 ======================= 2522 2523 Changes to Python's build process and to the C API include: 2524 2525 * The *idle*, *pydoc* and *2to3* scripts are now installed with a 2526 version-specific suffix on ``make altinstall`` (:issue:`10679`). 2527 2528 * The C functions that access the Unicode Database now accept and return 2529 characters from the full Unicode range, even on narrow unicode builds 2530 (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference 2531 in Python is that :func:`unicodedata.numeric` now returns the correct value 2532 for large code points, and :func:`repr` may consider more characters as 2533 printable. 2534 2535 (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.) 2536 2537 * Computed gotos are now enabled by default on supported compilers (which are 2538 detected by the configure script). They can still be disabled selectively by 2539 specifying ``--without-computed-gotos``. 2540 2541 (Contributed by Antoine Pitrou; :issue:`9203`.) 2542 2543 * The option ``--with-wctype-functions`` was removed. The built-in unicode 2544 database is now used for all functions. 2545 2546 (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.) 2547 2548 * Hash values are now values of a new type, :c:type:`Py_hash_t`, which is 2549 defined to be the same size as a pointer. Previously they were of type long, 2550 which on some 64-bit operating systems is still only 32 bits long. As a 2551 result of this fix, :class:`set` and :class:`dict` can now hold more than 2552 ``2**32`` entries on builds with 64-bit pointers (previously, they could grow 2553 to that size but their performance degraded catastrophically). 2554 2555 (Suggested by Raymond Hettinger and implemented by Benjamin Peterson; 2556 :issue:`9778`.) 2557 2558 * A new macro :c:macro:`Py_VA_COPY` copies the state of the variable argument 2559 list. It is equivalent to C99 *va_copy* but available on all Python platforms 2560 (:issue:`2443`). 2561 2562 * A new C API function :c:func:`PySys_SetArgvEx` allows an embedded interpreter 2563 to set :attr:`sys.argv` without also modifying :attr:`sys.path` 2564 (:issue:`5753`). 2565 2566 * :c:macro:`PyEval_CallObject` is now only available in macro form. The 2567 function declaration, which was kept for backwards compatibility reasons, is 2568 now removed -- the macro was introduced in 1997 (:issue:`8276`). 2569 2570 * There is a new function :c:func:`PyLong_AsLongLongAndOverflow` which 2571 is analogous to :c:func:`PyLong_AsLongAndOverflow`. They both serve to 2572 convert Python :class:`int` into a native fixed-width type while providing 2573 detection of cases where the conversion won't fit (:issue:`7767`). 2574 2575 * The :c:func:`PyUnicode_CompareWithASCIIString` function now returns *not 2576 equal* if the Python string is *NUL* terminated. 2577 2578 * There is a new function :c:func:`PyErr_NewExceptionWithDoc` that is 2579 like :c:func:`PyErr_NewException` but allows a docstring to be specified. 2580 This lets C exceptions have the same self-documenting capabilities as 2581 their pure Python counterparts (:issue:`7033`). 2582 2583 * When compiled with the ``--with-valgrind`` option, the pymalloc 2584 allocator will be automatically disabled when running under Valgrind. This 2585 gives improved memory leak detection when running under Valgrind, while taking 2586 advantage of pymalloc at other times (:issue:`2422`). 2587 2588 * Removed the ``O?`` format from the *PyArg_Parse* functions. The format is no 2589 longer used and it had never been documented (:issue:`8837`). 2590 2591 There were a number of other small changes to the C-API. See the 2592 :source:`Misc/NEWS` file for a complete list. 2593 2594 Also, there were a number of updates to the Mac OS X build, see 2595 :source:`Mac/BuildScript/README.txt` for details. For users running a 32/64-bit 2596 build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6. 2597 Accordingly, we recommend installing an updated alternative such as 2598 `ActiveState Tcl/Tk 8.5.9 <https://www.activestate.com/activetcl/downloads>`_\. 2599 See https://www.python.org/download/mac/tcltk/ for additional details. 2600 2601 Porting to Python 3.2 2602 ===================== 2603 2604 This section lists previously described changes and other bugfixes that may 2605 require changes to your code: 2606 2607 * The :mod:`configparser` module has a number of clean-ups. The major change is 2608 to replace the old :class:`ConfigParser` class with long-standing preferred 2609 alternative :class:`SafeConfigParser`. In addition there are a number of 2610 smaller incompatibilities: 2611 2612 * The interpolation syntax is now validated on 2613 :meth:`~configparser.ConfigParser.get` and 2614 :meth:`~configparser.ConfigParser.set` operations. In the default 2615 interpolation scheme, only two tokens with percent signs are valid: ``%(name)s`` 2616 and ``%%``, the latter being an escaped percent sign. 2617 2618 * The :meth:`~configparser.ConfigParser.set` and 2619 :meth:`~configparser.ConfigParser.add_section` methods now verify that 2620 values are actual strings. Formerly, unsupported types could be introduced 2621 unintentionally. 2622 2623 * Duplicate sections or options from a single source now raise either 2624 :exc:`~configparser.DuplicateSectionError` or 2625 :exc:`~configparser.DuplicateOptionError`. Formerly, duplicates would 2626 silently overwrite a previous entry. 2627 2628 * Inline comments are now disabled by default so now the **;** character 2629 can be safely used in values. 2630 2631 * Comments now can be indented. Consequently, for **;** or **#** to appear at 2632 the start of a line in multiline values, it has to be interpolated. This 2633 keeps comment prefix characters in values from being mistaken as comments. 2634 2635 * ``""`` is now a valid value and is no longer automatically converted to an 2636 empty string. For empty strings, use ``"option ="`` in a line. 2637 2638 * The :mod:`nntplib` module was reworked extensively, meaning that its APIs 2639 are often incompatible with the 3.1 APIs. 2640 2641 * :class:`bytearray` objects can no longer be used as filenames; instead, 2642 they should be converted to :class:`bytes`. 2643 2644 * The :meth:`array.tostring` and :meth:`array.fromstring` have been renamed to 2645 :meth:`array.tobytes` and :meth:`array.frombytes` for clarity. The old names 2646 have been deprecated. (See :issue:`8990`.) 2647 2648 * ``PyArg_Parse*()`` functions: 2649 2650 * "t#" format has been removed: use "s#" or "s*" instead 2651 * "w" and "w#" formats has been removed: use "w*" instead 2652 2653 * The :c:type:`PyCObject` type, deprecated in 3.1, has been removed. To wrap 2654 opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used 2655 instead; the new type has a well-defined interface for passing typing safety 2656 information and a less complicated signature for calling a destructor. 2657 2658 * The :func:`sys.setfilesystemencoding` function was removed because 2659 it had a flawed design. 2660 2661 * The :func:`random.seed` function and method now salt string seeds with an 2662 sha512 hash function. To access the previous version of *seed* in order to 2663 reproduce Python 3.1 sequences, set the *version* argument to *1*, 2664 ``random.seed(s, version=1)``. 2665 2666 * The previously deprecated :func:`string.maketrans` function has been removed 2667 in favor of the static methods :meth:`bytes.maketrans` and 2668 :meth:`bytearray.maketrans`. This change solves the confusion around which 2669 types were supported by the :mod:`string` module. Now, :class:`str`, 2670 :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and 2671 **translate** methods with intermediate translation tables of the appropriate 2672 type. 2673 2674 (Contributed by Georg Brandl; :issue:`5675`.) 2675 2676 * The previously deprecated :func:`contextlib.nested` function has been removed 2677 in favor of a plain :keyword:`with` statement which can accept multiple 2678 context managers. The latter technique is faster (because it is built-in), 2679 and it does a better job finalizing multiple context managers when one of them 2680 raises an exception:: 2681 2682 with open('mylog.txt') as infile, open('a.out', 'w') as outfile: 2683 for line in infile: 2684 if '<critical>' in line: 2685 outfile.write(line) 2686 2687 (Contributed by Georg Brandl and Mattias Brndstrm; 2688 `appspot issue 53094 <https://codereview.appspot.com/53094>`_.) 2689 2690 * :func:`struct.pack` now only allows bytes for the ``s`` string pack code. 2691 Formerly, it would accept text arguments and implicitly encode them to bytes 2692 using UTF-8. This was problematic because it made assumptions about the 2693 correct encoding and because a variable-length encoding can fail when writing 2694 to fixed length segment of a structure. 2695 2696 Code such as ``struct.pack('<6sHHBBB', 'GIF87a', x, y)`` should be rewritten 2697 with to use bytes instead of text, ``struct.pack('<6sHHBBB', b'GIF87a', x, y)``. 2698 2699 (Discovered by David Beazley and fixed by Victor Stinner; :issue:`10783`.) 2700 2701 * The :class:`xml.etree.ElementTree` class now raises an 2702 :exc:`xml.etree.ElementTree.ParseError` when a parse fails. Previously it 2703 raised an :exc:`xml.parsers.expat.ExpatError`. 2704 2705 * The new, longer :func:`str` value on floats may break doctests which rely on 2706 the old output format. 2707 2708 * In :class:`subprocess.Popen`, the default value for *close_fds* is now 2709 ``True`` under Unix; under Windows, it is ``True`` if the three standard 2710 streams are set to ``None``, ``False`` otherwise. Previously, *close_fds* 2711 was always ``False`` by default, which produced difficult to solve bugs 2712 or race conditions when open file descriptors would leak into the child 2713 process. 2714 2715 * Support for legacy HTTP 0.9 has been removed from :mod:`urllib.request` 2716 and :mod:`http.client`. Such support is still present on the server side 2717 (in :mod:`http.server`). 2718 2719 (Contributed by Antoine Pitrou, :issue:`10711`.) 2720 2721 * SSL sockets in timeout mode now raise :exc:`socket.timeout` when a timeout 2722 occurs, rather than a generic :exc:`~ssl.SSLError`. 2723 2724 (Contributed by Antoine Pitrou, :issue:`10272`.) 2725 2726 * The misleading functions :c:func:`PyEval_AcquireLock()` and 2727 :c:func:`PyEval_ReleaseLock()` have been officially deprecated. The 2728 thread-state aware APIs (such as :c:func:`PyEval_SaveThread()` 2729 and :c:func:`PyEval_RestoreThread()`) should be used instead. 2730 2731 * Due to security risks, :func:`asyncore.handle_accept` has been deprecated, and 2732 a new function, :func:`asyncore.handle_accepted`, was added to replace it. 2733 2734 (Contributed by Giampaolo Rodola in :issue:`6706`.) 2735 2736 * Due to the new :term:`GIL` implementation, :c:func:`PyEval_InitThreads()` 2737 cannot be called before :c:func:`Py_Initialize()` anymore. 2738
