1 :mod:`codecs` --- Codec registry and base classes 2 ================================================= 3 4 .. module:: codecs 5 :synopsis: Encode and decode data and streams. 6 7 .. moduleauthor:: Marc-Andr Lemburg <mal (a] lemburg.com> 8 .. sectionauthor:: Marc-Andr Lemburg <mal (a] lemburg.com> 9 .. sectionauthor:: Martin v. Lwis <martin (a] v.loewis.de> 10 11 **Source code:** :source:`Lib/codecs.py` 12 13 .. index:: 14 single: Unicode 15 single: Codecs 16 pair: Codecs; encode 17 pair: Codecs; decode 18 single: streams 19 pair: stackable; streams 20 21 -------------- 22 23 This module defines base classes for standard Python codecs (encoders and 24 decoders) and provides access to the internal Python codec registry, which 25 manages the codec and error handling lookup process. Most standard codecs 26 are :term:`text encodings <text encoding>`, which encode text to bytes, 27 but there are also codecs provided that encode text to text, and bytes to 28 bytes. Custom codecs may encode and decode between arbitrary types, but some 29 module features are restricted to use specifically with 30 :term:`text encodings <text encoding>`, or with codecs that encode to 31 :class:`bytes`. 32 33 The module defines the following functions for encoding and decoding with 34 any codec: 35 36 .. function:: encode(obj, encoding='utf-8', errors='strict') 37 38 Encodes *obj* using the codec registered for *encoding*. 39 40 *Errors* may be given to set the desired error handling scheme. The 41 default error handler is ``'strict'`` meaning that encoding errors raise 42 :exc:`ValueError` (or a more codec specific subclass, such as 43 :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more 44 information on codec error handling. 45 46 .. function:: decode(obj, encoding='utf-8', errors='strict') 47 48 Decodes *obj* using the codec registered for *encoding*. 49 50 *Errors* may be given to set the desired error handling scheme. The 51 default error handler is ``'strict'`` meaning that decoding errors raise 52 :exc:`ValueError` (or a more codec specific subclass, such as 53 :exc:`UnicodeDecodeError`). Refer to :ref:`codec-base-classes` for more 54 information on codec error handling. 55 56 The full details for each codec can also be looked up directly: 57 58 .. function:: lookup(encoding) 59 60 Looks up the codec info in the Python codec registry and returns a 61 :class:`CodecInfo` object as defined below. 62 63 Encodings are first looked up in the registry's cache. If not found, the list of 64 registered search functions is scanned. If no :class:`CodecInfo` object is 65 found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object 66 is stored in the cache and returned to the caller. 67 68 .. class:: CodecInfo(encode, decode, streamreader=None, streamwriter=None, incrementalencoder=None, incrementaldecoder=None, name=None) 69 70 Codec details when looking up the codec registry. The constructor 71 arguments are stored in attributes of the same name: 72 73 74 .. attribute:: name 75 76 The name of the encoding. 77 78 79 .. attribute:: encode 80 decode 81 82 The stateless encoding and decoding functions. These must be 83 functions or methods which have the same interface as 84 the :meth:`~Codec.encode` and :meth:`~Codec.decode` methods of Codec 85 instances (see :ref:`Codec Interface <codec-objects>`). 86 The functions or methods are expected to work in a stateless mode. 87 88 89 .. attribute:: incrementalencoder 90 incrementaldecoder 91 92 Incremental encoder and decoder classes or factory functions. 93 These have to provide the interface defined by the base classes 94 :class:`IncrementalEncoder` and :class:`IncrementalDecoder`, 95 respectively. Incremental codecs can maintain state. 96 97 98 .. attribute:: streamwriter 99 streamreader 100 101 Stream writer and reader classes or factory functions. These have to 102 provide the interface defined by the base classes 103 :class:`StreamWriter` and :class:`StreamReader`, respectively. 104 Stream codecs can maintain state. 105 106 To simplify access to the various codec components, the module provides 107 these additional functions which use :func:`lookup` for the codec lookup: 108 109 .. function:: getencoder(encoding) 110 111 Look up the codec for the given encoding and return its encoder function. 112 113 Raises a :exc:`LookupError` in case the encoding cannot be found. 114 115 116 .. function:: getdecoder(encoding) 117 118 Look up the codec for the given encoding and return its decoder function. 119 120 Raises a :exc:`LookupError` in case the encoding cannot be found. 121 122 123 .. function:: getincrementalencoder(encoding) 124 125 Look up the codec for the given encoding and return its incremental encoder 126 class or factory function. 127 128 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec 129 doesn't support an incremental encoder. 130 131 132 .. function:: getincrementaldecoder(encoding) 133 134 Look up the codec for the given encoding and return its incremental decoder 135 class or factory function. 136 137 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec 138 doesn't support an incremental decoder. 139 140 141 .. function:: getreader(encoding) 142 143 Look up the codec for the given encoding and return its :class:`StreamReader` 144 class or factory function. 145 146 Raises a :exc:`LookupError` in case the encoding cannot be found. 147 148 149 .. function:: getwriter(encoding) 150 151 Look up the codec for the given encoding and return its :class:`StreamWriter` 152 class or factory function. 153 154 Raises a :exc:`LookupError` in case the encoding cannot be found. 155 156 Custom codecs are made available by registering a suitable codec search 157 function: 158 159 .. function:: register(search_function) 160 161 Register a codec search function. Search functions are expected to take one 162 argument, being the encoding name in all lower case letters, and return a 163 :class:`CodecInfo` object. In case a search function cannot find 164 a given encoding, it should return ``None``. 165 166 .. note:: 167 168 Search function registration is not currently reversible, 169 which may cause problems in some cases, such as unit testing or 170 module reloading. 171 172 While the builtin :func:`open` and the associated :mod:`io` module are the 173 recommended approach for working with encoded text files, this module 174 provides additional utility functions and classes that allow the use of a 175 wider range of codecs when working with binary files: 176 177 .. function:: open(filename, mode='r', encoding=None, errors='strict', buffering=1) 178 179 Open an encoded file using the given *mode* and return an instance of 180 :class:`StreamReaderWriter`, providing transparent encoding/decoding. 181 The default file mode is ``'r'``, meaning to open the file in read mode. 182 183 .. note:: 184 185 Underlying encoded files are always opened in binary mode. 186 No automatic conversion of ``'\n'`` is done on reading and writing. 187 The *mode* argument may be any binary mode acceptable to the built-in 188 :func:`open` function; the ``'b'`` is automatically added. 189 190 *encoding* specifies the encoding which is to be used for the file. 191 Any encoding that encodes to and decodes from bytes is allowed, and 192 the data types supported by the file methods depend on the codec used. 193 194 *errors* may be given to define the error handling. It defaults to ``'strict'`` 195 which causes a :exc:`ValueError` to be raised in case an encoding error occurs. 196 197 *buffering* has the same meaning as for the built-in :func:`open` function. It 198 defaults to line buffered. 199 200 201 .. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict') 202 203 Return a :class:`StreamRecoder` instance, a wrapped version of *file* 204 which provides transparent transcoding. The original file is closed 205 when the wrapped version is closed. 206 207 Data written to the wrapped file is decoded according to the given 208 *data_encoding* and then written to the original file as bytes using 209 *file_encoding*. Bytes read from the original file are decoded 210 according to *file_encoding*, and the result is encoded 211 using *data_encoding*. 212 213 If *file_encoding* is not given, it defaults to *data_encoding*. 214 215 *errors* may be given to define the error handling. It defaults to 216 ``'strict'``, which causes :exc:`ValueError` to be raised in case an encoding 217 error occurs. 218 219 220 .. function:: iterencode(iterator, encoding, errors='strict', **kwargs) 221 222 Uses an incremental encoder to iteratively encode the input provided by 223 *iterator*. This function is a :term:`generator`. 224 The *errors* argument (as well as any 225 other keyword argument) is passed through to the incremental encoder. 226 227 This function requires that the codec accept text :class:`str` objects 228 to encode. Therefore it does not support bytes-to-bytes encoders such as 229 ``base64_codec``. 230 231 232 .. function:: iterdecode(iterator, encoding, errors='strict', **kwargs) 233 234 Uses an incremental decoder to iteratively decode the input provided by 235 *iterator*. This function is a :term:`generator`. 236 The *errors* argument (as well as any 237 other keyword argument) is passed through to the incremental decoder. 238 239 This function requires that the codec accept :class:`bytes` objects 240 to decode. Therefore it does not support text-to-text encoders such as 241 ``rot_13``, although ``rot_13`` may be used equivalently with 242 :func:`iterencode`. 243 244 245 The module also provides the following constants which are useful for reading 246 and writing to platform dependent files: 247 248 249 .. data:: BOM 250 BOM_BE 251 BOM_LE 252 BOM_UTF8 253 BOM_UTF16 254 BOM_UTF16_BE 255 BOM_UTF16_LE 256 BOM_UTF32 257 BOM_UTF32_BE 258 BOM_UTF32_LE 259 260 These constants define various byte sequences, 261 being Unicode byte order marks (BOMs) for several encodings. They are 262 used in UTF-16 and UTF-32 data streams to indicate the byte order used, 263 and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either 264 :const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's 265 native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`, 266 :const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for 267 :const:`BOM_UTF16_BE`. The others represent the BOM in UTF-8 and UTF-32 268 encodings. 269 270 271 .. _codec-base-classes: 272 273 Codec Base Classes 274 ------------------ 275 276 The :mod:`codecs` module defines a set of base classes which define the 277 interfaces for working with codec objects, and can also be used as the basis 278 for custom codec implementations. 279 280 Each codec has to define four interfaces to make it usable as codec in Python: 281 stateless encoder, stateless decoder, stream reader and stream writer. The 282 stream reader and writers typically reuse the stateless encoder/decoder to 283 implement the file protocols. Codec authors also need to define how the 284 codec will handle encoding and decoding errors. 285 286 287 .. _surrogateescape: 288 .. _error-handlers: 289 290 Error Handlers 291 ^^^^^^^^^^^^^^ 292 293 To simplify and standardize error handling, 294 codecs may implement different error handling schemes by 295 accepting the *errors* string argument. The following string values are 296 defined and implemented by all standard Python codecs: 297 298 .. tabularcolumns:: |l|L| 299 300 +-------------------------+-----------------------------------------------+ 301 | Value | Meaning | 302 +=========================+===============================================+ 303 | ``'strict'`` | Raise :exc:`UnicodeError` (or a subclass); | 304 | | this is the default. Implemented in | 305 | | :func:`strict_errors`. | 306 +-------------------------+-----------------------------------------------+ 307 | ``'ignore'`` | Ignore the malformed data and continue | 308 | | without further notice. Implemented in | 309 | | :func:`ignore_errors`. | 310 +-------------------------+-----------------------------------------------+ 311 312 The following error handlers are only applicable to 313 :term:`text encodings <text encoding>`: 314 315 .. index:: 316 single: ? (question mark); replacement character 317 single: \ (backslash); escape sequence 318 single: \x; escape sequence 319 single: \u; escape sequence 320 single: \U; escape sequence 321 single: \N; escape sequence 322 323 +-------------------------+-----------------------------------------------+ 324 | Value | Meaning | 325 +=========================+===============================================+ 326 | ``'replace'`` | Replace with a suitable replacement | 327 | | marker; Python will use the official | 328 | | ``U+FFFD`` REPLACEMENT CHARACTER for the | 329 | | built-in codecs on decoding, and '?' on | 330 | | encoding. Implemented in | 331 | | :func:`replace_errors`. | 332 +-------------------------+-----------------------------------------------+ 333 | ``'xmlcharrefreplace'`` | Replace with the appropriate XML character | 334 | | reference (only for encoding). Implemented | 335 | | in :func:`xmlcharrefreplace_errors`. | 336 +-------------------------+-----------------------------------------------+ 337 | ``'backslashreplace'`` | Replace with backslashed escape sequences. | 338 | | Implemented in | 339 | | :func:`backslashreplace_errors`. | 340 +-------------------------+-----------------------------------------------+ 341 | ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | 342 | | (only for encoding). Implemented in | 343 | | :func:`namereplace_errors`. | 344 +-------------------------+-----------------------------------------------+ 345 | ``'surrogateescape'`` | On decoding, replace byte with individual | 346 | | surrogate code ranging from ``U+DC80`` to | 347 | | ``U+DCFF``. This code will then be turned | 348 | | back into the same byte when the | 349 | | ``'surrogateescape'`` error handler is used | 350 | | when encoding the data. (See :pep:`383` for | 351 | | more.) | 352 +-------------------------+-----------------------------------------------+ 353 354 In addition, the following error handler is specific to the given codecs: 355 356 +-------------------+------------------------+-------------------------------------------+ 357 | Value | Codecs | Meaning | 358 +===================+========================+===========================================+ 359 |``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding of surrogate | 360 | | utf-16-be, utf-16-le, | codes. These codecs normally treat the | 361 | | utf-32-be, utf-32-le | presence of surrogates as an error. | 362 +-------------------+------------------------+-------------------------------------------+ 363 364 .. versionadded:: 3.1 365 The ``'surrogateescape'`` and ``'surrogatepass'`` error handlers. 366 367 .. versionchanged:: 3.4 368 The ``'surrogatepass'`` error handlers now works with utf-16\* and utf-32\* codecs. 369 370 .. versionadded:: 3.5 371 The ``'namereplace'`` error handler. 372 373 .. versionchanged:: 3.5 374 The ``'backslashreplace'`` error handlers now works with decoding and 375 translating. 376 377 The set of allowed values can be extended by registering a new named error 378 handler: 379 380 .. function:: register_error(name, error_handler) 381 382 Register the error handling function *error_handler* under the name *name*. 383 The *error_handler* argument will be called during encoding and decoding 384 in case of an error, when *name* is specified as the errors parameter. 385 386 For encoding, *error_handler* will be called with a :exc:`UnicodeEncodeError` 387 instance, which contains information about the location of the error. The 388 error handler must either raise this or a different exception, or return a 389 tuple with a replacement for the unencodable part of the input and a position 390 where encoding should continue. The replacement may be either :class:`str` or 391 :class:`bytes`. If the replacement is bytes, the encoder will simply copy 392 them into the output buffer. If the replacement is a string, the encoder will 393 encode the replacement. Encoding continues on original input at the 394 specified position. Negative position values will be treated as being 395 relative to the end of the input string. If the resulting position is out of 396 bound an :exc:`IndexError` will be raised. 397 398 Decoding and translating works similarly, except :exc:`UnicodeDecodeError` or 399 :exc:`UnicodeTranslateError` will be passed to the handler and that the 400 replacement from the error handler will be put into the output directly. 401 402 403 Previously registered error handlers (including the standard error handlers) 404 can be looked up by name: 405 406 .. function:: lookup_error(name) 407 408 Return the error handler previously registered under the name *name*. 409 410 Raises a :exc:`LookupError` in case the handler cannot be found. 411 412 The following standard error handlers are also made available as module level 413 functions: 414 415 .. function:: strict_errors(exception) 416 417 Implements the ``'strict'`` error handling: each encoding or 418 decoding error raises a :exc:`UnicodeError`. 419 420 421 .. function:: replace_errors(exception) 422 423 Implements the ``'replace'`` error handling (for :term:`text encodings 424 <text encoding>` only): substitutes ``'?'`` for encoding errors 425 (to be encoded by the codec), and ``'\ufffd'`` (the Unicode replacement 426 character) for decoding errors. 427 428 429 .. function:: ignore_errors(exception) 430 431 Implements the ``'ignore'`` error handling: malformed data is ignored and 432 encoding or decoding is continued without further notice. 433 434 435 .. function:: xmlcharrefreplace_errors(exception) 436 437 Implements the ``'xmlcharrefreplace'`` error handling (for encoding with 438 :term:`text encodings <text encoding>` only): the 439 unencodable character is replaced by an appropriate XML character reference. 440 441 442 .. function:: backslashreplace_errors(exception) 443 444 Implements the ``'backslashreplace'`` error handling (for 445 :term:`text encodings <text encoding>` only): malformed data is 446 replaced by a backslashed escape sequence. 447 448 .. function:: namereplace_errors(exception) 449 450 Implements the ``'namereplace'`` error handling (for encoding with 451 :term:`text encodings <text encoding>` only): the 452 unencodable character is replaced by a ``\N{...}`` escape sequence. 453 454 .. versionadded:: 3.5 455 456 457 .. _codec-objects: 458 459 Stateless Encoding and Decoding 460 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 461 462 The base :class:`Codec` class defines these methods which also define the 463 function interfaces of the stateless encoder and decoder: 464 465 466 .. method:: Codec.encode(input[, errors]) 467 468 Encodes the object *input* and returns a tuple (output object, length consumed). 469 For instance, :term:`text encoding` converts 470 a string object to a bytes object using a particular 471 character set encoding (e.g., ``cp1252`` or ``iso-8859-1``). 472 473 The *errors* argument defines the error handling to apply. 474 It defaults to ``'strict'`` handling. 475 476 The method may not store state in the :class:`Codec` instance. Use 477 :class:`StreamWriter` for codecs which have to keep state in order to make 478 encoding efficient. 479 480 The encoder must be able to handle zero length input and return an empty object 481 of the output object type in this situation. 482 483 484 .. method:: Codec.decode(input[, errors]) 485 486 Decodes the object *input* and returns a tuple (output object, length 487 consumed). For instance, for a :term:`text encoding`, decoding converts 488 a bytes object encoded using a particular 489 character set encoding to a string object. 490 491 For text encodings and bytes-to-bytes codecs, 492 *input* must be a bytes object or one which provides the read-only 493 buffer interface -- for example, buffer objects and memory mapped files. 494 495 The *errors* argument defines the error handling to apply. 496 It defaults to ``'strict'`` handling. 497 498 The method may not store state in the :class:`Codec` instance. Use 499 :class:`StreamReader` for codecs which have to keep state in order to make 500 decoding efficient. 501 502 The decoder must be able to handle zero length input and return an empty object 503 of the output object type in this situation. 504 505 506 Incremental Encoding and Decoding 507 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 508 509 The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide 510 the basic interface for incremental encoding and decoding. Encoding/decoding the 511 input isn't done with one call to the stateless encoder/decoder function, but 512 with multiple calls to the 513 :meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method of 514 the incremental encoder/decoder. The incremental encoder/decoder keeps track of 515 the encoding/decoding process during method calls. 516 517 The joined output of calls to the 518 :meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method is 519 the same as if all the single inputs were joined into one, and this input was 520 encoded/decoded with the stateless encoder/decoder. 521 522 523 .. _incremental-encoder-objects: 524 525 IncrementalEncoder Objects 526 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 527 528 The :class:`IncrementalEncoder` class is used for encoding an input in multiple 529 steps. It defines the following methods which every incremental encoder must 530 define in order to be compatible with the Python codec registry. 531 532 533 .. class:: IncrementalEncoder(errors='strict') 534 535 Constructor for an :class:`IncrementalEncoder` instance. 536 537 All incremental encoders must provide this constructor interface. They are free 538 to add additional keyword arguments, but only the ones defined here are used by 539 the Python codec registry. 540 541 The :class:`IncrementalEncoder` may implement different error handling schemes 542 by providing the *errors* keyword argument. See :ref:`error-handlers` for 543 possible values. 544 545 The *errors* argument will be assigned to an attribute of the same name. 546 Assigning to this attribute makes it possible to switch between different error 547 handling strategies during the lifetime of the :class:`IncrementalEncoder` 548 object. 549 550 551 .. method:: encode(object[, final]) 552 553 Encodes *object* (taking the current state of the encoder into account) 554 and returns the resulting encoded object. If this is the last call to 555 :meth:`encode` *final* must be true (the default is false). 556 557 558 .. method:: reset() 559 560 Reset the encoder to the initial state. The output is discarded: call 561 ``.encode(object, final=True)``, passing an empty byte or text string 562 if necessary, to reset the encoder and to get the output. 563 564 565 .. method:: getstate() 566 567 Return the current state of the encoder which must be an integer. The 568 implementation should make sure that ``0`` is the most common 569 state. (States that are more complicated than integers can be converted 570 into an integer by marshaling/pickling the state and encoding the bytes 571 of the resulting string into an integer). 572 573 574 .. method:: setstate(state) 575 576 Set the state of the encoder to *state*. *state* must be an encoder state 577 returned by :meth:`getstate`. 578 579 580 .. _incremental-decoder-objects: 581 582 IncrementalDecoder Objects 583 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 584 585 The :class:`IncrementalDecoder` class is used for decoding an input in multiple 586 steps. It defines the following methods which every incremental decoder must 587 define in order to be compatible with the Python codec registry. 588 589 590 .. class:: IncrementalDecoder(errors='strict') 591 592 Constructor for an :class:`IncrementalDecoder` instance. 593 594 All incremental decoders must provide this constructor interface. They are free 595 to add additional keyword arguments, but only the ones defined here are used by 596 the Python codec registry. 597 598 The :class:`IncrementalDecoder` may implement different error handling schemes 599 by providing the *errors* keyword argument. See :ref:`error-handlers` for 600 possible values. 601 602 The *errors* argument will be assigned to an attribute of the same name. 603 Assigning to this attribute makes it possible to switch between different error 604 handling strategies during the lifetime of the :class:`IncrementalDecoder` 605 object. 606 607 608 .. method:: decode(object[, final]) 609 610 Decodes *object* (taking the current state of the decoder into account) 611 and returns the resulting decoded object. If this is the last call to 612 :meth:`decode` *final* must be true (the default is false). If *final* is 613 true the decoder must decode the input completely and must flush all 614 buffers. If this isn't possible (e.g. because of incomplete byte sequences 615 at the end of the input) it must initiate error handling just like in the 616 stateless case (which might raise an exception). 617 618 619 .. method:: reset() 620 621 Reset the decoder to the initial state. 622 623 624 .. method:: getstate() 625 626 Return the current state of the decoder. This must be a tuple with two 627 items, the first must be the buffer containing the still undecoded 628 input. The second must be an integer and can be additional state 629 info. (The implementation should make sure that ``0`` is the most common 630 additional state info.) If this additional state info is ``0`` it must be 631 possible to set the decoder to the state which has no input buffered and 632 ``0`` as the additional state info, so that feeding the previously 633 buffered input to the decoder returns it to the previous state without 634 producing any output. (Additional state info that is more complicated than 635 integers can be converted into an integer by marshaling/pickling the info 636 and encoding the bytes of the resulting string into an integer.) 637 638 639 .. method:: setstate(state) 640 641 Set the state of the encoder to *state*. *state* must be a decoder state 642 returned by :meth:`getstate`. 643 644 645 Stream Encoding and Decoding 646 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 647 648 649 The :class:`StreamWriter` and :class:`StreamReader` classes provide generic 650 working interfaces which can be used to implement new encoding submodules very 651 easily. See :mod:`encodings.utf_8` for an example of how this is done. 652 653 654 .. _stream-writer-objects: 655 656 StreamWriter Objects 657 ~~~~~~~~~~~~~~~~~~~~ 658 659 The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the 660 following methods which every stream writer must define in order to be 661 compatible with the Python codec registry. 662 663 664 .. class:: StreamWriter(stream, errors='strict') 665 666 Constructor for a :class:`StreamWriter` instance. 667 668 All stream writers must provide this constructor interface. They are free to add 669 additional keyword arguments, but only the ones defined here are used by the 670 Python codec registry. 671 672 The *stream* argument must be a file-like object open for writing 673 text or binary data, as appropriate for the specific codec. 674 675 The :class:`StreamWriter` may implement different error handling schemes by 676 providing the *errors* keyword argument. See :ref:`error-handlers` for 677 the standard error handlers the underlying stream codec may support. 678 679 The *errors* argument will be assigned to an attribute of the same name. 680 Assigning to this attribute makes it possible to switch between different error 681 handling strategies during the lifetime of the :class:`StreamWriter` object. 682 683 .. method:: write(object) 684 685 Writes the object's contents encoded to the stream. 686 687 688 .. method:: writelines(list) 689 690 Writes the concatenated list of strings to the stream (possibly by reusing 691 the :meth:`write` method). The standard bytes-to-bytes codecs 692 do not support this method. 693 694 695 .. method:: reset() 696 697 Flushes and resets the codec buffers used for keeping state. 698 699 Calling this method should ensure that the data on the output is put into 700 a clean state that allows appending of new fresh data without having to 701 rescan the whole stream to recover state. 702 703 704 In addition to the above methods, the :class:`StreamWriter` must also inherit 705 all other methods and attributes from the underlying stream. 706 707 708 .. _stream-reader-objects: 709 710 StreamReader Objects 711 ~~~~~~~~~~~~~~~~~~~~ 712 713 The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the 714 following methods which every stream reader must define in order to be 715 compatible with the Python codec registry. 716 717 718 .. class:: StreamReader(stream, errors='strict') 719 720 Constructor for a :class:`StreamReader` instance. 721 722 All stream readers must provide this constructor interface. They are free to add 723 additional keyword arguments, but only the ones defined here are used by the 724 Python codec registry. 725 726 The *stream* argument must be a file-like object open for reading 727 text or binary data, as appropriate for the specific codec. 728 729 The :class:`StreamReader` may implement different error handling schemes by 730 providing the *errors* keyword argument. See :ref:`error-handlers` for 731 the standard error handlers the underlying stream codec may support. 732 733 The *errors* argument will be assigned to an attribute of the same name. 734 Assigning to this attribute makes it possible to switch between different error 735 handling strategies during the lifetime of the :class:`StreamReader` object. 736 737 The set of allowed values for the *errors* argument can be extended with 738 :func:`register_error`. 739 740 741 .. method:: read([size[, chars, [firstline]]]) 742 743 Decodes data from the stream and returns the resulting object. 744 745 The *chars* argument indicates the number of decoded 746 code points or bytes to return. The :func:`read` method will 747 never return more data than requested, but it might return less, 748 if there is not enough available. 749 750 The *size* argument indicates the approximate maximum 751 number of encoded bytes or code points to read 752 for decoding. The decoder can modify this setting as 753 appropriate. The default value -1 indicates to read and decode as much as 754 possible. This parameter is intended to 755 prevent having to decode huge files in one step. 756 757 The *firstline* flag indicates that 758 it would be sufficient to only return the first 759 line, if there are decoding errors on later lines. 760 761 The method should use a greedy read strategy meaning that it should read 762 as much data as is allowed within the definition of the encoding and the 763 given size, e.g. if optional encoding endings or state markers are 764 available on the stream, these should be read too. 765 766 767 .. method:: readline([size[, keepends]]) 768 769 Read one line from the input stream and return the decoded data. 770 771 *size*, if given, is passed as size argument to the stream's 772 :meth:`read` method. 773 774 If *keepends* is false line-endings will be stripped from the lines 775 returned. 776 777 778 .. method:: readlines([sizehint[, keepends]]) 779 780 Read all lines available on the input stream and return them as a list of 781 lines. 782 783 Line-endings are implemented using the codec's decoder method and are 784 included in the list entries if *keepends* is true. 785 786 *sizehint*, if given, is passed as the *size* argument to the stream's 787 :meth:`read` method. 788 789 790 .. method:: reset() 791 792 Resets the codec buffers used for keeping state. 793 794 Note that no stream repositioning should take place. This method is 795 primarily intended to be able to recover from decoding errors. 796 797 798 In addition to the above methods, the :class:`StreamReader` must also inherit 799 all other methods and attributes from the underlying stream. 800 801 .. _stream-reader-writer: 802 803 StreamReaderWriter Objects 804 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 805 806 The :class:`StreamReaderWriter` is a convenience class that allows wrapping 807 streams which work in both read and write modes. 808 809 The design is such that one can use the factory functions returned by the 810 :func:`lookup` function to construct the instance. 811 812 813 .. class:: StreamReaderWriter(stream, Reader, Writer, errors='strict') 814 815 Creates a :class:`StreamReaderWriter` instance. *stream* must be a file-like 816 object. *Reader* and *Writer* must be factory functions or classes providing the 817 :class:`StreamReader` and :class:`StreamWriter` interface resp. Error handling 818 is done in the same way as defined for the stream readers and writers. 819 820 :class:`StreamReaderWriter` instances define the combined interfaces of 821 :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other 822 methods and attributes from the underlying stream. 823 824 825 .. _stream-recoder-objects: 826 827 StreamRecoder Objects 828 ~~~~~~~~~~~~~~~~~~~~~ 829 830 The :class:`StreamRecoder` translates data from one encoding to another, 831 which is sometimes useful when dealing with different encoding environments. 832 833 The design is such that one can use the factory functions returned by the 834 :func:`lookup` function to construct the instance. 835 836 837 .. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors='strict') 838 839 Creates a :class:`StreamRecoder` instance which implements a two-way conversion: 840 *encode* and *decode* work on the frontend the data visible to 841 code calling :meth:`read` and :meth:`write`, while *Reader* and *Writer* 842 work on the backend the data in *stream*. 843 844 You can use these objects to do transparent transcodings from e.g. Latin-1 845 to UTF-8 and back. 846 847 The *stream* argument must be a file-like object. 848 849 The *encode* and *decode* arguments must 850 adhere to the :class:`Codec` interface. *Reader* and 851 *Writer* must be factory functions or classes providing objects of the 852 :class:`StreamReader` and :class:`StreamWriter` interface respectively. 853 854 Error handling is done in the same way as defined for the stream readers and 855 writers. 856 857 858 :class:`StreamRecoder` instances define the combined interfaces of 859 :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other 860 methods and attributes from the underlying stream. 861 862 863 .. _encodings-overview: 864 865 Encodings and Unicode 866 --------------------- 867 868 Strings are stored internally as sequences of code points in 869 range ``0x0``--``0x10FFFF``. (See :pep:`393` for 870 more details about the implementation.) 871 Once a string object is used outside of CPU and memory, endianness 872 and how these arrays are stored as bytes become an issue. As with other 873 codecs, serialising a string into a sequence of bytes is known as *encoding*, 874 and recreating the string from the sequence of bytes is known as *decoding*. 875 There are a variety of different text serialisation codecs, which are 876 collectivity referred to as :term:`text encodings <text encoding>`. 877 878 The simplest text encoding (called ``'latin-1'`` or ``'iso-8859-1'``) maps 879 the code points 0--255 to the bytes ``0x0``--``0xff``, which means that a string 880 object that contains code points above ``U+00FF`` can't be encoded with this 881 codec. Doing so will raise a :exc:`UnicodeEncodeError` that looks 882 like the following (although the details of the error message may differ): 883 ``UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in 884 position 3: ordinal not in range(256)``. 885 886 There's another group of encodings (the so called charmap encodings) that choose 887 a different subset of all Unicode code points and how these code points are 888 mapped to the bytes ``0x0``--``0xff``. To see how this is done simply open 889 e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on 890 Windows). There's a string constant with 256 characters that shows you which 891 character is mapped to which byte value. 892 893 All of these encodings can only encode 256 of the 1114112 code points 894 defined in Unicode. A simple and straightforward way that can store each Unicode 895 code point, is to store each code point as four consecutive bytes. There are two 896 possibilities: store the bytes in big endian or in little endian order. These 897 two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their 898 disadvantage is that if e.g. you use ``UTF-32-BE`` on a little endian machine you 899 will always have to swap bytes on encoding and decoding. ``UTF-32`` avoids this 900 problem: bytes will always be in natural endianness. When these bytes are read 901 by a CPU with a different endianness, then bytes have to be swapped though. To 902 be able to detect the endianness of a ``UTF-16`` or ``UTF-32`` byte sequence, 903 there's the so called BOM ("Byte Order Mark"). This is the Unicode character 904 ``U+FEFF``. This character can be prepended to every ``UTF-16`` or ``UTF-32`` 905 byte sequence. The byte swapped version of this character (``0xFFFE``) is an 906 illegal character that may not appear in a Unicode text. So when the 907 first character in an ``UTF-16`` or ``UTF-32`` byte sequence 908 appears to be a ``U+FFFE`` the bytes have to be swapped on decoding. 909 Unfortunately the character ``U+FEFF`` had a second purpose as 910 a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow 911 a word to be split. It can e.g. be used to give hints to a ligature algorithm. 912 With Unicode 4.0 using ``U+FEFF`` as a ``ZERO WIDTH NO-BREAK SPACE`` has been 913 deprecated (with ``U+2060`` (``WORD JOINER``) assuming this role). Nevertheless 914 Unicode software still must be able to handle ``U+FEFF`` in both roles: as a BOM 915 it's a device to determine the storage layout of the encoded bytes, and vanishes 916 once the byte sequence has been decoded into a string; as a ``ZERO WIDTH 917 NO-BREAK SPACE`` it's a normal character that will be decoded like any other. 918 919 There's another encoding that is able to encoding the full range of Unicode 920 characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues 921 with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two 922 parts: marker bits (the most significant bits) and payload bits. The marker bits 923 are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are 924 encoded like this (with x being payload bits, which when concatenated give the 925 Unicode character): 926 927 +-----------------------------------+----------------------------------------------+ 928 | Range | Encoding | 929 +===================================+==============================================+ 930 | ``U-00000000`` ... ``U-0000007F`` | 0xxxxxxx | 931 +-----------------------------------+----------------------------------------------+ 932 | ``U-00000080`` ... ``U-000007FF`` | 110xxxxx 10xxxxxx | 933 +-----------------------------------+----------------------------------------------+ 934 | ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx | 935 +-----------------------------------+----------------------------------------------+ 936 | ``U-00010000`` ... ``U-0010FFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | 937 +-----------------------------------+----------------------------------------------+ 938 939 The least significant bit of the Unicode character is the rightmost x bit. 940 941 As UTF-8 is an 8-bit encoding no BOM is required and any ``U+FEFF`` character in 942 the decoded string (even if it's the first character) is treated as a ``ZERO 943 WIDTH NO-BREAK SPACE``. 944 945 Without external information it's impossible to reliably determine which 946 encoding was used for encoding a string. Each charmap encoding can 947 decode any random byte sequence. However that's not possible with UTF-8, as 948 UTF-8 byte sequences have a structure that doesn't allow arbitrary byte 949 sequences. To increase the reliability with which a UTF-8 encoding can be 950 detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls 951 ``"utf-8-sig"``) for its Notepad program: Before any of the Unicode characters 952 is written to the file, a UTF-8 encoded BOM (which looks like this as a byte 953 sequence: ``0xef``, ``0xbb``, ``0xbf``) is written. As it's rather improbable 954 that any charmap encoded file starts with these byte values (which would e.g. 955 map to 956 957 | LATIN SMALL LETTER I WITH DIAERESIS 958 | RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK 959 | INVERTED QUESTION MARK 960 961 in iso-8859-1), this increases the probability that a ``utf-8-sig`` encoding can be 962 correctly guessed from the byte sequence. So here the BOM is not used to be able 963 to determine the byte order used for generating the byte sequence, but as a 964 signature that helps in guessing the encoding. On encoding the utf-8-sig codec 965 will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On 966 decoding ``utf-8-sig`` will skip those three bytes if they appear as the first 967 three bytes in the file. In UTF-8, the use of the BOM is discouraged and 968 should generally be avoided. 969 970 971 .. _standard-encodings: 972 973 Standard Encodings 974 ------------------ 975 976 Python comes with a number of codecs built-in, either implemented as C functions 977 or with dictionaries as mapping tables. The following table lists the codecs by 978 name, together with a few common aliases, and the languages for which the 979 encoding is likely used. Neither the list of aliases nor the list of languages 980 is meant to be exhaustive. Notice that spelling alternatives that only differ in 981 case or use a hyphen instead of an underscore are also valid aliases; therefore, 982 e.g. ``'utf-8'`` is a valid alias for the ``'utf_8'`` codec. 983 984 .. impl-detail:: 985 986 Some common encodings can bypass the codecs lookup machinery to 987 improve performance. These optimization opportunities are only 988 recognized by CPython for a limited set of (case insensitive) 989 aliases: utf-8, utf8, latin-1, latin1, iso-8859-1, iso8859-1, mbcs 990 (Windows only), ascii, us-ascii, utf-16, utf16, utf-32, utf32, and 991 the same using underscores instead of dashes. Using alternative 992 aliases for these encodings may result in slower execution. 993 994 .. versionchanged:: 3.6 995 Optimization opportunity recognized for us-ascii. 996 997 Many of the character sets support the same languages. They vary in individual 998 characters (e.g. whether the EURO SIGN is supported or not), and in the 999 assignment of characters to code positions. For the European languages in 1000 particular, the following variants typically exist: 1001 1002 * an ISO 8859 codeset 1003 1004 * a Microsoft Windows code page, which is typically derived from an 8859 codeset, 1005 but replaces control characters with additional graphic characters 1006 1007 * an IBM EBCDIC code page 1008 1009 * an IBM PC code page, which is ASCII compatible 1010 1011 .. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}| 1012 1013 +-----------------+--------------------------------+--------------------------------+ 1014 | Codec | Aliases | Languages | 1015 +=================+================================+================================+ 1016 | ascii | 646, us-ascii | English | 1017 +-----------------+--------------------------------+--------------------------------+ 1018 | big5 | big5-tw, csbig5 | Traditional Chinese | 1019 +-----------------+--------------------------------+--------------------------------+ 1020 | big5hkscs | big5-hkscs, hkscs | Traditional Chinese | 1021 +-----------------+--------------------------------+--------------------------------+ 1022 | cp037 | IBM037, IBM039 | English | 1023 +-----------------+--------------------------------+--------------------------------+ 1024 | cp273 | 273, IBM273, csIBM273 | German | 1025 | | | | 1026 | | | .. versionadded:: 3.4 | 1027 +-----------------+--------------------------------+--------------------------------+ 1028 | cp424 | EBCDIC-CP-HE, IBM424 | Hebrew | 1029 +-----------------+--------------------------------+--------------------------------+ 1030 | cp437 | 437, IBM437 | English | 1031 +-----------------+--------------------------------+--------------------------------+ 1032 | cp500 | EBCDIC-CP-BE, EBCDIC-CP-CH, | Western Europe | 1033 | | IBM500 | | 1034 +-----------------+--------------------------------+--------------------------------+ 1035 | cp720 | | Arabic | 1036 +-----------------+--------------------------------+--------------------------------+ 1037 | cp737 | | Greek | 1038 +-----------------+--------------------------------+--------------------------------+ 1039 | cp775 | IBM775 | Baltic languages | 1040 +-----------------+--------------------------------+--------------------------------+ 1041 | cp850 | 850, IBM850 | Western Europe | 1042 +-----------------+--------------------------------+--------------------------------+ 1043 | cp852 | 852, IBM852 | Central and Eastern Europe | 1044 +-----------------+--------------------------------+--------------------------------+ 1045 | cp855 | 855, IBM855 | Bulgarian, Byelorussian, | 1046 | | | Macedonian, Russian, Serbian | 1047 +-----------------+--------------------------------+--------------------------------+ 1048 | cp856 | | Hebrew | 1049 +-----------------+--------------------------------+--------------------------------+ 1050 | cp857 | 857, IBM857 | Turkish | 1051 +-----------------+--------------------------------+--------------------------------+ 1052 | cp858 | 858, IBM858 | Western Europe | 1053 +-----------------+--------------------------------+--------------------------------+ 1054 | cp860 | 860, IBM860 | Portuguese | 1055 +-----------------+--------------------------------+--------------------------------+ 1056 | cp861 | 861, CP-IS, IBM861 | Icelandic | 1057 +-----------------+--------------------------------+--------------------------------+ 1058 | cp862 | 862, IBM862 | Hebrew | 1059 +-----------------+--------------------------------+--------------------------------+ 1060 | cp863 | 863, IBM863 | Canadian | 1061 +-----------------+--------------------------------+--------------------------------+ 1062 | cp864 | IBM864 | Arabic | 1063 +-----------------+--------------------------------+--------------------------------+ 1064 | cp865 | 865, IBM865 | Danish, Norwegian | 1065 +-----------------+--------------------------------+--------------------------------+ 1066 | cp866 | 866, IBM866 | Russian | 1067 +-----------------+--------------------------------+--------------------------------+ 1068 | cp869 | 869, CP-GR, IBM869 | Greek | 1069 +-----------------+--------------------------------+--------------------------------+ 1070 | cp874 | | Thai | 1071 +-----------------+--------------------------------+--------------------------------+ 1072 | cp875 | | Greek | 1073 +-----------------+--------------------------------+--------------------------------+ 1074 | cp932 | 932, ms932, mskanji, ms-kanji | Japanese | 1075 +-----------------+--------------------------------+--------------------------------+ 1076 | cp949 | 949, ms949, uhc | Korean | 1077 +-----------------+--------------------------------+--------------------------------+ 1078 | cp950 | 950, ms950 | Traditional Chinese | 1079 +-----------------+--------------------------------+--------------------------------+ 1080 | cp1006 | | Urdu | 1081 +-----------------+--------------------------------+--------------------------------+ 1082 | cp1026 | ibm1026 | Turkish | 1083 +-----------------+--------------------------------+--------------------------------+ 1084 | cp1125 | 1125, ibm1125, cp866u, ruscii | Ukrainian | 1085 | | | | 1086 | | | .. versionadded:: 3.4 | 1087 +-----------------+--------------------------------+--------------------------------+ 1088 | cp1140 | ibm1140 | Western Europe | 1089 +-----------------+--------------------------------+--------------------------------+ 1090 | cp1250 | windows-1250 | Central and Eastern Europe | 1091 +-----------------+--------------------------------+--------------------------------+ 1092 | cp1251 | windows-1251 | Bulgarian, Byelorussian, | 1093 | | | Macedonian, Russian, Serbian | 1094 +-----------------+--------------------------------+--------------------------------+ 1095 | cp1252 | windows-1252 | Western Europe | 1096 +-----------------+--------------------------------+--------------------------------+ 1097 | cp1253 | windows-1253 | Greek | 1098 +-----------------+--------------------------------+--------------------------------+ 1099 | cp1254 | windows-1254 | Turkish | 1100 +-----------------+--------------------------------+--------------------------------+ 1101 | cp1255 | windows-1255 | Hebrew | 1102 +-----------------+--------------------------------+--------------------------------+ 1103 | cp1256 | windows-1256 | Arabic | 1104 +-----------------+--------------------------------+--------------------------------+ 1105 | cp1257 | windows-1257 | Baltic languages | 1106 +-----------------+--------------------------------+--------------------------------+ 1107 | cp1258 | windows-1258 | Vietnamese | 1108 +-----------------+--------------------------------+--------------------------------+ 1109 | cp65001 | | Windows only: Windows UTF-8 | 1110 | | | (``CP_UTF8``) | 1111 | | | | 1112 | | | .. versionadded:: 3.3 | 1113 +-----------------+--------------------------------+--------------------------------+ 1114 | euc_jp | eucjp, ujis, u-jis | Japanese | 1115 +-----------------+--------------------------------+--------------------------------+ 1116 | euc_jis_2004 | jisx0213, eucjis2004 | Japanese | 1117 +-----------------+--------------------------------+--------------------------------+ 1118 | euc_jisx0213 | eucjisx0213 | Japanese | 1119 +-----------------+--------------------------------+--------------------------------+ 1120 | euc_kr | euckr, korean, ksc5601, | Korean | 1121 | | ks_c-5601, ks_c-5601-1987, | | 1122 | | ksx1001, ks_x-1001 | | 1123 +-----------------+--------------------------------+--------------------------------+ 1124 | gb2312 | chinese, csiso58gb231280, | Simplified Chinese | 1125 | | euc-cn, euccn, eucgb2312-cn, | | 1126 | | gb2312-1980, gb2312-80, | | 1127 | | iso-ir-58 | | 1128 +-----------------+--------------------------------+--------------------------------+ 1129 | gbk | 936, cp936, ms936 | Unified Chinese | 1130 +-----------------+--------------------------------+--------------------------------+ 1131 | gb18030 | gb18030-2000 | Unified Chinese | 1132 +-----------------+--------------------------------+--------------------------------+ 1133 | hz | hzgb, hz-gb, hz-gb-2312 | Simplified Chinese | 1134 +-----------------+--------------------------------+--------------------------------+ 1135 | iso2022_jp | csiso2022jp, iso2022jp, | Japanese | 1136 | | iso-2022-jp | | 1137 +-----------------+--------------------------------+--------------------------------+ 1138 | iso2022_jp_1 | iso2022jp-1, iso-2022-jp-1 | Japanese | 1139 +-----------------+--------------------------------+--------------------------------+ 1140 | iso2022_jp_2 | iso2022jp-2, iso-2022-jp-2 | Japanese, Korean, Simplified | 1141 | | | Chinese, Western Europe, Greek | 1142 +-----------------+--------------------------------+--------------------------------+ 1143 | iso2022_jp_2004 | iso2022jp-2004, | Japanese | 1144 | | iso-2022-jp-2004 | | 1145 +-----------------+--------------------------------+--------------------------------+ 1146 | iso2022_jp_3 | iso2022jp-3, iso-2022-jp-3 | Japanese | 1147 +-----------------+--------------------------------+--------------------------------+ 1148 | iso2022_jp_ext | iso2022jp-ext, iso-2022-jp-ext | Japanese | 1149 +-----------------+--------------------------------+--------------------------------+ 1150 | iso2022_kr | csiso2022kr, iso2022kr, | Korean | 1151 | | iso-2022-kr | | 1152 +-----------------+--------------------------------+--------------------------------+ 1153 | latin_1 | iso-8859-1, iso8859-1, 8859, | West Europe | 1154 | | cp819, latin, latin1, L1 | | 1155 +-----------------+--------------------------------+--------------------------------+ 1156 | iso8859_2 | iso-8859-2, latin2, L2 | Central and Eastern Europe | 1157 +-----------------+--------------------------------+--------------------------------+ 1158 | iso8859_3 | iso-8859-3, latin3, L3 | Esperanto, Maltese | 1159 +-----------------+--------------------------------+--------------------------------+ 1160 | iso8859_4 | iso-8859-4, latin4, L4 | Baltic languages | 1161 +-----------------+--------------------------------+--------------------------------+ 1162 | iso8859_5 | iso-8859-5, cyrillic | Bulgarian, Byelorussian, | 1163 | | | Macedonian, Russian, Serbian | 1164 +-----------------+--------------------------------+--------------------------------+ 1165 | iso8859_6 | iso-8859-6, arabic | Arabic | 1166 +-----------------+--------------------------------+--------------------------------+ 1167 | iso8859_7 | iso-8859-7, greek, greek8 | Greek | 1168 +-----------------+--------------------------------+--------------------------------+ 1169 | iso8859_8 | iso-8859-8, hebrew | Hebrew | 1170 +-----------------+--------------------------------+--------------------------------+ 1171 | iso8859_9 | iso-8859-9, latin5, L5 | Turkish | 1172 +-----------------+--------------------------------+--------------------------------+ 1173 | iso8859_10 | iso-8859-10, latin6, L6 | Nordic languages | 1174 +-----------------+--------------------------------+--------------------------------+ 1175 | iso8859_11 | iso-8859-11, thai | Thai languages | 1176 +-----------------+--------------------------------+--------------------------------+ 1177 | iso8859_13 | iso-8859-13, latin7, L7 | Baltic languages | 1178 +-----------------+--------------------------------+--------------------------------+ 1179 | iso8859_14 | iso-8859-14, latin8, L8 | Celtic languages | 1180 +-----------------+--------------------------------+--------------------------------+ 1181 | iso8859_15 | iso-8859-15, latin9, L9 | Western Europe | 1182 +-----------------+--------------------------------+--------------------------------+ 1183 | iso8859_16 | iso-8859-16, latin10, L10 | South-Eastern Europe | 1184 +-----------------+--------------------------------+--------------------------------+ 1185 | johab | cp1361, ms1361 | Korean | 1186 +-----------------+--------------------------------+--------------------------------+ 1187 | koi8_r | | Russian | 1188 +-----------------+--------------------------------+--------------------------------+ 1189 | koi8_t | | Tajik | 1190 | | | | 1191 | | | .. versionadded:: 3.5 | 1192 +-----------------+--------------------------------+--------------------------------+ 1193 | koi8_u | | Ukrainian | 1194 +-----------------+--------------------------------+--------------------------------+ 1195 | kz1048 | kz_1048, strk1048_2002, rk1048 | Kazakh | 1196 | | | | 1197 | | | .. versionadded:: 3.5 | 1198 +-----------------+--------------------------------+--------------------------------+ 1199 | mac_cyrillic | maccyrillic | Bulgarian, Byelorussian, | 1200 | | | Macedonian, Russian, Serbian | 1201 +-----------------+--------------------------------+--------------------------------+ 1202 | mac_greek | macgreek | Greek | 1203 +-----------------+--------------------------------+--------------------------------+ 1204 | mac_iceland | maciceland | Icelandic | 1205 +-----------------+--------------------------------+--------------------------------+ 1206 | mac_latin2 | maclatin2, maccentraleurope | Central and Eastern Europe | 1207 +-----------------+--------------------------------+--------------------------------+ 1208 | mac_roman | macroman, macintosh | Western Europe | 1209 +-----------------+--------------------------------+--------------------------------+ 1210 | mac_turkish | macturkish | Turkish | 1211 +-----------------+--------------------------------+--------------------------------+ 1212 | ptcp154 | csptcp154, pt154, cp154, | Kazakh | 1213 | | cyrillic-asian | | 1214 +-----------------+--------------------------------+--------------------------------+ 1215 | shift_jis | csshiftjis, shiftjis, sjis, | Japanese | 1216 | | s_jis | | 1217 +-----------------+--------------------------------+--------------------------------+ 1218 | shift_jis_2004 | shiftjis2004, sjis_2004, | Japanese | 1219 | | sjis2004 | | 1220 +-----------------+--------------------------------+--------------------------------+ 1221 | shift_jisx0213 | shiftjisx0213, sjisx0213, | Japanese | 1222 | | s_jisx0213 | | 1223 +-----------------+--------------------------------+--------------------------------+ 1224 | utf_32 | U32, utf32 | all languages | 1225 +-----------------+--------------------------------+--------------------------------+ 1226 | utf_32_be | UTF-32BE | all languages | 1227 +-----------------+--------------------------------+--------------------------------+ 1228 | utf_32_le | UTF-32LE | all languages | 1229 +-----------------+--------------------------------+--------------------------------+ 1230 | utf_16 | U16, utf16 | all languages | 1231 +-----------------+--------------------------------+--------------------------------+ 1232 | utf_16_be | UTF-16BE | all languages | 1233 +-----------------+--------------------------------+--------------------------------+ 1234 | utf_16_le | UTF-16LE | all languages | 1235 +-----------------+--------------------------------+--------------------------------+ 1236 | utf_7 | U7, unicode-1-1-utf-7 | all languages | 1237 +-----------------+--------------------------------+--------------------------------+ 1238 | utf_8 | U8, UTF, utf8 | all languages | 1239 +-----------------+--------------------------------+--------------------------------+ 1240 | utf_8_sig | | all languages | 1241 +-----------------+--------------------------------+--------------------------------+ 1242 1243 .. versionchanged:: 3.4 1244 The utf-16\* and utf-32\* encoders no longer allow surrogate code points 1245 (``U+D800``--``U+DFFF``) to be encoded. 1246 The utf-32\* decoders no longer decode 1247 byte sequences that correspond to surrogate code points. 1248 1249 1250 Python Specific Encodings 1251 ------------------------- 1252 1253 A number of predefined codecs are specific to Python, so their codec names have 1254 no meaning outside Python. These are listed in the tables below based on the 1255 expected input and output types (note that while text encodings are the most 1256 common use case for codecs, the underlying codec infrastructure supports 1257 arbitrary data transforms rather than just text encodings). For asymmetric 1258 codecs, the stated purpose describes the encoding direction. 1259 1260 Text Encodings 1261 ^^^^^^^^^^^^^^ 1262 1263 The following codecs provide :class:`str` to :class:`bytes` encoding and 1264 :term:`bytes-like object` to :class:`str` decoding, similar to the Unicode text 1265 encodings. 1266 1267 .. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}| 1268 1269 +--------------------+---------+---------------------------+ 1270 | Codec | Aliases | Purpose | 1271 +====================+=========+===========================+ 1272 | idna | | Implements :rfc:`3490`, | 1273 | | | see also | 1274 | | | :mod:`encodings.idna`. | 1275 | | | Only ``errors='strict'`` | 1276 | | | is supported. | 1277 +--------------------+---------+---------------------------+ 1278 | mbcs | ansi, | Windows only: Encode | 1279 | | dbcs | operand according to the | 1280 | | | ANSI codepage (CP_ACP) | 1281 +--------------------+---------+---------------------------+ 1282 | oem | | Windows only: Encode | 1283 | | | operand according to the | 1284 | | | OEM codepage (CP_OEMCP) | 1285 | | | | 1286 | | | .. versionadded:: 3.6 | 1287 +--------------------+---------+---------------------------+ 1288 | palmos | | Encoding of PalmOS 3.5 | 1289 +--------------------+---------+---------------------------+ 1290 | punycode | | Implements :rfc:`3492`. | 1291 | | | Stateful codecs are not | 1292 | | | supported. | 1293 +--------------------+---------+---------------------------+ 1294 | raw_unicode_escape | | Latin-1 encoding with | 1295 | | | ``\uXXXX`` and | 1296 | | | ``\UXXXXXXXX`` for other | 1297 | | | code points. Existing | 1298 | | | backslashes are not | 1299 | | | escaped in any way. | 1300 | | | It is used in the Python | 1301 | | | pickle protocol. | 1302 +--------------------+---------+---------------------------+ 1303 | undefined | | Raise an exception for | 1304 | | | all conversions, even | 1305 | | | empty strings. The error | 1306 | | | handler is ignored. | 1307 +--------------------+---------+---------------------------+ 1308 | unicode_escape | | Encoding suitable as the | 1309 | | | contents of a Unicode | 1310 | | | literal in ASCII-encoded | 1311 | | | Python source code, | 1312 | | | except that quotes are | 1313 | | | not escaped. Decodes from | 1314 | | | Latin-1 source code. | 1315 | | | Beware that Python source | 1316 | | | code actually uses UTF-8 | 1317 | | | by default. | 1318 +--------------------+---------+---------------------------+ 1319 | unicode_internal | | Return the internal | 1320 | | | representation of the | 1321 | | | operand. Stateful codecs | 1322 | | | are not supported. | 1323 | | | | 1324 | | | .. deprecated:: 3.3 | 1325 | | | This representation is | 1326 | | | obsoleted by | 1327 | | | :pep:`393`. | 1328 +--------------------+---------+---------------------------+ 1329 1330 .. _binary-transforms: 1331 1332 Binary Transforms 1333 ^^^^^^^^^^^^^^^^^ 1334 1335 The following codecs provide binary transforms: :term:`bytes-like object` 1336 to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode` 1337 (which only produces :class:`str` output). 1338 1339 1340 .. tabularcolumns:: |l|L|L|L| 1341 1342 +----------------------+------------------+------------------------------+------------------------------+ 1343 | Codec | Aliases | Purpose | Encoder / decoder | 1344 +======================+==================+==============================+==============================+ 1345 | base64_codec [#b64]_ | base64, base_64 | Convert operand to multiline | :meth:`base64.encodebytes` / | 1346 | | | MIME base64 (the result | :meth:`base64.decodebytes` | 1347 | | | always includes a trailing | | 1348 | | | ``'\n'``) | | 1349 | | | | | 1350 | | | .. versionchanged:: 3.4 | | 1351 | | | accepts any | | 1352 | | | :term:`bytes-like object` | | 1353 | | | as input for encoding and | | 1354 | | | decoding | | 1355 +----------------------+------------------+------------------------------+------------------------------+ 1356 | bz2_codec | bz2 | Compress the operand | :meth:`bz2.compress` / | 1357 | | | using bz2 | :meth:`bz2.decompress` | 1358 +----------------------+------------------+------------------------------+------------------------------+ 1359 | hex_codec | hex | Convert operand to | :meth:`binascii.b2a_hex` / | 1360 | | | hexadecimal | :meth:`binascii.a2b_hex` | 1361 | | | representation, with two | | 1362 | | | digits per byte | | 1363 +----------------------+------------------+------------------------------+------------------------------+ 1364 | quopri_codec | quopri, | Convert operand to MIME | :meth:`quopri.encode` with | 1365 | | quotedprintable, | quoted printable | ``quotetabs=True`` / | 1366 | | quoted_printable | | :meth:`quopri.decode` | 1367 +----------------------+------------------+------------------------------+------------------------------+ 1368 | uu_codec | uu | Convert the operand using | :meth:`uu.encode` / | 1369 | | | uuencode | :meth:`uu.decode` | 1370 +----------------------+------------------+------------------------------+------------------------------+ 1371 | zlib_codec | zip, zlib | Compress the operand | :meth:`zlib.compress` / | 1372 | | | using gzip | :meth:`zlib.decompress` | 1373 +----------------------+------------------+------------------------------+------------------------------+ 1374 1375 .. [#b64] In addition to :term:`bytes-like objects <bytes-like object>`, 1376 ``'base64_codec'`` also accepts ASCII-only instances of :class:`str` for 1377 decoding 1378 1379 .. versionadded:: 3.2 1380 Restoration of the binary transforms. 1381 1382 .. versionchanged:: 3.4 1383 Restoration of the aliases for the binary transforms. 1384 1385 1386 .. _text-transforms: 1387 1388 Text Transforms 1389 ^^^^^^^^^^^^^^^ 1390 1391 The following codec provides a text transform: a :class:`str` to :class:`str` 1392 mapping. It is not supported by :meth:`str.encode` (which only produces 1393 :class:`bytes` output). 1394 1395 .. tabularcolumns:: |l|l|L| 1396 1397 +--------------------+---------+---------------------------+ 1398 | Codec | Aliases | Purpose | 1399 +====================+=========+===========================+ 1400 | rot_13 | rot13 | Returns the Caesar-cypher | 1401 | | | encryption of the operand | 1402 +--------------------+---------+---------------------------+ 1403 1404 .. versionadded:: 3.2 1405 Restoration of the ``rot_13`` text transform. 1406 1407 .. versionchanged:: 3.4 1408 Restoration of the ``rot13`` alias. 1409 1410 1411 :mod:`encodings.idna` --- Internationalized Domain Names in Applications 1412 ------------------------------------------------------------------------ 1413 1414 .. module:: encodings.idna 1415 :synopsis: Internationalized Domain Names implementation 1416 .. moduleauthor:: Martin v. Lwis 1417 1418 This module implements :rfc:`3490` (Internationalized Domain Names in 1419 Applications) and :rfc:`3492` (Nameprep: A Stringprep Profile for 1420 Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding 1421 and :mod:`stringprep`. 1422 1423 These RFCs together define a protocol to support non-ASCII characters in domain 1424 names. A domain name containing non-ASCII characters (such as 1425 ``www.Alliancefranaise.nu``) is converted into an ASCII-compatible encoding 1426 (ACE, such as ``www.xn--alliancefranaise-npb.nu``). The ACE form of the domain 1427 name is then used in all places where arbitrary characters are not allowed by 1428 the protocol, such as DNS queries, HTTP :mailheader:`Host` fields, and so 1429 on. This conversion is carried out in the application; if possible invisible to 1430 the user: The application should transparently convert Unicode domain labels to 1431 IDNA on the wire, and convert back ACE labels to Unicode before presenting them 1432 to the user. 1433 1434 Python supports this conversion in several ways: the ``idna`` codec performs 1435 conversion between Unicode and ACE, separating an input string into labels 1436 based on the separator characters defined in :rfc:`section 3.1 of RFC 3490 <3490#section-3.1>` 1437 and converting each label to ACE as required, and conversely separating an input 1438 byte string into labels based on the ``.`` separator and converting any ACE 1439 labels found into unicode. Furthermore, the :mod:`socket` module 1440 transparently converts Unicode host names to ACE, so that applications need not 1441 be concerned about converting host names themselves when they pass them to the 1442 socket module. On top of that, modules that have host names as function 1443 parameters, such as :mod:`http.client` and :mod:`ftplib`, accept Unicode host 1444 names (:mod:`http.client` then also transparently sends an IDNA hostname in the 1445 :mailheader:`Host` field if it sends that field at all). 1446 1447 When receiving host names from the wire (such as in reverse name lookup), no 1448 automatic conversion to Unicode is performed: Applications wishing to present 1449 such host names to the user should decode them to Unicode. 1450 1451 The module :mod:`encodings.idna` also implements the nameprep procedure, which 1452 performs certain normalizations on host names, to achieve case-insensitivity of 1453 international domain names, and to unify similar characters. The nameprep 1454 functions can be used directly if desired. 1455 1456 1457 .. function:: nameprep(label) 1458 1459 Return the nameprepped version of *label*. The implementation currently assumes 1460 query strings, so ``AllowUnassigned`` is true. 1461 1462 1463 .. function:: ToASCII(label) 1464 1465 Convert a label to ASCII, as specified in :rfc:`3490`. ``UseSTD3ASCIIRules`` is 1466 assumed to be false. 1467 1468 1469 .. function:: ToUnicode(label) 1470 1471 Convert a label to Unicode, as specified in :rfc:`3490`. 1472 1473 1474 :mod:`encodings.mbcs` --- Windows ANSI codepage 1475 ----------------------------------------------- 1476 1477 .. module:: encodings.mbcs 1478 :synopsis: Windows ANSI codepage 1479 1480 Encode operand according to the ANSI codepage (CP_ACP). 1481 1482 .. availability:: Windows only. 1483 1484 .. versionchanged:: 3.3 1485 Support any error handler. 1486 1487 .. versionchanged:: 3.2 1488 Before 3.2, the *errors* argument was ignored; ``'replace'`` was always used 1489 to encode, and ``'ignore'`` to decode. 1490 1491 1492 :mod:`encodings.utf_8_sig` --- UTF-8 codec with BOM signature 1493 ------------------------------------------------------------- 1494 1495 .. module:: encodings.utf_8_sig 1496 :synopsis: UTF-8 codec with BOM signature 1497 .. moduleauthor:: Walter Drwald 1498 1499 This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded 1500 BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this 1501 is only done once (on the first write to the byte stream). For decoding an 1502 optional UTF-8 encoded BOM at the start of the data will be skipped. 1503
