1 .. _tut-brieftourtwo: 2 3 ********************************************** 4 Brief Tour of the Standard Library --- Part II 5 ********************************************** 6 7 This second tour covers more advanced modules that support professional 8 programming needs. These modules rarely occur in small scripts. 9 10 11 .. _tut-output-formatting: 12 13 Output Formatting 14 ================= 15 16 The :mod:`reprlib` module provides a version of :func:`repr` customized for 17 abbreviated displays of large or deeply nested containers:: 18 19 >>> import reprlib 20 >>> reprlib.repr(set('supercalifragilisticexpialidocious')) 21 "{'a', 'c', 'd', 'e', 'f', 'g', ...}" 22 23 The :mod:`pprint` module offers more sophisticated control over printing both 24 built-in and user defined objects in a way that is readable by the interpreter. 25 When the result is longer than one line, the "pretty printer" adds line breaks 26 and indentation to more clearly reveal data structure:: 27 28 >>> import pprint 29 >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta', 30 ... 'yellow'], 'blue']]] 31 ... 32 >>> pprint.pprint(t, width=30) 33 [[[['black', 'cyan'], 34 'white', 35 ['green', 'red']], 36 [['magenta', 'yellow'], 37 'blue']]] 38 39 The :mod:`textwrap` module formats paragraphs of text to fit a given screen 40 width:: 41 42 >>> import textwrap 43 >>> doc = """The wrap() method is just like fill() except that it returns 44 ... a list of strings instead of one big string with newlines to separate 45 ... the wrapped lines.""" 46 ... 47 >>> print(textwrap.fill(doc, width=40)) 48 The wrap() method is just like fill() 49 except that it returns a list of strings 50 instead of one big string with newlines 51 to separate the wrapped lines. 52 53 The :mod:`locale` module accesses a database of culture specific data formats. 54 The grouping attribute of locale's format function provides a direct way of 55 formatting numbers with group separators:: 56 57 >>> import locale 58 >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252') 59 'English_United States.1252' 60 >>> conv = locale.localeconv() # get a mapping of conventions 61 >>> x = 1234567.8 62 >>> locale.format("%d", x, grouping=True) 63 '1,234,567' 64 >>> locale.format_string("%s%.*f", (conv['currency_symbol'], 65 ... conv['frac_digits'], x), grouping=True) 66 '$1,234,567.80' 67 68 69 .. _tut-templating: 70 71 Templating 72 ========== 73 74 The :mod:`string` module includes a versatile :class:`~string.Template` class 75 with a simplified syntax suitable for editing by end-users. This allows users 76 to customize their applications without having to alter the application. 77 78 The format uses placeholder names formed by ``$`` with valid Python identifiers 79 (alphanumeric characters and underscores). Surrounding the placeholder with 80 braces allows it to be followed by more alphanumeric letters with no intervening 81 spaces. Writing ``$$`` creates a single escaped ``$``:: 82 83 >>> from string import Template 84 >>> t = Template('${village}folk send $$10 to $cause.') 85 >>> t.substitute(village='Nottingham', cause='the ditch fund') 86 'Nottinghamfolk send $10 to the ditch fund.' 87 88 The :meth:`~string.Template.substitute` method raises a :exc:`KeyError` when a 89 placeholder is not supplied in a dictionary or a keyword argument. For 90 mail-merge style applications, user supplied data may be incomplete and the 91 :meth:`~string.Template.safe_substitute` method may be more appropriate --- 92 it will leave placeholders unchanged if data is missing:: 93 94 >>> t = Template('Return the $item to $owner.') 95 >>> d = dict(item='unladen swallow') 96 >>> t.substitute(d) 97 Traceback (most recent call last): 98 ... 99 KeyError: 'owner' 100 >>> t.safe_substitute(d) 101 'Return the unladen swallow to $owner.' 102 103 Template subclasses can specify a custom delimiter. For example, a batch 104 renaming utility for a photo browser may elect to use percent signs for 105 placeholders such as the current date, image sequence number, or file format:: 106 107 >>> import time, os.path 108 >>> photofiles = ['img_1074.jpg', 'img_1076.jpg', 'img_1077.jpg'] 109 >>> class BatchRename(Template): 110 ... delimiter = '%' 111 >>> fmt = input('Enter rename style (%d-date %n-seqnum %f-format): ') 112 Enter rename style (%d-date %n-seqnum %f-format): Ashley_%n%f 113 114 >>> t = BatchRename(fmt) 115 >>> date = time.strftime('%d%b%y') 116 >>> for i, filename in enumerate(photofiles): 117 ... base, ext = os.path.splitext(filename) 118 ... newname = t.substitute(d=date, n=i, f=ext) 119 ... print('{0} --> {1}'.format(filename, newname)) 120 121 img_1074.jpg --> Ashley_0.jpg 122 img_1076.jpg --> Ashley_1.jpg 123 img_1077.jpg --> Ashley_2.jpg 124 125 Another application for templating is separating program logic from the details 126 of multiple output formats. This makes it possible to substitute custom 127 templates for XML files, plain text reports, and HTML web reports. 128 129 130 .. _tut-binary-formats: 131 132 Working with Binary Data Record Layouts 133 ======================================= 134 135 The :mod:`struct` module provides :func:`~struct.pack` and 136 :func:`~struct.unpack` functions for working with variable length binary 137 record formats. The following example shows 138 how to loop through header information in a ZIP file without using the 139 :mod:`zipfile` module. Pack codes ``"H"`` and ``"I"`` represent two and four 140 byte unsigned numbers respectively. The ``"<"`` indicates that they are 141 standard size and in little-endian byte order:: 142 143 import struct 144 145 with open('myfile.zip', 'rb') as f: 146 data = f.read() 147 148 start = 0 149 for i in range(3): # show the first 3 file headers 150 start += 14 151 fields = struct.unpack('<IIIHH', data[start:start+16]) 152 crc32, comp_size, uncomp_size, filenamesize, extra_size = fields 153 154 start += 16 155 filename = data[start:start+filenamesize] 156 start += filenamesize 157 extra = data[start:start+extra_size] 158 print(filename, hex(crc32), comp_size, uncomp_size) 159 160 start += extra_size + comp_size # skip to the next header 161 162 163 .. _tut-multi-threading: 164 165 Multi-threading 166 =============== 167 168 Threading is a technique for decoupling tasks which are not sequentially 169 dependent. Threads can be used to improve the responsiveness of applications 170 that accept user input while other tasks run in the background. A related use 171 case is running I/O in parallel with computations in another thread. 172 173 The following code shows how the high level :mod:`threading` module can run 174 tasks in background while the main program continues to run:: 175 176 import threading, zipfile 177 178 class AsyncZip(threading.Thread): 179 def __init__(self, infile, outfile): 180 threading.Thread.__init__(self) 181 self.infile = infile 182 self.outfile = outfile 183 184 def run(self): 185 f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED) 186 f.write(self.infile) 187 f.close() 188 print('Finished background zip of:', self.infile) 189 190 background = AsyncZip('mydata.txt', 'myarchive.zip') 191 background.start() 192 print('The main program continues to run in foreground.') 193 194 background.join() # Wait for the background task to finish 195 print('Main program waited until background was done.') 196 197 The principal challenge of multi-threaded applications is coordinating threads 198 that share data or other resources. To that end, the threading module provides 199 a number of synchronization primitives including locks, events, condition 200 variables, and semaphores. 201 202 While those tools are powerful, minor design errors can result in problems that 203 are difficult to reproduce. So, the preferred approach to task coordination is 204 to concentrate all access to a resource in a single thread and then use the 205 :mod:`queue` module to feed that thread with requests from other threads. 206 Applications using :class:`~queue.Queue` objects for inter-thread communication and 207 coordination are easier to design, more readable, and more reliable. 208 209 210 .. _tut-logging: 211 212 Logging 213 ======= 214 215 The :mod:`logging` module offers a full featured and flexible logging system. 216 At its simplest, log messages are sent to a file or to ``sys.stderr``:: 217 218 import logging 219 logging.debug('Debugging information') 220 logging.info('Informational message') 221 logging.warning('Warning:config file %s not found', 'server.conf') 222 logging.error('Error occurred') 223 logging.critical('Critical error -- shutting down') 224 225 This produces the following output: 226 227 .. code-block:: none 228 229 WARNING:root:Warning:config file server.conf not found 230 ERROR:root:Error occurred 231 CRITICAL:root:Critical error -- shutting down 232 233 By default, informational and debugging messages are suppressed and the output 234 is sent to standard error. Other output options include routing messages 235 through email, datagrams, sockets, or to an HTTP Server. New filters can select 236 different routing based on message priority: :const:`~logging.DEBUG`, 237 :const:`~logging.INFO`, :const:`~logging.WARNING`, :const:`~logging.ERROR`, 238 and :const:`~logging.CRITICAL`. 239 240 The logging system can be configured directly from Python or can be loaded from 241 a user editable configuration file for customized logging without altering the 242 application. 243 244 245 .. _tut-weak-references: 246 247 Weak References 248 =============== 249 250 Python does automatic memory management (reference counting for most objects and 251 :term:`garbage collection` to eliminate cycles). The memory is freed shortly 252 after the last reference to it has been eliminated. 253 254 This approach works fine for most applications but occasionally there is a need 255 to track objects only as long as they are being used by something else. 256 Unfortunately, just tracking them creates a reference that makes them permanent. 257 The :mod:`weakref` module provides tools for tracking objects without creating a 258 reference. When the object is no longer needed, it is automatically removed 259 from a weakref table and a callback is triggered for weakref objects. Typical 260 applications include caching objects that are expensive to create:: 261 262 >>> import weakref, gc 263 >>> class A: 264 ... def __init__(self, value): 265 ... self.value = value 266 ... def __repr__(self): 267 ... return str(self.value) 268 ... 269 >>> a = A(10) # create a reference 270 >>> d = weakref.WeakValueDictionary() 271 >>> d['primary'] = a # does not create a reference 272 >>> d['primary'] # fetch the object if it is still alive 273 10 274 >>> del a # remove the one reference 275 >>> gc.collect() # run garbage collection right away 276 0 277 >>> d['primary'] # entry was automatically removed 278 Traceback (most recent call last): 279 File "<stdin>", line 1, in <module> 280 d['primary'] # entry was automatically removed 281 File "C:/python37/lib/weakref.py", line 46, in __getitem__ 282 o = self.data[key]() 283 KeyError: 'primary' 284 285 286 .. _tut-list-tools: 287 288 Tools for Working with Lists 289 ============================ 290 291 Many data structure needs can be met with the built-in list type. However, 292 sometimes there is a need for alternative implementations with different 293 performance trade-offs. 294 295 The :mod:`array` module provides an :class:`~array.array()` object that is like 296 a list that stores only homogeneous data and stores it more compactly. The 297 following example shows an array of numbers stored as two byte unsigned binary 298 numbers (typecode ``"H"``) rather than the usual 16 bytes per entry for regular 299 lists of Python int objects:: 300 301 >>> from array import array 302 >>> a = array('H', [4000, 10, 700, 22222]) 303 >>> sum(a) 304 26932 305 >>> a[1:3] 306 array('H', [10, 700]) 307 308 The :mod:`collections` module provides a :class:`~collections.deque()` object 309 that is like a list with faster appends and pops from the left side but slower 310 lookups in the middle. These objects are well suited for implementing queues 311 and breadth first tree searches:: 312 313 >>> from collections import deque 314 >>> d = deque(["task1", "task2", "task3"]) 315 >>> d.append("task4") 316 >>> print("Handling", d.popleft()) 317 Handling task1 318 319 :: 320 321 unsearched = deque([starting_node]) 322 def breadth_first_search(unsearched): 323 node = unsearched.popleft() 324 for m in gen_moves(node): 325 if is_goal(m): 326 return m 327 unsearched.append(m) 328 329 In addition to alternative list implementations, the library also offers other 330 tools such as the :mod:`bisect` module with functions for manipulating sorted 331 lists:: 332 333 >>> import bisect 334 >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')] 335 >>> bisect.insort(scores, (300, 'ruby')) 336 >>> scores 337 [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')] 338 339 The :mod:`heapq` module provides functions for implementing heaps based on 340 regular lists. The lowest valued entry is always kept at position zero. This 341 is useful for applications which repeatedly access the smallest element but do 342 not want to run a full list sort:: 343 344 >>> from heapq import heapify, heappop, heappush 345 >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0] 346 >>> heapify(data) # rearrange the list into heap order 347 >>> heappush(data, -5) # add a new entry 348 >>> [heappop(data) for i in range(3)] # fetch the three smallest entries 349 [-5, 0, 1] 350 351 352 .. _tut-decimal-fp: 353 354 Decimal Floating Point Arithmetic 355 ================================= 356 357 The :mod:`decimal` module offers a :class:`~decimal.Decimal` datatype for 358 decimal floating point arithmetic. Compared to the built-in :class:`float` 359 implementation of binary floating point, the class is especially helpful for 360 361 * financial applications and other uses which require exact decimal 362 representation, 363 * control over precision, 364 * control over rounding to meet legal or regulatory requirements, 365 * tracking of significant decimal places, or 366 * applications where the user expects the results to match calculations done by 367 hand. 368 369 For example, calculating a 5% tax on a 70 cent phone charge gives different 370 results in decimal floating point and binary floating point. The difference 371 becomes significant if the results are rounded to the nearest cent:: 372 373 >>> from decimal import * 374 >>> round(Decimal('0.70') * Decimal('1.05'), 2) 375 Decimal('0.74') 376 >>> round(.70 * 1.05, 2) 377 0.73 378 379 The :class:`~decimal.Decimal` result keeps a trailing zero, automatically 380 inferring four place significance from multiplicands with two place 381 significance. Decimal reproduces mathematics as done by hand and avoids 382 issues that can arise when binary floating point cannot exactly represent 383 decimal quantities. 384 385 Exact representation enables the :class:`~decimal.Decimal` class to perform 386 modulo calculations and equality tests that are unsuitable for binary floating 387 point:: 388 389 >>> Decimal('1.00') % Decimal('.10') 390 Decimal('0.00') 391 >>> 1.00 % 0.10 392 0.09999999999999995 393 394 >>> sum([Decimal('0.1')]*10) == Decimal('1.0') 395 True 396 >>> sum([0.1]*10) == 1.0 397 False 398 399 The :mod:`decimal` module provides arithmetic with as much precision as needed:: 400 401 >>> getcontext().prec = 36 402 >>> Decimal(1) / Decimal(7) 403 Decimal('0.142857142857142857142857142857142857') 404 405 406
