1 :mod:`shelve` --- Python object persistence 2 =========================================== 3 4 .. module:: shelve 5 :synopsis: Python object persistence. 6 7 **Source code:** :source:`Lib/shelve.py` 8 9 .. index:: module: pickle 10 11 -------------- 12 13 A "shelf" is a persistent, dictionary-like object. The difference with "dbm" 14 databases is that the values (not the keys!) in a shelf can be essentially 15 arbitrary Python objects --- anything that the :mod:`pickle` module can handle. 16 This includes most class instances, recursive data types, and objects containing 17 lots of shared sub-objects. The keys are ordinary strings. 18 19 20 .. function:: open(filename, flag='c', protocol=None, writeback=False) 21 22 Open a persistent dictionary. The filename specified is the base filename for 23 the underlying database. As a side-effect, an extension may be added to the 24 filename and more than one file may be created. By default, the underlying 25 database file is opened for reading and writing. The optional *flag* parameter 26 has the same interpretation as the *flag* parameter of :func:`dbm.open`. 27 28 By default, version 3 pickles are used to serialize values. The version of the 29 pickle protocol can be specified with the *protocol* parameter. 30 31 Because of Python semantics, a shelf cannot know when a mutable 32 persistent-dictionary entry is modified. By default modified objects are 33 written *only* when assigned to the shelf (see :ref:`shelve-example`). If the 34 optional *writeback* parameter is set to ``True``, all entries accessed are also 35 cached in memory, and written back on :meth:`~Shelf.sync` and 36 :meth:`~Shelf.close`; this can make it handier to mutate mutable entries in 37 the persistent dictionary, but, if many entries are accessed, it can consume 38 vast amounts of memory for the cache, and it can make the close operation 39 very slow since all accessed entries are written back (there is no way to 40 determine which accessed entries are mutable, nor which ones were actually 41 mutated). 42 43 .. note:: 44 45 Do not rely on the shelf being closed automatically; always call 46 :meth:`~Shelf.close` explicitly when you don't need it any more, or 47 use :func:`shelve.open` as a context manager:: 48 49 with shelve.open('spam') as db: 50 db['eggs'] = 'eggs' 51 52 .. warning:: 53 54 Because the :mod:`shelve` module is backed by :mod:`pickle`, it is insecure 55 to load a shelf from an untrusted source. Like with pickle, loading a shelf 56 can execute arbitrary code. 57 58 Shelf objects support all methods supported by dictionaries. This eases the 59 transition from dictionary based scripts to those requiring persistent storage. 60 61 Two additional methods are supported: 62 63 .. method:: Shelf.sync() 64 65 Write back all entries in the cache if the shelf was opened with *writeback* 66 set to :const:`True`. Also empty the cache and synchronize the persistent 67 dictionary on disk, if feasible. This is called automatically when the shelf 68 is closed with :meth:`close`. 69 70 .. method:: Shelf.close() 71 72 Synchronize and close the persistent *dict* object. Operations on a closed 73 shelf will fail with a :exc:`ValueError`. 74 75 76 .. seealso:: 77 78 `Persistent dictionary recipe <https://code.activestate.com/recipes/576642/>`_ 79 with widely supported storage formats and having the speed of native 80 dictionaries. 81 82 83 Restrictions 84 ------------ 85 86 .. index:: 87 module: dbm.ndbm 88 module: dbm.gnu 89 90 * The choice of which database package will be used (such as :mod:`dbm.ndbm` or 91 :mod:`dbm.gnu`) depends on which interface is available. Therefore it is not 92 safe to open the database directly using :mod:`dbm`. The database is also 93 (unfortunately) subject to the limitations of :mod:`dbm`, if it is used --- 94 this means that (the pickled representation of) the objects stored in the 95 database should be fairly small, and in rare cases key collisions may cause 96 the database to refuse updates. 97 98 * The :mod:`shelve` module does not support *concurrent* read/write access to 99 shelved objects. (Multiple simultaneous read accesses are safe.) When a 100 program has a shelf open for writing, no other program should have it open for 101 reading or writing. Unix file locking can be used to solve this, but this 102 differs across Unix versions and requires knowledge about the database 103 implementation used. 104 105 106 .. class:: Shelf(dict, protocol=None, writeback=False, keyencoding='utf-8') 107 108 A subclass of :class:`collections.abc.MutableMapping` which stores pickled 109 values in the *dict* object. 110 111 By default, version 3 pickles are used to serialize values. The version of the 112 pickle protocol can be specified with the *protocol* parameter. See the 113 :mod:`pickle` documentation for a discussion of the pickle protocols. 114 115 If the *writeback* parameter is ``True``, the object will hold a cache of all 116 entries accessed and write them back to the *dict* at sync and close times. 117 This allows natural operations on mutable entries, but can consume much more 118 memory and make sync and close take a long time. 119 120 The *keyencoding* parameter is the encoding used to encode keys before they 121 are used with the underlying dict. 122 123 A :class:`Shelf` object can also be used as a context manager, in which 124 case it will be automatically closed when the :keyword:`with` block ends. 125 126 .. versionchanged:: 3.2 127 Added the *keyencoding* parameter; previously, keys were always encoded in 128 UTF-8. 129 130 .. versionchanged:: 3.4 131 Added context manager support. 132 133 134 .. class:: BsdDbShelf(dict, protocol=None, writeback=False, keyencoding='utf-8') 135 136 A subclass of :class:`Shelf` which exposes :meth:`first`, :meth:`!next`, 137 :meth:`previous`, :meth:`last` and :meth:`set_location` which are available 138 in the third-party :mod:`bsddb` module from `pybsddb 139 <https://www.jcea.es/programacion/pybsddb.htm>`_ but not in other database 140 modules. The *dict* object passed to the constructor must support those 141 methods. This is generally accomplished by calling one of 142 :func:`bsddb.hashopen`, :func:`bsddb.btopen` or :func:`bsddb.rnopen`. The 143 optional *protocol*, *writeback*, and *keyencoding* parameters have the same 144 interpretation as for the :class:`Shelf` class. 145 146 147 .. class:: DbfilenameShelf(filename, flag='c', protocol=None, writeback=False) 148 149 A subclass of :class:`Shelf` which accepts a *filename* instead of a dict-like 150 object. The underlying file will be opened using :func:`dbm.open`. By 151 default, the file will be created and opened for both read and write. The 152 optional *flag* parameter has the same interpretation as for the :func:`.open` 153 function. The optional *protocol* and *writeback* parameters have the same 154 interpretation as for the :class:`Shelf` class. 155 156 157 .. _shelve-example: 158 159 Example 160 ------- 161 162 To summarize the interface (``key`` is a string, ``data`` is an arbitrary 163 object):: 164 165 import shelve 166 167 d = shelve.open(filename) # open -- file may get suffix added by low-level 168 # library 169 170 d[key] = data # store data at key (overwrites old data if 171 # using an existing key) 172 data = d[key] # retrieve a COPY of data at key (raise KeyError 173 # if no such key) 174 del d[key] # delete data stored at key (raises KeyError 175 # if no such key) 176 177 flag = key in d # true if the key exists 178 klist = list(d.keys()) # a list of all existing keys (slow!) 179 180 # as d was opened WITHOUT writeback=True, beware: 181 d['xx'] = [0, 1, 2] # this works as expected, but... 182 d['xx'].append(3) # *this doesn't!* -- d['xx'] is STILL [0, 1, 2]! 183 184 # having opened d without writeback=True, you need to code carefully: 185 temp = d['xx'] # extracts the copy 186 temp.append(5) # mutates the copy 187 d['xx'] = temp # stores the copy right back, to persist it 188 189 # or, d=shelve.open(filename,writeback=True) would let you just code 190 # d['xx'].append(5) and have it work as expected, BUT it would also 191 # consume more memory and make the d.close() operation slower. 192 193 d.close() # close it 194 195 196 .. seealso:: 197 198 Module :mod:`dbm` 199 Generic interface to ``dbm``-style databases. 200 201 Module :mod:`pickle` 202 Object serialization used by :mod:`shelve`. 203 204
