Storage (zarr.storage)

This module contains storage classes for use with Zarr arrays and groups. However, note that any object implementing the MutableMapping interface can be used as a Zarr array store.

zarr.storage.init_array(store, shape, chunks=None, dtype=None, compressor='default', fill_value=None, order='C', overwrite=False, path=None, chunk_store=None, filters=None)

initialize an array store with the given configuration.

Parameters:

store : MutableMapping

A mapping that supports string keys and bytes-like values.

shape : int or tuple of ints

Array shape.

chunks : int or tuple of ints, optional

Chunk shape. If not provided, will be guessed from shape and dtype.

dtype : string or dtype, optional

NumPy dtype.

compressor : Codec, optional

Primary compressor.

fill_value : object

Default value to use for uninitialized portions of the array.

order : {‘C’, ‘F’}, optional

Memory layout to be used within each chunk.

overwrite : bool, optional

If True, erase all data in store prior to initialisation.

path : string, optional

Path under which array is stored.

chunk_store : MutableMapping, optional

Separate storage for chunks. If not provided, store will be used for storage of both chunks and metadata.

filters : sequence, optional

Sequence of filters to use to encode chunk data prior to compression.

Notes

The initialisation process involves normalising all array metadata, encoding as JSON and storing under the ‘.zarray’ key. User attributes are also initialized and stored as JSON under the ‘.zattrs’ key.

Examples

Initialize an array store:

>>> from zarr.storage import init_array
>>> store = dict()
>>> init_array(store, shape=(10000, 10000), chunks=(1000, 1000))
>>> sorted(store.keys())
['.zarray', '.zattrs']

Array metadata is stored as JSON:

>>> print(str(store['.zarray'], 'ascii'))
{
    "chunks": [
        1000,
        1000
    ],
    "compressor": {
        "blocksize": 0,
        "clevel": 5,
        "cname": "lz4",
        "id": "blosc",
        "shuffle": 1
    },
    "dtype": "<f8",
    "fill_value": null,
    "filters": null,
    "order": "C",
    "shape": [
        10000,
        10000
    ],
    "zarr_format": 2
}

User-defined attributes are also stored as JSON, initially empty:

>>> print(str(store['.zattrs'], 'ascii'))
{}

Initialize an array using a storage path:

>>> store = dict()
>>> init_array(store, shape=100000000, chunks=1000000, dtype='i1',
...            path='foo')
>>> sorted(store.keys())
['.zattrs', '.zgroup', 'foo/.zarray', 'foo/.zattrs']
>>> print(str(store['foo/.zarray'], 'ascii'))
{
    "chunks": [
        1000000
    ],
    "compressor": {
        "blocksize": 0,
        "clevel": 5,
        "cname": "lz4",
        "id": "blosc",
        "shuffle": 1
    },
    "dtype": "|i1",
    "fill_value": null,
    "filters": null,
    "order": "C",
    "shape": [
        100000000
    ],
    "zarr_format": 2
}
zarr.storage.init_group(store, overwrite=False, path=None, chunk_store=None)

initialize a group store.

Parameters:

store : MutableMapping

A mapping that supports string keys and byte sequence values.

overwrite : bool, optional

If True, erase all data in store prior to initialisation.

path : string, optional

Path under which array is stored.

chunk_store : MutableMapping, optional

Separate storage for chunks. If not provided, store will be used for storage of both chunks and metadata.

class zarr.storage.DictStore(cls=<type 'dict'>)

Extended mutable mapping interface to a hierarchy of dicts.

Examples

>>> import zarr
>>> store = zarr.DictStore()
>>> store['foo'] = b'bar'
>>> store['foo']
b'bar'
>>> store['a/b/c'] = b'xxx'
>>> store['a/b/c']
b'xxx'
>>> sorted(store.keys())
['a/b/c', 'foo']
>>> store.listdir()
['a', 'foo']
>>> store.listdir('a/b')
['c']
>>> store.rmdir('a')
>>> sorted(store.keys())
['foo']
class zarr.storage.DirectoryStore(path)

Mutable Mapping interface to a directory. Keys must be strings, values must be bytes-like objects.

Parameters:

path : string

Location of directory.

Examples

>>> import zarr
>>> store = zarr.DirectoryStore('example_store')
>>> store['foo'] = b'bar'
>>> store['foo']
b'bar'
>>> open('example_store/foo', 'rb').read()
b'bar'
>>> store['a/b/c'] = b'xxx'
>>> store['a/b/c']
b'xxx'
>>> open('example_store/a/b/c', 'rb').read()
b'xxx'
>>> sorted(store.keys())
['a/b/c', 'foo']
>>> store.listdir()
['a', 'foo']
>>> store.listdir('a/b')
['c']
>>> store.rmdir('a')
>>> sorted(store.keys())
['foo']
>>> import os
>>> os.path.exists('example_store/a')
False
class zarr.storage.TempStore(suffix='', prefix='zarr', dir=None)

Directory store using a temporary directory for storage.

class zarr.storage.ZipStore(path, compression=0, allowZip64=True, mode='a')

Mutable Mapping interface to a Zip file. Keys must be strings, values must be bytes-like objects.

Parameters:

path : string

Location of file.

compression : integer, optional

Compression method to use when writing to the archive.

allowZip64 : bool, optional

If True (the default) will create ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GiB. If False will raise an exception when the ZIP file would require ZIP64 extensions.

mode : string, optional

One of ‘r’ to read an existing file, ‘w’ to truncate and write a new file, ‘a’ to append to an existing file, or ‘x’ to exclusively create and write a new file.

Notes

When modifying a ZipStore the close() method must be called otherwise essential data will not be written to the underlying zip file. The ZipStore class also supports the context manager protocol, which ensures the close() method is called on leaving the with statement.

Examples

>>> import zarr
>>> store = zarr.ZipStore('example.zip', mode='w')
>>> store['foo'] = b'bar'
>>> store['foo']
b'bar'
>>> store['a/b/c'] = b'xxx'
>>> store['a/b/c']
b'xxx'
>>> sorted(store.keys())
['a/b/c', 'foo']
>>> store.close()
>>> import zipfile
>>> zf = zipfile.ZipFile('example.zip', mode='r')
>>> sorted(zf.namelist())
['a/b/c', 'foo']
close()

Closes the underlying zip file, ensuring all records are written.

flush()

Closes the underlying zip file, ensuring all records are written, then re-opens the file for further modifications.

zarr.storage.migrate_1to2(store)

Migrate array metadata in store from Zarr format version 1 to version 2.

Parameters:

store : MutableMapping

Store to be migrated.

Notes

Version 1 did not support hierarchies, so this migration function will look for a single array in store and migrate the array metadata to version 2.