zarr.storage

Submodules

• zarr.storage.common
• zarr.storage.local
• zarr.storage.logging
• zarr.storage.memory
• zarr.storage.remote
• zarr.storage.zip

Attributes

StoreLike

Classes

LocalStore	Local file system store.
LoggingStore	Store wrapper that logs all calls to the wrapped store.
MemoryStore	In-memory store for testing purposes.
RemoteStore	A remote Store based on FSSpec
StorePath	Path-like interface for a Store.
ZipStore	Storage class using a ZIP file.

Functions

make_store_path(→ StorePath)

Convert a StoreLike object into a StorePath object.

Package Contents

class zarr.storage.LocalStore(root: pathlib.Path | str, *, read_only: bool = False)

Bases: zarr.abc.store.Store

Local file system store.

Parameters:

rootstr or Path

Directory to use as root of store.

read_onlybool

Whether the store is read-only

Attributes:

supports_writes

supports_deletes

supports_partial_writes

supports_listing

root

async clear() → None

Clear the store.

Remove all keys and values from the store.

async delete(key: str) → None

Remove a key from the store

Parameters:

keystr

async exists(key: str) → bool

Check if a key exists in the store.

Parameters:

keystr

Returns:

bool

async get(

key: str,

prototype: zarr.core.buffer.BufferPrototype | None = None,

byte_range: tuple[int | None, int | None] | None = None,

) → zarr.core.buffer.Buffer | None

Retrieve the value associated with a given key.

Parameters:

keystr

byte_rangetuple[int | None, int | None], optional

Returns:

Buffer

async get_partial_values(

prototype: zarr.core.buffer.BufferPrototype,

key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRangeRequest]],

) → list[zarr.core.buffer.Buffer | None]

Retrieve possibly partial values from given key_ranges.

Parameters:

key_rangesIterable[tuple[str, tuple[int | None, int | None]]]

Ordered set of key, range pairs, a key may occur multiple times with different ranges

Returns:

list of values, in the order of the key_ranges, may contain null/none for missing keys

async getsize(key: str) → int

Return the size, in bytes, of a value in a Store.

Parameters:

keystr

Returns:

nbytesint

The size of the value (in bytes).

Raises:

FileNotFoundError

When the given key does not exist in the store.

async list() → collections.abc.AsyncIterator[str]

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

async list_dir(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async list_prefix(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async set(key: str, value: zarr.core.buffer.Buffer) → None

Store a (key, value) pair.

Parameters:

keystr

valueBuffer

async set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) → None

Store a key to value if the key is not already present.

Parameters:

keystr

valueBuffer

async set_partial_values(

key_start_values: collections.abc.Iterable[tuple[str, int, bytes | bytearray | memoryview]],

) → None

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]

set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

root: pathlib.Path

supports_deletes: bool = True

Does the store support deletes?

supports_listing: bool = True

Does the store support listing?

supports_partial_writes: bool = True

Does the store support partial writes?

supports_writes: bool = True

Does the store support writes?

class zarr.storage.LoggingStore(

store: zarr.abc.store.Store,

log_level: str = 'DEBUG',

log_handler: logging.Handler | None = None,

)

Bases: zarr.abc.store.Store

Store wrapper that logs all calls to the wrapped store.

Parameters:

storeStore

Store to wrap

log_levelstr

Log level

log_handlerlogging.Handler

Log handler

Attributes:

counterdict

Counter of number of times each method has been called

async clear() → None

Clear the store.

Remove all keys and values from the store.

async delete(key: str) → None

Remove a key from the store

Parameters:

keystr

async delete_dir(prefix: str) → None

Remove all keys and prefixes in the store that begin with a given prefix.

async exists(key: str) → bool

Check if a key exists in the store.

Parameters:

keystr

Returns:

bool

async get(

key: str,

prototype: zarr.core.buffer.BufferPrototype,

byte_range: tuple[int | None, int | None] | None = None,

) → zarr.core.buffer.Buffer | None

Retrieve the value associated with a given key.

Parameters:

keystr

byte_rangetuple[int | None, int | None], optional

Returns:

Buffer

async get_partial_values(

prototype: zarr.core.buffer.BufferPrototype,

key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRangeRequest]],

) → list[zarr.core.buffer.Buffer | None]

Retrieve possibly partial values from given key_ranges.

Parameters:

key_rangesIterable[tuple[str, tuple[int | None, int | None]]]

Ordered set of key, range pairs, a key may occur multiple times with different ranges

Returns:

list of values, in the order of the key_ranges, may contain null/none for missing keys

async getsize(key: str) → int

Return the size, in bytes, of a value in a Store.

Parameters:

keystr

Returns:

nbytesint

The size of the value (in bytes).

Raises:

FileNotFoundError

When the given key does not exist in the store.

async getsize_prefix(prefix: str) → int

Return the size, in bytes, of all values under a prefix.

Parameters:

prefixstr

The prefix of the directory to measure.

Returns:

nbytesint

The sum of the sizes of the values in the directory (in bytes).

See also

zarr.Array.nbytes_stored

Store.getsize

Notes

getsize_prefix is just provided as a potentially faster alternative to listing all the keys under a prefix calling Store.getsize() on each.

In general, prefix should be the path of an Array or Group in the Store. Implementations may differ on the behavior when some other prefix is provided.

async is_empty(prefix: str = '') → bool

Check if the directory is empty.

Parameters:

prefixstr

Prefix of keys to check.

Returns:

bool

True if the store is empty, False otherwise.

async list() → collections.abc.AsyncIterator[str]

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

async list_dir(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async list_prefix(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

log(hint: Any = '') → collections.abc.Generator[None, None, None]

Context manager to log method calls

Each call to the wrapped store is logged to the configured logger and added to the counter dict.

async set(key: str, value: zarr.core.buffer.Buffer) → None

Store a (key, value) pair.

Parameters:

keystr

valueBuffer

async set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) → None

Store a key to value if the key is not already present.

Parameters:

keystr

valueBuffer

async set_partial_values(

key_start_values: collections.abc.Iterable[tuple[str, int, bytes | bytearray | memoryview]],

) → None

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]

set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

counter: collections.defaultdict[str, int]

log_handler = None

log_level = 'DEBUG'

property read_only: bool

Is the store read-only?

property supports_deletes: bool

Does the store support deletes?

property supports_listing: bool

Does the store support listing?

property supports_partial_writes: bool

Does the store support partial writes?

property supports_writes: bool

Does the store support writes?

class zarr.storage.MemoryStore(

store_dict: collections.abc.MutableMapping[str, zarr.core.buffer.Buffer] | None = None,

*,

read_only: bool = False,

)

Bases: zarr.abc.store.Store

In-memory store for testing purposes.

Parameters:

store_dictdict

Initial data

read_onlybool

Whether the store is read-only

Attributes:

supports_writes

supports_deletes

supports_partial_writes

supports_listing

async clear() → None

Clear the store.

Remove all keys and values from the store.

async delete(key: str) → None

Remove a key from the store

Parameters:

keystr

async exists(key: str) → bool

Check if a key exists in the store.

Parameters:

keystr

Returns:

bool

async get(

key: str,

prototype: zarr.core.buffer.BufferPrototype,

byte_range: tuple[int | None, int | None] | None = None,

) → zarr.core.buffer.Buffer | None

Retrieve the value associated with a given key.

Parameters:

keystr

byte_rangetuple[int | None, int | None], optional

Returns:

Buffer

async get_partial_values(

prototype: zarr.core.buffer.BufferPrototype,

key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRangeRequest]],

) → list[zarr.core.buffer.Buffer | None]

Retrieve possibly partial values from given key_ranges.

Parameters:

key_rangesIterable[tuple[str, tuple[int | None, int | None]]]

Ordered set of key, range pairs, a key may occur multiple times with different ranges

Returns:

list of values, in the order of the key_ranges, may contain null/none for missing keys

async list() → collections.abc.AsyncIterator[str]

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

async list_dir(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async list_prefix(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async set(

key: str,

value: zarr.core.buffer.Buffer,

byte_range: tuple[int, int] | None = None,

) → None

Store a (key, value) pair.

Parameters:

keystr

valueBuffer

async set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) → None

Store a key to value if the key is not already present.

Parameters:

keystr

valueBuffer

abstract set_partial_values(

key_start_values: collections.abc.Iterable[tuple[str, int, bytes]],

) → None

Async:

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]

set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

supports_deletes: bool = True

Does the store support deletes?

supports_listing: bool = True

Does the store support listing?

supports_partial_writes: bool = True

Does the store support partial writes?

supports_writes: bool = True

Does the store support writes?

class zarr.storage.RemoteStore(

fs: fsspec.asyn.AsyncFileSystem,

read_only: bool = False,

path: str = '/',

allowed_exceptions: tuple[type[Exception], Ellipsis] = ALLOWED_EXCEPTIONS,

)

Bases: zarr.abc.store.Store

A remote Store based on FSSpec

Parameters:

fsAsyncFileSystem

The Async FSSpec filesystem to use with this store.

read_onlybool

Whether the store is read-only

pathstr

The root path of the store. This should be a relative path and must not include the filesystem scheme.

allowed_exceptionstuple[type[Exception], …]

When fetching data, these cases will be deemed to correspond to missing keys.

Attributes:

fs

allowed_exceptions

supports_writes

supports_deletes

supports_partial_writes

supports_listing

Raises:

TypeError

If the Filesystem does not support async operations.

ValueError

If the path argument includes a scheme.

Warns:

UserWarning

If the file system (fs) was not created with asynchronous=True.

See also

RemoteStore.from_upath

RemoteStore.from_url

async clear() → None

Clear the store.

Remove all keys and values from the store.

async delete(key: str) → None

Remove a key from the store

Parameters:

keystr

async exists(key: str) → bool

Check if a key exists in the store.

Parameters:

keystr

Returns:

bool

classmethod from_upath(

upath: Any,

read_only: bool = False,

allowed_exceptions: tuple[type[Exception], Ellipsis] = ALLOWED_EXCEPTIONS,

) → RemoteStore

Create a RemoteStore from an upath object.

Parameters:

upathUPath

The upath to the root of the store.

read_onlybool

Whether the store is read-only, defaults to False.

allowed_exceptionstuple, optional

The exceptions that are allowed to be raised when accessing the store. Defaults to ALLOWED_EXCEPTIONS.

Returns:

RemoteStore

classmethod from_url(

url: str,

storage_options: dict[str, Any] | None = None,

read_only: bool = False,

allowed_exceptions: tuple[type[Exception], Ellipsis] = ALLOWED_EXCEPTIONS,

) → RemoteStore

Create a RemoteStore from a URL.

Parameters:

urlstr

The URL to the root of the store.

storage_optionsdict, optional

The options to pass to fsspec when creating the filesystem.

read_onlybool

Whether the store is read-only, defaults to False.

allowed_exceptionstuple, optional

The exceptions that are allowed to be raised when accessing the store. Defaults to ALLOWED_EXCEPTIONS.

Returns:

RemoteStore

async get(

key: str,

prototype: zarr.core.buffer.BufferPrototype,

byte_range: zarr.abc.store.ByteRangeRequest | None = None,

) → zarr.core.buffer.Buffer | None

Retrieve the value associated with a given key.

Parameters:

keystr

byte_rangetuple[int | None, int | None], optional

Returns:

Buffer

async get_partial_values(

prototype: zarr.core.buffer.BufferPrototype,

key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRangeRequest]],

) → list[zarr.core.buffer.Buffer | None]

Retrieve possibly partial values from given key_ranges.

Parameters:

key_rangesIterable[tuple[str, tuple[int | None, int | None]]]

Ordered set of key, range pairs, a key may occur multiple times with different ranges

Returns:

list of values, in the order of the key_ranges, may contain null/none for missing keys

async getsize(key: str) → int

Return the size, in bytes, of a value in a Store.

Parameters:

keystr

Returns:

nbytesint

The size of the value (in bytes).

Raises:

FileNotFoundError

When the given key does not exist in the store.

async list() → collections.abc.AsyncIterator[str]

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

async list_dir(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async list_prefix(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async set(

key: str,

value: zarr.core.buffer.Buffer,

byte_range: tuple[int, int] | None = None,

) → None

Store a (key, value) pair.

Parameters:

keystr

valueBuffer

abstract set_partial_values(

key_start_values: collections.abc.Iterable[tuple[str, int, zarr.core.common.BytesLike]],

) → None

Async:

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]

set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

allowed_exceptions: tuple[type[Exception], Ellipsis]

fs: fsspec.asyn.AsyncFileSystem

path = '/'

supports_deletes: bool = True

Does the store support deletes?

supports_listing: bool = True

Does the store support listing?

supports_partial_writes: bool = False

Does the store support partial writes?

supports_writes: bool = True

Does the store support writes?

class zarr.storage.StorePath(store: zarr.abc.store.Store, path: str = '')

Path-like interface for a Store.

Parameters:

storeStore

The store to use.

pathstr

The path within the store.

async delete() → None

Delete the key from the store.

Raises:

NotImplementedError

If the store does not support deletion.

async delete_dir() → None

Delete all keys with the given prefix from the store.

async exists() → bool

Check if the key exists in the store.

Returns:

bool

True if the key exists in the store, False otherwise.

async get(

prototype: zarr.core.buffer.BufferPrototype | None = None,

byte_range: zarr.abc.store.ByteRangeRequest | None = None,

) → zarr.core.buffer.Buffer | None

Read bytes from the store.

Parameters:

prototypeBufferPrototype, optional

The buffer prototype to use when reading the bytes.

byte_rangeByteRangeRequest, optional

The range of bytes to read.

Returns:

bufferBuffer or None

The read bytes, or None if the key does not exist.

async is_empty() → bool

Check if any keys exist in the store with the given prefix.

Returns:

bool

True if no keys exist in the store with the given prefix, False otherwise.

classmethod open(

store: zarr.abc.store.Store,

path: str,

mode: zarr.core.common.AccessModeLiteral | None = None,

) → StorePath

Async:

Open StorePath based on the provided mode.

• If the mode is ‘w-’ and the StorePath contains keys, raise a FileExistsError.

• If the mode is ‘w’, delete all keys nested within the StorePath

• If the mode is ‘a’, ‘r’, or ‘r+’, do nothing

Parameters:

modeAccessModeLiteral

The mode to use when initializing the store path.

Raises:

FileExistsError

If the mode is ‘w-’ and the store path already exists.

async set(

value: zarr.core.buffer.Buffer,

byte_range: zarr.abc.store.ByteRangeRequest | None = None,

) → None

Write bytes to the store.

Parameters:

valueBuffer

The buffer to write.

byte_rangeByteRangeRequest, optional

The range of bytes to write. If None, the entire buffer is written.

Raises:

NotImplementedError

If byte_range is not None, because Store.set does not support partial writes yet.

async set_if_not_exists(default: zarr.core.buffer.Buffer) → None

Store a key to value if the key is not already present.

Parameters:

defaultBuffer

The buffer to store if the key is not already present.

path: str

property read_only: bool

store: zarr.abc.store.Store

class zarr.storage.ZipStore(

path: pathlib.Path | str,

*,

mode: ZipStoreAccessModeLiteral = 'r',

read_only: bool | None = None,

compression: int = zipfile.ZIP_STORED,

allowZip64: bool = True,

)

Bases: zarr.abc.store.Store

Storage class using a ZIP file.

Parameters:

pathstr

Location of file.

modestr, optional

One of ‘r’ to read an existing file, ‘w’ to truncate and write a new file, ‘a’ to append to an existing file, or ‘x’ to exclusively create and write a new file.

compressionint, optional

Compression method to use when writing to the archive.

allowZip64bool, optional

If True (the default) will create ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GiB. If False will raise an exception when the ZIP file would require ZIP64 extensions.

Attributes:

allowed_exceptions

supports_writes

supports_deletes

supports_partial_writes

supports_listing

path

compression

allowZip64

async clear() → None

Clear the store.

Remove all keys and values from the store.

close() → None

Close the store.

async delete(key: str) → None

Remove a key from the store

Parameters:

keystr

async delete_dir(prefix: str) → None

Remove all keys and prefixes in the store that begin with a given prefix.

async exists(key: str) → bool

Check if a key exists in the store.

Parameters:

keystr

Returns:

bool

async get(

key: str,

prototype: zarr.core.buffer.BufferPrototype,

byte_range: zarr.abc.store.ByteRangeRequest | None = None,

) → zarr.core.buffer.Buffer | None

Retrieve the value associated with a given key.

Parameters:

keystr

byte_rangetuple[int | None, int | None], optional

Returns:

Buffer

async get_partial_values(

prototype: zarr.core.buffer.BufferPrototype,

key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRangeRequest]],

) → list[zarr.core.buffer.Buffer | None]

Retrieve possibly partial values from given key_ranges.

Parameters:

key_rangesIterable[tuple[str, tuple[int | None, int | None]]]

Ordered set of key, range pairs, a key may occur multiple times with different ranges

Returns:

list of values, in the order of the key_ranges, may contain null/none for missing keys

async list() → collections.abc.AsyncIterator[str]

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

async list_dir(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async list_prefix(prefix: str) → collections.abc.AsyncIterator[str]

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async set(key: str, value: zarr.core.buffer.Buffer) → None

Store a (key, value) pair.

Parameters:

keystr

valueBuffer

async set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) → None

Store a key to value if the key is not already present.

Parameters:

keystr

valueBuffer

abstract set_partial_values(

key_start_values: collections.abc.Iterable[tuple[str, int, bytes]],

) → None

Async:

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]

set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

allowZip64: bool

compression: int

path: pathlib.Path

supports_deletes: bool = False

Does the store support deletes?

supports_listing: bool = True

Does the store support listing?

supports_partial_writes: bool = False

Does the store support partial writes?

supports_writes: bool = True

Does the store support writes?

async zarr.storage.make_store_path(

store_like: StoreLike | None,

*,

path: str | None = '',

mode: zarr.core.common.AccessModeLiteral | None = None,

storage_options: dict[str, Any] | None = None,

) → StorePath

Convert a StoreLike object into a StorePath object.

This function takes a StoreLike object and returns a StorePath object. The StoreLike object can be a Store, StorePath, Path, str, or dict[str, Buffer]. If the StoreLike object is a Store or StorePath, it is converted to a StorePath object. If the StoreLike object is a Path or str, it is converted to a LocalStore object and then to a StorePath object. If the StoreLike object is a dict[str, Buffer], it is converted to a MemoryStore object and then to a StorePath object.

If the StoreLike object is None, a MemoryStore object is created and converted to a StorePath object.

If the StoreLike object is a str and starts with a protocol, it is converted to a RemoteStore object and then to a StorePath object.

If the StoreLike object is a dict[str, Buffer] and the mode is not None, the MemoryStore object is created with the given mode.

If the StoreLike object is a str and starts with a protocol, the RemoteStore object is created with the given mode and storage options.

Parameters:

store_likeStoreLike | None

The object to convert to a StorePath object.

pathstr | None, optional

The path to use when creating the StorePath object. If None, the default path is the empty string.

modeStoreAccessMode | None, optional

The mode to use when creating the StorePath object. If None, the default mode is ‘r’.

storage_optionsdict[str, Any] | None, optional

The storage options to use when creating the RemoteStore object. If None, the default storage options are used.

Returns:

StorePath

The converted StorePath object.

Raises:

TypeError

If the StoreLike object is not one of the supported types.

zarr.storage.StoreLike