zarr.storage#

Attributes#

StoreLike

Classes#

`FsspecStore`	A remote Store based on FSSpec
`GpuMemoryStore`	A GPU only memory store that stores every chunk in GPU memory irrespective
`LocalStore`	Local file system store.
`LoggingStore`	Store wrapper that logs all calls to the wrapped store.
`MemoryStore`	In-memory store.
`StorePath`	Path-like interface for a Store.
`WrapperStore`	A store class that wraps an existing `Store` instance.
`ZipStore`	Storage class using a ZIP file.

Package Contents#

class zarr.storage.FsspecStore( fs: fsspec.asyn.AsyncFileSystem, read_only: bool = False, path: str = '/', allowed_exceptions: tuple[type[Exception], Ellipsis] = ALLOWED_EXCEPTIONS, )[source]#

Bases: zarr.abc.store.Store

A remote Store based on FSSpec

Parameters:

fsAsyncFileSystem: The Async FSSpec filesystem to use with this store.
read_onlybool: Whether the store is read-only
pathstr: The root path of the store. This should be a relative path and must not include the filesystem scheme.
allowed_exceptionstuple[type[Exception], …]: When fetching data, these cases will be deemed to correspond to missing keys.

Attributes:

fs
allowed_exceptions
supports_writes
supports_deletes
supports_partial_writes
supports_listing

Raises:

TypeError: If the Filesystem does not support async operations.
ValueError: If the path argument includes a scheme.

Warns:

UserWarning: If the file system (fs) was not created with asynchronous=True.

See also

zarr.Array.nbytes_stored
Store.getsize

Notes

getsize_prefix is just provided as a potentially faster alternative to listing all the keys under a prefix calling Store.getsize() on each.

In general, prefix should be the path of an Array or Group in the Store. Implementations may differ on the behavior when some other prefix is provided.

async is_empty(prefix: str = '') → bool[source]#

Check if the directory is empty.

Parameters:

prefixstr: Prefix of keys to check.

Returns:

bool: True if the store is empty, False otherwise.

async list() → collections.abc.AsyncGenerator[str, None][source]#

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

async list_dir(prefix: str) → collections.abc.AsyncGenerator[str, None][source]#

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async list_prefix(prefix: str) → collections.abc.AsyncGenerator[str, None][source]#

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

log(hint: Any = '') → collections.abc.Generator[None, None, None][source]#

Context manager to log method calls

Each call to the wrapped store is logged to the configured logger and added to the counter dict.

async set(key: str, value: zarr.core.buffer.Buffer) → None[source]#

Store a (key, value) pair.

Parameters:

keystr
valueBuffer

async set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) → None[source]#

Store a key to value if the key is not already present.

Parameters:

keystr
valueBuffer

async set_partial_values( key_start_values: collections.abc.Iterable[tuple[str, int, bytes | bytearray | memoryview]], ) → None[source]#

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]: set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

counter: collections.defaultdict[str, int]#

log_handler = None#

log_level = 'DEBUG'#

property read_only: bool#: Is the store read-only?

property supports_deletes: bool#: Does the store support deletes?

property supports_listing: bool#: Does the store support listing?

property supports_partial_writes: bool#: Does the store support partial writes?

property supports_writes: bool#: Does the store support writes?

class zarr.storage.MemoryStore( store_dict: collections.abc.MutableMapping[str, zarr.core.buffer.Buffer] | None = None, *, read_only: bool = False, )[source]#

Bases: zarr.abc.store.Store

In-memory store.

Parameters:

store_dictdict: Initial data
read_onlybool: Whether the store is read-only

Attributes:

supports_writes
supports_deletes
supports_partial_writes
supports_listing

async clear() → None[source]#

Clear the store.

Remove all keys and values from the store.

async delete(key: str) → None[source]#

Remove a key from the store

Parameters:

keystr

async exists(key: str) → bool[source]#

Check if a key exists in the store.

Parameters:

keystr

Returns:

bool

async get( key: str, prototype: zarr.core.buffer.BufferPrototype, byte_range: zarr.abc.store.ByteRequest | None = None, ) → zarr.core.buffer.Buffer | None[source]#

Retrieve the value associated with a given key.

Parameters:

keystr

byte_rangeByteRequest, optional

ByteRequest may be one of the following. If not provided, all data associated with the key is retrieved.

RangeByteRequest(int, int): Request a specific range of bytes in the form (start, end). The end is exclusive. If the given range is zero-length or starts after the end of the object, an error will be returned. Additionally, if the range ends after the end of the object, the entire remainder of the object will be returned. Otherwise, the exact requested range will be returned.
OffsetByteRequest(int): Request all bytes starting from a given byte offset. This is equivalent to bytes={int}- as an HTTP header.
SuffixByteRequest(int): Request the last int bytes. Note that here, int is the size of the request, not the byte offset. This is equivalent to bytes=-{int} as an HTTP header.

Returns:

Buffer

async get_partial_values( prototype: zarr.core.buffer.BufferPrototype, key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRequest | None]], ) → list[zarr.core.buffer.Buffer | None][source]#

Retrieve possibly partial values from given key_ranges.

Parameters:

key_rangesIterable[tuple[str, tuple[int | None, int | None]]]: Ordered set of key, range pairs, a key may occur multiple times with different ranges

Returns:

list of values, in the order of the key_ranges, may contain null/none for missing keys

async list() → collections.abc.AsyncIterator[str][source]#

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

async list_dir(prefix: str) → collections.abc.AsyncIterator[str][source]#

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async list_prefix(prefix: str) → collections.abc.AsyncIterator[str][source]#

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async set( key: str, value: zarr.core.buffer.Buffer, byte_range: tuple[int, int] | None = None, ) → None[source]#

Store a (key, value) pair.

Parameters:

keystr
valueBuffer

async set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) → None[source]#

Store a key to value if the key is not already present.

Parameters:

keystr
valueBuffer

abstract set_partial_values( key_start_values: collections.abc.Iterable[tuple[str, int, bytes]], ) → None[source]#

Async:

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]: set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

supports_deletes: bool = True#: Does the store support deletes?

supports_listing: bool = True#: Does the store support listing?

supports_partial_writes: bool = True#: Does the store support partial writes?

supports_writes: bool = True#: Does the store support writes?

class zarr.storage.StorePath(store: zarr.abc.store.Store, path: str = '')[source]#

Path-like interface for a Store.

Parameters:

storeStore: The store to use.
pathstr: The path within the store.

async delete() → None[source]#

Delete the key from the store.

Raises:

NotImplementedError: If the store does not support deletion.

async delete_dir() → None[source]#: Delete all keys with the given prefix from the store.

async exists() → bool[source]#

Check if the key exists in the store.

Returns:

bool: True if the key exists in the store, False otherwise.

async get( prototype: zarr.core.buffer.BufferPrototype | None = None, byte_range: zarr.abc.store.ByteRequest | None = None, ) → zarr.core.buffer.Buffer | None[source]#

Read bytes from the store.

Parameters:

prototypeBufferPrototype, optional: The buffer prototype to use when reading the bytes.
byte_rangeByteRequest, optional: The range of bytes to read.

Returns:

bufferBuffer or None: The read bytes, or None if the key does not exist.

async is_empty() → bool[source]#

Check if any keys exist in the store with the given prefix.

Returns:

bool: True if no keys exist in the store with the given prefix, False otherwise.

classmethod open( store: zarr.abc.store.Store, path: str, mode: zarr.core.common.AccessModeLiteral | None = None, ) → StorePath[source]#

Async:

Open StorePath based on the provided mode.

If the mode is ‘w-’ and the StorePath contains keys, raise a FileExistsError.
If the mode is ‘w’, delete all keys nested within the StorePath
If the mode is ‘a’, ‘r’, or ‘r+’, do nothing

Parameters:

modeAccessModeLiteral: The mode to use when initializing the store path.

Raises:

FileExistsError: If the mode is ‘w-’ and the store path already exists.

async set( value: zarr.core.buffer.Buffer, byte_range: zarr.abc.store.ByteRequest | None = None, ) → None[source]#

Write bytes to the store.

Parameters:

valueBuffer: The buffer to write.
byte_rangeByteRequest, optional: The range of bytes to write. If None, the entire buffer is written.

Raises:

NotImplementedError: If byte_range is not None, because Store.set does not support partial writes yet.

async set_if_not_exists(default: zarr.core.buffer.Buffer) → None[source]#

Store a key to value if the key is not already present.

Parameters:

defaultBuffer: The buffer to store if the key is not already present.

path: str#

property read_only: bool#

store: zarr.abc.store.Store#

class zarr.storage.WrapperStore(store: T_Store)[source]#

Bases: zarr.abc.store.Store, Generic[T_Store]

A store class that wraps an existing Store instance. By default all of the store methods are delegated to the wrapped store instance, which is accessible via the ._store attribute of this class.

Use this class to modify or extend the behavior of the other store classes.

async clear() → None[source]#

Clear the store.

Remove all keys and values from the store.

close() → None[source]#: Close the store.

async delete(key: str) → None[source]#

Remove a key from the store

Parameters:

keystr

async delete_dir(prefix: str) → None[source]#: Remove all keys and prefixes in the store that begin with a given prefix.

async exists(key: str) → bool[source]#

Check if a key exists in the store.

Parameters:

keystr

Returns:

bool

async get( key: str, prototype: zarr.core.buffer.BufferPrototype, byte_range: zarr.abc.store.ByteRequest | None = None, ) → zarr.core.buffer.Buffer | None[source]#

Retrieve the value associated with a given key.

Parameters:

keystr

byte_rangeByteRequest, optional

ByteRequest may be one of the following. If not provided, all data associated with the key is retrieved.

RangeByteRequest(int, int): Request a specific range of bytes in the form (start, end). The end is exclusive. If the given range is zero-length or starts after the end of the object, an error will be returned. Additionally, if the range ends after the end of the object, the entire remainder of the object will be returned. Otherwise, the exact requested range will be returned.
OffsetByteRequest(int): Request all bytes starting from a given byte offset. This is equivalent to bytes={int}- as an HTTP header.
SuffixByteRequest(int): Request the last int bytes. Note that here, int is the size of the request, not the byte offset. This is equivalent to bytes=-{int} as an HTTP header.

Returns:

Buffer

async get_partial_values( prototype: zarr.core.buffer.BufferPrototype, key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRequest | None]], ) → list[zarr.core.buffer.Buffer | None][source]#

Retrieve possibly partial values from given key_ranges.

Parameters:

key_rangesIterable[tuple[str, tuple[int | None, int | None]]]: Ordered set of key, range pairs, a key may occur multiple times with different ranges

Returns:

list of values, in the order of the key_ranges, may contain null/none for missing keys

async is_empty(prefix: str) → bool[source]#

Check if the directory is empty.

Parameters:

prefixstr: Prefix of keys to check.

Returns:

bool: True if the store is empty, False otherwise.

list() → collections.abc.AsyncIterator[str][source]#

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

list_dir(prefix: str) → collections.abc.AsyncIterator[str][source]#

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

list_prefix(prefix: str) → collections.abc.AsyncIterator[str][source]#

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

classmethod open(store_cls: type[T_Store], *args: Any, **kwargs: Any) → Self[source]#

Async:

Create and open the store.

Parameters:

*argsAny: Positional arguments to pass to the store constructor.
**kwargsAny: Keyword arguments to pass to the store constructor.

Returns:

Store: The opened store instance.

async set(key: str, value: zarr.core.buffer.Buffer) → None[source]#

Store a (key, value) pair.

Parameters:

keystr
valueBuffer

async set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) → None[source]#

Store a key to value if the key is not already present.

Parameters:

keystr
valueBuffer

async set_partial_values( key_start_values: collections.abc.Iterable[tuple[str, int, zarr.core.common.BytesLike]], ) → None[source]#

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]: set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

property read_only: bool#: Is the store read-only?

property supports_deletes: bool#: Does the store support deletes?

property supports_listing: bool#: Does the store support listing?

property supports_partial_writes: bool#: Does the store support partial writes?

property supports_writes: bool#: Does the store support writes?

class zarr.storage.ZipStore( path: pathlib.Path | str, *, mode: ZipStoreAccessModeLiteral = 'r', read_only: bool | None = None, compression: int = zipfile.ZIP_STORED, allowZip64: bool = True, )[source]#

Bases: zarr.abc.store.Store

Storage class using a ZIP file.

Parameters:

pathstr: Location of file.
modestr, optional: One of ‘r’ to read an existing file, ‘w’ to truncate and write a new file, ‘a’ to append to an existing file, or ‘x’ to exclusively create and write a new file.
compressionint, optional: Compression method to use when writing to the archive.
allowZip64bool, optional: If True (the default) will create ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GiB. If False will raise an exception when the ZIP file would require ZIP64 extensions.

Attributes:

allowed_exceptions
supports_writes
supports_deletes
supports_partial_writes
supports_listing
path
compression
allowZip64

async clear() → None[source]#

Clear the store.

Remove all keys and values from the store.

close() → None[source]#: Close the store.

async delete(key: str) → None[source]#

Remove a key from the store

Parameters:

keystr

async delete_dir(prefix: str) → None[source]#: Remove all keys and prefixes in the store that begin with a given prefix.

async exists(key: str) → bool[source]#

Check if a key exists in the store.

Parameters:

keystr

Returns:

bool

async get( key: str, prototype: zarr.core.buffer.BufferPrototype, byte_range: zarr.abc.store.ByteRequest | None = None, ) → zarr.core.buffer.Buffer | None[source]#

Retrieve the value associated with a given key.

Parameters:

keystr

byte_rangeByteRequest, optional

ByteRequest may be one of the following. If not provided, all data associated with the key is retrieved.

RangeByteRequest(int, int): Request a specific range of bytes in the form (start, end). The end is exclusive. If the given range is zero-length or starts after the end of the object, an error will be returned. Additionally, if the range ends after the end of the object, the entire remainder of the object will be returned. Otherwise, the exact requested range will be returned.
OffsetByteRequest(int): Request all bytes starting from a given byte offset. This is equivalent to bytes={int}- as an HTTP header.
SuffixByteRequest(int): Request the last int bytes. Note that here, int is the size of the request, not the byte offset. This is equivalent to bytes=-{int} as an HTTP header.

Returns:

Buffer

async get_partial_values( prototype: zarr.core.buffer.BufferPrototype, key_ranges: collections.abc.Iterable[tuple[str, zarr.abc.store.ByteRequest | None]], ) → list[zarr.core.buffer.Buffer | None][source]#

Retrieve possibly partial values from given key_ranges.

Parameters:

key_rangesIterable[tuple[str, tuple[int | None, int | None]]]: Ordered set of key, range pairs, a key may occur multiple times with different ranges

Returns:

list of values, in the order of the key_ranges, may contain null/none for missing keys

async list() → collections.abc.AsyncIterator[str][source]#

Retrieve all keys in the store.

Returns:

AsyncIterator[str]

async list_dir(prefix: str) → collections.abc.AsyncIterator[str][source]#

Retrieve all keys and prefixes with a given prefix and which do not contain the character “/” after the given prefix.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async list_prefix(prefix: str) → collections.abc.AsyncIterator[str][source]#

Retrieve all keys in the store that begin with a given prefix. Keys are returned relative to the root of the store.

Parameters:

prefixstr

Returns:

AsyncIterator[str]

async set(key: str, value: zarr.core.buffer.Buffer) → None[source]#

Store a (key, value) pair.

Parameters:

keystr
valueBuffer

async set_if_not_exists(key: str, value: zarr.core.buffer.Buffer) → None[source]#

Store a key to value if the key is not already present.

Parameters:

keystr
valueBuffer

abstract set_partial_values( key_start_values: collections.abc.Iterable[tuple[str, int, bytes]], ) → None[source]#

Async:

Store values at a given key, starting at byte range_start.

Parameters:

key_start_valueslist[tuple[str, int, BytesLike]]: set of key, range_start, values triples, a key may occur multiple times with different range_starts, range_starts (considering the length of the respective values) must not specify overlapping ranges for the same key

allowZip64: bool#

compression: int#

path: pathlib.Path#

supports_deletes: bool = False#: Does the store support deletes?

supports_listing: bool = True#: Does the store support listing?

supports_partial_writes: bool = False#: Does the store support partial writes?

supports_writes: bool = True#: Does the store support writes?

zarr.storage.StoreLike#

zarr.storage#

Attributes#

Classes#

Package Contents#

This Page