Roadmap#

  • Status: active

  • Author: Joe Hamman

  • Created On: October 31, 2023

  • Input from:

    • Davis Bennett / @d-v-b

    • Norman Rzepka / @normanrz

    • Deepak Cherian @dcherian

    • Brian Davis / @monodeldiablo

    • Oliver McCormack / @olimcc

    • Ryan Abernathey / @rabernat

    • Jack Kelly / @JackKelly

    • Martin Durrant / @martindurant

Note

This document was written in the early stages of the 3.0 refactor. Some aspects of the design have changed since this was originally written. Questions and discussion about the contents of this document should be directed to this GitHub Discussion.

Introduction#

This document lays out a design proposal for version 3.0 of the Zarr-Python package. A specific focus of the design is to bring Zarr-Python’s API up to date with the Zarr V3 specification, with the hope of enabling the development of the many features and extensions that motivated the V3 Spec. The ideas presented here are expected to result in a major release of Zarr-Python (version 3.0) including significant a number of breaking API changes. For clarity, “V3” will be used to describe the version of the Zarr specification and “3.0” will be used to describe the release tag of the Zarr-Python project.

Current status of V3 in Zarr-Python#

During the development of the V3 Specification, a prototype implementation was added to the Zarr-Python library. Since that implementation, the V3 spec evolved in significant ways and as a result, the Zarr-Python library is now out of sync with the approved spec. Downstream libraries (e.g. Xarray) have added support for this implementation and will need to migrate to the accepted spec when its available in Zarr-Python.

Goals#

  • Provide a complete implementation of Zarr V3 through the Zarr-Python API

  • Clear the way for exciting extensions / ZEPs (i.e. sharding, variable chunking, etc.)

  • Provide a developer API that can be used to implement and register V3 extensions

  • Improve the performance of Zarr-Python by streamlining the interface between the Store layer and higher level APIs (e.g. Groups and Arrays)

  • Clean up the internal and user facing APIs

  • Improve code quality and robustness (e.g. achieve 100% type hint coverage)

  • Align the Zarr-Python array API with the array API Standard

Examples of what 3.0 will enable?#

  1. Reading and writing V3 spec-compliant groups and arrays

  2. V3 extensions including sharding and variable chunking.

  3. Improved performance by leveraging concurrency when creating/reading/writing to stores (imagine a create_hierarchy(zarr_objects) function).

  4. User-developed extensions (e.g. storage-transformers) can be registered with Zarr-Python at runtime

Non-goals (of this document)#

  • Implementation of any unaccepted Zarr V3 extensions

  • Major revisions to the Zarr V3 spec

Requirements#

  1. Read and write spec compliant V2 and V3 data

  2. Limit unnecessary traffic to/from the store

  3. Cleanly define the Array/Group/Store abstractions

  4. Cleanly define how V2 will be supported going forward

  5. Provide a clear roadmap to help users upgrade to 3.0

  6. Developer tools / hooks for registering extensions

Design#

Async API#

Zarr-Python is an IO library. As such, supporting concurrent action against the storage layer is critical to achieving acceptable performance. The Zarr-Python 2 was not designed with asynchronous computation in mind and as a result has struggled to effectively leverage the benefits of concurrency. At one point, getitems and setitems support was added to the Zarr store model but that is only used for operating on a set of chunks in a single variable.

With Zarr-Python 3.0, we have the opportunity to revisit this design. The proposal here is as follows:

  1. The Store interface will be entirely async.

  2. On top of the async Store interface, we will provide an AsyncArray and AsyncGroup interface.

  3. Finally, the primary user facing API will be synchronous Array and Group classes that wrap the async equivalents.

Examples

  • Store

    class Store:
    ...
    async def get(self, key: str) -> bytes:
        ...
    async def get_partial_values(self, key_ranges: List[Tuple[str, Tuple[int, Optional[int]]]]) -> bytes:
        ...
# (no sync interface here)

  • Array

    class AsyncArray:
    ...

    async def getitem(self, selection: Selection) -> np.ndarray:
       # the core logic for getitem goes here

class Array:
    _async_array: AsyncArray

    def __getitem__(self, selection: Selection) -> np.ndarray:
        return sync(self._async_array.getitem(selection))

  • Group

    class AsyncGroup:
    ...

    async def create_group(self, path: str, **kwargs) -> AsyncGroup:
       # the core logic for create_group goes here

class Group:
    _async_group: AsyncGroup

    def create_group(self, path: str, **kwargs) -> Group:
        return sync(self._async_group.create_group(path, **kwargs))

    Internal Synchronization API

With the Store and core AsyncArray/ AsyncGroup classes being predominantly async, Zarr-Python will need an internal API to provide a synchronous API. The proposal here is to use the approach in fsspec to provide a high-level sync function that takes an awaitable and runs it in its managed IO Loop / thread.

FAQ 1. Why two levels of Arrays/groups? a. First, this is an intentional decision and departure from the current Zarrita implementation b. The idea is that users rarely want to mix interfaces. Either they are working within an async context (currently quite rare) or they are in a typical synchronous context. c. Splitting the two will allow us to clearly define behavior on the AsyncObj and simply wrap it in the SyncObj. 2. What if a store is only has a synchronous backend? a. First off, this is expected to be a fairly rare occurrence. Most storage backends have async interfaces. b. But in the event a storage backend doesn’t have a async interface, there is nothing wrong with putting synchronous code in async methods. There are approaches to enabling concurrent action through wrappers like AsyncIO’s loop.run_in_executor (ref 1, ref 2, ref 3, ref 4.
3. Will Zarr help manage the async contexts encouraged by some libraries (e.g. AioBotoCore)? a. Many async IO libraries require entering an async context before interacting with the API. We expect some experimentation to be needed here but the initial design will follow something close to what fsspec does (example in s3fs). 4. Why not provide a synchronous Store interface? a. We could but this design is simpler. It would mean supporting it in the AsyncGroup and AsyncArray classes which, may be more trouble than its worth. Storage backends that do not have an async API will be encouraged to wrap blocking calls in an async wrapper (e.g. loop.run_in_executor).

Store API#

The Store API is specified directly in the V3 specification. All V3 stores should implement this abstract API, omitting Write and List support as needed. As described above, all stores will be expected to expose the required methods as async methods.

Example

class ReadWriteStore:
        ...
    async def get(self, key: str) -> bytes:
        ...

    async def get_partial_values(self, key_ranges: List[Tuple[str, int, int]) -> bytes:
        ...

    async def set(self, key: str, value: Union[bytes, bytearray, memoryview]) -> None:
        ...  # required for writable stores

    async def set_partial_values(self, key_start_values: List[Tuple[str, int, Union[bytes, bytearray, memoryview]]]) -> None:
        ...  # required for writable stores

    async def list(self) -> List[str]:
        ...  # required for listable stores

    async def list_prefix(self, prefix: str) -> List[str]:
        ...  # required for listable stores

    async def list_dir(self, prefix: str) -> List[str]:
        ...  # required for listable stores

    # additional (optional methods)
    async def getsize(self, prefix: str) -> int:
        ...

    async def rename(self, src: str, dest: str) -> None
        ...

Recognizing that there are many Zarr applications today that rely on the MutableMapping interface supported by Zarr-Python 2, a wrapper store will be developed to allow existing stores to plug directly into this API.

Array API#

The user facing array interface will implement a subset of the Array API Standard. Most of the computational parts of the Array API Standard don’t fit into Zarr right now. That’s okay. What matters most is that we ensure we can give downstream applications a compliant API.

Note, Zarr already does most of this so this is more about formalizing the relationship than a substantial change in API.

Included

Not Included

Unknown / Maybe Possible

Attributes

dtype

mT

device

ndim

T

shape

size

Methods

__getitem__

__array_namespace__

to_device

__setitem__

__abs__

__bool__

__eq__

__add__

__complex__

__bool__

__and__

__dlpack__

__floordiv__

__dlpack_device__

__ge__

__float__

__gt__

__index__

__invert__

__int__

__le__

__lshift__

__lt__

__matmul__

__mod__

__mul__

__ne__

__neg__

__or__

__pos__

__pow__

__rshift__

__sub__

__truediv__

__xor__

Creation functions (zarr.creation)

zeros

arange

zeros_like

asarray

ones

eye

ones_like

from_dlpack

full

linspace

full_like

meshgrid

empty

tril

empty_like

triu

In addition to the core array API defined above, the Array class should have the following Zarr specific properties:

  • .metadata (see Metadata Interface below)

  • .attrs - (pulled from metadata object)

  • .info - (repolicated from existing property †)

† In Zarr-Python 2, the info property listed the store to identify initialized chunks. By default this will be turned off in 3.0 but will be configurable.

Indexing

Zarr-Python currently supports __getitem__ style indexing and the special oindex and vindex indexers. These are not part of the current Array API standard (see data-apis/array-api#669) but they have been proposed as a NEP. Zarr-Python will maintain these in 3.0.

We are also exploring a new high-level indexing API that will enabled optimized batch/concurrent loading of many chunks. We expect this to be important to enable performant loading of data in the context of sharding. See this discussion for more detail.

Concurrent indexing across multiple arrays will be possible using the AsyncArray API.

Async and Sync Array APIs

Most the logic to support Zarr Arrays will live in the AsyncArray class. There are a few notable differences that should be called out.

Sync Method

Async Method

__getitem__

getitem

__setitem__

setitem

__eq__

equals

Metadata interface

Zarr-Python 2.* closely mirrors the V2 spec metadata schema in the Array and Group classes. In 3.0, we plan to move the underlying metadata representation to a separate interface (e.g. Array.metadata). This interface will return either a V2ArrayMetadata or V3ArrayMetadata object (both will inherit from a parent ArrayMetadataABC class. The V2ArrayMetadata and V3ArrayMetadata classes will be responsible for producing valid JSON representations of their metadata, and yielding a consistent view to the Array or Group class.

Group API#

The main question is how closely we should follow the existing Zarr-Python implementation / MutableMapping interface. The table below shows the primary Group methods in Zarr-Python 2 and attempts to identify if and how they would be implemented in 3.0.

V2 Group Methods

AsyncGroup

Group

h5py_compat.Group

__len__

length

__len__

__len__

__iter__

__aiter__

__iter__

__iter__

__contains__

contains

__contains__

__contains__

__getitem__

getitem

__getitem__

__getitem__

__enter__

N/A

N/A

__enter__

__exit__

N/A

N/A

__exit__

group_keys

group_keys

group_keys

N/A

groups

groups

groups

N/A

array_keys

array_key

array_keys

N/A

arrays

arrays

arrays

N/A

visit

?

?

visit

visitkeys

?

?

?

visitvalues

?

?

?

visititems

?

?

visititems

tree

tree

tree

Both

create_group

create_group

create_group

create_group

require_group

N/A

N/A

require_group

create_groups

?

?

N/A

require_groups

?

?

?

create_dataset

N/A

N/A

create_dataset

require_dataset

N/A

N/A

require_dataset

create

create_array

create_array

N/A

empty

empty

empty

N/A

zeros

zeros

zeros

N/A

ones

ones

ones

N/A

full

full

full

N/A

array

create_array

create_array

N/A

empty_like

empty_like

empty_like

N/A

zeros_like

zeros_like

zeros_like

N/A

ones_like

ones_like

ones_like

N/A

full_like

full_like

full_like

N/A

move

move

move

move

``zarr.h5compat.Group`` – Zarr-Python 2.* made an attempt to align its API with that of h5py. With 3.0, we will relax this alignment in favor of providing an explicit compatibility module (zarr.h5py_compat). This module will expose the Group and Dataset APIs that map to Zarr-Python’s Group and Array objects.

Creation API#

Zarr-Python 2.* bundles together the creation and serialization of Zarr objects. Zarr-Python 3.* will make it possible to create objects in memory separate from serializing them. This will specifically enable writing hierarchies of Zarr objects in a single batch step. For example:

arr1 = Array(shape=(10, 10), path="foo/bar", dtype="i4", store=store)
arr2 = Array(shape=(10, 10), path="foo/spam", dtype="f8", store=store)

arr1.save()
arr2.save()

# or equivalently

zarr.save_many([arr1 ,arr2])

Note: this batch creation API likely needs additional design effort prior to implementation.

Plugin API#

Zarr V3 was designed to be extensible at multiple layers. Zarr-Python will support these extensions through a combination of Abstract Base Classes (ABCs) and Entrypoints.

ABCs

Zarr V3 will expose Abstract base classes for the following objects:

  • Store, ReadStore, ReadWriteStore, ReadListStore, and ReadWriteListStore

  • BaseArray, SynchronousArray, and AsynchronousArray

  • BaseGroup, SynchronousGroup, and AsynchronousGroup

  • Codec, ArrayArrayCodec, ArrayBytesCodec, BytesBytesCodec

Entrypoints

Lots more thinking here but the idea here is to provide entrypoints for data type, chunk grid, chunk key encoding, codecs, storage_transformers and stores. These might look something like:

entry_points="""
    [zarr.codecs]
    blosc_codec=codec_plugin:make_blosc_codec
    zlib_codec=codec_plugin:make_zlib_codec
"""

Python type hints and static analysis#

Target 100% Mypy coverage in 3.0 source.

Observability#

A persistent problem in Zarr-Python is diagnosing problems that span many parts of the stack. To address this in 3.0, we will add a basic logging framework that can be used to debug behavior at various levels of the stack. We propose to add the separate loggers for the following namespaces:

  • array

  • group

  • store

  • codec

These should be documented such that users know how to activate them and developers know how to use them when developing extensions.

Dependencies#

Today, Zarr-Python has the following required dependencies:

dependencies = [
    'asciitree',
    'numpy>=1.20,!=1.21.0',
    'fasteners',
    'numcodecs>=0.10.0',
]

What other dependencies should be considered?

  1. Attrs - Zarrita makes extensive use of the Attrs library

  2. Fsspec - Zarrita has a hard dependency on Fsspec. This could be easily relaxed though.

Breaking changes relative to Zarr-Python 2.*#

  1. H5py compat moved to a stand alone module?

  2. Group.__getitem__ support moved to Group.members.__getitem__?

  3. Others?

Open questions#

  1. How to treat V2

    1. Note: Zarrita currently implements a separate V2Array and V3Array classes. This feels less than ideal.

    2. We could easily convert metadata from v2 to the V3 Array, but what about writing?

    3. Ideally, we don’t have completely separate code paths. But if its too complicated to support both within one interface, its probably better.

  2. How and when to remove the current implementation of V3.

    1. It’s hidden behind a hard-to-use feature flag so we probably don’t need to do anything.

  3. How to model runtime configuration?

  4. Which extensions belong in Zarr-Python and which belong in separate packages?

    1. We don’t need to take a strong position on this here. It’s likely that someone will want to put Sharding in. That will be useful to develop in parallel because it will give us a good test case for the plugin interface.

Testing#

Zarr-python 3.0 adds a major new dimension to Zarr: Async support. This also comes with a compatibility risk, we will need to thoroughly test support in key execution environments. Testing plan: - Reuse the existing test suite for testing the v3 API. - xfail tests that expose breaking changes with 3.0 - breaking change description. This will help identify additional and/or unintentional breaking changes - Rework tests that were only testing internal APIs. - Add a set of functional / integration tests targeting real-world workflows in various contexts (e.g. w/ Dask)

Development process#

Zarr-Python 3.0 will introduce a number of new APIs and breaking changes to existing APIs. In order to facilitate ongoing support for Zarr-Python 2.*, we will take on the following development process:

  • Create a v3 branch that can be use for developing the core functionality apart from the main branch. This will allow us to support ongoing work and bug fixes on the main branch.

  • Put the 3.0 APIs inside a zarr.v3 module. Imports from this namespace will all be new APIs that users can develop and test against once the v3 branch is merged to main.

  • Kickstart the process by pulling in the current state of zarrita - which has many of the features described in this design.

  • Release a series of 2.* releases with the v3 namespace

  • When v3 is complete, move contents of v3 to the package root

Milestones

Below are a set of specific milestones leading toward the completion of this process. As work begins, we expect this list to grow in specificity.

  1. Port current version of Zarrita to Zarr-Python

  2. Formalize Async interface by splitting Array and Group objects into Sync and Async versions

  3. Implement “fancy” indexing operations on the AsyncArray

  4. Implement an abstract base class for the Store interface and a wrapper Store to make use of existing MutableMapping stores.

  5. Rework the existing unit test suite to use the v3 namespace.

  6. Develop a plugin interface for extensions

  7. Develop a set of functional and integration tests

  8. Work with downstream libraries (Xarray, Dask, etc.) to test new APIs

TODOs#

The following subjects are not covered in detail above but perhaps should be. Including them here so they are not forgotten.

  1. [Store] Should Zarr provide an API for caching objects after first read/list/etc. Read only stores?

  2. [Array] buffer protocol support

  3. [Array] meta_array support

  4. [Extensions] Define how Zarr-Python will consume the various plugin types

  5. [Misc] H5py compatibility requires a bit more work and a champion to drive it forward.

  6. [Misc] Define chunk_store API in 3.0

  7. [Misc] Define synchronizer API in 3.0

References#

  1. Zarr-Python repository

  2. Zarr core specification (version 3.0) — Zarr specs documentation

  3. Zarrita repository

  4. Async-Zarr

  5. Zarr-Python Discussion Topic