Storage guide¶

Zarr-Python supports multiple storage backends, including: local file systems, Zip files, remote stores via fsspec (S3, HTTP, etc.), and in-memory stores. In Zarr-Python 3, stores must implement the abstract store API from zarr.abc.store.Store.

Note

Unlike Zarr-Python 2 where the store interface was built around a generic MutableMapping API, Zarr-Python 3 utilizes a custom store API that utilizes Python's AsyncIO library.

Implicit Store Creation¶

In most cases, it is not required to create a Store object explicitly. Passing a string (or other StoreLike value) to Zarr's top level API will result in the store being created automatically:

import zarr

# Implicitly creates a writable LocalStore
group = zarr.create_group(store='data/foo/bar')
print(group)

<Group file://data/foo/bar>

# Implicitly creates a read-only FsspecStore
# Note: requires s3fs to be installed
group = zarr.open_group(
   store='s3://noaa-nwm-retro-v2-zarr-pds',
   mode='r',
   storage_options={'anon': True}
)
print(group)

<Group <FsspecStore(S3FileSystem, noaa-nwm-retro-v2-zarr-pds)>>

# Implicitly creates a MemoryStore
data = {}
group = zarr.create_group(store=data)
print(group)

<Group memory://123241913053632>

StoreLike¶

StoreLike values can be:

a Path or string indicating a location on the local file system. This will create a local store:

group = zarr.open_group(store='data/foo/bar')
print(group)

<Group file://data/foo/bar>

from pathlib import Path
group = zarr.open_group(store=Path('data/foo/bar'))
print(group)

<Group file://data/foo/bar>

an FSSpec URI string, indicating a remote store location:

# Note: requires s3fs to be installed
group = zarr.open_group(
   store='s3://noaa-nwm-retro-v2-zarr-pds',
   mode='r',
   storage_options={'anon': True}
)
print(group)

<Group <FsspecStore(S3FileSystem, noaa-nwm-retro-v2-zarr-pds)>>

an empty dictionary or None, which will create a new memory store:

group = zarr.create_group(store={})
print(group)

<Group memory://123241873102784>

group = zarr.create_group(store=None)
print(group)

<Group memory://123241873193408>

a dictionary of string to Buffer mappings. This will create a memory store, using this dictionary as the store_dict argument.
an FSSpec FSMap object, which will create an FsspecStore.
a Store or StorePath - see explicit store creation below.

Explicit Store Creation¶

In some cases, it may be helpful to create a store instance directly. Zarr-Python offers built-in stores: zarr.storage.LocalStore, zarr.storage.FsspecStore, zarr.storage.ZipStore, zarr.storage.MemoryStore, and zarr.storage.ObjectStore.

Local Store¶

The zarr.storage.LocalStore stores data in a nested set of directories on a local filesystem:

store = zarr.storage.LocalStore('data/foo/bar', read_only=True)
group = zarr.open_group(store=store, mode='r')
print(group)

<Group file://data/foo/bar>

Zip Store¶

The zarr.storage.ZipStore stores the contents of a Zarr hierarchy in a single Zip file. The Zip Store specification is currently in draft form:

store = zarr.storage.ZipStore('data.zip', mode='w')
array = zarr.create_array(store=store, shape=(2,), dtype='float64')
print(array)

<Array zip://data.zip shape=(2,) dtype=float64>

Remote Store¶

The zarr.storage.FsspecStore stores the contents of a Zarr hierarchy following the same logical layout as the LocalStore, except the store is assumed to be on a remote storage system such as cloud object storage (e.g. AWS S3, Google Cloud Storage, Azure Blob Store). The zarr.storage.FsspecStore is backed by fsspec and can support any backend that implements the AbstractFileSystem API. storage_options can be used to configure the fsspec backend:

# Note: requires s3fs to be installed
store = zarr.storage.FsspecStore.from_url(
   's3://noaa-nwm-retro-v2-zarr-pds',
   read_only=True,
   storage_options={'anon': True}
)
group = zarr.open_group(store=store, mode='r')
print(group)

<Group <FsspecStore(S3FileSystem, noaa-nwm-retro-v2-zarr-pds)>>

The type of filesystem (e.g. S3, https, etc..) is inferred from the scheme of the url (e.g. s3 for "s3://noaa-nwm-retro-v2-zarr-pds"). In case a specific filesystem is needed, one can explicitly create it. For example to create an S3 filesystem (note that the filesystem must be created with asynchronous=True):

# Note: requires s3fs to be installed
import fsspec
fs = fsspec.filesystem('s3', anon=True, asynchronous=True)
store = zarr.storage.FsspecStore(fs, path='noaa-nwm-retro-v2-zarr-pds', read_only=True)
print(store)

<FsspecStore(S3FileSystem, noaa-nwm-retro-v2-zarr-pds)>

When using an S3-compatible service other than AWS, pass the service endpoint to the filesystem via client_kwargs={'endpoint_url': 'https://...'}.

Memory Store¶

The zarr.storage.MemoryStore stores Zarr data (metadata and chunks) in an in-memory dictionary:

data = {}
store = zarr.storage.MemoryStore(data)
array = zarr.create_array(store=store, shape=(2,), dtype='float64')
print(array)

<Array memory://123241780252736 shape=(2,) dtype=float64>

Object Store¶

zarr.storage.ObjectStore stores the contents of the Zarr hierarchy using any ObjectStore storage implementation, including AWS S3 (obstore.store.S3Store), Google Cloud Storage (obstore.store.GCSStore), and Azure Blob Storage (obstore.store.AzureStore). This store is backed by obstore, which builds on the production quality Rust library object_store.

from zarr.storage import ObjectStore
from obstore.store import MemoryStore

store = ObjectStore(MemoryStore())
array = zarr.create_array(store=store, shape=(2,), dtype='float64')
print(array)

<Array object_store://<obstore.store.MemoryStore object at 0x70168c1845d0> shape=(2,) dtype=float64>

Here's an example of using ObjectStore for accessing remote data:

from zarr.storage import ObjectStore
from obstore.store import S3Store

s3_store = S3Store('noaa-nwm-retro-v2-zarr-pds', skip_signature=True, region="us-west-2")
store = zarr.storage.ObjectStore(store=s3_store, read_only=True)
group = zarr.open_group(store=store, mode='r')
print(group.info)

Name        : 
Type        : Group
Zarr format : 2
Read-only   : True
Store type  : ObjectStore
No. members : 12
No. arrays  : 12
No. groups  : 0

Warning

The zarr.storage.ObjectStore class is experimental.

Developing custom stores¶

Zarr-Python zarr.abc.store.Store API is meant to be extended. The Store Abstract Base Class includes all of the methods needed to be a fully operational store in Zarr Python. Zarr also provides a test harness for custom stores: zarr.testing.store.StoreTests. See the Custom stores section of the extending guide for more on implementing your own store.