Consolidated metadata¶
Warning
The Consolidated Metadata feature in Zarr-Python is considered experimental for v3 stores. zarr-specs#309 has proposed a formal extension to the v3 specification to support consolidated metadata.
Zarr-Python implements the Consolidated Metadata feature for both the v2 and v3 formats. Consolidated metadata can reduce the time needed to load the metadata for an entire hierarchy, especially when the metadata is being served over a network. Consolidated metadata essentially stores all the metadata for a hierarchy in the metadata of the root Group.
Usage¶
If consolidated metadata is present in a Zarr Group's metadata then it is used
by default. The initial read to open the group will need to communicate with
the store (reading from a file for a zarr.storage.LocalStore, making a
network request for a zarr.storage.FsspecStore). After that, any subsequent
metadata reads to get child Group or Array nodes will not require reads from the store.
In Python, the consolidated metadata is available on the .consolidated_metadata
attribute of the GroupMetadata object.
import zarr
import warnings
warnings.filterwarnings("ignore", category=UserWarning)
group = zarr.create_group(store="memory://consolidated-metadata-demo")
print(group)
array = group.create_array(shape=(1,), name='a', dtype='float64')
print(array)
<Group <FsspecStore(AsyncFileSystemWrapper, /consolidated-metadata-demo)>>
<Array <FsspecStore(AsyncFileSystemWrapper, /consolidated-metadata-demo)>/a shape=(1,) dtype=float64>
<Array <FsspecStore(AsyncFileSystemWrapper, /consolidated-metadata-demo)>/b shape=(2, 2) dtype=float64>
<Array <FsspecStore(AsyncFileSystemWrapper, /consolidated-metadata-demo)>/c shape=(3, 3, 3) dtype=float64>
If we open that group, the Group's metadata includes a ConsolidatedMetadata object
holding the metadata for every child node, which can be used:
from pprint import pprint
import io
consolidated = zarr.open_group(store="memory://consolidated-metadata-demo")
consolidated_metadata = consolidated.metadata.consolidated_metadata.metadata
output = io.StringIO()
pprint(dict(sorted(consolidated_metadata.items())), stream=output, width=60)
print(output.getvalue())
{'a': ArrayV3Metadata(shape=(1,),
data_type=Float64(endianness='little'),
chunk_grid=RegularChunkGridMetadata(chunk_shape=(1,)),
chunk_key_encoding=DefaultChunkKeyEncoding(separator='/'),
fill_value=np.float64(0.0),
codecs=(BytesCodec(endian='little'),
ZstdCodec(level=0,
checksum=False)),
attributes={},
dimension_names=None,
zarr_format=3,
node_type='array',
storage_transformers=(),
extra_fields={}),
'b': ArrayV3Metadata(shape=(2, 2),
data_type=Float64(endianness='little'),
chunk_grid=RegularChunkGridMetadata(chunk_shape=(2,
2)),
chunk_key_encoding=DefaultChunkKeyEncoding(separator='/'),
fill_value=np.float64(0.0),
codecs=(BytesCodec(endian='little'),
ZstdCodec(level=0,
checksum=False)),
attributes={},
dimension_names=None,
zarr_format=3,
node_type='array',
storage_transformers=(),
extra_fields={}),
'c': ArrayV3Metadata(shape=(3, 3, 3),
data_type=Float64(endianness='little'),
chunk_grid=RegularChunkGridMetadata(chunk_shape=(3,
3,
3)),
chunk_key_encoding=DefaultChunkKeyEncoding(separator='/'),
fill_value=np.float64(0.0),
codecs=(BytesCodec(endian='little'),
ZstdCodec(level=0,
checksum=False)),
attributes={},
dimension_names=None,
zarr_format=3,
node_type='array',
storage_transformers=(),
extra_fields={})}
Operations on the group to get children automatically use the consolidated metadata:
<Array <FsspecStore(AsyncFileSystemWrapper, /consolidated-metadata-demo)>/a shape=(1,) dtype=float64>
With nested groups, the consolidated metadata is available on the children, recursively:
child = group.create_group('child', attributes={'kind': 'child'})
grandchild = child.create_group('grandchild', attributes={'kind': 'grandchild'})
consolidated = zarr.consolidate_metadata("memory://consolidated-metadata-demo")
output = io.StringIO()
pprint(consolidated['child'].metadata.consolidated_metadata, stream=output, width=60)
print(output.getvalue())
ConsolidatedMetadata(metadata={'grandchild': GroupMetadata(attributes={'kind': 'grandchild'},
zarr_format=3,
consolidated_metadata=ConsolidatedMetadata(metadata={},
kind='inline',
must_understand=False),
node_type='group')},
kind='inline',
must_understand=False)
Added in version 3.1.1
The keys in the consolidated metadata are sorted prior to writing. Keys are
sorted in ascending order by path depth, where a path is defined as a sequence
of strings joined by "/". For keys with the same path length, lexicographic
order is used to break the tie. This behavior ensures deterministic metadata
output for a given group.
Controlling the use of consolidated metadata¶
By default, zarr.open_group uses consolidated metadata if it is present, and
falls back to reading metadata from the store otherwise. This behavior can be
controlled with the use_consolidated keyword. Pass use_consolidated=False to
ignore consolidated metadata and always read the metadata of child nodes directly
from the store:
group = zarr.open_group(store="memory://consolidated-metadata-demo", use_consolidated=False)
print(group.metadata.consolidated_metadata)
Passing use_consolidated=True instead raises an error if consolidated metadata is
not found, which is useful when reading over a network, where relying on many
per-node metadata requests would be slow.
Synchronization and Concurrency¶
Consolidated metadata is intended for read-heavy use cases on slowly changing hierarchies. For hierarchies where new nodes are constantly being added, removed, or modified, consolidated metadata may not be desirable.
- It will add some overhead to each update operation, since the metadata would need to be re-consolidated to keep it in sync with the store.
- Readers using consolidated metadata will regularly see a "past" version
of the metadata, at the time they read the root node with its consolidated
metadata. Readers who need the latest view of a changing hierarchy can pass
use_consolidated=Falsetozarr.open_groupto always read child metadata directly from the store.
Stores Without Support for Consolidated Metadata¶
Some stores may want to opt out of the consolidated metadata mechanism. This may be for several reasons like:
- They want to maintain read-write consistency, which is challenging with consolidated metadata.
- They have their own consolidated metadata mechanism.
- They offer good enough performance without need for consolidation.
This type of store can declare it doesn't want consolidation by implementing
Store.supports_consolidated_metadata and returning False. For stores that don't support
consolidation, Zarr will:
- Raise an error on
consolidate_metadatacalls, maintaining the store in its unconsolidated state. - Raise an error in
AsyncGroup.open(..., use_consolidated=True) - Not use consolidated metadata in
AsyncGroup.open(..., use_consolidated=None)