Release notes ============= .. # Copy the warning statement _under_ the latest release version # and unindent for pre-releases. .. warning:: Pre-release! Use :command:`pip install --pre zarr` to evaluate this release. .. # Unindent the section between releases in order # to document your changes. On releases it will be # re-indented so that it does not show up in the notes. .. note:: Zarr-Python 2.18.* is expected be the final release in the 2.* series. Work on Zarr-Python 3.0 is underway. See `GH1777 `_ for more details on the upcoming 3.0 release. .. release_3.0.0-alpha: 3.0.0-alpha ----------- .. warning:: Zarr-Python 3.0.0-alpha is a pre-release of the upcoming 3.0 release. This release is not feature complete or expected to be ready for production applications. .. note:: The complete release notes for 3.0 have not been added to this document yet. See the `3.0.0-alpha `_ release on GitHub for a record of changes included in this release. Enhancements ~~~~~~~~~~~~ * Implement listing of the sub-arrays and sub-groups for a V3 ``Group``. By :user:`Davis Bennett ` :issue:`1726`. * Bootstrap v3 branch with zarrita. By :user:`Joe Hamman ` :issue:`1584`. * Extensible codecs for V3. By :user:`Norman Rzepka ` :issue:`1588`. * Don't import from tests. By :user:`Davis Bennett ` :issue:`1601`. * Listable V3 Stores. By :user:`Joe Hamman ` :issue:`1634`. * Codecs without array metadata. By :user:`Norman Rzepka ` :issue:`1632`. * fix sync group class methods. By :user:`Joe Hamman ` :issue:`1652`. * implement eq for LocalStore. By :user:`Charoula Kyriakides ` :issue:`1792`. * V3 reorg. By :user:`Joe Hamman ` :issue:`1809`. * [v3] Sync with futures. By :user:`Davis Bennett ` :issue:`1804`. * implement group.members. By :user:`Davis Bennett ` :issue:`1726`. * Remove implicit groups. By :user:`Joe Hamman ` :issue:`1827`. * feature(store): ``list_*`` -> AsyncGenerators. By :user:`Joe Hamman ` :issue:`1844`. * Test codec entrypoints. By :user:`Norman Rzepka ` :issue:`1835`. * Remove extra v3 sync module. By :user:`Max Jones ` :issue:`1856`. * Use donfig for V3 configuration. By :user:`Max Jones ` :issue:`1655`. * groundwork for V3 group tests. By :user:`Davis Bennett ` :issue:`1743`. * [v3] First step to generalizes ndarray and bytes. By :user:`Mads R. B. Kristensen ` :issue:`1826`. * Reworked codec pipelines. By :user:`Norman Rzepka ` :issue:`1670`. * Followup on codecs. By :user:`Norman Rzepka ` :issue:`1889`. * Protocols for Buffer and NDBuffer. By :user:`Mads R. B. Kristensen ` :issue:`1899`. * [V3] Expand store tests. By :user:`Davis Bennett ` :issue:`1900`. * [v3] Feature: Store open mode. By :user:`Joe Hamman ` :issue:`1911`. * fix(types): Group.info -> NotImplementedError. By :user:`Joe Hamman ` :issue:`1936`. * feature(typing): add py.typed file to package root. By :user:`Joe Hamman ` :issue:`1935`. * Support all indexing variants. By :user:`Norman Rzepka ` :issue:`1917`. * Feature: group and array name properties. By :user:`Joe Hamman ` :issue:`1940`. * implement .chunks on v3 arrays. By :user:`Ryan Abernathey ` :issue:`1929`. * Fixes bug in transpose. By :user:`Norman Rzepka ` :issue:`1949`. * Buffer Prototype Argument. By :user:`Mads R. B. Kristensen ` :issue:`1910`. * Feature: Top level V3 API. By :user:`Joe Hamman ` :issue:`1884`. * Basic working FsspecStore. By :user:`Martin Durant `; :issue:`1785`. Typing ~~~~~~ * Resolve Mypy errors in v3 branch. By :user:`Daniel Jahn ` :issue:`1692`. * Allow dmypy to be run on v3 branch. By :user:`David Stansby ` :issue:`1780`. * Remove unused typing ignore comments. By :user:`David Stansby ` :issue:`1781`. * Check untyped defs on v3. By :user:`David Stansby ` :issue:`1784`. * [v3] Enable some more strict mypy options. By :user:`David Stansby ` :issue:`1793`. * [v3] Disallow generic Any typing. By :user:`David Stansby ` :issue:`1794`. * Disallow incomplete type definitions. By :user:`David Stansby ` :issue:`1814`. * Disallow untyped calls. By :user:`David Stansby ` :issue:`1811`. * Fix some untyped calls. By :user:`David Stansby ` :issue:`1865`. * Disallow untyped defs. By :user:`David Stansby ` :issue:`1834`. * Add more typing to zarr.group. By :user:`David Stansby ` :issue:`1870`. * Fix any generics in zarr.array. By :user:`David Stansby ` :issue:`1861`. * Remove some unused mypy overrides. By :user:`David Stansby ` :issue:`1894`. * Finish typing zarr.metadata. By :user:`David Stansby ` :issue:`1880`. * Disallow implicit re-exports. By :user:`David Stansby ` :issue:`1908`. * Make typing strict. By :user:`David Stansby ` :issue:`1879`. * Enable extra mypy error codes. By :user:`David Stansby ` :issue:`1909`. * Enable warn_unreachable for mypy. By :user:`David Stansby ` :issue:`1937`. * Fix final typing errors. By :user:`David Stansby ` :issue:`1939`. Maintenance ~~~~~~~~~~~ * Remedy a situation where ``zarr-python`` was importing ``DummyStorageTransformer`` from the test suite. The dependency relationship is now reversed: the test suite imports this class from ``zarr-python``. By :user:`Davis Bennett ` :issue:`1601`. * [V3] Update minimum supported Python and Numpy versions. By :user:`Joe Hamman ` :issue:`1638` * use src layout and use hatch for packaging. By :user:`Davis Bennett ` :issue:`1592`. * temporarily disable mypy in v3 directory. By :user:`Joe Hamman ` :issue:`1649`. * create hatch test env. By :user:`Ryan Abernathey ` :issue:`1650`. * removed unused environments and workflows. By :user:`Ryan Abernathey ` :issue:`1651`. * Add env variables to sprint setup instructions. By :user:`Max Jones ` :issue:`1654`. * Add test matrix for V3. By :user:`Max Jones ` :issue:`1656`. * Remove attrs. By :user:`Davis Bennett ` :issue:`1660`. * Specify hatch envs using GitHub actions matrix for v3 tests. By :user:`Max Jones ` :issue:`1728`. * black -> ruff format + cleanup. By :user:`Saransh Chopra ` :issue:`1639`. * Remove old v3. By :user:`Davis Bennett ` :issue:`1742`. * V3 update pre commit. By :user:`Joe Hamman ` :issue:`1808`. * remove windows testing on v3 branch. By :user:`Joe Hamman ` :issue:`1817`. * fix: add mypy to test dependencies. By :user:`Davis Bennett ` :issue:`1789`. * chore(ci): add numpy 2 release candidate to test matrix. By :user:`Joe Hamman ` :issue:`1828`. * fix dependencies. By :user:`Norman Rzepka ` :issue:`1840`. * Add pytest to mypy dependencies. By :user:`David Stansby ` :issue:`1846`. * chore(pre-commit): update pre-commit versions and remove attrs dep mypy section. By :user:`Joe Hamman ` :issue:`1848`. * Enable some ruff rules (RUF) and fix issues. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1869`. * Configure Ruff to apply flake8-bugbear/isort/pyupgrade. By :user:`Norman Rzepka ` :issue:`1890`. * chore(ci): remove mypy from test action in favor of pre-commit action. By :user:`Joe Hamman ` :issue:`1887`. * Enable ruff/flake8-raise rules (RSE) and fix issues. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1872`. * Apply assorted ruff/refurb rules (FURB). By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1873`. * Enable ruff/flake8-implicit-str-concat rules (ISC) and fix issues. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1868`. * Add numpy to mypy pre-commit check env. By :user:`David Stansby ` :issue:`1893`. * remove fixture files from src. By :user:`Davis Bennett ` :issue:`1897`. * Fix list of packages in mypy pre-commit environment. By :user:`David Stansby ` :issue:`1907`. * Run sphinx directly on readthedocs. By :user:`David Stansby ` :issue:`1919`. * Apply preview ruff rules. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1942`. * Enable and apply ruff rule RUF009. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1941`. Documentation ~~~~~~~~~~~~~ * Specify docs hatch env for v3 branch. By :user:`Max Jones ` :issue:`1655`. * Development installation/contributing docs updates. By :user:`Alden Keefe Sampson ` :issue:`1643`. * chore: update project settings per scientific python repo-review. By :user:`Joe Hamman ` :issue:`1863`. * doc: update release notes for 3.0.0.alpha. By :user:`Joe Hamman ` :issue:`1959`. .. _release_2.18.3: 2.18.3 ------ Enhancements ~~~~~~~~~~~~ * Added support for creating a copy of data when converting a `zarr.Array` to a numpy array. By :user:`David Stansby ` (:issue:`2106`) and :user:`Joe Hamman ` (:issue:`2123`). Maintenance ~~~~~~~~~~~ * Removed support for Python 3.9. By :user:`David Stansby ` (:issue:`2074`). * Fix a regression when using orthogonal indexing with a scalar. By :user:`Deepak Cherian ` :issue:`1931` * Added compatibility with NumPy 2.1. By :user:`David Stansby ` * Bump minimum NumPy version to 1.24. :user:`Joe Hamman ` (:issue:`2127`). Deprecations ~~~~~~~~~~~~ * Deprecate :class:`zarr.n5.N5Store` and :class:`zarr.n5.N5FSStore`. These stores are slated to be removed in Zarr Python 3.0. By :user:`Joe Hamman ` :issue:`2085`. .. _release_2.18.2: 2.18.2 ------ Enhancements ~~~~~~~~~~~~ * Add Zstd codec to old V3 code path. By :user:`Ryan Abernathey ` .. _release_2.18.1: 2.18.1 ------ Maintenance ~~~~~~~~~~~ * Fix a regression when getting or setting a single value from arrays with size-1 chunks. By :user:`Deepak Cherian ` :issue:`1874` .. _release_2.18.0: 2.18.0 ------ Enhancements ~~~~~~~~~~~~ * Performance improvement for reading and writing chunks if any of the dimensions is size 1. By :user:`Deepak Cherian ` :issue:`1730`. Maintenance ~~~~~~~~~~~ * Enable ruff/bugbear rules (B) and fix issues. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1702`. * Minor updates to use `np.inf` instead of `np.PINF` / `np.NINF` in preparation for NumPy 2.0.0 release. By :user:`Joe Hamman ` :issue:`1842`. Deprecations ~~~~~~~~~~~~ * Deprecate experimental v3 support by issuing a `FutureWarning`. Also updated docs to warn about using the experimental v3 version. By :user:`Joe Hamman ` :issue:`1802` and :issue:`1807`. * Deprecate the following stores: :class:`zarr.storage.DBMStore`, :class:`zarr.storage.LMDBStore`, :class:`zarr.storage.SQLiteStore`, :class:`zarr.storage.MongoDBStore`, :class:`zarr.storage.RedisStore`, and :class:`zarr.storage.ABSStore`. These stores are slated to be removed from Zarr-Python in version 3.0. By :user:`Joe Hamman ` :issue:`1801`. .. _release_2.17.2: 2.17.2 ------ Enhancements ~~~~~~~~~~~~ * [v3] Dramatically reduce number of ``__contains__`` requests in favor of optimistically calling `__getitem__` and handling any error that may arise. By :user:`Deepak Cherian ` :issue:`1741`. * [v3] Reuse the downloaded array metadata when creating an ``Array``. By :user:`Deepak Cherian ` :issue:`1734`. * Optimize ``Array.info`` so that it calls `getsize` only once. By :user:`Deepak Cherian ` :issue:`1733`. * Override IPython ``_repr_*_`` methods to avoid expensive lookups against object stores. By :user:`Deepak Cherian ` :issue:`1716`. * FSStore now raises rather than return bad data. By :user:`Martin Durant ` and :user:`Ian Carroll ` :issue:`1604`. * Avoid redundant ``__contains__``. By :user:`Deepak Cherian ` :issue:`1739`. Docs ~~~~ * Fix link to GCSMap in ``tutorial.rst``. By :user:`Daniel Jahn ` :issue:`1689`. * Endorse `SPEC0000 `_ and state version support policy in ``installation.rst``. By :user:`Sanket Verma ` :issue:`1665`. * Migrate v1 and v2 specification to `Zarr-Specs `_. By :user:`Sanket Verma ` :issue:`1582`. Maintenance ~~~~~~~~~~~ * Add CI test environment for Python 3.12 By :user:`Joe Hamman ` :issue:`1719`. * Bump minimum supported NumPy version to 1.23 (per spec 0000) By :user:`Joe Hamman ` :issue:`1719`. * Minor fixes: Using ``is`` instead of ``type`` and removing unnecessary ``None``. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1737`. * Fix tests failure related to Pytest 8. By :user:`David Stansby ` :issue:`1714`. .. _release_2.17.1: 2.17.1 ------ Enhancements ~~~~~~~~~~~~ * Change occurrences of % and format() to f-strings. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1423`. * Proper argument for numpy.reshape. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1425`. * Add typing to dimension separator arguments. By :user:`David Stansby ` :issue:`1620`. Docs ~~~~ * ZIP related tweaks. By :user:`Davis Bennett ` :issue:`1641`. Maintenance ~~~~~~~~~~~ * Update config.yml with Zulip. By :user:`Josh Moore `. * Replace Gitter with the new Zulip Chat link. By :user:`Sanket Verma ` :issue:`1685`. * Fix RTD build. By :user:`Sanket Verma ` :issue:`1694`. .. _release_2.17.0: 2.17.0 ------ Enhancements ~~~~~~~~~~~~ * Added type hints to ``zarr.creation.create()``. By :user:`David Stansby ` :issue:`1536`. * Pyodide support: Don't require fasteners on Emscripten. By :user:`Hood Chatham ` :issue:`1663`. Docs ~~~~ * Minor correction and changes in documentation. By :user:`Sanket Verma ` :issue:`1509`. * Fix typo in documentation. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1554` * The documentation build now fails if there are any warnings. By :user:`David Stansby ` :issue:`1548`. * Add links to ``numcodecs`` docs in the tutorial. By :user:`David Stansby ` :issue:`1535`. * Enable offline formats for documentation builds. By :user:`Sanket Verma ` :issue:`1551`. * Minor tweak to advanced indexing tutorial examples. By :user:`Ross Barnowski ` :issue:`1550`. * Automatically document array members using sphinx-automodapi. By :user:`David Stansby ` :issue:`1547`. * Add a markdown file documenting the current and former core-developer team. By :user:`Joe Hamman ` :issue:`1628`. * Add Norman Rzepka to core-dev team. By :user:`Joe Hamman ` :issue:`1630`. * Added section about accessing ZIP archives on s3. By :user:`Jeff Peck ` :issue:`1613`, :issue:`1615`, and :user:`Davis Bennett ` :issue:`1641`. * Add V3 roadmap and design document. By :user:`Joe Hamman ` :issue:`1583`. Maintenance ~~~~~~~~~~~ * Drop Python 3.8 and NumPy 1.20 By :user:`Josh Moore `; :issue:`1557`. * Cache result of ``FSStore._fsspec_installed()``. By :user:`Janick Martinez Esturo ` :issue:`1581`. * Extend copyright notice to 2023. By :user:`Jack Kelly ` :issue:`1528`. * Change occurrence of ``io.open()`` into ``open()``. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1421`. * Preserve ``dimension_separator`` when resizing arrays. By :user:`Ziwen Liu ` :issue:`1533`. * Initialise some sets in tests with set literals instead of list literals. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1534`. * Allow ``black`` code formatter to be run with any Python version. By :user:`David Stansby ` :issue:`1549`. * Remove ``sphinx-rtd-theme`` dependency from ``pyproject.toml``. By :user:`Sanket Verma ` :issue:`1563`. * Remove ``CODE_OF_CONDUCT.md`` file from the Zarr-Python repository. By :user:`Sanket Verma ` :issue:`1572`. * Bump version of black in pre-commit. By :user:`David Stansby ` :issue:`1559`. * Use list comprehension where applicable. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1555`. * Use format specification mini-language to format string. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1558`. * Single startswith() call instead of multiple ones. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1556`. * Move codespell options around. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1196`. * Remove unused mypy ignore comments. By :user:`David Stansby ` :issue:`1602`. .. _release_2.16.1: 2.16.1 ------ Maintenance ~~~~~~~~~~~ * Require ``setuptools_scm`` version ``1.5.4``\+ By :user:`John A. Kirkham ` :issue:`1477`. * Add ``docs`` requirements to ``pyproject.toml`` By :user:`John A. Kirkham ` :issue:`1494`. * Fixed caching issue in ``LRUStoreCache``. By :user:`Mads R. B. Kristensen ` :issue:`1499`. .. _release_2.16.0: 2.16.0 ------ Enhancements ~~~~~~~~~~~~ * Allow for partial codec specification in V3 array metadata. By :user:`Joe Hamman ` :issue:`1443`. * Add ``__contains__`` method to ``KVStore``. By :user:`Christoph Gohlke ` :issue:`1454`. * **Block Indexing**: Implemented blockwise (chunk blocks) indexing to ``zarr.Array``. By :user:`Altay Sansal ` :issue:`1428` Maintenance ~~~~~~~~~~~ * Refactor the core array tests to reduce code duplication. By :user:`Davis Bennett ` :issue:`1462`. * Style the codebase with ``ruff`` and ``black``. By :user:`Davis Bennett ` :issue:`1459` * Ensure that chunks is tuple of ints upon array creation. By :user:`Philipp Hanslovsky ` :issue:`1461` .. _release_2.15.0: 2.15.0 ------ Enhancements ~~~~~~~~~~~~ * Implement more extensive fallback of getitem/setitem for orthogonal indexing. By :user:`Andreas Albert ` :issue:`1029`. * Getitems supports ``meta_array``. By :user:`Mads R. B. Kristensen ` :issue:`1131`. * ``open_array()`` now takes the ``meta_array`` argument. By :user:`Mads R. B. Kristensen ` :issue:`1396`. Maintenance ~~~~~~~~~~~ * Remove ``codecov`` from GitHub actions. By :user:`John A. Kirkham ` :issue:`1391`. * Replace ``np.product`` with ``np.prod`` due to deprecation. By :user:`James Bourbeau ` :issue:`1405`. * Activate Py 3.11 builds. By :user:`Joe Hamman ` :issue:`1415`. Documentation ~~~~~~~~~~~~~ * Add API reference for V3 Implementation in the docs. By :user:`Sanket Verma ` :issue:`1345`. Bug fixes ~~~~~~~~~ * Fix the conda-forge error. Read :issue:`1347` for detailed info. By :user:`Josh Moore ` :issue:`1364` and :issue:`1367`. * Fix ``ReadOnlyError`` when opening V3 store via fsspec reference file system. By :user:`Joe Hamman ` :issue:`1383`. * Fix ``normalize_fill_value`` for structured arrays. By :user:`Alan Du ` :issue:`1397`. .. _release_2.14.2: 2.14.2 ------ Bug fixes ~~~~~~~~~ * Ensure ``zarr.group`` uses writeable mode to fix issue with :issue:`1304`. By :user:`Brandur Thorgrimsson ` :issue:`1354`. .. _release_2.14.1: 2.14.1 ------ Documentation ~~~~~~~~~~~~~ * Fix API links. By :user:`Josh Moore ` :issue:`1346`. * Fix unit tests which prevented the conda-forge release. By :user:`Josh Moore ` :issue:`1348`. .. _release_2.14.0: 2.14.0 ------ Major changes ~~~~~~~~~~~~~ * Improve Zarr V3 support, adding partial store read/write and storage transformers. Add new features from the `v3 spec `_: * storage transformers * `get_partial_values` and `set_partial_values` * efficient `get_partial_values` implementation for `FSStoreV3` * sharding storage transformer By :user:`Jonathan Striebel `; :issue:`1096`, :issue:`1111`. * N5 nows supports Blosc. Remove warnings emitted when using N5Store or N5FSStore with a blosc-compressed array. By :user:`Davis Bennett `; :issue:`1331`. Bug fixes ~~~~~~~~~ * Allow reading utf-8 encoded json files By :user:`Nathan Zimmerberg ` :issue:`1308`. * Ensure contiguous data is give to ``FSStore``. Only copying if needed. By :user:`Mads R. B. Kristensen ` :issue:`1285`. * NestedDirectoryStore.listdir now returns chunk keys with the correct '/' dimension_separator. By :user:`Brett Graham ` :issue:`1334`. * N5Store/N5FSStore dtype returns zarr Stores readable dtype. By :user:`Marwan Zouinkhi ` :issue:`1339`. .. _release_2.13.6: 2.13.6 ------ Maintenance ~~~~~~~~~~~ * Bump gh-action-pypi-publish to 1.6.4. By :user:`Josh Moore ` :issue:`1320`. .. _release_2.13.5: 2.13.5 ------ Bug fixes ~~~~~~~~~ * Ensure ``zarr.create`` uses writeable mode to fix issue with :issue:`1304`. By :user:`James Bourbeau ` :issue:`1309`. .. _release_2.13.4: 2.13.4 ------ Appreciation ~~~~~~~~~~~~~ Special thanks to Outreachy participants for contributing to most of the maintenance PRs. Please read the blog post summarising the contribution phase and welcoming new Outreachy interns: https://zarr.dev/blog/welcoming-outreachy-2022-interns/ Enhancements ~~~~~~~~~~~~ * Handle fsspec.FSMap using FSStore store. By :user:`Rafal Wojdyla ` :issue:`1304`. Bug fixes ~~~~~~~~~ * Fix bug that caused double counting of groups in ``groups()`` and ``group_keys()`` methods with V3 stores. By :user:`Ryan Abernathey ` :issue:`1228`. * Remove unnecessary calling of `contains_array` for key that ended in `.array.json`. By :user:`Joe Hamman ` :issue:`1149`. * Fix bug that caused double counting of groups in ``groups()`` and ``group_keys()`` methods with V3 stores. By :user:`Ryan Abernathey ` :issue:`1228`. Documentation ~~~~~~~~~~~~~ * Fix minor indexing errors in tutorial and specification examples of documentation. By :user:`Kola Babalola ` :issue:`1277`. * Add `requirements_rtfd.txt` in `contributing.rst`. By :user:`AWA BRANDON AWA ` :issue:`1243`. * Add documentation for find/findall using visit. By :user:`Weddy Gikunda ` :issue:`1241`. * Refresh of the main landing page. By :user:`Josh Moore ` :issue:`1173`. Maintenance ~~~~~~~~~~~ * Migrate to ``pyproject.toml`` and remove redundant infrastructure. By :user:`Saransh Chopra ` :issue:`1158`. * Require ``setuptools`` 64.0.0+ By :user:`Saransh Chopra ` :issue:`1193`. * Pin action versions (pypi-publish, setup-miniconda) for dependabot By :user:`Saransh Chopra ` :issue:`1205`. * Remove ``tox`` support By :user:`Saransh Chopra ` :issue:`1219`. * Add workflow to label PRs with "needs release notes". By :user:`Saransh Chopra ` :issue:`1239`. * Simplify if/else statement. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1227`. * Get coverage up to 100%. By :user:`John Kirkham ` :issue:`1264`. * Migrate coverage to ``pyproject.toml``. By :user:`John Kirkham ` :issue:`1250`. * Use ``conda-incubator/setup-miniconda@v2.2.0``. By :user:`John Kirkham ` :issue:`1263`. * Delete unused files. By :user:`John Kirkham ` :issue:`1251`. * Skip labeller for bot PRs. By :user:`Saransh Chopra ` :issue:`1271`. * Restore Flake8 configuration. By :user:`John Kirkham ` :issue:`1249`. * Add missing newline at EOF. By :user:`Dimitri Papadopoulos` :issue:`1253`. * Add `license_files` to `pyproject.toml`. By :user:`John Kirkham ` :issue:`1247`. * Adding `pyupgrade` suggestions. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1225`. * Fixed some linting errors. By :user:`Weddy Gikunda ` :issue:`1226`. * Added the link to main website in readthedocs sidebar. By :user:`Stephanie_nkwatoh ` :issue:`1216`. * Remove redundant wheel dependency in `pyproject.toml`. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1233`. * Turned on `isloated_build` in `tox.ini` file. By :user:`AWA BRANDON AWA ` :issue:`1210`. * Fixed `flake8` alert and avoid duplication of `Zarr Developers`. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1203`. * Bump to NumPy 1.20+ in `environment.yml`. By :user:`John Kirkham ` :issue:`1201`. * Bump to NumPy 1.20 in `pyproject.toml`. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1192`. * Remove LGTM (`.lgtm.yml`) configuration file. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1191`. * Codespell will skip `fixture` in pre-commit. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1197`. * Add msgpack in `requirements_rtfd.txt`. By :user:`Emmanuel Bolarinwa ` :issue:`1188`. * Added license to docs fixed a typo from `_spec_v2` to `_spec_v3`. By :user:`AWA BRANDON AWA ` :issue:`1182`. * Fixed installation link in `README.md`. By :user:`AWA BRANDON AWA ` :issue:`1177`. * Fixed typos in `installation.rst` and `release.rst`. By :user:`Chizoba Nweke ` :issue:`1178`. * Set `docs/conf.py` language to `en`. By :user:`AWA BRANDON AWA ` :issue:`1174`. * Added `installation.rst` to the docs. By :user:`AWA BRANDON AWA ` :issue:`1170`. * Adjustment of year to `2015-2018` to `2015-2022` in the docs. By :user:`Emmanuel Bolarinwa ` :issue:`1165`. * Updated `Forking the repository` section in `contributing.rst`. By :user:`AWA BRANDON AWA ` :issue:`1171`. * Updated GitHub actions. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1134`. * Update web links: `http:// → https://`. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1313`. .. _release_2.13.3: 2.13.3 ------ * Improve performance of slice selections with steps by omitting chunks with no relevant data. By :user:`Richard Shaw ` :issue:`843`. .. _release_2.13.2: 2.13.2 ------ * Fix test failure on conda-forge builds (again). By :user:`Josh Moore `; see `zarr-feedstock#65 `_. .. _release_2.13.1: 2.13.1 ------ * Fix test failure on conda-forge builds. By :user:`Josh Moore `; see `zarr-feedstock#65 `_. .. _release_2.13.0: 2.13.0 ------ Major changes ~~~~~~~~~~~~~ * **Support of alternative array classes** by introducing a new argument, meta_array, that specifies the type/class of the underlying array. The meta_array argument can be any class instance that can be used as the like argument in NumPy (see `NEP 35 `_). enabling support for CuPy through, for example, the creation of a CuPy CPU compressor. By :user:`Mads R. B. Kristensen ` :issue:`934`. * **Remove support for Python 3.7** in concert with NumPy dependency. By :user:`Davis Bennett ` :issue:`1067`. * **Zarr v3: add support for the default root path** rather than requiring that all API users pass an explicit path. By :user:`Gregory R. Lee ` :issue:`1085`, :issue:`1142`. Bug fixes ~~~~~~~~~ * Remove/relax erroneous "meta" path check (**regression**). By :user:`Gregory R. Lee ` :issue:`1123`. * Cast all attribute keys to strings (and issue deprecation warning). By :user:`Mattia Almansi ` :issue:`1066`. * Fix bug in N5 storage that prevented arrays located in the root of the hierarchy from bearing the `n5` keyword. Along with fixing this bug, new tests were added for N5 routines that had previously been excluded from testing, and type annotations were added to the N5 codebase. By :user:`Davis Bennett ` :issue:`1092`. * Fix bug in LRUEStoreCache in which the current size wasn't reset on invalidation. By :user:`BGCMHou ` and :user:`Josh Moore ` :issue:`1076`, :issue:`1077`. * Remove erroneous check that disallowed array keys starting with "meta". By :user:`Gregory R. Lee ` :issue:`1105`. Documentation ~~~~~~~~~~~~~ * Typo fixes to close quotes. By :user:`Pavithra Eswaramoorthy ` * Added copy button to documentation. By :user:`Altay Sansal ` :issue:`1124`. Maintenance ~~~~~~~~~~~ * Simplify release docs. By :user:`Josh Moore ` :issue:`1119`. * Pin werkzeug to prevent test hangs. By :user:`Davis Bennett ` :issue:`1098`. * Fix a few DeepSource.io alerts By :user:`Dimitri Papadopoulos Orfanos ` :issue:`1080`. * Fix URLs. By :user:`Dimitri Papadopoulos Orfanos `, :issue:`1074`. * Fix spelling. By :user:`Dimitri Papadopoulos Orfanos `, :issue:`1073`. * Update GitHub issue templates with `YAML` format. By :user:`Saransh Chopra ` :issue:`1079`. * Remove option to return None from _ensure_store. By :user:`Gregory Lee ` :issue:`1068`. * Fix a typo of "integers". By :user:`Richard Scott ` :issue:`1056`. .. _release_2.12.0: 2.12.0 ------ Enhancements ~~~~~~~~~~~~ * **Add support for reading and writing Zarr V3.** The new `zarr._store.v3` package has the necessary classes and functions for evaluating Zarr V3. Since the format is not yet finalized, the classes and functions are not automatically imported into the regular `zarr` name space. Setting the `ZARR_V3_EXPERIMENTAL_API` environment variable will activate them. By :user:`Gregory Lee `; :issue:`898`, :issue:`1006`, and :issue:`1007` as well as by :user:`Josh Moore ` :issue:`1032`. * **Create FSStore from an existing fsspec filesystem**. If you have created an fsspec filesystem outside of Zarr, you can now pass it as a keyword argument to ``FSStore``. By :user:`Ryan Abernathey `; :issue:`911`. * Add numpy encoder class for json.dumps By :user:`Eric Prestat `; :issue:`933`. * Appending performance improvement to Zarr arrays, e.g., when writing to S3. By :user:`hailiangzhang `; :issue:`1014`. * Add number encoder for ``json.dumps`` to support numpy integers in ``chunks`` arguments. By :user:`Eric Prestat ` :issue:`697`. Bug fixes ~~~~~~~~~ * Fix bug that made it impossible to create an ``FSStore`` on unlistable filesystems (e.g. some HTTP servers). By :user:`Ryan Abernathey `; :issue:`993`. Documentation ~~~~~~~~~~~~~ * Update resize doc to clarify surprising behavior. By :user:`hailiangzhang `; :issue:`1022`. Maintenance ~~~~~~~~~~~ * Added Pre-commit configuration, incl. Yaml Check. By :user:`Shivank Chaudhary `; :issue:`1015`, :issue:`1016`. * Fix URL to renamed file in Blosc repo. By :user:`Andrew Thomas ` :issue:`1028`. * Activate Py 3.10 builds. By :user:`Josh Moore ` :issue:`1027`. * Make all unignored zarr warnings errors. By :user:`Josh Moore ` :issue:`1021`. .. _release_2.11.3: 2.11.3 ------ Bug fixes ~~~~~~~~~ * Fix missing case to fully revert change to default write_empty_chunks. By :user:`Tom White `; :issue:`1005`. .. _release_2.11.2: 2.11.2 ------ Bug fixes ~~~~~~~~~ * Changes the default value of ``write_empty_chunks`` to ``True`` to prevent unanticipated data losses when the data types do not have a proper default value when empty chunks are read back in. By :user:`Vyas Ramasubramani `; :issue:`965`, :issue:`1001`. .. _release_2.11.1: 2.11.1 ------ Bug fixes ~~~~~~~~~ * Fix bug where indexing with a scalar numpy value returned a single-value array. By :user:`Ben Jeffery ` :issue:`967`. * Removed `clobber` argument from `normalize_store_arg`. This enables to change data within an opened consolidated group using mode `"r+"` (i.e region write). By :user:`Tobias Kölling ` :issue:`975`. .. _release_2.11.0: 2.11.0 ------ Enhancements ~~~~~~~~~~~~ * **Sparse changes with performance impact!** One of the advantages of the Zarr format is that it is sparse, which means that chunks with no data (more precisely, with data equal to the fill value, which is usually 0) don't need to be written to disk at all. They will simply be assumed to be empty at read time. However, until this release, the Zarr library would write these empty chunks to disk anyway. This changes in this version: a small performance penalty at write time leads to significant speedups at read time and in filesystem operations in the case of sparse arrays. To revert to the old behavior, pass the argument ``write_empty_chunks=True`` to the array creation function. By :user:`Juan Nunez-Iglesias `; :issue:`853` and :user:`Davis Bennett `; :issue:`738`. * **Fancy indexing**. Zarr arrays now support NumPy-style fancy indexing with arrays of integer coordinates. This is equivalent to using zarr.Array.vindex. Mixing slices and integer arrays is not supported. By :user:`Juan Nunez-Iglesias `; :issue:`725`. * **New base class**. This release of Zarr Python introduces a new ``BaseStore`` class that all provided store classes implemented in Zarr Python now inherit from. This is done as part of refactoring to enable future support of the Zarr version 3 spec. Existing third-party stores that are a MutableMapping (e.g. dict) can be converted to a new-style key/value store inheriting from ``BaseStore`` by passing them as the argument to the new ``zarr.storage.KVStore`` class. For backwards compatibility, various higher-level array creation and convenience functions still accept plain Python dicts or other mutable mappings for the ``store`` argument, but will internally convert these to a ``KVStore``. By :user:`Gregory Lee `; :issue:`839`, :issue:`789`, and :issue:`950`. * Allow to assign array ``fill_values`` and update metadata accordingly. By :user:`Ryan Abernathey `, :issue:`662`. * Allow to update array fill_values By :user:`Matthias Bussonnier ` :issue:`665`. Bug fixes ~~~~~~~~~ * Fix bug where the checksum of zipfiles is wrong By :user:`Oren Watson ` :issue:`930`. * Fix consolidate_metadata with FSStore. By :user:`Joe Hamman ` :issue:`916`. * Unguarded next inside generator. By :user:`Dimitri Papadopoulos Orfanos ` :issue:`889`. Documentation ~~~~~~~~~~~~~ * Update docs creation of dev env. By :user:`Ray Bell ` :issue:`921`. * Update docs to use ``python -m pytest``. By :user:`Ray Bell ` :issue:`923`. * Fix versionadded tag in zarr.Array docstring. By :user:`Juan Nunez-Iglesias ` :issue:`852`. * Doctest seem to be stricter now, updating tostring() to tobytes(). By :user:`John Kirkham ` :issue:`907`. * Minor doc fix. By :user:`Mads R. B. Kristensen ` :issue:`937`. Maintenance ~~~~~~~~~~~ * Upgrade MongoDB in test env. By :user:`Joe Hamman ` :issue:`939`. * Pass dimension_separator on fixture generation. By :user:`Josh Moore ` :issue:`858`. * Activate Python 3.9 in GitHub Actions. By :user:`Josh Moore ` :issue:`859`. * Drop shortcut ``fsspec[s3]`` for dependency. By :user:`Josh Moore ` :issue:`920`. * and a swath of code-linting improvements by :user:`Dimitri Papadopoulos Orfanos `: - Unnecessary comprehension (:issue:`899`) - Unnecessary ``None`` provided as default (:issue:`900`) - use an if ``expression`` instead of `and`/`or` (:issue:`888`) - Remove unnecessary literal (:issue:`891`) - Decorate a few method with `@staticmethod` (:issue:`885`) - Drop unneeded ``return`` (:issue:`884`) - Drop explicit ``object`` inheritance from ``class``-es (:issue:`886`) - Unnecessary comprehension (:issue:`883`) - Codespell configuration (:issue:`882`) - Fix typos found by codespell (:issue:`880`) - Proper C-style formatting for integer (:issue:`913`) - Add LGTM.com / DeepSource.io configuration files (:issue:`909`) .. _release_2.10.3: 2.10.3 ------ Bug fixes ~~~~~~~~~ * N5 keywords now emit UserWarning instead of raising a ValueError. By :user:`Boaz Mohar `; :issue:`860`. * blocks_to_decompress not used in read_part function. By :user:`Boaz Mohar `; :issue:`861`. * defines blocksize for array, updates hexdigest values. By :user:`Andrew Fulton `; :issue:`867`. * Fix test failure on Debian and conda-forge builds. By :user:`Josh Moore `; :issue:`871`. .. _release_2.10.2: 2.10.2 ------ Bug fixes ~~~~~~~~~ * Fix NestedDirectoryStore datasets without dimension_separator metadata. By :user:`Josh Moore `; :issue:`850`. .. _release_2.10.1: 2.10.1 ------ Bug fixes ~~~~~~~~~ * Fix regression by setting normalize_keys=False in fsstore constructor. By :user:`Davis Bennett `; :issue:`842`. .. _release_2.10.0: 2.10.0 ------ Enhancements ~~~~~~~~~~~~ * Add N5FSStore. By :user:`Davis Bennett `; :issue:`793`. Bug fixes ~~~~~~~~~ * Ignore None dim_separators in save_array. By :user:`Josh Moore `; :issue:`831`. .. _release_2.9.5: 2.9.5 ----- Bug fixes ~~~~~~~~~ * Fix FSStore.listdir behavior for nested directories. By :user:`Gregory Lee `; :issue:`802`. .. _release_2.9.4: 2.9.4 ----- Bug fixes ~~~~~~~~~ * Fix structured arrays that contain objects By :user: `Attila Bergou `; :issue: `806` .. _release_2.9.3: 2.9.3 ----- Maintenance ~~~~~~~~~~~ * Mark the fact that some tests that require ``fsspec``, without compromising the code coverage score. By :user:`Ben Williams `; :issue:`823`. * Only inspect alternate node type if desired isn't present. By :user:`Trevor Manz `; :issue:`696`. .. _release_2.9.2: 2.9.2 ----- Maintenance ~~~~~~~~~~~ * Correct conda-forge deployment of Zarr by fixing some Zarr tests. By :user:`Ben Williams `; :issue:`821`. .. _release_2.9.1: 2.9.1 ----- Maintenance ~~~~~~~~~~~ * Correct conda-forge deployment of Zarr. By :user:`Josh Moore `; :issue:`819`. .. _release_2.9.0: 2.9.0 ----- This release of Zarr Python is the first release of Zarr to not support Python 3.6. Enhancements ~~~~~~~~~~~~ * Update ABSStore for compatibility with newer `azure.storage.blob`. By :user:`Tom Augspurger `; :issue:`759`. * Pathlib support. By :user:`Chris Barnes `; :issue:`768`. Documentation ~~~~~~~~~~~~~ * Clarify that arbitrary key/value pairs are OK for attributes. By :user:`Stephan Hoyer `; :issue:`751`. * Clarify how to manually convert a DirectoryStore to a ZipStore. By :user:`pmav99 `; :issue:`763`. Bug fixes ~~~~~~~~~ * Fix dimension_separator support. By :user:`Josh Moore `; :issue:`775`. * Extract ABSStore to zarr._storage.absstore. By :user:`Josh Moore `; :issue:`781`. * avoid NumPy 1.21.0 due to https://github.com/numpy/numpy/issues/19325 By :user:`Gregory Lee `; :issue:`791`. Maintenance ~~~~~~~~~~~ * Drop 3.6 builds. By :user:`Josh Moore `; :issue:`774`, :issue:`778`. * Fix build with Sphinx 4. By :user:`Elliott Sales de Andrade `; :issue:`799`. * TST: add missing assert in test_hexdigest. By :user:`Gregory Lee `; :issue:`801`. .. _release_2.8.3: 2.8.3 ----- Bug fixes ~~~~~~~~~ * FSStore: default to normalize_keys=False By :user:`Josh Moore `; :issue:`755`. * ABSStore: compatibility with ``azure.storage.python>=12`` By :user:`Tom Augspurger `; :issue:`618` .. _release_2.8.2: 2.8.2 ----- Documentation ~~~~~~~~~~~~~ * Add section on rechunking to tutorial By :user:`David Baddeley `; :issue:`730`. Bug fixes ~~~~~~~~~ * Expand FSStore tests and fix implementation issues By :user:`Davis Bennett `; :issue:`709`. Maintenance ~~~~~~~~~~~ * Updated ipytree warning for jlab3 By :user:`Ian Hunt-Isaak `; :issue:`721`. * b170a48a - (issue-728, copy-nested) Updated ipytree warning for jlab3 (#721) (3 weeks ago) * Activate dependabot By :user:`Josh Moore `; :issue:`734`. * Update Python classifiers (Zarr is stable!) By :user:`Josh Moore `; :issue:`731`. .. _release_2.8.1: 2.8.1 ----- Bug fixes ~~~~~~~~~ * raise an error if create_dataset's dimension_separator is inconsistent By :user:`Gregory R. Lee `; :issue:`724`. .. _release_2.8.0: 2.8.0 ----- V2 Specification Update ~~~~~~~~~~~~~~~~~~~~~~~ * Introduce optional dimension_separator .zarray key for nested chunks. By :user:`Josh Moore `; :issue:`715`, :issue:`716`. .. _release_2.7.1: 2.7.1 ----- Bug fixes ~~~~~~~~~ * Update Array to respect FSStore's key_separator (#718) By :user:`Gregory R. Lee `; :issue:`718`. .. _release_2.7.0: 2.7.0 ----- Enhancements ~~~~~~~~~~~~ * Start stop for iterator (`islice()`) By :user:`Sebastian Grill `; :issue:`621`. * Add capability to partially read and decompress chunks By :user:`Andrew Fulton `; :issue:`667`. Bug fixes ~~~~~~~~~ * Make DirectoryStore __setitem__ resilient against antivirus file locking By :user:`Eric Younkin `; :issue:`698`. * Compare test data's content generally By :user:`John Kirkham `; :issue:`436`. * Fix dtype usage in zarr/meta.py By :user:`Josh Moore `; :issue:`700`. * Fix FSStore key_seperator usage By :user:`Josh Moore `; :issue:`669`. * Simplify text handling in DB Store By :user:`John Kirkham `; :issue:`670`. * GitHub Actions migration By :user:`Matthias Bussonnier `; :issue:`641`, :issue:`671`, :issue:`674`, :issue:`676`, :issue:`677`, :issue:`678`, :issue:`679`, :issue:`680`, :issue:`682`, :issue:`684`, :issue:`685`, :issue:`686`, :issue:`687`, :issue:`695`, :issue:`706`. .. _release_2.6.1: 2.6.1 ----- * Minor build fix By :user:`Matthias Bussonnier `; :issue:`666`. .. _release_2.6.0: 2.6.0 ----- This release of Zarr Python is the first release of Zarr to not support Python 3.5. * End Python 3.5 support. By :user:`Chris Barnes `; :issue:`602`. * Fix ``open_group/open_array`` to allow opening of read-only store with ``mode='r'`` :issue:`269` * Add `Array` tests for FSStore. By :user:`Andrew Fulton `; :issue: `644`. * fix a bug in which ``attrs`` would not be copied on the root when using ``copy_all``; :issue:`613` * Fix ``FileNotFoundError`` with dask/s3fs :issue:`649` * Fix flaky fixture in test_storage.py :issue:`652` * Fix FSStore getitems fails with arrays that have a 0 length shape dimension :issue:`644` * Use async to fetch/write result concurrently when possible. :issue:`536`, See `this comment `_ for some performance analysis showing order of magnitude faster response in some benchmark. See `this link `_ for the full list of closed and merged PR tagged with the 2.6 milestone. * Add ability to partially read and decompress arrays, see :issue:`667`. It is only available to chunks stored using fsspec and using Blosc as a compressor. For certain analysis case when only a small portion of chunks is needed it can be advantageous to only access and decompress part of the chunks. Doing partial read and decompression add high latency to many of the operation so should be used only when the subset of the data is small compared to the full chunks and is stored contiguously (that is to say either last dimensions for C layout, firsts for F). Pass ``partial_decompress=True`` as argument when creating an ``Array``, or when using ``open_array``. No option exists yet to apply partial read and decompress on a per-operation basis. .. _release_2.5.0: 2.5.0 ----- This release will be the last to support Python 3.5, next version of Zarr will be Python 3.6+. * `DirectoryStore` now uses `os.scandir`, which should make listing large store faster, :issue:`563` * Remove a few remaining Python 2-isms. By :user:`Poruri Sai Rahul `; :issue:`393`. * Fix minor bug in `N5Store`. By :user:`gsakkis`, :issue:`550`. * Improve error message in Jupyter when trying to use the ``ipytree`` widget without ``ipytree`` installed. By :user:`Zain Patel `; :issue:`537` * Add typing information to many of the core functions :issue:`589` * Explicitly close stores during testing. By :user:`Elliott Sales de Andrade `; :issue:`442` * Many of the convenience functions to emit errors (``err_*`` from ``zarr.errors`` have been replaced by ``ValueError`` subclasses. The corresponding ``err_*`` function have been removed. :issue:`590`, :issue:`614`) * Improve consistency of terminology regarding arrays and datasets in the documentation. By :user:`Josh Moore `; :issue:`571`. * Added support for generic URL opening by ``fsspec``, where the URLs have the form "protocol://[server]/path" or can be chained URls with "::" separators. The additional argument ``storage_options`` is passed to the backend, see the ``fsspec`` docs. By :user:`Martin Durant `; :issue:`546` * Added support for fetching multiple items via ``getitems`` method of a store, if it exists. This allows for concurrent fetching of data blocks from stores that implement this; presently HTTP, S3, GCS. Currently only applies to reading. By :user:`Martin Durant `; :issue:`606` * Efficient iteration expanded with option to pass start and stop index via ``array.islice``. By :user:`Sebastian Grill `, :issue:`615`. .. _release_2.4.0: 2.4.0 ----- Enhancements ~~~~~~~~~~~~ * Add key normalization option for ``DirectoryStore``, ``NestedDirectoryStore``, ``TempStore``, and ``N5Store``. By :user:`James Bourbeau `; :issue:`459`. * Add ``recurse`` keyword to ``Group.array_keys`` and ``Group.arrays`` methods. By :user:`James Bourbeau `; :issue:`458`. * Use uniform chunking for all dimensions when specifying ``chunks`` as an integer. Also adds support for specifying ``-1`` to chunk across an entire dimension. By :user:`James Bourbeau `; :issue:`456`. * Rename ``DictStore`` to ``MemoryStore``. By :user:`James Bourbeau `; :issue:`455`. * Rewrite ``.tree()`` pretty representation to use ``ipytree``. Allows it to work in both the Jupyter Notebook and JupyterLab. By :user:`John Kirkham `; :issue:`450`. * Do not rename Blosc parameters in n5 backend and add `blocksize` parameter, compatible with n5-blosc. By :user:`axtimwalde`, :issue:`485`. * Update ``DirectoryStore`` to create files with more permissive permissions. By :user:`Eduardo Gonzalez ` and :user:`James Bourbeau `; :issue:`493` * Use ``math.ceil`` for scalars. By :user:`John Kirkham `; :issue:`500`. * Ensure contiguous data using ``astype``. By :user:`John Kirkham `; :issue:`513`. * Refactor out ``_tofile``/``_fromfile`` from ``DirectoryStore``. By :user:`John Kirkham `; :issue:`503`. * Add ``__enter__``/``__exit__`` methods to ``Group`` for ``h5py.File`` compatibility. By :user:`Chris Barnes `; :issue:`509`. Bug fixes ~~~~~~~~~ * Fix Sqlite Store Wrong Modification. By :user:`Tommy Tran `; :issue:`440`. * Add intermediate step (using ``zipfile.ZipInfo`` object) to write inside ``ZipStore`` to solve too restrictive permission issue. By :user:`Raphael Dussin `; :issue:`505`. * Fix '/' prepend bug in ``ABSStore``. By :user:`Shikhar Goenka `; :issue:`525`. Documentation ~~~~~~~~~~~~~ * Fix hyperlink in ``README.md``. By :user:`Anderson Banihirwe `; :issue:`531`. * Replace "nuimber" with "number". By :user:`John Kirkham `; :issue:`512`. * Fix azure link rendering in tutorial. By :user:`James Bourbeau `; :issue:`507`. * Update ``README`` file to be more detailed. By :user:`Zain Patel `; :issue:`495`. * Import blosc from numcodecs in tutorial. By :user:`James Bourbeau `; :issue:`491`. * Adds logo to docs. By :user:`James Bourbeau `; :issue:`462`. * Fix N5 link in tutorial. By :user:`James Bourbeau `; :issue:`480`. * Fix typo in code snippet. By :user:`Joe Jevnik `; :issue:`461`. * Fix URLs to point to zarr-python By :user:`John Kirkham `; :issue:`453`. Maintenance ~~~~~~~~~~~ * Add documentation build to CI. By :user:`James Bourbeau `; :issue:`516`. * Use ``ensure_ndarray`` in a few more places. By :user:`John Kirkham `; :issue:`506`. * Support Python 3.8. By :user:`John Kirkham `; :issue:`499`. * Require Numcodecs 0.6.4+ to use text handling functionality from it. By :user:`John Kirkham `; :issue:`497`. * Updates tests to use ``pytest.importorskip``. By :user:`James Bourbeau `; :issue:`492` * Removed support for Python 2. By :user:`jhamman`; :issue:`393`, :issue:`470`. * Upgrade dependencies in the test matrices and resolve a compatibility issue with testing against the Azure Storage Emulator. By :user:`alimanfoo`; :issue:`468`, :issue:`467`. * Use ``unittest.mock`` on Python 3. By :user:`Elliott Sales de Andrade `; :issue:`426`. * Drop ``decode`` from ``ConsolidatedMetadataStore``. By :user:`John Kirkham `; :issue:`452`. .. _release_2.3.2: 2.3.2 ----- Enhancements ~~~~~~~~~~~~ * Use ``scandir`` in ``DirectoryStore``'s ``getsize`` method. By :user:`John Kirkham `; :issue:`431`. Bug fixes ~~~~~~~~~ * Add and use utility functions to simplify reading and writing JSON. By :user:`John Kirkham `; :issue:`429`, :issue:`430`. * Fix ``collections``'s ``DeprecationWarning``\ s. By :user:`John Kirkham `; :issue:`432`. * Fix tests on big endian machines. By :user:`Elliott Sales de Andrade `; :issue:`427`. .. _release_2.3.1: 2.3.1 ----- Bug fixes ~~~~~~~~~ * Makes ``azure-storage-blob`` optional for testing. By :user:`John Kirkham `; :issue:`419`, :issue:`420`. .. _release_2.3.0: 2.3.0 ----- Enhancements ~~~~~~~~~~~~ * New storage backend, backed by Azure Blob Storage, class :class:`zarr.storage.ABSStore`. All data is stored as block blobs. By :user:`Shikhar Goenka `, :user:`Tim Crone ` and :user:`Zain Patel `; :issue:`345`. * Add "consolidated" metadata as an experimental feature: use :func:`zarr.convenience.consolidate_metadata` to copy all metadata from the various metadata keys within a dataset hierarchy under a single key, and :func:`zarr.convenience.open_consolidated` to use this single key. This can greatly cut down the number of calls to the storage backend, and so remove a lot of overhead for reading remote data. By :user:`Martin Durant `, :user:`Alistair Miles `, :user:`Ryan Abernathey `, :issue:`268`, :issue:`332`, :issue:`338`. * Support has been added for structured arrays with sub-array shape and/or nested fields. By :user:`Tarik Onalan `, :issue:`111`, :issue:`296`. * Adds the SQLite-backed :class:`zarr.storage.SQLiteStore` class enabling an SQLite database to be used as the backing store for an array or group. By :user:`John Kirkham `, :issue:`368`, :issue:`365`. * Efficient iteration over arrays by decompressing chunkwise. By :user:`Jerome Kelleher `, :issue:`398`, :issue:`399`. * Adds the Redis-backed :class:`zarr.storage.RedisStore` class enabling a Redis database to be used as the backing store for an array or group. By :user:`Joe Hamman `, :issue:`299`, :issue:`372`. * Adds the MongoDB-backed :class:`zarr.storage.MongoDBStore` class enabling a MongoDB database to be used as the backing store for an array or group. By :user:`Noah D Brenowitz `, :user:`Joe Hamman `, :issue:`299`, :issue:`372`, :issue:`401`. * **New storage class for N5 containers**. The :class:`zarr.n5.N5Store` has been added, which uses :class:`zarr.storage.NestedDirectoryStore` to support reading and writing from and to N5 containers. By :user:`Jan Funke ` and :user:`John Kirkham `. Bug fixes ~~~~~~~~~ * The implementation of the :class:`zarr.storage.DirectoryStore` class has been modified to ensure that writes are atomic and there are no race conditions where a chunk might appear transiently missing during a write operation. By :user:`sbalmer `, :issue:`327`, :issue:`263`. * Avoid raising in :class:`zarr.storage.DirectoryStore`'s ``__setitem__`` when file already exists. By :user:`Justin Swaney `, :issue:`272`, :issue:`318`. * The required version of the `Numcodecs`_ package has been upgraded to 0.6.2, which has enabled some code simplification and fixes a failing test involving msgpack encoding. By :user:`John Kirkham `, :issue:`361`, :issue:`360`, :issue:`352`, :issue:`355`, :issue:`324`. * Failing tests related to pickling/unpickling have been fixed. By :user:`Ryan Williams `, :issue:`273`, :issue:`308`. * Corrects handling of ``NaT`` in ``datetime64`` and ``timedelta64`` in various compressors (by :user:`John Kirkham `; :issue:`344`). * Ensure ``DictStore`` contains only ``bytes`` to facilitate comparisons and protect against writes. By :user:`John Kirkham `, :issue:`350`. * Test and fix an issue (w.r.t. fill values) when storing complex data to ``Array``. By :user:`John Kirkham `, :issue:`363`. * Always use a ``tuple`` when indexing a NumPy ``ndarray``. By :user:`John Kirkham `, :issue:`376`. * Ensure when ``Array`` uses a ``dict``-based chunk store that it only contains ``bytes`` to facilitate comparisons and protect against writes. Drop the copy for the no filter/compressor case as this handles that case. By :user:`John Kirkham `, :issue:`359`. Maintenance ~~~~~~~~~~~ * Simplify directory creation and removal in ``DirectoryStore.rename``. By :user:`John Kirkham `, :issue:`249`. * CI and test environments have been upgraded to include Python 3.7, drop Python 3.4, and upgrade all pinned package requirements. :user:`Alistair Miles `, :issue:`308`. * Start using pyup.io to maintain dependencies. :user:`Alistair Miles `, :issue:`326`. * Configure flake8 line limit generally. :user:`John Kirkham `, :issue:`335`. * Add missing coverage pragmas. :user:`John Kirkham `, :issue:`343`, :issue:`355`. * Fix missing backslash in docs. :user:`John Kirkham `, :issue:`254`, :issue:`353`. * Include tests for stores' ``popitem`` and ``pop`` methods. By :user:`John Kirkham `, :issue:`378`, :issue:`380`. * Include tests for different compressors, endianness, and attributes. By :user:`John Kirkham `, :issue:`378`, :issue:`380`. * Test validity of stores' contents. By :user:`John Kirkham `, :issue:`359`, :issue:`408`. .. _release_2.2.0: 2.2.0 ----- Enhancements ~~~~~~~~~~~~ * **Advanced indexing**. The ``Array`` class has several new methods and properties that enable a selection of items in an array to be retrieved or updated. See the :ref:`tutorial_indexing` tutorial section for more information. There is also a `notebook `_ with extended examples and performance benchmarks. :issue:`78`, :issue:`89`, :issue:`112`, :issue:`172`. * **New package for compressor and filter codecs**. The classes previously defined in the :mod:`zarr.codecs` module have been factored out into a separate package called `Numcodecs`_. The `Numcodecs`_ package also includes several new codec classes not previously available in Zarr, including compressor codecs for Zstd and LZ4. This change is backwards-compatible with existing code, as all codec classes defined by Numcodecs are imported into the :mod:`zarr.codecs` namespace. However, it is recommended to import codecs from the new package, see the tutorial sections on :ref:`tutorial_compress` and :ref:`tutorial_filters` for examples. With contributions by :user:`John Kirkham `; :issue:`74`, :issue:`102`, :issue:`120`, :issue:`123`, :issue:`139`. * **New storage class for DBM-style databases**. The :class:`zarr.storage.DBMStore` class enables any DBM-style database such as gdbm, ndbm or Berkeley DB, to be used as the backing store for an array or group. See the tutorial section on :ref:`tutorial_storage` for some examples. :issue:`133`, :issue:`186`. * **New storage class for LMDB databases**. The :class:`zarr.storage.LMDBStore` class enables an LMDB "Lightning" database to be used as the backing store for an array or group. :issue:`192`. * **New storage class using a nested directory structure for chunk files**. The :class:`zarr.storage.NestedDirectoryStore` has been added, which is similar to the existing :class:`zarr.storage.DirectoryStore` class but nests chunk files for multidimensional arrays into sub-directories. :issue:`155`, :issue:`177`. * **New tree() method for printing hierarchies**. The ``Group`` class has a new :func:`zarr.hierarchy.Group.tree` method which enables a tree representation of a group hierarchy to be printed. Also provides an interactive tree representation when used within a Jupyter notebook. See the :ref:`tutorial_diagnostics` tutorial section for examples. By :user:`John Kirkham `; :issue:`82`, :issue:`140`, :issue:`184`. * **Visitor API**. The ``Group`` class now implements the h5py visitor API, see docs for the :func:`zarr.hierarchy.Group.visit`, :func:`zarr.hierarchy.Group.visititems` and :func:`zarr.hierarchy.Group.visitvalues` methods. By :user:`John Kirkham `, :issue:`92`, :issue:`122`. * **Viewing an array as a different dtype**. The ``Array`` class has a new :func:`zarr.Array.astype` method, which is a convenience that enables an array to be viewed as a different dtype. By :user:`John Kirkham `, :issue:`94`, :issue:`96`. * **New open(), save(), load() convenience functions**. The function :func:`zarr.convenience.open` provides a convenient way to open a persistent array or group, using either a ``DirectoryStore`` or ``ZipStore`` as the backing store. The functions :func:`zarr.convenience.save` and :func:`zarr.convenience.load` are also available and provide a convenient way to save an entire NumPy array to disk and load back into memory later. See the tutorial section :ref:`tutorial_persist` for examples. :issue:`104`, :issue:`105`, :issue:`141`, :issue:`181`. * **IPython completions**. The ``Group`` class now implements ``__dir__()`` and ``_ipython_key_completions_()`` which enables tab-completion for group members to be used in any IPython interactive environment. :issue:`170`. * **New info property; changes to __repr__**. The ``Group`` and ``Array`` classes have a new ``info`` property which can be used to print diagnostic information, including compression ratio where available. See the tutorial section on :ref:`tutorial_diagnostics` for examples. The string representation (``__repr__``) of these classes has been simplified to ensure it is cheap and quick to compute in all circumstances. :issue:`83`, :issue:`115`, :issue:`132`, :issue:`148`. * **Chunk options**. When creating an array, ``chunks=False`` can be specified, which will result in an array with a single chunk only. Alternatively, ``chunks=True`` will trigger an automatic chunk shape guess. See :ref:`tutorial_chunks` for more on the ``chunks`` parameter. :issue:`106`, :issue:`107`, :issue:`183`. * **Zero-dimensional arrays** and are now supported; by :user:`Prakhar Goel `, :issue:`154`, :issue:`161`. * **Arrays with one or more zero-length dimensions** are now fully supported; by :user:`Prakhar Goel `, :issue:`150`, :issue:`154`, :issue:`160`. * **The .zattrs key is now optional** and will now only be created when the first custom attribute is set; :issue:`121`, :issue:`200`. * **New Group.move() method** supports moving a sub-group or array to a different location within the same hierarchy. By :user:`John Kirkham `, :issue:`191`, :issue:`193`, :issue:`196`. * **ZipStore is now thread-safe**; :issue:`194`, :issue:`192`. * **New Array.hexdigest() method** computes an ``Array``'s hash with ``hashlib``. By :user:`John Kirkham `, :issue:`98`, :issue:`203`. * **Improved support for object arrays**. In previous versions of Zarr, creating an array with ``dtype=object`` was possible but could under certain circumstances lead to unexpected errors and/or segmentation faults. To make it easier to properly configure an object array, a new ``object_codec`` parameter has been added to array creation functions. See the tutorial section on :ref:`tutorial_objects` for more information and examples. Also, runtime checks have been added in both Zarr and Numcodecs so that segmentation faults are no longer possible, even with a badly configured array. This API change is backwards compatible and previous code that created an object array and provided an object codec via the ``filters`` parameter will continue to work, however a warning will be raised to encourage use of the ``object_codec`` parameter. :issue:`208`, :issue:`212`. * **Added support for datetime64 and timedelta64 data types**; :issue:`85`, :issue:`215`. * **Array and group attributes are now cached by default** to improve performance with slow stores, e.g., stores accessing data via the network; :issue:`220`, :issue:`218`, :issue:`204`. * **New LRUStoreCache class**. The class :class:`zarr.storage.LRUStoreCache` has been added and provides a means to locally cache data in memory from a store that may be slow, e.g., a store that retrieves data from a remote server via the network; :issue:`223`. * **New copy functions**. The new functions :func:`zarr.convenience.copy` and :func:`zarr.convenience.copy_all` provide a way to copy groups and/or arrays between HDF5 and Zarr, or between two Zarr groups. The :func:`zarr.convenience.copy_store` provides a more efficient way to copy data directly between two Zarr stores. :issue:`87`, :issue:`113`, :issue:`137`, :issue:`217`. Bug fixes ~~~~~~~~~ * Fixed bug where ``read_only`` keyword argument was ignored when creating an array; :issue:`151`, :issue:`179`. * Fixed bugs when using a ``ZipStore`` opened in 'w' mode; :issue:`158`, :issue:`182`. * Fill values can now be provided for fixed-length string arrays; :issue:`165`, :issue:`176`. * Fixed a bug where the number of chunks initialized could be counted incorrectly; :issue:`97`, :issue:`174`. * Fixed a bug related to the use of an ellipsis (...) in indexing statements; :issue:`93`, :issue:`168`, :issue:`172`. * Fixed a bug preventing use of other integer types for indexing; :issue:`143`, :issue:`147`. Documentation ~~~~~~~~~~~~~ * Some changes have been made to the Zarr Specification v2 document to clarify ambiguities and add some missing information. These changes do not break compatibility with any of the material as previously implemented, and so the changes have been made in-place in the document without incrementing the document version number. See the section on changes in the specification document for more information. * A new :ref:`tutorial_indexing` section has been added to the tutorial. * A new :ref:`tutorial_strings` section has been added to the tutorial (:issue:`135`, :issue:`175`). * The :ref:`tutorial_chunks` tutorial section has been reorganised and updated. * The :ref:`tutorial_persist` and :ref:`tutorial_storage` tutorial sections have been updated with new examples (:issue:`100`, :issue:`101`, :issue:`103`). * A new tutorial section on :ref:`tutorial_pickle` has been added (:issue:`91`). * A new tutorial section on :ref:`tutorial_datetime` has been added. * A new tutorial section on :ref:`tutorial_diagnostics` has been added. * The tutorial sections on :ref:`tutorial_sync` and :ref:`tutorial_tips_blosc` have been updated to provide information about how to avoid program hangs when using the Blosc compressor with multiple processes (:issue:`199`, :issue:`201`). Maintenance ~~~~~~~~~~~ * A data fixture has been included in the test suite to ensure data format compatibility is maintained; :issue:`83`, :issue:`146`. * The test suite has been migrated from nosetests to pytest; :issue:`189`, :issue:`225`. * Various continuous integration updates and improvements; :issue:`118`, :issue:`124`, :issue:`125`, :issue:`126`, :issue:`109`, :issue:`114`, :issue:`171`. * Bump numcodecs dependency to 0.5.3, completely remove nose dependency, :issue:`237`. * Fix compatibility issues with NumPy 1.14 regarding fill values for structured arrays, :issue:`222`, :issue:`238`, :issue:`239`. Acknowledgments ~~~~~~~~~~~~~~~ Code was contributed to this release by :user:`Alistair Miles `, :user:`John Kirkham ` and :user:`Prakhar Goel `. Documentation was contributed to this release by :user:`Mamy Ratsimbazafy ` and :user:`Charles Noyes `. Thank you to :user:`John Kirkham `, :user:`Stephan Hoyer `, :user:`Francesc Alted `, and :user:`Matthew Rocklin ` for code reviews and/or comments on pull requests. .. _release_2.1.4: 2.1.4 ----- * Resolved an issue where calling ``hasattr`` on a ``Group`` object erroneously returned a ``KeyError``. By :user:`Vincent Schut `; :issue:`88`, :issue:`95`. .. _release_2.1.3: 2.1.3 ----- * Resolved an issue with :func:`zarr.creation.array` where dtype was given as None (:issue:`80`). .. _release_2.1.2: 2.1.2 ----- * Resolved an issue when no compression is used and chunks are stored in memory (:issue:`79`). .. _release_2.1.1: 2.1.1 ----- Various minor improvements, including: ``Group`` objects support member access via dot notation (``__getattr__``); fixed metadata caching for ``Array.shape`` property and derivatives; added ``Array.ndim`` property; fixed ``Array.__array__`` method arguments; fixed bug in pickling ``Array`` state; fixed bug in pickling ``ThreadSynchronizer``. .. _release_2.1.0: 2.1.0 ----- * Group objects now support member deletion via ``del`` statement (:issue:`65`). * Added :class:`zarr.storage.TempStore` class for convenience to provide storage via a temporary directory (:issue:`59`). * Fixed performance issues with :class:`zarr.storage.ZipStore` class (:issue:`66`). * The Blosc extension has been modified to return bytes instead of array objects from compress and decompress function calls. This should improve compatibility and also provides a small performance increase for compressing high compression ratio data (:issue:`55`). * Added ``overwrite`` keyword argument to array and group creation methods on the :class:`zarr.hierarchy.Group` class (:issue:`71`). * Added ``cache_metadata`` keyword argument to array creation methods. * The functions :func:`zarr.creation.open_array` and :func:`zarr.hierarchy.open_group` now accept any store as first argument (:issue:`56`). .. _release_2.0.1: 2.0.1 ----- The bundled Blosc library has been upgraded to version 1.11.1. .. _release_2.0.0: 2.0.0 ----- Hierarchies ~~~~~~~~~~~ Support has been added for organizing arrays into hierarchies via groups. See the tutorial section on :ref:`tutorial_groups` and the :mod:`zarr.hierarchy` API docs for more information. Filters ~~~~~~~ Support has been added for configuring filters to preprocess chunk data prior to compression. See the tutorial section on :ref:`tutorial_filters` and the :mod:`zarr.codecs` API docs for more information. Other changes ~~~~~~~~~~~~~ To accommodate support for hierarchies and filters, the Zarr metadata format has been modified. See the :ref:`spec_v2` for more information. To migrate an array stored using Zarr version 1.x, use the :func:`zarr.storage.migrate_1to2` function. The bundled Blosc library has been upgraded to version 1.11.0. Acknowledgments ~~~~~~~~~~~~~~~ Thanks to :user:`Matthew Rocklin `, :user:`Stephan Hoyer ` and :user:`Francesc Alted ` for contributions and comments. .. _release_1.1.0: 1.1.0 ----- * The bundled Blosc library has been upgraded to version 1.10.0. The 'zstd' internal compression library is now available within Blosc. See the tutorial section on :ref:`tutorial_compress` for an example. * When using the Blosc compressor, the default internal compression library is now 'lz4'. * The default number of internal threads for the Blosc compressor has been increased to a maximum of 8 (previously 4). * Added convenience functions :func:`zarr.blosc.list_compressors` and :func:`zarr.blosc.get_nthreads`. .. _release_1.0.0: 1.0.0 ----- This release includes a complete re-organization of the code base. The major version number has been bumped to indicate that there have been backwards-incompatible changes to the API and the on-disk storage format. However, Zarr is still in an early stage of development, so please do not take the version number as an indicator of maturity. Storage ~~~~~~~ The main motivation for re-organizing the code was to create an abstraction layer between the core array logic and data storage (:issue:`21`). In this release, any object that implements the ``MutableMapping`` interface can be used as an array store. See the tutorial sections on :ref:`tutorial_persist` and :ref:`tutorial_storage`, the :ref:`spec_v1`, and the :mod:`zarr.storage` module documentation for more information. Please note also that the file organization and file name conventions used when storing a Zarr array in a directory on the file system have changed. Persistent Zarr arrays created using previous versions of the software will not be compatible with this version. See the :mod:`zarr.storage` API docs and the :ref:`spec_v1` for more information. Compression ~~~~~~~~~~~ An abstraction layer has also been created between the core array logic and the code for compressing and decompressing array chunks. This release still bundles the c-blosc library and uses Blosc as the default compressor, however other compressors including zlib, BZ2 and LZMA are also now supported via the Python standard library. New compressors can also be dynamically registered for use with Zarr. See the tutorial sections on :ref:`tutorial_compress` and :ref:`tutorial_tips_blosc`, the :ref:`spec_v1`, and the :mod:`zarr.compressors` module documentation for more information. Synchronization ~~~~~~~~~~~~~~~ The synchronization code has also been refactored to create a layer of abstraction, enabling Zarr arrays to be used in parallel computations with a number of alternative synchronization methods. For more information see the tutorial section on :ref:`tutorial_sync` and the :mod:`zarr.sync` module documentation. Changes to the Blosc extension ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NumPy is no longer a build dependency for the :mod:`zarr.blosc` Cython extension, so setup.py will run even if NumPy is not already installed, and should automatically install NumPy as a runtime dependency. Manual installation of NumPy prior to installing Zarr is still recommended, however, as the automatic installation of NumPy may fail or be sub-optimal on some platforms. Some optimizations have been made within the :mod:`zarr.blosc` extension to avoid unnecessary memory copies, giving a ~10-20% performance improvement for multi-threaded compression operations. The :mod:`zarr.blosc` extension now automatically detects whether it is running within a single-threaded or multi-threaded program and adapts its internal behaviour accordingly (:issue:`27`). There is no need for the user to make any API calls to switch Blosc between contextual and non-contextual (global lock) mode. See also the tutorial section on :ref:`tutorial_tips_blosc`. Other changes ~~~~~~~~~~~~~ The internal code for managing chunks has been rewritten to be more efficient. Now no state is maintained for chunks outside of the array store, meaning that chunks do not carry any extra memory overhead not accounted for by the store. This negates the need for the "lazy" option present in the previous release, and this has been removed. The memory layout within chunks can now be set as either "C" (row-major) or "F" (column-major), which can help to provide better compression for some data (:issue:`7`). See the tutorial section on :ref:`tutorial_chunks_order` for more information. A bug has been fixed within the ``__getitem__`` and ``__setitem__`` machinery for slicing arrays, to properly handle getting and setting partial slices. Acknowledgments ~~~~~~~~~~~~~~~ Thanks to :user:`Matthew Rocklin `, :user:`Stephan Hoyer `, :user:`Francesc Alted `, :user:`Anthony Scopatz ` and :user:`Martin Durant ` for contributions and comments. .. _release_0.4.0: 0.4.0 ----- See `v0.4.0 release notes on GitHub `_. .. _release_0.3.0: 0.3.0 ----- See `v0.3.0 release notes on GitHub `_. .. _Numcodecs: https://numcodecs.readthedocs.io/