{"added":null,"apiVersion":1,"behindSince":null,"buildDepends":[],"categories":["filesystems","python"],"comment":"Functions to make reference descriptions for ReferenceFileSystem","compare":null,"dependentCount":null,"description":"Kerchunk is a library that provides a unified way to represent a variety of\nchunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient\naccess to the data from traditional file systems or cloud object storage. It\nalso provides a flexible way to create virtual datasets from multiple files. It\ndoes this by extracting the byte ranges, compression information and other\ninformation about the data and storing this metadata in a new, separate object.\nThis means that you can create a virtual aggregate dataset over potentially many\nsource files, for efficient, parallel and cloud-friendly in-situ access without\nhaving to copy or translate the originals. It is a gateway to in-the-cloud\nmassive data processing while the data providers still insist on using legacy\nformats for archival storage.\n\nWe provide the following things:\n- completely serverless architecture\n- metadata consolidation, so you can understand a many-file dataset (metadata\n  plus physical storage) in a single read\n- read from all of the storage backends supported by fsspec, including object\n  storage (s3, gcs, abfs, alibaba), http, cloud user storage (dropbox, gdrive)\n  and network protocols (ftp, ssh, hdfs, smb...)\n- loading of various file types (currently netcdf4/HDF, grib2, tiff, fits,\n  zarr), potentially heterogeneous within a single dataset, without a need to go\n  via the specific driver (e.g., no need for h5py)\n- asynchronous concurrent fetch of many data chunks in one go, amortizing the\n  cost of latency\n- parallel access with a library like zarr without any locks\n- logical datasets viewing many (\u003e~millions) data files, and direct\n  access/subselection to them via coordinate indexing across an arbitrary number\n  of dimensions","flavors":[],"html":"https://daemonports.org/dragonfly/filesystems/py-kerchunk/","kind":"port","lastUpdate":null,"license":["MIT"],"licenses":["MIT"],"maintainer":"sunpoet@FreeBSD.org","name":"py312-kerchunk","newest":true,"newestVersion":"0.2.10","noVulnData":false,"origin":"filesystems/py-kerchunk","otherSystems":[{"maintainer":"sunpoet@FreeBSD.org","newest":true,"origin":"filesystems/py-kerchunk","path":"/filesystems/py-kerchunk/","system":"FreeBSD","updated":"2026-08-19","version":"0.2.10"},{"maintainer":"sunpoet@FreeBSD.org","newest":true,"noVulnData":true,"origin":"filesystems/py-kerchunk","path":"","system":"HardenedBSD","version":"0.2.10"},{"maintainer":"sunpoet@FreeBSD.org","newest":true,"noVulnData":true,"origin":"filesystems/py-kerchunk","path":"","system":"GhostBSD","version":"0.2.10"}],"packages":null,"pkgName":"py312-kerchunk-0.2.10","related":null,"repository":"DPorts","requiredBy":[],"requiredByCount":0,"runDepends":["devel/py-ujson","devel/py-zarr","filesystems/py-fsspec","lang/python312","math/py-numpy","misc/py-numcodecs"],"status":null,"system":"dragonfly","systemLabel":"DragonFly","version":"0.2.10","versions":[{"date":"2026-11-11","version":"0.2.10"}],"vulnerabilities":null,"www":"https://fsspec.github.io/kerchunk/"}
