filesystems/py-kerchunk 0.2.10 DragonFly
Functions to make reference descriptions for ReferenceFileSystem
- System
- DragonFly
- Repository
- DPorts
- Maintainer
- [email protected] (as in FreeBSD)
- License
- MIT
- Categories
- filesystems, python
- Package size
- 138.0 KiB, 708.3 KiB installed
Description
Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files. It does this by extracting the byte ranges, compression information and other information about the data and storing this metadata in a new, separate object. This means that you can create a virtual aggregate dataset over potentially many source files, for efficient, parallel and cloud-friendly in-situ access without having to copy or translate the originals. It is a gateway to in-the-cloud massive data processing while the data providers still insist on using legacy formats for archival storage.
We provide the following things:
- completely serverless architecture
- metadata consolidation, so you can understand a many-file dataset (metadata plus physical storage) in a single read
- read from all of the storage backends supported by fsspec, including object storage (s3, gcs, abfs, alibaba), http, cloud user storage (dropbox, gdrive) and network protocols (ftp, ssh, hdfs, smb...)
- loading of various file types (currently netcdf4/HDF, grib2, tiff, fits, zarr), potentially heterogeneous within a single dataset, without a need to go via the specific driver (e.g., no need for h5py)
- asynchronous concurrent fetch of many data chunks in one go, amortizing the cost of latency
- parallel access with a library like zarr without any locks
- logical datasets viewing many (>~millions) data files, and direct access/subselection to them via coordinate indexing across an arbitrary number of dimensions
In the other systems
The newest version among them is 0.2.10, as upstream numbers it.
| System | Port | Version | Vulnerabilities | Status | Maintainer | Last update |
|---|---|---|---|---|---|---|
| DragonFly | filesystems/py-kerchunk | 0.2.10 newest | none known | - | [email protected] | - |
| FreeBSD | filesystems/py-kerchunk | 0.2.10 newest | none known | - | [email protected] | 2026-08-19 |
| HardenedBSD | filesystems/py-kerchunk | 0.2.10 newest | no data | - | [email protected] | - |
| GhostBSD | filesystems/py-kerchunk | 0.2.10 newest | no data | - | [email protected] | - |
Version history 1
- 0.2.10
Upstream's version as each commit to the port's Makefile set it, or as this site first saw it in the system's packages; the tree's own revisions are not versions here.
Install
Binary package:
pkg install py312-kerchunk
From DPorts:
cd /usr/dports/filesystems/py-kerchunk && make install clean
Dependency graph
Ports that need it on the left, what it needs to run on the right; at most 8 of each. "Dependencies" below lists them all.
Dependencies
Run dependencies 6
No port depends on this one.