Datasets¶

chronicle.datasets turns a file or directory of training data into a durable, provenance-stamped asset you can link as an experiment- or variation-level input and load back later.

Datasets are plain assets — there's no separate dataset table. "dataset" is an ordinary free-form asset_type, search-indexed on metadata only (name + provenance record, never the bytes), so a dataset travels the same presigned-PUT flow as any other binary asset (see the assets guide).

Three ways to register a dataset¶

Method	Bytes live where	Copies bytes?	Returns
`upload`	local file/dir	yes — presigned PUT per component	`DatasetRef`
`register`	you PUT them yourself	yes — you drive the PUTs	`AssetUploadInfo`
`register_by_reference`	external `gs://` / `s3://` URI	no — registers in place	created asset dict

upload — local bytes¶

upload hashes the bytes, computes a provenance record, presigns, PUTs each component, and finalizes — one call. A single file is one component; a directory becomes one component per file (the way to shard GB-scale data, since the S3 fallback caps a single PUT).

ref = chronicle.datasets.upload(
    "./data/burgers_2d.npz",
    name="burgers-2d-re1000",
    source="generated by solvers/burgers.py @ a1b2c3d",
    provenance={"pde_family": "burgers", "n_dims": 2, "num_samples": 50_000, "precision": "fp32"},
    link_experiment="exp_7Qk",          # link as an experiment input in the same call
)
print(ref.asset_id, ref.asset_uri, ref.components)

provenance carries domain facts you know (shape, rows, schema, generator config); the SDK merges them on top of the verifiable facts it computes itself — per-component sha256 + size_bytes, total size, content type, and the uploading SDK version. Pass output_of={"experiment_id": ..., "variation": ..., "run": ...} instead to record the run that produced the dataset (this requires Write on that experiment server-side). upload returns a DatasetRef: asset_id, asset_uri, name, asset_type, components, provenance.

Owning scope

organization_id / team_id / visibility set the owning scope, same model as experiments.create. Omitting organization_id on a researcher-initiated upload uses the client's default org; pass methodic.PERSONAL to force a personal-scope upload.

register — drive the PUTs yourself¶

When you want to handle the component transfer (custom transport, resumable retries, externally generated data), register creates the asset record and presigned URLs without uploading — you own the upload + finalize steps:

info = chronicle.datasets.register(
    components=["train.safetensors", "val.safetensors", "meta.json"],
    name="navier-stokes-v3",
    provenance={"pde_family": "navier_stokes", "n_dims": 3},
)
for comp, url in info.upload_urls.items():
    chronicle.assets.upload_component(url, local_path=Path(f"./out/{comp}"), content_type="application/octet-stream")
chronicle.assets.finalize(info.asset_id)

provenance is stored verbatim under asset_config.provenance.

register_by_reference — external URI, no copy¶

For data already sitting in a bucket (uploaded out-of-band), register it in place. The asset is created ready in one call — no upload step. metadata lands under asset_config.metadata and is projected to the promoted columns + Vertex search facets; size_bytes is author-declared:

asset = chronicle.datasets.register_by_reference(
    "gs://methodic-pde-corpus/heat-equation/2d/",
    name="heat-2d-corpus",
    size_bytes=4_200_000_000,
    metadata={"pde_family": "heat", "n_dims": 2, "geometry": "rectangular"},
)

Metadata vs provenance¶

Two distinct things ride on a dataset asset. Provenance is the verifiable lineage record (hashes, sizes, source, generator config) — read it back with chronicle.datasets.provenance(asset_id), which returns the stored record or None. Metadata is a mutable descriptive annotation: update_metadata changes it without touching the immutable content (uri / sha256 / components), and recomputes the promoted columns + Vertex projection. Requires Write on the asset.

chronicle.datasets.update_metadata(asset_id, metadata={"precision": "fp64", "license": "CC-BY-4.0"})

Linking as an input¶

link attaches a dataset as an experiment- or variation-level input. Either target must be open — the server returns 409 once it's committed.

chronicle.datasets.link(asset_id, "exp_7Qk")                       # experiment-level input
chronicle.datasets.link(asset_id, "exp_7Qk", variation=2)          # variation-level input

ACL propagation

Experiment-level linking stamps the experiment's auto-roles onto the asset by default (propagate_acl=True) so experiment members can read it — pass propagate_acl=False for a sensitive dataset that shouldn't inherit experiment ACLs. Variation-level linking ignores propagate_acl: the server does not stamp variation inputs. Use allow_invalid_assets=True to link a deprecated/invalidated asset.

Finding and loading¶

list does cheap Postgres-side filtering over the promoted columns; for paginated, semantic discovery use chronicle.search — list is the lightweight filter, not the primary search surface.

hits = chronicle.datasets.list(
    pde_family="navier_stokes", min_dims=2, precision="fp32",
    min_size=1_000_000, order_by="num_samples",     # order_by ∈ size_bytes, created_at, num_samples
)

Filter	Matches on
`n_dims` / `min_dims` / `max_dims`	spatial dimensionality
`precision`	declared float precision
`pde_family`	PDE family tag
`geometry`	domain geometry
`min_size` / `max_size`	`size_bytes` range
`owner`	owning principal

Then inspect or pull the bytes. get(asset_id, include_presigned=True) returns metadata plus presigned read URLs in one call; load streams every component to disk and returns the destination directory:

meta = chronicle.datasets.get(asset_id)             # asset metadata dict
chronicle.datasets.load(asset_id, dest="./data")    # downloads every component → ./data