Dataset operations

Versioning basics

At minimum, every training snapshot needs:

• A content hash (SHA-256 of the normalized file or manifest of row IDs)
• Row count and byte size after dedup
• Git commit or pipeline run ID that built it
• Filter and dedup config (thresholds, blocklists, date ranges)
• Parent version if this is a delta on top of an earlier set

Tools like DVC, LakeFS, or plain object storage with immutable prefixes all work. The habit matters more than the brand: never overwrite train.jsonl in place. Write train-v2026-06-29-a3f9.jsonl or store by hash.

For LLM fine-tuning, a million tokens of JSONL is often 2–8 MB compressed. Storage is cheap. Debugging an unversioned file is not.

Dataset cards and lineage

A dataset card documents what is inside and how it was built. Google's Data Cards playbook and Hugging Face dataset READMEs follow the same idea: provenance, collection process, known biases, sensitive fields, and intended use.

Include:

• Sources (logs, vendors, synthetic generator + model version)
• Labeling rubric version and agreement stats from pilots
• Dedup method and how many rows were removed
• Known gaps ("weak on non-English," "no medical cases")
• License and retention constraints

Lineage links upstream and downstream: raw export → filtered → labeled → deduped → split → training run → deployed model ID. When a user reports a bad answer, you should trace which data version taught that behavior.

Privacy and compliance review

Before a dataset enters training or a shared bucket:

1. Scan for PII, secrets, and internal identifiers.
2. Confirm consent and retention policy for logged data.
3. Restrict access (who can download, who can add rows).
4. Document redaction rules (what was stripped, what was generalized).

Privacy review is not a one-time gate. A refresh that pulls new logs needs the same checks. Automated scanners catch obvious emails and API keys; policy review catches contextual leaks (customer names in free text, account numbers in ticket subjects).

Refresh cadence and slice tracking

Production traffic shifts. Plan refreshes on a schedule tied to risk:

• High-risk domains (finance, health, safety): review failures weekly, full refresh monthly or on policy change.
• Stable domains: quarterly refresh plus ad-hoc batches from active learning.

Track slices in metadata: intent, locale, source, labeler, synthetic flag. Eval dashboards should read the same slice keys. When aggregate accuracy is flat but the "refund_escalation" slice drops, you want to see it before users do.

Feed production failures back through the pipeline from lesson 02: filter, label, dedup, version, train, eval, deploy. Close the loop or the model freezes in time.

Linking data version to model version

Your model card or release notes should list:

• Base model checkpoint
• Training data hash and dataset card link
• Eval set version and pass/fail per slice
• Training config (LR, epochs, LoRA rank if applicable)

In production, store the data version in the same config object as the model endpoint. When you A/B test, you are often testing model + data + prompt together. Name that bundle explicitly.

Where this course goes next

You now have the full data path: what counts as a dataset, how to label it, when to use synthetic data, how to deduplicate and check contamination, and how to operate datasets in production.

Next stops in the curriculum:

• Prepare a fine-tuning dataset applies this to JSONL and training jobs.
• Task evals and golden sets defines how you measure before you ship.
• Build preference datasets when alignment needs pairwise judgment.

Quick check