diff --git a/.gitignore b/.gitignore index d0ecfbc..5876499 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,5 @@ __pycache__/ # Structure outputs / trajectories (large; keep out of git history) *.sdf +# PyTorch checkpoints (large; regenerate locally; do not push to git) +*.pt diff --git a/GUIDELINES.md b/GUIDELINES.md index b674d3e..e4d5c47 100644 --- a/GUIDELINES.md +++ b/GUIDELINES.md @@ -9,8 +9,8 @@ Make overfitting robust and measurable, targeting `mean_rmsd_100 <= 1.0`. 1. **Branch per line of work**: create a branch (e.g. `attempt/`) before changing `train.py` for a new experiment. 2. Modify code/config, run training, write `reports/latest_eval.json`. 3. Append one line to `README.md` attempt log for every attempt (success and failure). -4. **Never commit `train.py` and `README.md` in the same git commit.** After a training run: (a) commit code/eval artifacts (`train.py`, `reports/latest_eval.json`, checkpoints, etc.) without `README.md`; then (b) make a **docs-only** commit that touches **only** `README.md` (attempt log line). Pre-commit enforces this split so `README.md` can be cherry-picked to `main` without dragging `train.py` along. Feature-branch commits are not blocked by the mean-RMSD performance gate. -5. When a branch is ready to land: **merge (or cherry-pick) into `main`**. The performance gate and `BEST_PRACTICE.json` / best-artifact refresh run only on **`main`** when `train.py` is part of the commit. +4. **Never commit `train.py` and `README.md` in the same git commit.** After a training run: (a) commit code/eval artifacts (`train.py`, `reports/latest_eval.json`, etc.) without `README.md` (PyTorch `*.pt` checkpoints are local-only, gitignored); then (b) make a **docs-only** commit that touches **only** `README.md` (attempt log line). Pre-commit enforces this split so `README.md` can be cherry-picked to `main` without dragging `train.py` along. Feature-branch commits are not blocked by the mean-RMSD performance gate. +5. When a branch is ready to land: **merge (or cherry-pick) into `main`**. The performance gate and `BEST_PRACTICE.json` refresh run only on **`main`** when `train.py` is part of the commit (checkpoints are not committed). 6. **`README.md` attempt log must also live on `main`**: if you only merged code later or abandoned a `train.py` merge, still **bring new `## Attempt Log` lines onto `main`** soon after (docs-only commit is fine—stage **only** `README.md` so the mean-RMSD gate does not run). Cherry-pick the README hunk from the branch or copy the lines; do not leave the canonical log only on a feature branch. 7. **Mandatory best-update integration**: if any feature-branch attempt records a strictly better `mean_rmsd_100` than the current `main` anchor, treat it as merge-ready work. Merge/cherry-pick it into `main` promptly (do not keep a known best only on a feature branch), then continue new experiments from a fresh branch off updated `main`. 8. **Per-attempt logging+commit is mandatory**: every experiment run must immediately append its result to `README.md`, then record it in git **before** the next run—but as **separate commits** from rule 4: code commit first, `README.md`-only commit second (same attempt, two commits minimum when both files change). diff --git a/README.md b/README.md index 42882db..72bb4d8 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,8 @@ This repository is intentionally pinned to CUDA 12.6 PyTorch wheels and matching - Every attempt must update this README (append a short entry in `## Attempt Log`). - Attempt log is mandatory for both successful and failed trials. -- **Branch-first attempts**: do training experiments on a **feature branch**; **commit each attempt** as **two commits** when both change: (1) `train.py` plus eval artifacts (`reports/latest_eval.json`, checkpoints, …) **without** `README.md`; (2) a **docs-only** commit with **only** `README.md` (attempt log). Pre-commit blocks staging `train.py` and `README.md` together. Pre-commit does **not** enforce the mean-RMSD improvement rule on feature branches. -- **Main is the gate**: merging or committing to **`main`** with `train.py` staged triggers the performance gate (strictly better `mean_rmsd_100`, staged `latest_eval`, README log, auto-update of `BEST_PRACTICE.json` and best artifacts). Land work via merge or **cherry-pick** of the commits you still trust after re-evaluation. +- **Branch-first attempts**: do training experiments on a **feature branch**; **commit each attempt** as **two commits** when both change: (1) `train.py` plus eval artifacts (`reports/latest_eval.json`, …) **without** `README.md` (PyTorch `*.pt` checkpoints are **not** tracked—local only); (2) a **docs-only** commit with **only** `README.md` (attempt log). Pre-commit blocks staging `train.py` and `README.md` together. Pre-commit does **not** enforce the mean-RMSD improvement rule on feature branches. +- **Main is the gate**: merging or committing to **`main`** with `train.py` staged triggers the performance gate (strictly better `mean_rmsd_100`, staged `latest_eval`, README log, auto-update of `BEST_PRACTICE.json`; checkpoints remain local). Land work via merge or **cherry-pick** of the commits you still trust after re-evaluation. - **`## Attempt Log` on `main`**: new log lines written on a feature branch must be **replicated on `main`** (docs-only `README.md` commit if `train.py` is not landing yet). See `GUIDELINES.md` workflow step 6. - Flow-matching training time must stay random (middle-time supervision is mandatory). - Independent attempts must be research-level changes (architecture/training strategy/loss design). Pure hyperparameter-only runs are not counted as standalone attempts. @@ -34,10 +34,9 @@ This repository is intentionally pinned to CUDA 12.6 PyTorch wheels and matching - `GUIDELINES.md`: operating rules and workflow. - `BEST_PRACTICE.json`: current best-known metric and config. - `reports/latest_eval.json`: most recent measured metric. -- `artifacts/latest_eval_best_model.pt`: checkpoint from latest run that produced `latest_eval`. -- `artifacts/best_model.pt`: best checkpoint from latest improved run. -- `reports/trajectories/`: trajectory SDFs are **gitignored** (`*.sdf`); regenerate locally after training when needed. -- `scripts/precommit_performance_gate.py`: flow-matching token check on any branch when `train.py` is staged; **mean-RMSD gate and best-artifact refresh only on `main`**. +- `artifacts/*.pt`: checkpoints are **gitignored**; written locally by `train.py` / hooks (`latest_eval_best_model.pt`, `best_model.pt`). +- `reports/trajectories/`: trajectory SDFs are **gitignored** (`*.sdf`); regenerate locally (`python scripts/update_best_artifacts.py` after training when needed). +- `scripts/precommit_performance_gate.py`: flow-matching token check on any branch when `train.py` is staged; **mean-RMSD gate and `BEST_PRACTICE.json` refresh only on `main`** (does not stage `.pt` / `.sdf`). ## Attempt Log @@ -107,3 +106,4 @@ This repository is intentionally pinned to CUDA 12.6 PyTorch wheels and matching - 2026-04-17: Fast-forward merged `attempt/post-main-doc-sync` into `main`; committed `BEST_PRACTICE.json` + `artifacts/best_model.pt` sync to anchor `mean_rmsd_100=2.350750` (command matches no-EMA residual-geodesic run). - 2026-04-17: Branch `attempt/graph-readout-geodesic` (off updated `main`): identical budget vs anchor but `--graph-readout attention` reached `mean_rmsd_100=2.716856` with early `train_mse=1000` wall; worse than anchor `2.350750`; branch not for merging to `main`. - 2026-04-17: Fixed `scripts/update_best_artifacts.py` to build `RFMModel` with the same flags as training (notably `gcn_residual`); best-practice trajectories had looked static because the forward pass did not match the checkpoint. Checkpoints now store RFM metadata; old runs infer from `BEST_PRACTICE.json` command when keys are absent. +- 2026-04-17: Removed all `*.pt` and `*.sdf` from git history (`git-filter-repo`); added `*.pt` to `.gitignore`; pre-commit no longer stages checkpoints. `git-filter-repo` drops `origin`—re-add with `git remote add origin ` before push. Pushes to existing remotes need **`git push --force-with-lease`** once because history was rewritten. diff --git a/scripts/precommit_performance_gate.py b/scripts/precommit_performance_gate.py index f9301b9..14a5dc3 100644 --- a/scripts/precommit_performance_gate.py +++ b/scripts/precommit_performance_gate.py @@ -124,11 +124,9 @@ def run_main_gate(previous_best: float, mode: str) -> int: if mode == "pre-commit": subprocess.check_call(["git", "add", str(BEST)], cwd=ROOT) subprocess.check_call(["git", "add", "reports/trajectories"], cwd=ROOT) - subprocess.check_call(["git", "add", "artifacts/best_model.pt"], cwd=ROOT) - subprocess.check_call(["git", "add", "artifacts/latest_eval_best_model.pt"], cwd=ROOT) print( f"{prefix} PASS: improved mean_rmsd_100 {previous_best:.6f} -> {latest_rmsd:.6f}; " - "BEST_PRACTICE.json and best artifacts refreshed", + "BEST_PRACTICE.json refreshed; checkpoints (*.pt) and *.sdf stay local (gitignored, not staged).", file=sys.stderr, ) return 0