LoRA Speedrun 公开排行榜：6分05秒微调Qwen2.5-1.5B达GSM8K 61.1%准确率

How fast can you LoRA-fine-tune Qwen2.5-1.5B to ≥ 57% on GSM8K — on a single L40S?

This is modded-nanogpt for fine-tuning: a frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is independently re-run 3× with fresh seeds on identical hardware before it counts.

Attempting and verifying are free: official timing runs on a Modal L40S sandbox, and Modal's free monthly compute credits cover full runs — so anyone can compete, and anyone can re-verify any record with one command.

Leaderboard

Track 1 — GSM8K · Qwen2.5-1.5B · target ≥ 57.0% · 1× L40S

Current record: 6m 05s by @Saivineeth147 — Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy.

Track 2 — SQuAD v1.1 · SmolLM2-1.7B · target ≥ 75.5% · 1× L40S

Current record: 11m 08s by @Saivineeth147 — Track 2 baseline: plain LoRA r=16 on SQuAD, first 20k examples, 1 epoch, full-sequence loss. No tricks.

Full history with verification reports: records/RECORDS.md

The tracks

Two frozen tracks, deliberately different model families and task types — so a technique only proves general by winning on both. Same hardware, caps, and verification everywhere.

Machine-readable specs: spec.yaml · spec-t2.yaml. Full rules: TASK.md.

You control everything else: LoRA rank and placement, quantization, learning-rate schedules, sequence packing, data subset selection and ordering, custom kernels, when to stop. Train on 1,000 well-chosen examples for 90 seconds if you can make it clear the bar.

Why this exists

LoRA/QLoRA is how most real-world fine-tuning actually happens, and the technique space exploded — DoRA, rsLoRA, PiSSA, LoRA+, NEFTune, Unsloth kernels, rank-adaptive methods — but there's no adversarial, apples-to-apples arena where these ideas race each other in public. Papers report numbers on different models, data, and hardware; nothing is comparable.

The nanoGPT speedrun fixed this for pretraining and produced real science (Muon came out of it). This repo does the same for parameter-efficient fine-tuning: one frozen task, one GPU, wall-clock time, receipts required.

Quickstart

Official-hardware run (free). Make a Modal account, then:

git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun
pip install modal pyyaml && modal setup      # one-time browser auth

python harness/modal_verify.py --prefetch    # one-time: cache model + data in a volume

# one timed, evaluated attempt of the baseline on the exact spec hardware:
python harness/modal_verify.py --submission submissions/000-baseline --runs 1

# full record-style verification (3 fresh seeds, all must pass):
python harness/modal_verify.py --submission submissions/000-baseline --runs 3

Local iteration (optional). Any 24 GB+ card runs the baseline for fast experimenting — bash scripts/setup_gpu.sh, then python harness/run_submission.py submissions/000-baseline --runs 1. Local times aren't official; the leaderboard clock is the Modal L40S.

Then copy submissions/TEMPLATE/, make it faster, and open a PR. See CONTRIBUTING.md.

How records get verified

A real verification, replayed from the logs (seed 463953844, time-compressed): train → integrity + adapter audit → eval → 3-seed verdict.

1. You open a PR with your training script, config, notes, and self-reported numbers.
2. CI statically validates it, and an automated Claude security screen reviews the diff (exfiltration attempts, network use, harness tampering, test-set contact) and posts its findings publicly.
3. A maintainer reviews the code, then comments /verify — which re-runs your submission 3× with fresh seeds in a network-blocked Modal sandbox on the spec L40S. All 3 runs must clear the target; official time is the mean.
4. The harness audits the adapter param count and re-verifies model/data content hashes (anti-tampering), and the verification report is posted on the PR and committed to records/verifications/ with the accept/reject reasoning.

Full protocol, rubric, and threat model: JUDGING.md · SECURITY.md.

Ideas nobody has claimed yet

Taken so far: sequence packing + completion-only masking (record #1).

1-epoch aggressive-LR schedules · data pruning (train on the hardest 2k examples?) · block-diagonal/varlen packing attention · QLoRA NF4 vs bf16 tradeoff · rsLoRA / DoRA / PiSSA init · LoRA+ (asymmetric LR for A/B) · NEFTune noise · curriculum ordering · rank/placement search (MLP-only vs attention-only) · torch.compile · Unsloth kernels · Liger kernels · fused cross-entropy · smarter warmup for short runs

Claim one, beat 6m 05s, get your name on the board.

FAQ

Won't techniques overfit to one model + one task? That's exactly why there are two tracks with different model families and task types — and more will follow the same freeze-and-calibrate protocol. A trick that only wins on one track is a record, but the techniques worth trusting are the ones that transfer. The track system makes that an empirical question instead of an argument.

Why Qwen2.5-1.5B? Isn't it pretrained on math? Probably, like every modern base model. It doesn't matter: the target is an anchor, not a claim about mathematical discovery. The race is the interesting part — same reason nanoGPT speedrunning targets an arbitrary val loss. (Track 2 uses a different family, SmolLM2, partly for this reason.)

Why wall-clock instead of FLOPs or steps? Because wall-clock is what you pay for, and it forces kernels, data loading, and algorithms to compete in the same currency. Same rule as modded-nanogpt.

Why an L40S on Modal instead of a 4090 or H100? Three reasons. It's one consistent datacenter SKU, so times are actually comparable (rented consumer cards vary host-to-host). It's free to use via Modal's monthly credits, so competing and re-verifying costs nothing. And submissions are strangers' code — Modal sandboxes run them network-blocked and secretless. (The L40S is the same AD102 silicon as the 4090, so consumer-GPU tricks transfer.)

Can I train on other data / distill from a bigger model? No. GSM8K train split only, no teacher models, no synthetic data. See TASK.md for the full banned list.

Multiple GPUs? No. One L40S. That's the point.

License

MIT. Records, reports, and write-ups are as public as the code.