ProteinGymLLM
PaperCode ↗

Run PG-LLM

Score a new model on the benchmark.

Quickstart

git clone https://github.com/rohitarorayyc/proteingym-llm.git
cd proteingym-llm
python -m pip install -e .
pgllm-data
pgllm-init

pgllm-data downloads and verifies the frozen 217-assay split bundle. Add the provider key to the private .env created by pgllm-init.

What pgllm-run does

For every model, assay, set size, and seed, the runner records the exact prompt and split hashes, returned model ID, provider usage, reasoning, raw response, stop reason, and request fingerprint. Only complete, schema-valid rankings enter the scored tree. Failed calls and interrupted streams remain immutable attempts outside it.

pgllm-score aggregates assay → protein → functional category, with equal weight at each higher level. The same scorer is used for the public leaderboard.

Example: Kimi K3

The included Kimi profile runs the primary benchmark: N=50, maximum reasoning, and all three fixed seeds. Begin with one assay.

export MOONSHOT_API_KEY=your-key-here
export MOONSHOT_BASE_URL=https://api.moonshot.ai/v1

# Validate the endpoint and profile
pgllm-models --registry examples/kimi_k3.json --models kimi-k3

# One assay test
pgllm-run --registry examples/kimi_k3.json \
  --models kimi-k3 \
  --assays A0A247D711_LISMN_Stadelmann_2021 \
  --sizes 50 --seeds 1 --concurrency 1

pgllm-status --models kimi-k3 --sizes 50 \
  --seeds 1 --assays A0A247D711_LISMN_Stadelmann_2021

After the test is clean, launch the full benchmark.

pgllm-run --registry examples/kimi_k3.json \
  --models kimi-k3 --sizes 50 --seeds 1 2 3 \
  --timeout 7200 --retries 1

pgllm-status --models kimi-k3 --sizes 50
pgllm-score --models kimi-k3 --sizes 50

Models and providers

Copy the OpenAI-compatible registry template and declare the endpoint's model ID, reasoning setting, context window, output ceiling, and metadata requirements.

cp examples/internal_model.json my_model.json
export LAB_API_KEY=...
export LAB_BASE_URL=https://your-endpoint.example/v1

pgllm-models --registry my_model.json --models lab-model
pgllm-run --registry my_model.json \
  --models lab-model --sizes 50 --seeds 1 2 3
pgllm-score --models lab-model --sizes 50

Responses-style and Chat Completions-style transports are supported.

Submit to the leaderboard

Retain all three primary seeds, run the scorer and integrity checks, then export the publication bundle.

pgllm-score --models lab-model --sizes 50 --breakdown
pgllm-export --models lab-model --sizes 50 \
  --output artifacts/lab-model-n50.publication.jsonl.gz

Send the bundle, registry profile, exact command, and provider/model version to rohitarora@g.harvard.edu, or open a GitHub issue. The complete command reference is in the repository.