DryLab Discovery at the OpenADMET PXR challenge

DryLab Discovery is our autonomous research system for data-driven modeling: a closed-loop AI scientist that forms its own hypotheses, writes and executes code, audits its own mistakes, learns from failed branches, and maintains a fully traceable scientific record. Here we report on how it performed under blind, competitive pressure against elite human data-science teams.

The landscape of autonomous AI research

We have been developing DryLab internally since September 2024, before the recent wave of autonomous research agents popularized by figures like Andrej Karpathy. Today, that landscape is emerging across three distinct branches:

DryLab sits at the intersection of the second and third branches. It is built to treat ML engineering as a search problem, but is specifically optimized for a targeted, highly flexible biochemical search space.

Before this open challenge, we validated DryLab on BioML-bench (Miller et al., 2025), a rigorous, third-party benchmark for end-to-end biomedical ML tasks, where it reached top-quartile to state-of-the-art performance across the majority of these benchmarks (Section 7). We had not yet tested it against top-tier human data scientists in a live, high-pressure environment, so we entered DryLab into the OpenADMET PXR Activity Track — a highly competitive challenge drawing roughly 400 submissions in Round 1 and 100 in Round 2. Our sincere thanks to the teams at Octant AI and UCSF for designing and running such a rigorous, open challenge — the exact kind of benchmark the whole field benefits from.

The challenge design

Every team got the same task: predict how strongly a molecule activates PXR from its chemical structure alone. Activation strength is reported as pEC₅₀ — the higher the value, the more potent the activator.

The 513 test molecules were divided into two sets whose measured labels were withheld, and the challenge ran in two phases around that split:

• Phase 1Teams submitted predictions for Set 1 (253 molecules) and were scored on a live leaderboard, with the true labels kept hidden. This is where you tuned a model without ever seeing the answers.
• Phase 2The Set 1 labels were then released as extra training data, and the final ranking was decided on Set 2 (260 molecules) — a fully blind set no team ever got feedback on.

So Set 1 is the slice that becomes visible partway through, and Set 2 is the one that stays hidden until the end and settles the standings. Both are drawn from the same campaign but, as Section 1 shows, not from the same potency mix — a detail that ends up mattering more than it sounds.

A benchmark that punishes shortcuts

Many molecular benchmarks reward interpolation: train and test share close chemistry, so a model that memorizes patterns across both the training and validation sets can look strong without generalizing. Early versions of the engine ran straight into this. The loop would generate a plausible hypothesis, run error analysis on the validation set, and build a fix around whatever it found — and because the validation set was in view the whole time, "whatever it found" often included patterns that were true of that split and nothing else.

Frontier models don’t save you here. Opus 4.8, GPT-5.5, even Fable 5 lack the discernment to distinguish an idea with staying power from one that merely looks appealing. Their validation error analyses often read sharp but prove shallow under scrutiny. Even if the model recognized a genuine pattern and committed to exploiting it, the fix wouldn’t generalize. As the run progressed, things only degraded further: the obvious, structurally grounded error patterns were resolved early, leaving behind patterns that existed solely within the validation set. The system couldn't tell the difference. Yet on benchmarks, it continued to score well—propping up an illusion of progress.

PXR was less forgiving, because the assay campaign looked like lead optimization: start broad, confirm actives, then order analogs around hit scaffolds. Many compounds are close enough to tempt a similarity model, but the errors that matter sit in activity cliffs, potency tails, and small scaffold-dependent shifts.

There is a second, quieter way this campaign structure punishes a shortcut, and it shows up the moment you line up the label distributions. The training compounds, the Set 1 compounds released mid-challenge, and the Set 2 compounds held back for the final ranking are not drawn from the same potency mix. As the campaign moved from a broad opening screen toward confirmed analogs, the compounds got more potent — so the test sets are shifted toward exactly the high-activity end where the model has the least training support.

Reading it as a cumulative curve makes the ordering exact: at every potency level, more of the training set is inactive than Set 1, and more of Set 1 is inactive than Set 2. Set 1 sits cleanly between the training data and Set 2. That is why Set 1 turned out to be a usable proxy once its labels were released — it is shifted the same way Set 2 is — and also why it is an imperfect one, because it does not shift quite as far.

Put in the challenge's own potency bands, the shift is large. The potent fraction (pEC50 ≥ 5.5) roughly triples between training and Set 2, while the deep-inactive fraction is cut in half. A model that faithfully learns the training prior is, before it makes a single representational mistake, already biased toward calling Set 2 compounds weaker than they are.

That combination — a potency shift toward the tails plus a representation that compresses them — has a specific, visible signature in the final model's errors. It does not miss uniformly. Measured against the Set 2 labels after they were released, it over-predicts the deep inactives by more than a full log unit and under-predicts the strong actives by about four tenths, while the middle stays close to right. The errors fan out from the center rather than sitting off to one side.

The two-phase protocol added a further layer. Before Set 1 was released, scaffold-aware validation was the best available proxy. After Set 1 opened, the task mirrored a real project, where new data arrives, part of the test set becomes visible, and you have to decide which lessons are safe to apply to the still-hidden compounds — knowing, as the distributions above show, that the hidden compounds are shifted a little further out than the ones you can see.

Not a normal autoresearch run

Standard AutoML systems — H2O, Bayesian search — search a fixed set of model families, descriptors, hyperparameters, and ensemble weights, which works when the right option is already on the options. In this challenge, most of the moves that helped were not fixed set of directions but the kind of hypothesis a data scientist would form about the endgoal:

• inactive compounds are being predicted too active because ensembling averages their predictions upward, so the model needs a gated floor correction;
• the high-potency tail is being compressed, so training should be reweighted only where the model is already confident;
• 3D information should matter most for unfamiliar scaffolds, not for every compound;
• external PXR data should be used as a narrow correction, not as a second training set with mismatched assays.

Rather than sweeping a search space, the engine treated each candidate as a hypothesis to confirm or refute: one iteration picked a branch, named a single mechanism, changed the code, audited the result, and wrote the verdict to memory so the next proposal had to account for what had already failed.

Why a linear loop was not enough

Before this design, we tried the obvious things: autonomous-research scripts, their variants, even a loop wrapping Codex and Claude Code. Each needed too much human babysitting and stalled after a while. The core problem is that a linear loop of idea iterations does not explore. Once the LLM is conditioned on a long history of previous iterations, its next proposals collapse toward that history, and the search stops branching out.

So we moved to tree search, which fixed exploration but was slow. Standard MCTS and UCT need many rollouts down a single branch before their value estimates are good enough to decide whether the branch is worth chasing. The PXR task makes that worse, because verifying one hypothesis means a full training run that can take hours.

Parallel preliminary exploration

Our fix splits the roles. The idea-generation LLM never reads the full iteration history; instead it hands out several cheap scouting missions at once, each to a separate agent in its own sandbox. A scout runs a small probe — a few training steps, a quick error slice, or a literature check — and reports back one line. The lead reads only those one-line verdicts, so its context stays short and proposals keep diverging.

A scout that lands just under the bar does not report "failed" — it returns a sharpened version of the idea. When the system proposed fine-tuning Uni-Mol v2, it copied the epoch count from public configs. A scout ran a short probe, saw the loss still dropping steeply at that budget, and returned "right idea, wrong epochs" with a corrected number. That direction stayed alive instead of getting dropped.

The breakthroughs behind the model

The baseline was Morgan fingerprints into a tree regressor, which reached only 0.6384 full-513 RAE. The failure mode was regression toward the mean, because sparse fingerprints did not separate close compounds and the model compressed the potency tails.

From there it built a multi-view stack of count fingerprints, physicochemical descriptors, D-MPNN graphs, Uni-Mol features, 3D descriptors, residual correctors, and calibrated ensemble arms. Most did not survive. The handful of models we actually submitted to the public board — scored on full-513 RAE — came down to four checkpoints:

However after these breakthroughs, the internal proxy validation that derive the search process had become easy to overfit, so a good out-of-fold number was no longer enough on its own. After v19, a candidate had to explain where its gain came from and pass targeted checks before it was accepted rather than just outperform the validation numbers of previous iterations. This somewhat alleviate the overfitting problem.

After the set 1 is released, we let the system run fully autonomously in 7 days — and on PXR that spine is 27 nodes long, carrying the internal out-of-fold RAE from the 0.634 baseline down to 0.4901. Most were small, hard-won plateau steps; a handful were genuine changes of mechanism. Those are the ones worth naming.

• 0.634Morgan-fingerprint baseline. LightGBM on 2048-bit fingerprints with L1 loss. Refuted as a finish line but kept as the floor: the bottleneck was representation, not the loss function.
• 0.5355Add a 3D conformer view (UniMol-2). Bolting a 3D transformer onto the 2D ensemble cut error 13.9% at a 76% blend weight — not by decorrelating, but by making fewer large mistakes on hard compounds. The single biggest jump of the run.
• 0.5175Borrow the HTS screen (transfer learning). A fold-aware model trained on 10,870 high-throughput compounds, fed in as an extra feature, added signal without scaffold overfitting.
• 0.5074A second foundation graph model (CheMeleon D-MPNN). Strongest standalone arm at RAE 0.536; took 36% blend weight even at r=0.86 with UniMol, pulling the blend from 0.518 to 0.507.
• 0.4998Per-arm calibration. Linear blending inherited each arm's variance compression; calibrating arms individually — variance-matching for CatBoost, isotonic for the neural arms — first broke below 0.50.
• 0.496Orthogonal graph blending on HTS data. A Chemprop D-MPNN on the screen data produced errors genuinely decorrelated from the stacker, unlike the fingerprint arms that kept re-voting the same signal.
• 0.492ADMET-AI residual stacking. Fold-aware feature selection over external ADMET predictions added orthogonal biological signal without target leakage.
• 0.4901Distributional arm against mean-regression. An arm predicting a potency distribution rather than a point value counteracted the tail-compression bias that dogged every earlier model — the best internal score of the run.

The pattern across the spine is consistent: the moves that held were the ones adding a genuinely new view — 3D structure, an external assay, a distributional target — while the ones that stalled were re-encodings of signal the ensemble already had. Small refinements between these steps mattered too, but only as consolidation. And note the two rulers: this internal spine ends at 0.4901 out-of-fold, whereas the four submitted checkpoints above are scored on full-513 RAE and top out at 0.5856. Different metrics on different splits; the point of the internal number is the shape of the climb, not a leaderboard-comparable value.

That whole trajectory — the spine above and every branch that did not make it — is one search. The figure below is the actual run, every experiment the engine kept or rejected on PXR, pulled straight from its record. Click any node to read its hypothesis, verdict, and metrics; hover a node tagged preliminary to fan out the parallel probes that were tried and eliminated beneath it.

What we learned about PXR from Drylab Discovery

Because each iteration ended with a written verdict, the run produced a structured record of failures under hard evaluation — not just the final model but also the dead ends that narrow what to try on the next project. Here are few things we learned:

Activity cliffs set a floor for 2D models

The largest errors were structured rather than random. They came from near-neighbor compounds with very different measured potency, where a 2D fingerprint is locally confident for the wrong reason. Sort the training compounds by the severity of their local activity cliff — how much measured potency changes among their nearest structural neighbors — and the model's out-of-fold error increases monotonically. In the smoothest quarter of chemistry, the Morgan-fingerprint model is off by 0.36 log units; in the cliffiest quarter, it is off by 0.98. Fine-tuning a 3D representation helps but only partially: it never closes the gap because the information a cliff destroys is not present in any representation to recover.

The model was biased at both ends

The base stack over-predicted deep inactives and under-predicted strong actives, while the middle stayed comparatively stable — the same fan seen against Set 2 in Figure 4: about +1.17 on the deepest inactives, −0.40 on the potent tail, and near zero through the middle two bands. The inactive shrink and active push corrected the same shape error, and broader versions disrupted the stable middle, so the effective fixes were deliberately narrow.

External data helped only when used narrowly

Surprisingly to us, public PXR data was not automatically useful. Assay scales varied; some labels came from single-concentration screens instead of measured pEC₅₀; some sources added coverage but lacked reliable level information. When added broadly, an external-data arm actually hurts the bands it is not designed for — it degrades the mid and active regions the base model already handles. The useful form was narrow: a ChEMBL/PubChem active-region delta, applied only when the base model and an independent gate agreed the compound was active, where it recovers part of the deep tail without causing losses elsewhere.

Physics signals were real but redundant

3D and structure-derived features performed best on unfamiliar scaffolds. Docking-like features, shape overlap, pocket-interaction features, and cofold-derived signals often aligned. In a correlation matrix, the physics arms mostly represented variations of one shared signal — free-conformer and pose-based views correlated 0.7–0.9 with the Uni-Mol arm and with each other. Stacking many weakly distinct physics views diluted the ensemble instead of enriching it. A better approach was using one gated physics signal where 2D similarity was least reliable.

Frozen foundation embeddings were terrible

The engine tried frozen foundation embeddings and found them mostly redundant or worse. For that experiment — frozen embeddings feeding shallow models — the conclusion was reasonable: as a leak-safe OOF arm, the frozen Uni-Mol view lands around 0.55 RAE, no better than the other 3D arms and far from a decisive lever. The strongest final systems fine-tuned 3D foundation models around the target and used structure-aware active specialists. AIDD-LiLab's public report, for example, describes a Uni-Mol v2 system with free-conformer and bound-pose 3D pillars, multi-fidelity transfer, and gated active-region components. Our run tested the foundation model as a feature extractor but never as a trainable core — the one direction, as Section 6 details, that actually moved the blind number.

As usual, do not try to rely on public leaderboard

On the final private activity leaderboard, OpenADMET ranked our entry, quockhanh212, 23rd by MAE at 0.4357, compared to 0.4061 for the winner, matcha-croissant, and 0.4092 for AIDD-LiLab. The top ten were separated by about 0.016 MAE, and OpenADMET's analysis found that many leading entries were not statistically distinct under its testing.

The gap to the strongest systems is real, but the rank overstates it. Our entry sat in a crowded band where small calibration and tail errors shifted many places at once. The labels stayed hidden until the end, so 23rd is 23rd, and that is the value of a blind board.

The misses

After Set 2 labels were released, two failure modes became clear — both more interesting than a simple performance gap because the engine had the evidence in hand but tried to not chasing these two promising directions.

1 · The missed calibration opportunity

Our final model suffered from a systematic, unbalanced error: it over-predicted the activity of the most inactive compounds — the "deepest inactives" — and under-predicted the potency of the most active compounds. This dual-tail bias emerged early in Set 1 and persisted unchecked into Set 2.

Retrospectively, a simple post-hoc calibration fit on Set 1, evaluated after Set 2 labels were released, gives about 0.4092 MAE on Set 2, close to the second-place number. This is a diagnosis rather than a valid blind-challenge result, but the system had the evidence to make the call at the time: it found both tail biases, explored inactive and active corrections, and had access to Set 1 labels. It rejected the calibration because it resembled Set-1 tuning too closely (the generalization problem from feedback signals of validation set).

2 · Underinvesting in 3D fine-tuning

Our most significant technical gap was a failure to commit to end-to-end 3D fine-tuning. The top-performing teams did not merely append static 3D descriptors to their feature sets; they trained fully structure-aware, target-focused models from end to end, often using low-fidelity assay data, cofolded poses, and active-specialist gating.

Our run moved in that direction but did not fully commit. It used Uni-Mol and related features as arms, tested frozen embeddings, and explored cofold and shape-derived signals, but it did not allocate the compute to fine-tune a 3D core on the target. This matters more than the calibration miss because the calibration only affects the score on this challenge, while the fine-tuning gap remains a capability the next run must still develop.

The BioML benchmark

While PXR represented a live evaluation, we prior validated our engine on BioML-bench — an independent, third-party benchmark developed by Miller et al. (science-machine, bioRxiv 2025). As the first suite designed to evaluate AI agents on end-to-end biomedical machine learning rather than static question-answering or narrow bioinformatics, BioML-bench builds based on procedures from OpenAI's MLE-bench.

The evaluation framework presents an agent with a raw dataset and a text-based task description. The agent must autonomously parse the objectives, engineer a data pipeline, implement machine learning models, and output predictions, which are then scored against task-specific metrics. We did not modify the benchmark in any way; our sole contribution was running our engine, Drylab, natively through their open-source harness. The resulting performance provides the empirical foundation for our claim of achieving state-of-the-art results on several tasks.

What is in the suite

BioML-bench spans four biomedical domains. The tasks are drawn from established sources — Polaris and TDCommons for drug discovery, Open Problems for single-cell omics, ProteinGym for protein engineering, and Kaggle competitions for biomedical imaging — and each keeps its own primary metric, so a result lands on a real leaderboard rather than an isolated number:

• Drug discoveryPolaris / TDCommons — ADMET and bioactivity tasks (EGFR bioactivity, human liver microsomal clearance, plasma protein binding, solubility, lipophilicity, Caco-2 permeability, CYP2D6, hERG, blood–brain-barrier penetration). Metrics are task-specific: PR-AUC or ROC-AUC for classification, Pearson r or MAE for regression.
• Single-cell omicsOpen Problems — modality prediction (CITE-seq GEX→ADT), perturbation-response prediction, cell–cell communication, spatially-variable genes, and label projection. Metrics include RMSE, mean-rowwise-RMSE, and odds ratio.
• Protein eng.ProteinGym — deep mutational-scanning fitness prediction across substitution and indel assays, scored by average Spearman between predicted and measured variant effects.
• Biomedical imagingKaggle (3D / medical) — pulmonary-fibrosis progression, histopathologic cancer detection, brain-tumor radiogenomic classification, and GI-tract segmentation. Metrics range from Laplace log-likelihood to ROC-AUC and Dice / Hausdorff.

How a task is scored

The procedure mirrors the PXR run, which is why the benchmark is a fair proxy for it. The engine is handed a task description, the training split, and a held-out test set with hidden labels. It runs its own loop — hypotheses, code, evaluation, memory — for a fixed iteration budget (typically 50–100 iterations per task, sometimes with two parallel searches). Its final predictions are scored on the task's primary metric only, then converted to a leaderboard percentile: the fraction of official human submissions it beats. We report the non-penalized percentile, meaning a task the engine did not finish is dropped rather than counted as zero. No human touches the modeling loop once the task starts.

Baselines

The comparison agents come with the benchmark: the BioML-bench paper evaluates four open-source agents — two general ML-engineering coders and two biomedical specialists — and reports that agents on average underperform the human baselines, with biomedical specialization giving no consistent edge. We add Drylab as a fifth entrant, scored the same way, against the same human leaderboards:

• Dummy — a naive baseline (constant / trivial predictor) that fixes the floor of each leaderboard.
• MLAgentBench — the ReAct-style ML-experimentation agent of Huang et al. (2024).
• AIDE — Weco AI's tree-search ML-engineering agent (Jiang et al., 2025), the strongest general-purpose coder in the group.
• STELLA — the self-evolving biomedical agent of Jin et al. (2025).
• Biomni — Stanford's general-purpose biomedical agent (Huang et al., 2025).

Two things stand out. First, the domains where Drylab's lead is largest — drug discovery and protein engineering — are the ones closest to PXR: supervised prediction on molecular or sequence data with a real held-out test set. Second, no single prior agent holds second place across domains — consistent with the paper's own finding that scaffolding and strategy diversity, not biomedical specialization, drive agent performance here.

Beyond PXR

OpenADMET PXR and BioML-bench are both solid benchmarks for data-driven modeling, and across them the Drylab Discovery Engine reached competitive-to-leading performance — faster and at significantly lower cost than traditional methods and human teams, with no human steering the modeling. The natural next question is the opposite one: how much a human in the loop, intervening at the right moments, could speed the engine's convergence rather than replace it.

We are also keen to apply the engine beyond prediction: to hypothesis generation from public data silos and datasets, and to asset design such as binder design and virtual drug screening, where a library is iterated computationally along several directions in parallel until it converges on a small set of well-reasoned candidates.

This redefines what the engine is for. Previously, prediction focused on grading molecules that already exist. Design, on the other hand, proposes which molecules to create next—an inherently more challenging and impactful aspect of discovery. The mechanism behind Drylab Discovery remains unchanged: each proposed design includes a hypothesis and supporting evidence. As a result, the output is a curated shortlist accompanied by a written rationale that chemists can readily evaluate and act upon. In this context, parallel exploration becomes even more critical. The design space is vast, with objectives that are broader and more flexible than a single test metric. This aligns more closely with how a discovery team operates—exploring multiple directions simultaneously and retaining the most promising options. The cost advantage still applies: by allowing the engine to perform computational screening and design at a fraction of the cost of a wet-lab cycle, teams can reserve real experiments for candidates that have already passed a computational filter, backed by a clear rationale. This approach becomes even more powerful when integrated with Lab-in-the-loop data. Here, each node in the exploration tree incorporates real wet-lab feedback, allowing the engine to dynamically update its beliefs and evidence with every iteration. These new applications are incredibly exciting to consider for the future of molecular discovery.