At BigHat, we combine machine learning, a high-throughput lab, and human expertise to generate more and better data, design better models and produce life-saving antibody therapeutics, faster. Our scientists are brilliant protein engineers, and we design and characterize thousands of antibodies at a time in our Design-Build-Test Loop (DBTL). We want insights on all of this data, we aim to leverage every data point to design new variants and experiments, but we generate far more data than our protein scientists can easily sift through by hand. Scaling the human teams surrounding our programs, platform, and data to meet the demands of our programs would force BigHat to an intractable scale, paradoxically throttling the number of therapeutics we could create.
We wondered whether LLM-based agents could augment data and laboratory scientists by assisting with data-related questions. An LLM could enable our scientists to self-serve the sophisticated data insights that we use to make decisions about the antibodies we want to advance in a program. However, without guard rails, we wouldn’t know whether the proposed insights were simply confidently wrong, as happens frequently enough with even the most impressive LLMs. This calls for transparency into what the LLM actually does, infrastructure to evaluate it at scale, and a process that makes data-driven decisions by LLM trustworthy.
During an internal LLM-Day Hackathon, I wrote a Claude-based agent that integrates directly with Reccy, our lab information system and unified data layer. It enables scientists to ask and answer data science questions in plain English and allows for quick review and verification of LLM claims to ensure our decisions stay on track. I took the concept of Generative Adversarial Networks (GANs) as inspiration. Some of the best AI systems improve by fighting themselves: a generator that produces, and a discriminator that attacks. We call our agent the Generative Agentic Data Scientist (GADS): it pairs a solver that produces analyses with an evolving set of test questions designed to expose where it fails.
The Best Failure
We built a simple Claude-based LangGraph state machine using AWS Step Functions and a dashboard to track solver performance across 20 benchmark questions our scientists routinely ask. A LangFuse RL management system would then score (or solicit scores where they are manually assessed) each answer as correct, partially correct, an error, or, most concerning of all, "wrong but no error" — a confident, well-formatted answer that happens to be incorrect. Each run would accumulate new guidance that would be used to tune the tools, prompts and graph for our agent.

We then looped this process. After each iteration, Claude recorded performance and kept improving the solver, and once it plateaued, a fresh instance of Claude generated harder questions to escalate the benchmark. Exact accuracy climbed quickly from roughly 25% to over 90%, then collapsed at round 11, when SQL-joins broke the system. The culprit: without the metadata and schema that describe the dataset, the anchors for the joins were missing. The broken joins earned the project the hackathon's "Best Failure" award since it no longer produced real outputs at the conclusion of the hackathon. But the enthusiasm it drew made it clear there was real demand for this at BigHat; the crash made it clear the approach needed more structure.
Iterations without Regressions
The core principle became structured iteration with no silent regressions. The work now lives in GitHub: every commit triggers a suite of eval functions that re-run the benchmark, so any change (a prompt edit, an architectural tweak, a new tool) is scored against the same questions before it can land. Claude works the loop directly: it proposes a change, reads which evals fail and why, and keeps iterating until the suite passes and the branch is ready to open as a PR. This mirrors BigHat's DBTL cycle: run the benchmark, analyze what failed, implement fixes, run again to confirm we improved without breaking something else. Lately we've handed much of the prompt tuning to GEPA, an open-source prompt optimizer, so gains come from the benchmark rather than our guesses. In GEPA terms, the benchmark is the discriminator; the solver is the generator. Just as with antibody engineering, optimizing one property often regresses another, but eventually iterative optimization converges, and after a few more days we had something good enough to put in front of real users.
Real Life Is the Best Adversary
The most effective adversary, it turned out, was real life. The questions real scientists asked were far harder than anything Claude had generated for the benchmark: project abbreviations the system had never seen, follow-up questions we do not yet support, and queries requiring domain knowledge it lacks. This has led us to iteratively improve the background information, prompting, and tools to allow GADS to improve on new sets of benchmark questions that push on exactly these gaps.
We also want the results of GADS to have increasingly deterministic elements. When a scientist asks GADS about their project data, it returns a table, a chart, and the code that generated the result. The code is assembled from a vetted set of Python functions rather than written “freehand”. This means that a data scientist can verify how trusted building blocks were combined instead of auditing arbitrary code (or worse, raw agent traces), which is a far smaller, more tractable review. Each surprising failure then becomes a new test case added to the benchmark, so the next version must handle the messy, ambiguous questions that arise in real use.
Conclusions and What’s Next
The quality of our evaluation matters more than the raw accuracy of the solver: as scientists come to trust GADS with the easy questions, the ones they bring get harder on their own, a kind of natural selection that keeps the benchmark sharp. Regressions remain the dominant challenge.
The hardest and most important problem now is transparency. As questions grow more complex, the solver's mistakes get subtler and easier to miss, which turns trust into a design problem of making every step legible to our data science experts. That is where we are focused, because if experts can verify results at a glance instead of reviewing everything by hand, the pace of our data-driven decisions can finally match the pace of our science.
We are currently working on several extensions: multi-turn conversations so users can ask follow-up questions without starting from scratch, faster execution, and a feedback loop where users can rate results, giving us data we can use to keep improving the system. We are also considering the best way to integrate GADS analysis directly in Autopilot, our automated design loop for antibodies.