SANE Schema-aware Natural-language Evaluation of Biological Data
2026-06-03 • Computation and Language
Computation and Language
AI summaryⓘ
The authors developed a new method called SANE to test how well large language models can turn natural language into database queries, specifically for complex scientific data. They showed that with careful guidance and knowledge of the database structure, these models can generate accurate queries without any extra training. Most mistakes happen when the questions aren't clear, leading the model to ask for clarification instead of making wrong queries. This work suggests that language models can help non-experts access complicated databases if given the right context.
high-throughput microscopySQLlarge language modelstext-to-SQLschema-aware promptingfew-shot learningdatabase queriesnatural language processingbenchmarkingquery disambiguation
Authors
Rolf Gattung, Martin Krueger, Markus Reischl
Abstract
High-throughput microscopy generates large, structured datasets capturing cellular responses to pharmacological perturbations, but accessing these datasets typically requires SQL expertise. Large language models offer a natural-language alternative, yet their tendency to hallucinate raises concerns about result reliability . We present SANE Schema-Aware Natural-language Evaluation, a novel paradigm for domain-specific text-to-SQL evaluation: schema-grounded, automatically generated benchmarks tied to real and specific experimental structure. SANE makes evaluation more scalable, systematic, and reproducible. Using SANE, we evaluate a few-shot large language model and show that, under constrained schemas with structured prompting and guardrails, accurate query generation is achievable without any model training or fine-tuning. Most failures stem from ambiguous or underspecified inputs and manifest as overly cautious clarification requests or answers to queries that should first be disambiguated, rather than incorrect SQL generation. These results indicate that few-shot large language models can provide reliable database access in well-defined domains when combined with schema-aware prompting.