AI summaryⓘ
The authors describe a method called Policy Iteration with Human Feedback (PIHF) that helps improve how language models solve tasks by combining human expert review with automated learning. They use pretrained language models to execute tasks and continuously update policies based on feedback from experts and a language-model critic. This approach was tested on diagnosing very rare diseases and showed substantial improvements in accuracy across different language models of varying sizes. The experts oversee revisions to make sure improvements are real and can undo changes if needed. The authors suggest this method is effective for refining AI policies in specialized, challenging domains like rare-disease diagnosis.
Generative PretrainingPolicy IterationHuman FeedbackPretrained Language ModelIn-Context LearningRecall@1Rare Disease DiagnosisNatural-Language PolicyExpert ReviewLanguage-Model Critic
Authors
Minh-Ha Nguyen, Cathy Shyr
Abstract
Generative pretraining established reusable task representations; later work on language-based task conditioning and in-context learning showed that a fixed model could adapt its behavior from instructions and demonstrations. Policy Iteration with Human Feedback (PIHF) builds on this development and the recurrent evaluate-and-improve structure of generalized policy iteration. PIHF uses a pretrained language model as its execution substrate and moves persistent revision to a versioned natural-language policy and tool set. A language-model critic and clinical expert review complete-panel reasoning and tool-use trajectories to localize recurrent failures and form candidate revisions; the expert may reinterpret the evidence and retains authority over admission and rollback, while Recall@1 and Recall@5 validate outcomes after candidate execution. Across cumulative ablations and ultra-rare-disease benchmarks, a PIHF-derived policy improved Recall@1 in one proprietary executor and three open-weight executors spanning 3 to 49 billion active parameters. Gains were 32.7 percentage points for GPT-5.4 and 31.1 points for Qwen3.6-35B, a difference of 1.7 points. These results support the feasibility of using pretrained language models as fixed-weight execution substrates for expert-guided policy development in rare-disease diagnosis.