Rounds: Benchmarks and Claims

Authors: Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias, Thaís Gaudêncio
Published (record): 2026-10-22
Topic in the record: clinical llm evaluation
Atlas status: interesting

Abstract, as recorded: Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated. Relevance was defined by evidence spans mapped to all containing chunks, treating overlapping chunks with equivalent evidence as relevant. Retrieval was evaluated using R-Precision, Recall@K, MRR, and nDCG@K. Separately, generation was assessed on a 512-token index at K=1 to 5 using DeepEval claim-level metrics, with deepseek-v4-pro as the LLM judge, and BLEU/ROUGE metrics. Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing retrieval rankings. Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased recognized evidence without changing rankings, highlighting annotation sensitivity. The small development set and retrieval-informed curation require independent validation, including radiologist assessment of evidence sufficiency.

Atlas curation note, as recorded: - `2610.31606v1` — *How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions* — isomorphism: MAPS vs TERRITORY / CONSTRAINTS. Expanding ground-truth curation from 25→37 evidence spans and 36→51 chunk targets raises recognised evidence without changing retrieval rankings at all (Recall@3 / R-Precision / MRR@3 / nDCG@3 = 0.94). The score is a function of the annotation map, not just the retriever — a clean confession that evaluation rewrites what counts as found.

Authors: Bastien Le Guellec, Su Hwan Kim, Raphael Bentegeac, Aghiles Hamroun
Published (record): 2026-10-22
Topic in the record: clinical llm evaluation
Atlas status: interesting

Abstract, as recorded: With the growth of scientific output, the peer review system is approaching a breaking point: reviewers are increasingly hard to secure, and turnaround times lengthen accordingly. LLM-assisted or LLM-generated peer review has been proposed as a remedy, but how such models would treat manuscripts already published in top-tier journals, the closest thing the current system has to a gold standard of scientific validity, is not known. Here we assembled a corpus of 200 articles, each released under a CC BY 4.0 license and published in 2026, from 20 general-science, life-science, and clinical-medicine journals including Nature, Science, and Cell, and asked four current LLMs (two proprietary, two open-weight) to produce independent peer-review reports of each one under a blinded protocol. Their decisions were overwhelmingly harsh and starkly inconsistent with each other: 605 of 800 reviews (75.6%) recommended major revision or rejection, but the rate at which any single model did so ranged from 17.5% to 100% (p < 0.001), disagreement on manuscripts that had, by construction, already survived real editorial review at the journals in question. When the same manuscripts were doctored with one deliberately planted, field-specific methodological or statistical flaw, detection rates varied from 17.0% to 60.5% across models (p < 0.001). Recommending revision for papers that had already passed editorial review (low specificity) while missing flaws deliberately placed in front of them (low sensitivity), these models produce output that should be read for what it is: an uneven, framing- and model-dependent signal.

Atlas curation note, as recorded: - `2610.31414v1` — *Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals* — isomorphism: VERIFICATION / CONSILIENCE FAILURE. Four LLMs review 200 already-published top-tier papers: 75.6% recommend major revision/rejection, rates per model 17.5–100%, and planted-flaw detection only 17.0–60.5% — low specificity and low sensitivity at once. Agreement is not reliability; a reviewer ensemble that disagrees this wildly is a signal about the reviewers, not the manuscripts. Direct rhyme with builder-never-grades-own-work.

Authors: Yitong Bai, Chengyi Zhang, Ziyang Wang
Published (record): 2026-10-22
Topic in the record: clinical llm evaluation
Atlas status: interesting

Abstract, as recorded: In clinical systems built from several large language model (LLM) agents, one agent often chairs the discussion and makes the final decision. Whether the chair's persona prompt changes that decision, or the agreement between repeated runs that some systems treat as reliability, has not been tested with the case held constant. We ask whether the persona changes the decision or only its timing, how it affects agreement between runs, and which behaviors appear when the prompt describes only personality traits. A simulated department of seven agents discussed 20 published cases of degenerative cervical myelopathy, with the case reports replayed unchanged in every run. Only the prompts of the chair and its deputy changed (no persona, or a consultative, autocratic or avoidant one), written either as rules of conduct or as personality traits. With personas written as rules, the chair chose to operate now in 93% of runs when autocratic and in 2% when avoidant, against 57% without a persona. Surgery was still planned in nearly all decisions, so the personas mainly changed whether to operate now or after more tests. These effects held with a second model and reworded prompts. The autocratic and avoidant personas made the three runs of a case agree more often, but only about as often as chance predicts, and Fleiss $κ$ did not rise. Prompts that described only personality traits kept more than half of the autocratic and avoidant effects on operating now. They reproduced quick decisions, delays and extra tests, but not ignoring red flags, dropping backup plans or passing the decision to the family. Studies should report and vary the chair's persona as they do the model, and correct agreement between runs for chance before calling it reliability.

Atlas curation note, as recorded: - `2610.31325v1` — *Consistent but Biased: Chair Personas Trade Noise for Bias in Multi-Agent Clinical Decisions* — isomorphism: SEAMS / CONSTRAINTS / MULTI-AGENT MEMORY. Holding 20 cervical myelopathy cases constant, only the chair/deputy persona changes: autocratic chair operates now in 93% of runs, avoidant in 2%, vs 57% no-persona. Personas raise run-to-run agreement only to chance level (Fleiss κ does not rise) — consistency traded for bias. Persona is a hidden constraint that must be reported/varied like the model itself; a direct Agent Net warning.

Authors: Bayram Ayadi, Florian Geissler, Johanna Schmidhuber
Published (record): 2026-10-22
Topic in the record: clinical llm evaluation
Atlas status: interesting

Abstract, as recorded: As wearable and digital health systems increasingly generate longitudinal physiological records, communicating these observations in patient-accessible language remains a significant challenge. Large Language Models (LLMs) offer a promising mechanism for translating structured clinical information into readable narratives, but this process may introduce semantic drift, where clinically precise meaning is distorted during simplification. We investigate evidence-grounded health summarization for menstrual health monitoring using a constrained generation framework that combines structured evidence conditioning with Retrieval-Augmented Generation (RAG) using a prompt-based constrained generation framework. We evaluate semantic alignment using Natural Language Inference (NLI), atomic factuality metrics, and readability measures. Across three LLMs and 4,500 generated summaries, explicit grounding constraints consistently improve clinical alignment, particularly by improving uncertainty preservation and, once faithfulness is evaluated against retrieved evidence, generally preserving or improving faithfulness, while effects on omission remain model-dependent. We also find that improved clinical alignment comes at the cost of increased reading difficulty and medical terminology. These findings highlight the need to jointly evaluate faithfulness, uncertainty preservation, and readability when developing patient-facing medical LLM systems.

Atlas curation note, as recorded: - `2610.31467v1` — *From Structured Records to Patient Narratives: Preserving Clinical Alignment in LLM-Generated Health Summaries* — isomorphism: CONSTRAINTS / TRANSLATION SEAM / REPAIR. Structured records → patient narratives drifts semantically; explicit evidence-grounding + RAG constraints improve uncertainty preservation and faithfulness across 4,500 summaries, at the cost of readability/terminology. Fidelity is bought with friction at the translation seam.

Authors: Miguel Fernandez-de-Retana, Unai Zulaika, Ruben Sanchez-Corcuera, Aritz Bilbao-Jayo
Published (record): 2026-10-22
Topic in the record: clinical llm evaluation
Atlas status: interesting

Abstract, as recorded: Accuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more: whether the model recognizes when it is likely to be wrong, whether medical fine-tuning improves this ability, whether its correct answers rest on sound clinical reasoning, or whether it actually uses the images it is given. We examine these questions for four open models of 8 to 27 billion parameters on the Spanish MIR residency examination, scoring 1,552 questions from eight editions under the exam's own rules. Answering every question, the three larger models would rank among roughly the first 600 to 3,000 candidates, with 41 to 43 specialties still open to them. Abstention, however, never pays, because even their least confident answers are right more often than the 25% break-even. Medical fine-tuning reduced overconfidence at the cost of accuracy, without improving how well confidence separates right from wrong answers. Moreover, 13-38% of correct answers rejected a distractor for a clinically unsound reason that confidence did not reflect, and removing or swapping the provided contextual images left accuracy unchanged.

Atlas curation note, as recorded: - `2610.31586v1` — *Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination* — isomorphism: SEAMS / MAP vs TERRITORY. Models would rank 600–3000 on MIR, yet 13–38% of correct answers reject a distractor for a clinically unsound reason that confidence does not flag, and removing/swapping images leaves accuracy unchanged. Accuracy, calibration, reasoning, and image use are four different seams that a single exam score collapses.

Authors: Aditya Tanna, Abhishek Jindal
Published (record): 2026-10-05
Topic in the record: human-in-the-loop clinical
Atlas status: interesting

Abstract, as recorded: Reinforcement learning post-training for language models relies on two reward designs: human preferences (RLHF, DPO) and binary verifiers (RLVR). Clinical question answering fits neither. Near-correct answers differ by a single substituted entity, and no executable check decides clinical correctness. We instantiate a soft verifier from a maintained controlled vocabulary: UMLS Concept Unique Identifier overlap (via scispaCy, set-level F1) gives a graded, externally specified reward computed without a model in the loop. We combine it inside GRPO with an entropy-normalised LLM judge, which covers the safety and evidence axes overlap cannot see, and a small consistency penalty on padding and repetition that keeps early-training samples scorable. This three-term composite improves over SFT on Phi-3-mini (3.8B) over MedQA by 2.9% on EM (0.700 vs 0.680) and 39% on Token-F1 (0.202 vs 0.145); on Llama-3.2-3B the corresponding gains are 14% on EM and 35% on Token-F1. We report Token-F1 as the primary metric because it credits partially-correct clinical content that EM discards at this open-generation scale. Main-table results are means over 3 seeds with standard deviations below 0.005. The method transfers to PubMedQA, where training on the PubMedQA train set with the same composite reward improves Token-F1 over SFT by 22% on Phi-3-mini and 17% on Llama-3.2-3B without retuning. A reward ablation on Phi-3, varying the judge-ontology split at a fixed consistency weight, attributes 3 EM points to the ontology term, the contribution that catches entity substitutions the judge cannot. Three negative findings constrain the design: DPO under random negatives underperforms SFT for strong-prior models but helps the weakest-prior one; PPO under a sparse neural reward diverges; GRPO with KL-in-loss collapses at 7B.

Atlas curation note, as recorded: - `2610.06360v1` — *Ontology Concept Overlap as a Training Signal: Knowledge-Grounded Reinforcement Learning for Clinical Question Answering* — isomorphism: CONSTRAINTS / EXTERNAL VERIFIER / CONSILIENCE. Clinical QA fits neither RLHF nor binary RLVR, so a soft verifier is built from UMLS concept overlap (graded, externally specified, no model in the loop) + entropy-normalised LLM judge + consistency penalty. The ontology term alone catches entity substitutions the judge cannot (+3 EM). Two independent seams catching different errors is consilience as a reward design.

Authors: Gabriel Ocana-Santero, Marko Tvrdic
Published (record): 2026-10-07
Topic in the record: human-in-the-loop clinical
Atlas status: interesting

Abstract, as recorded: Drug discovery for neurological disease has traditionally centered on the molecules altered by disease. But the molecules that cause pathology are not necessarily the best points from which to reverse it. Here, we ask which otherwise unaltered molecular control points can be engaged to restore pathological neural circuits toward functional states. We present CircuitATLAS, a provenance-grounded systems-neuroscience knowledge graph and agentic framework for target discovery in circuitopathies. It structures literature-derived relationships across diseases, phenotypes, electrophysiology, circuits, brain regions, cell types and molecular effectors, while deliberately excluding direct disease-gene and disease-protein edges to reduce shortcut reasoning. The graph contains 3.83 million nodes and 7.66 million edges, including 5.31 million LLM-extracted relations, and incorporates structured datasets such as the Human Cell Atlas and new multimodal in vivo measurements. We then introduce an agentic workflow that reasons from measurable disease phenotypes through their circuit and cellular substrates to molecular interventions, therapeutic feasibility and clinical constraints. Finally, we introduce a human-governed in vivo lab-in-the-loop linking hypothesis generation to experimental iteration. Within this framework an agent nominated ATP1A3, the neuronal alpha3 Na+/K+-ATPase, as a control point on cortical excitability; interneuron-restricted expression of ATP1A3 abolished the beta- and gamma-band response to a focal 4-aminopyridine challenge in vivo, and the validated target was then carried into a structure-guided small-molecule campaign terminating in a defined assay to resolve the direction of modulation. CircuitATLAS thus provides a framework for discovering therapeutics based not only on what is molecularly disrupted in disease, but on what can be controlled to restore circuit function.

Atlas curation note, as recorded: - `2610.09643v1` — *CircuitATLAS: Agentic reasoning over a systems neuroscience knowledge graph for target discovery in circuitopathies* — isomorphism: MAPS / REPAIR VIA CONTROL / CONSTRAINTS BY EXCLUSION. 3.83M-node / 7.66M-edge provenance-grounded graph deliberately excludes direct disease-gene/protein edges to block shortcut reasoning, then reasons phenotype → circuit → cell → molecular control point. Repair target is not the broken molecule but an unaltered control point (validated: ATP1A3) that restores circuit function — a human-governed lab-in-the-loop closes it.

Authors: Sixu Chen, Mingrui Yang, Qiang Guan, Xiaojuan Li
Published (record): 2026-10-08
Topic in the record: human-in-the-loop clinical
Atlas status: weak

Abstract, as recorded: Knee osteoarthritis (KOA) progression prediction can support patient monitoring, requiring the integration of multimodal data and multidomain expertise. Moreover, isolated risk estimates provide limited insight underlying a prediction. To automate the progression assessment workflow and reduce manual effort while providing interpretable findings and supporting evidence, we present OA-MAP, an autonomous multi-agent framework for evidence-grounded assessment of structural and pain progression in KOA. The system incorporates modality-specific agents including MRI, X-ray, and clinical agents, together with a coordinator agent. This framework can autonomously recruit specialist agents, select tools for prediction and analysis, and retrieve literature as external evidence based on user request and available patient information. An uncertainty-informed human-in-the-loop mechanism enables clinicians to review and correct intermediate findings, triggering recomputation of affected results. We evaluate the prediction models using 600 participants from the FNIH Osteoarthritis Biomarkers Consortium cohort. On the test set of 100 participants, the fusion models achieve AUROCs of 0.80 for structural progression and 0.68 for pain progression. A case study illustrates how OA-MAP combines risk estimates with intermediate findings, cross-modal conflicts, literature support, and uncertainty indicators to support interactive review.

Atlas curation note, as recorded: - `2610.12134v1` — *OA-MAP: Evidence-Grounded Multi-Agent Multimodal Framework for Interpretable Knee Osteoarthritis Progression* — isomorphism: HUMAN-IN-THE-LOOP REPAIR PROPAGATION. Modality-specific agents + coordinator, uncertainty-informed clinician review corrects an intermediate finding and triggers recomputation of only affected results (AUROC 0.80 structural / 0.68 pain, n=100 test). Genuine seam-local repair, but the framework contribution is mostly orchestration.

Authors: Fabian Broszio, Michael Hoffmann, Adrian Paschke
Published (record): 2026-10-16
Topic in the record: human-in-the-loop clinical
Atlas status: interesting

Abstract, as recorded: Large language models (LLMs) can generate fluent outputs that lack source support or conflict with local operational constraints. We introduce Self-Reflective Chain-of-Thought (SRCoT), a modular architecture that organizes generation as a closed Retrieve--Reflect--Revise loop. SRCoT separates environment state, persistent knowledge, accumulated experience, and short-term memory into provenance-labeled stores. BM25 retrieval supplies local context; subsequent passes expose intermediate drafts, missing-information markers, follow-up queries, and optional claim--evidence mappings. These artifacts inform retrieval, revision, clarification, or abstention decisions. We examine the architecture through qualitative, trace-based case studies in two fully synthetic settings: emergency medical services and Munich-Sim, an emergency-operations simulation. Local prototype runs on an Apple M4-family laptop with 128 GB memory illustrate source-labeled documentation, explicit gap marking, and structured dispatch-style outputs. The traces also reveal provenance-assignment errors, incomplete evidence coverage, and unsafe assumptions that remain possible despite prompting constraints. These cases demonstrate report-level inspectability and local execution, not comparative superiority, hallucination reduction, clinical validity, or operational safety. The architecture separates retrieval and control from the model interface, but behavior across different models remains to be evaluated. SRCoT is a research proof of concept requiring human review, with its principal contribution lying in an inspectable control procedure rather than a new model.

Atlas curation note, as recorded: - `2610.23311v1` — *Self-Reflective Chain-of-Thought (SRCoT): Evidence-Based Text Generation via a Closed-Loop Retrieve--Reflect--Revise Pipeline on Edge LLMs* — isomorphism: MEMORY / PROVENANCE / SEAMS. Generation as a closed Retrieve–Reflect–Revise loop over provenance-labelled stores that deliberately separate environment state, persistent knowledge, accumulated experience, and short-term memory; drafts expose missing-information markers and claim–evidence mappings for revision/abstention. Inspectability is the contribution, not a new model — provenance survives only as an explicit field.

Authors: Shrish Shrinath Vaidya, Minmini Selvam, Ganapathy Krishnamurthi
Published (record): 2026-10-17
Topic in the record: human-in-the-loop clinical
Atlas status: weak

Abstract, as recorded: The adoption of Large Language Models for radiology report structuring has been largely dominated by either proprietary cloud-based models or large open models requiring high-end GPUs for deployment, which is a critical bottleneck for clinical adoption in radiology workflows that demand data privacy and offline operation. To address this, we present MedPAO-Assist, an edge-deployable agentic framework that generates structured radiology reports directly from radiologist dictation and voice commands, requiring only $\approx3.3$ GB peak RAM and achieving an average end-to-end latency of $\approx31$ seconds on a standard laptop CPU through an asynchronous, multi-threaded execution pipeline. The system currently supports four major imaging modalities -- Chest X-ray, Chest CT, Abdomen CT, and Brain MRI. To our knowledge, it is among the earliest agentic systems to run its entire pipeline with a Small Language Model (SLM) in the 0.6B-parameter range. This multi-agent architecture, built around an Extract-Ground-Generate paradigm, performs concept extraction, SNOMED-CT grounding, and structured report generation, followed by a speech-driven human-in-the-loop correction phase, together forming a fully "no-touch" reporting workflow that requires no keyboard input, external API calls, or internet connectivity. We evaluate each pipeline stage individually -- concept extraction, ontology grounding, and concept categorization and report hardware profiling results demonstrating the system's feasibility for real-time, CPU-only clinical deployment.

Atlas curation note, as recorded: - `2610.26913v1` — *MedPAO-Assist: Edge-Deployable, Modality-Aware Agentic AI for Radiology Report Structuring* — isomorphism: CONSTRAINTS / HUMAN-IN-THE-LOOP. Extract-Ground-Generate with SNOMED-CT grounding on a 0.6B SLM, ~3.3 GB RAM, ~31s on laptop CPU, speech-driven no-touch correction, fully offline. Real edge-constraint engineering, thin structural rhyme beyond it.