publications
in reverse chronological order. * denotes equal contribution.
2026
- Paint It, Black: Evaluating LLMs for De-Identification of Multimodal Clinical DocumentsFrancesco Zangrillo*, Alessio Cocchieri*, Giacomo Frisoni, Davide Freddi, Luca Ragazzi, Alberto Aguzzi, Diego Raimondo, Renato Seracchioli, and Gianluca MoroIn Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), 2026
Secondary use of clinical data requires removing Protected Health Information (PHI) under regulations such as HIPAA. While de-identification is well studied for text, clinical records are often structured PDFs, where PHI appears across complex layouts, tables, and headers. We introduce Paint-It-Black (PIB), the first benchmark for end-to-end, pixel-level PHI de-identification in clinical PDFs. PIB contains physician-validated synthetic documents with diverse layouts, document types, and controlled visual degradations, annotated with HIPAA-aligned labels and bounding boxes. We evaluate locally deployable Multimodal Large Language Models (MLLMs) and find that even the strongest remain below the reliability required for clinical use. A conventional OCR+NER baseline performs substantially worse than the best MLLMs, highlighting the inherently multimodal nature of clinical PHI de-identification.
@inproceedings{zangrillo2026paintitblack, author = {Zangrillo, Francesco and Cocchieri, Alessio and Frisoni, Giacomo and Freddi, Davide and Ragazzi, Luca and Aguzzi, Alberto and Raimondo, Diego and Seracchioli, Renato and Moro, Gianluca}, title = {Paint It, Black: Evaluating LLMs for De-Identification of Multimodal Clinical Documents}, booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026)}, publisher = {Association for Computational Linguistics}, year = {2026} } - Lost in Choice: Evaluating LLMs Under Extreme Multiple-Choice ScalingAlessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, and Gianluca MoroIn Findings of the Association for Computational Linguistics: EMNLP 2026, 2026
Existing multiple-choice evaluations use small option sets and thus conflate several sources of failure when candidate spaces grow. We provide a controlled factorial decomposition of large-choice degradation across four axes: whether a question requires comparing its options, candidate cardinality (up to 1,000), spatial layout, and gold-answer position. Across six reasoning LLMs, comparative questions degrade far more than questions solvable without inspecting the options; dispersion amplifies this gap; and accuracy often collapses at late-middle and terminal positions rather than recovering in the classic lost-in-the-middle U-shape. We call this asymmetric, structure-dependent failure mode Lost in Choice. A replication on 500 real ToolBench tool-selection trajectories per condition reproduces the same hardest position on two additional model–task evaluations. Finally, an audit of reasoning traces yields a taxonomy of the heuristics and shortcuts models adopt as candidate spaces expand.
@inproceedings{cocchieri2026lostinchoice, author = {Cocchieri, Alessio and Ragazzi, Luca and Tagliavini, Giuseppe and Moro, Gianluca}, title = {Lost in Choice: Evaluating LLMs Under Extreme Multiple-Choice Scaling}, booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026}, publisher = {Association for Computational Linguistics}, year = {2026} } - LLMs (Almost) Never Abstain Under Medical UncertaintyAlessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, and Gianluca MoroIn Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), 2026
Medical multiple-choice question answering (MCQA) benchmarks implicitly assume that large language models (LLMs) should always commit to an answer. However, in clinical practice, uncertainty is pervasive and abstaining is often the safest action. We introduce MedQAbstain, a benchmark explicitly designed to evaluate medical abstention under uncertainty. MedQAbstain repurposes standard medical MCQA datasets by removing the gold answer and introducing an explicit “I abstain” option, framed as a safety-critical decision with clinical consequences. The benchmark supports systematic analysis across abstention regimes, distractor complexity, and input modalities, and elicits self-reported model confidence to study calibration. Across all settings, we find that state-of-the-art LLMs systematically overcommit, rarely abstaining even when the question itself is hidden. These results reveal a fundamental mismatch between LLM behavior and clinical norms, highlighting abstention as a critical but overlooked dimension of medical decision-making evaluation.
@inproceedings{DBLP:conf/acl/CocchieriRTM26, author = {Alessio Cocchieri and Luca Ragazzi and Giuseppe Tagliavini and Gianluca Moro}, editor = {Maria Liakata and Viviane P. Moreira and Jiajun Zhang and David Jurgens}, title = {LLMs (Almost) Never Abstain Under Medical Uncertainty}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), {ACL} 2026, San Diego, California, United States, July 2-7, 2026}, pages = {29573--29613}, publisher = {Association for Computational Linguistics}, year = {2026}, url = {https://doi.org/10.18653/v1/2026.acl-long.1365}, doi = {10.18653/V1/2026.ACL-LONG.1365}, timestamp = {Thu, 30 Jul 2026 01:00:00 +0200}, biburl = {https://dblp.org/rec/conf/acl/CocchieriRTM26.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} } - Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?Lorenzo Molfetta, Alessio Cocchieri, Luca Ragazzi, Ilaria Bartolini, Marco Patella, and Gianluca MoroIn Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), 2026
In medicine, claims remain valid when supported by empirical evidence grounded in stable biological reality. In law, by contrast, truth is contingent, defined by jurisdiction, temporal validity, and the hierarchy of authoritative sources. The recent success of large language models (LLMs) on medical licensing examinations has encouraged an expectation of comparable legal competence. This analogy, however, obscures a critical distinction between domains. Unlike in medicine, legal performance often depends less on inference than on determining when external authority is applicable, valid, and non-contradictory. We introduce a comparative diagnostic framework evaluating legal reasoning against medical baselines along four axes (knowledge recall, grounding, confidence, and robustness), uncovering a sharp domain asymmetry when applied to a new benchmark that encodes temporal validity and normative relationships. While medical LLMs reliably benefit from verified sources, legal LLMs struggle to assess when retrieved citations are useful or misleading, exhibiting overconfidence in perturbed contexts and sensitivity to superficial formatting cues. Increased model scale amplifies this tendency, revealing that stronger instruction following can coincide with weaker resistance to authoritative perturbations. These findings show that LLMs treat law as unstructured text rather than binding precedent, while revealing a tendency to over-trust authoritative but false information when external references conflict with a model’s internal knowledge.
@inproceedings{DBLP:conf/acl/MolfettaCRBPM26, author = {Lorenzo Molfetta and Alessio Cocchieri and Luca Ragazzi and Ilaria Bartolini and Marco Patella and Gianluca Moro}, editor = {Maria Liakata and Viviane P. Moreira and Jiajun Zhang and David Jurgens}, title = {Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), {ACL} 2026, San Diego, California, United States, July 2-7, 2026}, pages = {10865--10886}, publisher = {Association for Computational Linguistics}, year = {2026}, url = {https://doi.org/10.18653/v1/2026.acl-long.497}, doi = {10.18653/V1/2026.ACL-LONG.497}, timestamp = {Thu, 30 Jul 2026 01:00:00 +0200}, biburl = {https://dblp.org/rec/conf/acl/MolfettaCRBPM26.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} } - Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programmingAlessio Cocchieri, Luca Ragazzi, Gianluca Aguzzi, Giacomo Frisoni, Lorenzo Molfetta, Gianluca Moro, and Mirko ViroliJournal of Systems and Software, 2026
Current code generation benchmarks largely overlook object-oriented programming (OOP) skills, leaving open whether large language models (LLMs) can effectively apply OOP principles to structured programming tasks. The dominance and permissiveness of Python in most benchmarks further obscure model weaknesses in core OOP principles. We introduce the Java Academic Benchmark (JAB), based on 103 authentic Java exams collected over a decade at a major university, together with 506 expert-written JUnit tests, to rigorously evaluate advanced Java programming competence with a strong focus on OOP. To complement execution-based scoring, we propose KODE, an LLM-as-a-Judge framework that assesses OOP adherence across four pedagogical dimensions. We evaluate 27 LLMs under two resolution strategies: single-attempt (one-shot) and agentic (iterative refinement with compiler and test feedback). Results reveal that, under our evaluation protocol, larger closed models match or surpass bachelor-level OOP students – especially under agentic resolution – while smaller open models lag behind. JAB’s class-level design enables fine-grained error analysis, exposing recurring misconceptions.
@article{cocchieri2026jab, author = {Cocchieri, Alessio and Ragazzi, Luca and Aguzzi, Gianluca and Frisoni, Giacomo and Molfetta, Lorenzo and Moro, Gianluca and Viroli, Mirko}, title = {Java academic benchmark: Exam-based evaluation of {LLMs} on object-oriented programming}, journal = {Journal of Systems and Software}, volume = {242}, pages = {113033}, year = {2026}, doi = {10.1016/j.jss.2026.113033} } - OpenBioNER-v2: A suite of lightweight models for zero-shot medical named entity recognition via type descriptionsAlessio Cocchieri, Giacomo Frisoni, Francesco Zangrillo, Luca Ragazzi, Marcos Martı́nez Galindo, Giuseppe Tagliavini, and Gianluca MoroExpert Systems with Applications, 2026
Named entity recognition (NER) in medicine is challenging due to specialized terminology, inconsistent annotation guidelines, and the continuous emergence of new entity types—requiring models that can adapt to unseen targets. Large language models (LLMs) exhibit strong generalization but are impractical for scalable deployment, whereas recent encoder-only approaches leverage entity names for zero-shot inference but struggle with disambiguation in complex domains. We introduce OpenBioNER-v2, a family of lightweight transformer encoders (15M–110M parameters) designed for zero-shot recognition of biomedical and clinical entities by conditioning on natural language descriptions of target types. Our cross-encoder architecture jointly models input text and entity-type descriptions, enabling semantic matches. Pretrained on LLM-generated silver annotations and multi-view descriptions covering thousands of medical types, OpenBioNER-v2 achieves state-of-the-art results across 11 benchmarks—including a new dataset for personal de-identification. Variants with ≤56M parameters outperform both large and small language models, such as UniversalNER and GliNER. Ablation studies reveal effective strategies for formulating descriptions. All data, code, and model checkpoints are publicly released under open-science principles.
@article{DBLP:journals/eswa/CocchieriFZRGTM26, author = {Alessio Cocchieri and Giacomo Frisoni and Francesco Zangrillo and Luca Ragazzi and Marcos Mart{\'{\i}}nez Galindo and Giuseppe Tagliavini and Gianluca Moro}, title = {OpenBioNER-v2: {A} suite of lightweight models for zero-shot medical named entity recognition via type descriptions}, journal = {Expert Syst. Appl.}, volume = {318}, pages = {131725}, year = {2026}, url = {https://doi.org/10.1016/j.eswa.2026.131725}, doi = {10.1016/J.ESWA.2026.131725}, timestamp = {Thu, 25 Jun 2026 01:00:00 +0200}, biburl = {https://dblp.org/rec/journals/eswa/CocchieriFZRGTM26.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} } - ReMedQA: Are We Done With Medical Multiple-Choice Benchmarks?Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, and Gianluca MoroIn Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026), 2026
Medical multiple-choice question answering (MCQA) benchmarks show that models achieve near-human accuracy, with some benchmarks approaching saturation—leading to claims of clinical readiness. Yet a single accuracy score is a poor proxy for competence: models that change answers under minor input perturbations cannot be considered reliable. We argue that reliability underpins accuracy—only consistent predictions make correctness meaningful. We release ReMedQA, a new benchmark that augments three standard medical MCQA datasets with open-ended answers and systematically perturbed options. Building on this, we introduce ReAcc and ReCon, two reliability metrics: ReAcc measures the proportion of questions answered correctly across all variations, while ReCon measures the proportion answered consistently regardless of correctness. Our evaluation shows that high MCQA accuracy masks low reliability: models remain sensitive to format and perturbation changes, and domain specialization offers no robustness gain. MCQA underestimates smaller models while inflating large ones that exploit structural cues— with some exceeding 50% accuracy even when the original questions are hidden. This shows that, despite near-saturated accuracy, we are not yet done with medical MCQA benchmarks.
@inproceedings{DBLP:conf/eacl/CocchieriRTM26, author = {Alessio Cocchieri and Luca Ragazzi and Giuseppe Tagliavini and Gianluca Moro}, editor = {Vera Demberg and Kentaro Inui and Llu{\'{\i}}s Marquez}, title = {ReMedQA: Are We Done With Medical Multiple-Choice Benchmarks?}, booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, {EACL} 2026 - Volume 1: Long Papers, Rabat, Morocco, March 24-29, 2026}, pages = {2706--2738}, publisher = {Association for Computational Linguistics}, year = {2026}, url = {https://doi.org/10.18653/v1/2026.eacl-long.124}, doi = {10.18653/V1/2026.EACL-LONG.124}, timestamp = {Wed, 10 Jun 2026 01:00:00 +0200}, biburl = {https://dblp.org/rec/conf/eacl/CocchieriRTM26.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} }
2025
- ZeroNER: Fueling Zero-Shot Named Entity Recognition via Entity Type DescriptionsAlessio Cocchieri, Marcos Martı́nez Galindo, Giacomo Frisoni, Gianluca Moro, Claudio Sartori, and Giuseppe TagliaviniIn Findings of the Association for Computational Linguistics: ACL 2025, 2025
What happens when a named entity recognition (NER) system encounters entities it has never seen before? In practical applications, models must generalize to unseen entity types where labeled training data is either unavailable or severely limited—a challenge that demands zero-shot learning capabilities. While large language models (LLMs) offer extensive parametric knowledge, they fall short in cost-effectiveness compared to specialized small encoders. Existing zero-shot methods predominantly adopt a relaxed definition of the term with potential leakage issues and rely on entity type names for generalization, overlooking the value of richer descriptions for disambiguation. In this work, we introduce ZE- RONER, a description-driven framework that enhances hard zero-shot NER in low-resource settings. By leveraging general-domain annotations and entity type descriptions with LLM supervision, ZERONER enables a BERT-based student model to successfully identify unseen entity types. Evaluated on three real-world benchmarks, ZERONER consistently outperforms LLMs by up to 16% in F1 score, and surpasses lightweight baselines that use type names alone. Our analysis further reveals that LLMs derive significant benefits from incorporating type descriptions in the prompts.
@inproceedings{DBLP:conf/acl/CocchieriGFM0T25, author = {Alessio Cocchieri and Marcos Mart{\'{\i}}nez Galindo and Giacomo Frisoni and Gianluca Moro and Claudio Sartori and Giuseppe Tagliavini}, editor = {Wanxiang Che and Joyce Nabende and Ekaterina Shutova and Mohammad Taher Pilehvar}, title = {ZeroNER: Fueling Zero-Shot Named Entity Recognition via Entity Type Descriptions}, booktitle = {Findings of the Association for Computational Linguistics, {ACL} 2025, Vienna, Austria, July 27 - August 1, 2025}, series = {Findings of {ACL}}, volume = {{ACL} 2025}, pages = {15594--15616}, publisher = {Association for Computational Linguistics}, year = {2025}, url = {https://doi.org/10.18653/v1/2025.findings-acl.805}, doi = {10.18653/V1/2025.FINDINGS-ACL.805}, timestamp = {Wed, 10 Jun 2026 01:00:00 +0200}, biburl = {https://dblp.org/rec/conf/acl/CocchieriGFM0T25.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} } - "What do you call a dog that is incontrovertibly true? Dogma": Testing LLM Generalization through HumorAlessio Cocchieri, Luca Ragazzi, Paolo Italiani, Giuseppe Tagliavini, and Gianluca MoroIn Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), 2025
Humor, requiring creativity and contextual understanding, is a hallmark of human intelligence, showcasing adaptability across linguistic scenarios. While recent advances in large language models (LLMs) demonstrate strong reasoning on various benchmarks, it remains unclear whether they truly adapt to new tasks like humans (i.e., generalize) or merely replicate memorized content. To explore this, we introduce PHUNNY, a new humor-based question-answering benchmark designed to assess LLMs’ reasoning through carefully crafted puns. Our dataset is manually curated to ensure novelty and minimize data contamination, providing a robust evaluation of LLMs’ linguistic comprehension. Experiments on pun comprehension, resolution, and generation reveal that most LLMs struggle with generalization, even on simple tasks, consistently underperforming the human baseline. Additionally, our detailed error analysis provides valuable insights to guide future research.
@inproceedings{DBLP:conf/acl/CocchieriRITM25, author = {Alessio Cocchieri and Luca Ragazzi and Paolo Italiani and Giuseppe Tagliavini and Gianluca Moro}, editor = {Wanxiang Che and Joyce Nabende and Ekaterina Shutova and Mohammad Taher Pilehvar}, title = {"What do you call a dog that is incontrovertibly true? Dogma": Testing {LLM} Generalization through Humor}, booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), {ACL} 2025, Vienna, Austria, July 27 - August 1, 2025}, pages = {22922--22937}, publisher = {Association for Computational Linguistics}, year = {2025}, url = {https://doi.org/10.18653/v1/2025.acl-long.1117}, doi = {10.18653/V1/2025.ACL-LONG.1117}, timestamp = {Wed, 10 Jun 2026 01:00:00 +0200}, biburl = {https://dblp.org/rec/conf/acl/CocchieriRITM25.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} } - FEAST: Retrieval-Augmented Multi-Hierarchical Food Classification for the FoodEx2 SystemLorenzo Molfetta, Alessio Cocchieri, Stefano Fantazzini, Giacomo Frisoni, Luca Ragazzi, and Gianluca MoroIn 28th European Conference on Artificial Intelligence (ECAI 2025), 2025
Hierarchical text classification (HTC) and extreme multi-label classification (XML) tasks face compounded challenges from complex label interdependencies, data sparsity, and extreme output dimensions. These challenges are exemplified in the European Food Safety Authority’s FoodEx2 system–a standardized food classification framework essential for food consumption monitoring and contaminant exposure assessment across Europe. FoodEx2 coding transforms natural language food descriptions into a set of codes from multiple standardized hierarchies, but faces implementation barriers due to its complex structure. Given a food description (e.g., “organic yogurt”), the system identifies its base term (“yogurt”), all the applicable facet categories (e.g., “production method”), and then, every relevant facet descriptors to each category (e.g., “organic production”). While existing models perform adequately on well-balanced and semantically dense hierarchies, no work has been applied on the practical constraints imposed by the FoodEx2 system. The limited literature addressing such real-world scenarios further compounds these challenges. We propose FEAST (Food Embedding And Semantic Taxonomy), a novel retrieval-augmented framework that decomposes FoodEx2 classification into a three-stage approach: (1) base term identification, (2) multi-label facet prediction, and (3) facet descriptor assignment. By leveraging the system’s hierarchical structure to guide training and performing deep metric learning, FEAST learns discriminative embeddings that mitigate data sparsity and improve generalization on rare and fine-grained labels. Evaluated on the multilingual FoodEx2 benchmark, FEAST outperforms the prior European’s CNN baseline F1 scores by 12–38% on rare classes.
@inproceedings{DBLP:conf/ecai/MolfettaCFFRM25, author = {Lorenzo Molfetta and Alessio Cocchieri and Stefano Fantazzini and Giacomo Frisoni and Luca Ragazzi and Gianluca Moro}, editor = {In{\^{e}}s Lynce and Nello Murano and Mauro Vallati and Serena Villata and Federico Chesani and Michela Milano and Andrea Omicini and Mehdi Dastani}, title = {{FEAST:} Retrieval-Augmented Multi-Hierarchical Food Classification for the FoodEx2 System}, booktitle = {{ECAI} 2025 - 28th European Conference on Artificial Intelligence, 25-30 October 2025, Bologna, Italy - Including 14th Conference on Prestigious Applications of Intelligent Systems {(PAIS} 2025)}, series = {Frontiers in Artificial Intelligence and Applications}, volume = {413}, pages = {4169--4176}, publisher = {{IOS} Press}, year = {2025}, url = {https://doi.org/10.3233/FAIA251309}, doi = {10.3233/FAIA251309}, timestamp = {Thu, 05 Mar 2026 00:00:00 +0100}, biburl = {https://dblp.org/rec/conf/ecai/MolfettaCFFRM25.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} } - Can Large Language Models Win the International Mathematical Games?Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, Lorenzo Tordi, Antonella Carbonaro, and Gianluca MoroIn Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), 2025
Recent advances in large language models (LLMs) have demonstrated strong mathematical reasoning abilities, even in visual contexts, with some models surpassing human performance on existing benchmarks. However, these benchmarks lack structured age categorization, clearly defined skill requirements, and–crucially–were not designed to assess human performance in international competitions. To address these limitations, we introduce MATHGAMES, a new benchmark of 2,183 high-quality mathematical problems (both text-only and multimodal) in an open-ended format, sourced from an international mathematical games championships. Spanning seven age groups and a skill-based taxonomy, MATHGAMES enables a structured evaluation of LLMs’ mathematical and logical reasoning abilities. Our experiments reveal a substantial gap between state-of-the-art LLMs and human participants–even 11-year-olds consistently outperform some of the strongest models–highlighting the need for advancements. Further, our detailed error analysis offers valuable insights to guide future research. The data is publicly available at https://disi-unibo-nlp.github.io/math-games/.
@inproceedings{DBLP:conf/emnlp/CocchieriRTTCM25, author = {Alessio Cocchieri and Luca Ragazzi and Giuseppe Tagliavini and Lorenzo Tordi and Antonella Carbonaro and Gianluca Moro}, editor = {Christos Christodoulopoulos and Tanmoy Chakraborty and Carolyn Rose and Violet Peng}, title = {Can Large Language Models Win the International Mathematical Games?}, booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, {EMNLP} 2025, Suzhou, China, November 4-9, 2025}, pages = {9645--9671}, publisher = {Association for Computational Linguistics}, year = {2025}, url = {https://doi.org/10.18653/v1/2025.emnlp-main.488}, doi = {10.18653/V1/2025.EMNLP-MAIN.488}, timestamp = {Wed, 10 Jun 2026 11:17:00 +0200}, biburl = {https://dblp.org/rec/conf/emnlp/CocchieriRTTCM25.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} } - OpenBioNER: Lightweight Open-Domain Biomedical Named Entity Recognition Through Entity Type DescriptionAlessio Cocchieri, Giacomo Frisoni, Marcos Martı́nez Galindo, Gianluca Moro, Giuseppe Tagliavini, and Francesco CandoliIn Findings of the Association for Computational Linguistics: NAACL 2025, 2025
Biomedical Named Entity Recognition (BioNER) faces significant challenges in real-world applications due to limited annotated data and the constant emergence of new entity types, making zero-shot learning capabilities crucial. While Large Language Models (LLMs) possess extensive domain knowledge necessary for specialized fields like biomedicine, their computational costs often make them impractical. To address these challenges, we introduce OPENBIONER, a lightweight BERT-based cross-encoder architecture that can identify any biomedical entity using only its description, eliminating the need for retraining on new, unseen entity types. Through comprehensive evaluation on established biomedical benchmarks, we demonstrate that OPENBIONER surpasses state-of-the-art baselines, including specialized 7B NER LLMs and GPT-4o, achieving up to 10% higher F1 scores while using 110M parameters only. Moreover, OPENBIONER outperforms existing small-scale models that match textual spans with entity types rather than descriptions, both in terms of accuracy and computational efficiency.
@inproceedings{DBLP:conf/naacl/CocchieriFGMTC25, author = {Alessio Cocchieri and Giacomo Frisoni and Marcos Mart{\'{\i}}nez Galindo and Gianluca Moro and Giuseppe Tagliavini and Francesco Candoli}, editor = {Luis Chiruzzo and Alan Ritter and Lu Wang}, title = {OpenBioNER: Lightweight Open-Domain Biomedical Named Entity Recognition Through Entity Type Description}, booktitle = {Findings of the Association for Computational Linguistics: {NAACL} 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025}, series = {Findings of {ACL}}, volume = {{NAACL} 2025}, pages = {818--837}, publisher = {Association for Computational Linguistics}, year = {2025}, url = {https://doi.org/10.18653/v1/2025.findings-naacl.47}, doi = {10.18653/V1/2025.FINDINGS-NAACL.47}, timestamp = {Tue, 27 Jan 2026 20:27:02 +0100}, biburl = {https://dblp.org/rec/conf/naacl/CocchieriFGMTC25.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} }
2024
- To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question AnsweringGiacomo Frisoni*, Alessio Cocchieri*, Alex Presepi, Gianluca Moro, and Zaiqiao MengIn Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024
Medical open-domain question answering demands substantial access to specialized knowledge. Recent efforts have sought to decouple knowledge from model parameters, counteracting architectural scaling and allowing for training on common low-resource hardware. The retrieve-then-read paradigm has become ubiquitous, with model predictions grounded on relevant knowledge pieces from external repositories such as PubMed, textbooks, and UMLS. An alternative path, still under-explored but made possible by the advent of domain-specific large language models, entails constructing artificial contexts through prompting. As a result, "to generate or to retrieve" is the modern equivalent of Hamlet’s dilemma. This paper presents MEDGENIE, the first generate-then-read framework for multiple-choice question answering in medicine. We conduct extensive experiments on MedQA-USMLE, MedMCQA, and MMLU, incorporating a practical perspective by assuming a maximum of 24GB VRAM. MEDGENIE sets a new state-of-the-art in the open-book setting of each testbed, allowing a small-scale reader to outcompete zero-shot closed-book 175B baselines while using up to 706× fewer parameters. Our findings reveal that generated passages are more effective than retrieved ones in attaining higher accuracy.
@inproceedings{DBLP:conf/acl/FrisoniCPMM24, author = {Giacomo Frisoni and Alessio Cocchieri and Alex Presepi and Gianluca Moro and Zaiqiao Meng}, editor = {Lun{-}Wei Ku and Andre Martins and Vivek Srikumar}, title = {To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering}, booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), {ACL} 2024, Bangkok, Thailand, August 11-16, 2024}, pages = {9878--9919}, publisher = {Association for Computational Linguistics}, year = {2024}, url = {https://doi.org/10.18653/v1/2024.acl-long.533}, doi = {10.18653/V1/2024.ACL-LONG.533}, timestamp = {Tue, 05 Aug 2025 01:00:00 +0200}, biburl = {https://dblp.org/rec/conf/acl/FrisoniCPMM24.bib}, bibsource = {dblp computer science bibliography, https://dblp.org} }
