--- title: "The AI Committee — paper by Vallabhaneni et al. (2026)" description: "The authors present the AI Committee (AIC), an open-source, model-agnostic multi-agent… 92.7 recall. 8 references, cited by 1 paper here." authors: ["Sunith Vallabhaneni", "Thomas Berkane", "Maimuna S. Majumder"] year: 2026 venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)" source: "https://aclanthology.org/2026.eacl-demo.41/" doi: "10.18653/v1/2026.eacl-demo.41" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/vallabhaneni-2026-ai-committee-multi-agent-framework/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vallabhaneni-2026-ai-committee-multi-agent-framework/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/18/) / The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data # The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data **Authors:** Sunith Vallabhaneni, Thomas Berkane and Maimuna S. Majumder · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.eacl-demo.41/) · [PDF](https://aclanthology.org/2026.eacl-demo.41.pdf) · [DOI 10.18653/v1/2026.eacl-demo.41](https://doi.org/10.18653/v1/2026.eacl-demo.41) · [OpenAlex](https://openalex.org/W7140087519) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read ## Summary LLM-based web agents can now build research datasets from web pages, but their output often contains invented or missing values, misread page content, and invalid entries that are slow to catch by hand. The authors present the AI Committee (AIC), an open-source, model-agnostic multi-agent system that checks and repairs tabular web-sourced data without fine-tuning. The user supplies a dataset with a source URL per row, a short description of its purpose, and the fields to check. AIC first derives dataset-specific rules and negative examples from a small sample through in-context learning. It then routes each record through specialised agents: relevancy screening, source scrutiny, chain-of-thought fact-checking, and integrity and format checks. A remediation loop fixes rejected rows and pulls new records from the source pages. On three LLM-collected datasets with human-cleaned ground truth, AIC outperformed a single-prompt monolithic agent and a rule-based cleaner. The cheapest backbone tested, gpt-4o-mini, gave the best F1 at a fraction of the cost, which suits research groups with limited budgets. ## Key points - **Evaluation data:** three web-collected datasets: - Natural Disasters: 125 points on Haiti and Cameroon. - Police Misconduct: 95 points from the US. - COVID-19 Tracing: 83 points on contact-tracing app downloads per US state. Three public health researchers annotated the ground truth under an IAA protocol. - **Pipeline components:** a ContextGenerator, a Relevancy Assessor, source scrutiny, a FactCheckerAgent, an ArbiterAgent, and a DataRemediationAgent. - The ContextGenerator infers each field's semantics and negative examples from 10 random rows. - The FactCheckerAgent combines CoT reasoning with a semantic audit. - The ArbiterAgent passes reasons for rejection to the DataRemediationAgent, which corrects the records. - A schema generated from the user's field list enforces structure. - **Main results:** AIC with gpt-4o-mini reached F1 85.1 (precision 92.7, recall 78.7) at about $0.30 per batch. - The gpt-5 monolithic baseline scored F1 71.0 at $4.95, making AIC more than 16× cheaper. - The rule-based baseline scored F1 71.4 and had the lowest recall (58.2). - AIC with gpt-5 reached 100% precision but with lower recall and much higher cost and latency. - **Ablation:** every agent contributes to performance. - Removing the formatter caused the largest F1 drop (−28.2). - Removing relevancy screening cost −12.0 F1, and removing remediation cost −6.4. - **Availability:** code, the three annotated datasets, and the IAA protocol are released under Apache 2.0 on GitHub (sunith-v/theAICommittee). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) — University, Cambridge, United States · 126 works · 45,130 citations - [Boston Children's Hospital](https://md.research.elamur.ai/free/en/hospitals/boston-children-s-hospital/) — Hospital, Boston, United States · 6 works · 678 citations 📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 134 works ## Cite Cite as: Sunith Vallabhaneni, Thomas Berkane and Maimuna S. Majumder. 2026. The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{vallabhaneni2026ai, title = {{The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data}}, author = {Sunith Vallabhaneni, Thomas Berkane and Maimuna S. Majumder}, booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)}, year = {2026}, doi = {10.18653/v1/2026.eacl-demo.41}, url = {https://doi.org/10.18653/v1/2026.eacl-demo.41} } ``` - [← previous Text-to-Text Automatic Story Generation: A Survey](https://md.research.elamur.ai/free/en/papers/ma-2026-text-text-automatic-story-generation/) - [next → The Argonauts at SemEval-2026 Task 9: Multilingual Polarization Detection and Classification Using LLM Prompting and Transformer Fine-Tuning](https://md.research.elamur.ai/free/en/papers/mahmud-2026-argonauts-semeval-2026-task-9/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/18/) ## References 8 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Tommaso Bendinelli, Artur Dox, and Christian Holz. 2025. Exploring LLM agents for cleaning tabular machine learning datasets. arXiv preprint arXiv:2503.06664. Proceedings of the ICLR 2025 Workshop on Foundation Models in the Wild. — [arXiv:2503.06664](https://arxiv.org/abs/2503.06664) [find](https://scholar.google.com/scholar?q=Tommaso+Bendinelli%2C+Artur+Dox%2C+and+Christian+Holz%2E+2025%2E+Exploring+LLM+agents+for+cleaning+tabular+machine+learning+datasets%2E+arXiv+preprint+arXiv%3A2503%2E06664%2E+Proceedings+of+the+ICLR+2025+Workshop+on+) 2. Thomas Berkane, Marie-Laure Charpignon, and Maimuna S. Majumder. 2025. LLM-based web data collection for research dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 12610–12622. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thomas+Berkane%2C+Marie%2DLaure+Charpignon%2C+and+Maimuna+S%2E+Majumder%2E+2025%2E+LLM%2Dbased+web+data+collection+for+research+dataset+creation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+) 3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. 4. Wenhao Huang, Zhouhong Gu, Chenghao Peng, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Liqian Wen, and Zulong Chen. 2024. AutoScraper: A progressive understanding web agent for web scraper generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2371–238… [find](https://scholar.google.com/scholar?q=Wenhao+Huang%2C+Zhouhong+Gu%2C+Chenghao+Peng%2C+Zhixu+Li%2C+Jiaqing+Liang%2C+Yanghua+Xiao%2C+Liqian+Wen%2C+and+Zulong+Chen%2E+2024%2E+AutoScraper%3A+A+progressive+understanding+web+agent+for+web+scraper+generation%2E+In+Pr) 5. Tianyi Ma, Yiyue Qian, Zheyuan Zhang, Zehong Wang, Xiaoye Qian, Feifan Bai, Yifan Ding, Xuwei Luo, Shinan Zhang, Keerthiram Murugesan, Chuxu Zhang, and Yanfang Ye. 2025. Auto-Data: A multi-agent system for open web data collection. arXiv preprint arXiv:2505.15859. — [arXiv:2505.15859](https://arxiv.org/abs/2505.15859) [find](https://scholar.google.com/scholar?q=Tianyi+Ma%2C+Yiyue+Qian%2C+Zheyuan+Zhang%2C+Zehong+Wang%2C+Xiaoye+Qian%2C+Feifan+Bai%2C+Yifan+Ding%2C+Xuwei+Luo%2C+Shinan+Zhang%2C+Keerthiram+Murugesan%2C+Chuxu+Zhang%2C+and+Yanfang+Ye%2E+2025%2E+Auto%2DData%3A+A+multi%2Dagent+syste) 6. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her) 7. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, volume 35, pages 24824– 24837. 8. Shuo Zhang, Zezhou Huang, and Eugene Wu. 2024. Data cleaning using large language models. arXiv preprint arXiv:2410.15547. — [arXiv:2410.15547](https://arxiv.org/abs/2410.15547) [find](https://scholar.google.com/scholar?q=Shuo+Zhang%2C+Zezhou+Huang%2C+and+Eugene+Wu%2E+2024%2E+Data+cleaning+using+large+language+models%2E+arXiv+preprint+arXiv%3A2410%2E15547%2E+%E2%80%94+%5BarXiv%3A2410%2E15547%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E15547%29) *Updated 31 October 2026*