Publication AI Automatons: AI Systems Intended to Imitate Humans Alexandra Olteanu, Solon Barocas, Su Lin Blodgett, Lisa Egede, Alicia DeVrio, Myra Cheng March 2025 Video Project
Publication Tip of the Tongue Query Elicitation for Simulated Evaluation Yifan He, To Eun Kim, Fernando Diaz, Jaime Arguello, Bhaskar Mitra International ACM SIGIR Conference on Research and Development in Information Retrieval | February 2025
Publication Judging the Judges: A Collection of LLM-Generated Relevance Judgements Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L. A. Clarke, Guglielmo Faggioli, Bhaskar Mitra, Paul Thomas, Emine Yilmaz February 2025
Publication Recall, Robustness, and Lexicographic Evaluation Fernando Diaz, Michael D. Ekstrand, Bhaskar Mitra ACM Transactions on Recommender Systems | February 2025
Publication Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs ICML 2025 | January 2025
Publication JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra The Web Conference | January 2025
Publication SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval Hossein A. Rahmani, Xi Wang, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas The Web Conference | January 2025
Publication Sociotechnical Implications of Generative Artificial Intelligence for Information Access Bhaskar Mitra, Henriette Cramer, Olya Gurevich Information Access in the Era of Generative AI, 7 | Published by Springer | 2025
Publication Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach November 2024 November 2024
Research