A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look

Authors: Upadhyay, Shivani, Pradeep, Ronak, Thakur, Nandan, Campos, Daniel, Craswell, Nick, Soboroff, Ian, Dang, Hoa Trang, Lin, Jimmy

Subject Terms: Computer Science - Information Retrieval, Computer Science - Computation and Language

Access URL: http://arxiv.org/abs/2411.08275

LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?

Authors: Takehi, Rikiya, Voorhees, Ellen M., Sakai, Tetsuya, Soboroff, Ian

Subject Terms: Computer Science - Information Retrieval

Access URL: http://arxiv.org/abs/2411.06877

Don't Use LLMs to Make Relevance Judgments

Authors: Soboroff, Ian

Subject Terms: Computer Science - Information Retrieval

Access URL: http://arxiv.org/abs/2409.15133

QueryBuilder: Human-in-the-Loop Query Development for Information Retrieval

Authors: Kandula, Hemanth, Karakos, Damianos, Qiu, Haoling, Rozonoyer, Benjamin, Soboroff, Ian, Tarlin, Lee, Min, Bonan

Subject Terms: Computer Science - Information Retrieval, Computer Science - Computation and Language, Computer Science - Machine Learning

Access URL: http://arxiv.org/abs/2409.04667

Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance

Authors: Banerjee, Somnath, Halder, Avik, Mandal, Rajarshi, Layek, Sayan, Soboroff, Ian, Hazra, Rima, Mukherjee, Animesh

Subject Terms: Computer Science - Computation and Language

Access URL: http://arxiv.org/abs/2406.11139

On the Evaluation of Machine-Generated Reports

Authors: Mayfield, James, Yang, Eugene, Lawrie, Dawn, MacAvaney, Sean, McNamee, Paul, Oard, Douglas W., Soldaini, Luca, Soboroff, Ian, Weller, Orion, Kayi, Efsun, Sanders, Kate, Mason, Marc, Hibbler, Noah

Subject Terms: Computer Science - Computation and Language, Computer Science - Information Retrieval

Access URL: http://arxiv.org/abs/2405.00982

Corrected Evaluation Results of the NTCIR WWW-2, WWW-3, and WWW-4 English Subtasks

Authors: Sakai, Tetsuya, Tao, Sijie, Maistro, Maria, Chu, Zhumin, Li, Yujing, Chen, Nuo, Ferro, Nicola, Wang, Junjie, Soboroff, Ian, Liu, Yiqun

Subject Terms: Computer Science - Information Retrieval

Access URL: http://arxiv.org/abs/2210.10266

Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?

Authors: Voorhees, Ellen M., Soboroff, Ian, Lin, Jimmy

Subject Terms: Computer Science - Information Retrieval

Access URL: http://arxiv.org/abs/2201.11086

Podcast Metadata and Content: Episode Relevance andAttractiveness in Ad Hoc Search

Authors: Carterette, Ben, Jones, Rosie, Jones, Gareth F., Eskevich, Maria, Reddy, Sravana, Clifton, Ann, Yu, Yongze, Karlgren, Jussi, Soboroff, Ian

Subject Terms: Computer Science - Information Retrieval

Access URL: http://arxiv.org/abs/2108.11460

Searching for Scientific Evidence in a Pandemic: An Overview of TREC-COVID

Authors: Roberts, Kirk, Alam, Tasmeer, Bedrick, Steven, Demner-Fushman, Dina, Lo, Kyle, Soboroff, Ian, Voorhees, Ellen, Wang, Lucy Lu, Hersh, William R

Subject Terms: Computer Science - Information Retrieval

Access URL: http://arxiv.org/abs/2104.09632

TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime

Authors: Craswell, Nick, Mitra, Bhaskar, Yilmaz, Emine, Campos, Daniel, Voorhees, Ellen M., Soboroff, Ian

Subject Terms: Computer Science - Information Retrieval, Computer Science - Artificial Intelligence, Computer Science - Machine Learning

Access URL: http://arxiv.org/abs/2104.09399

How to Measure the Reproducibility of System-oriented IR Experiments

Authors: Breuer, Timo, Ferro, Nicola, Fuhr, Norbert, Maistro, Maria, Sakai, Tetsuya, Schaer, Philipp, Soboroff, Ian

Subject Terms: Computer Science - Information Retrieval

Access URL: http://arxiv.org/abs/2010.13447

TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection

Authors: Voorhees, Ellen, Alam, Tasmeer, Bedrick, Steven, Demner-Fushman, Dina, Hersh, William R, Lo, Kyle, Roberts, Kirk, Soboroff, Ian, Wang, Lucy Lu

Subject Terms: Computer Science - Information Retrieval, H.3.0

Access URL: http://arxiv.org/abs/2005.04474

HLVU : A New Challenge to Test Deep Understanding of Movies the Way Humans do

Authors: Curtis, Keith, Awad, George, Rajput, Shahzad, Soboroff, Ian

Subject Terms: Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Computer Vision and Pattern Recognition

Access URL: http://arxiv.org/abs/2005.00463

Introduction to the second issue on machine translation for low-resource languages

Authors: Liu, Chao-Hong, Karakanta, Alina, Tong, Audrey N., Aulov, Oleg, Soboroff, Ian M., Washington, Jonathan, Zhao, Xiaobing

Source: Machine Translation, 2021 Apr 01. 35(1), 1-2.

Access URL: https://www.jstor.org/stable/48758484

Introduction to the Special Issue on Machine Translation for Low-Resource Languages

Authors: Liu, Chao-Hong, Karakanta, Alina, Tong, Audrey N., Aulov, Oleg, Soboroff, Ian M., Washington, Jonathan, Zhao, Xiaobing

Source: Machine Translation, 2020 Dec 01. 34(4), 247-249.

Access URL: https://www.jstor.org/stable/48740623

Building Better Search Engines by Measuring Search Quality

Authors: Voorhees, Ellen M., Over, Paul, Soboroff, Ian

Source: IT Professional IT Prof. IT Professional. 16(2):22-30 Jan, 2014

Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm
Labs of the evaluation forum (International Conference)

Authors: Ferro, Nicola, Fuhr, Norbert, Maistro, Maria, Sakai, Tetsuya, Soboroff, Ian

Source: Lecture notes in computer science. 11696:287-300

Availability: http://explore.bl.uk/primo_library/libweb/action/display.do?tabs=detailsTab&gathStatTab=true&ct=display&fn=search&doc=ETOCCN605436211&indx=1&recIds=ETOCCN605436211

CENTRE@CLEF 2019
Information retrieval (European conference)

Authors: Ferro, Nicola, Fuhr, Norbert, Maistro, Maria, Sakai, Tetsuya, Soboroff, Ian

Source: Lecture notes in computer science. 11438:283-290

Availability: http://explore.bl.uk/primo_library/libweb/action/display.do?tabs=detailsTab&gathStatTab=true&ct=display&fn=search&doc=ETOCCN605087782&indx=1&recIds=ETOCCN605087782

On enhancing the robustness of timeline summarization test collections

Authors: McCreadie, Richard, Rajput, Shahzad, Soboroff, Ian, Macdonald, Craig, Ounis, Iadh

Source: In Information Processing and Management September 2019 56(5):1815-1836