Bibliographic Details
| Title: |
Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers |
| Authors: |
Guo, Fang, Li, Wenyu, Zhuang, Honglei, Luo, Yun, Li, Yafu, Zhu, Qi, Yan, Le, Zhang, Yue |
| Publication Year: |
2024 |
| Collection: |
Computer Science |
| Subject Terms: |
Computer Science - Information Retrieval, Computer Science - Artificial Intelligence |
| More Details: |
The most recent pointwise Large Language Model (LLM) rankers have achieved remarkable ranking results. However, these rankers are hindered by two major drawbacks: (1) they fail to follow a standardized comparison guidance during the ranking process, and (2) they struggle with comprehensive considerations when dealing with complicated passages. To address these shortcomings, we propose to build a ranker that generates ranking scores based on a set of criteria from various perspectives. These criteria are intended to direct each perspective in providing a distinct yet synergistic evaluation. Our research, which examines eight datasets from the BEIR benchmark demonstrates that incorporating this multi-perspective criteria ensemble approach markedly enhanced the performance of pointwise LLM rankers. |
| Document Type: |
Working Paper |
| Access URL: |
http://arxiv.org/abs/2404.11960 |
| Accession Number: |
edsarx.2404.11960 |
| Database: |
arXiv |
