SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
| Title: | SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information |
|---|---|
| Authors: | Sun, Jiashuo, Zhang, Jihai, Zhou, Yucheng, Su, Zhaochen, Qu, Xiaoye, Cheng, Yu |
| Publication Year: | 2024 |
| Collection: | Computer Science |
| Subject Terms: | Computer Science - Computer Vision and Pattern Recognition |
| More Details: | Large Vision-Language Models (LVLMs) have become pivotal at the intersection of computer vision and natural language processing. However, the full potential of LVLMs Retrieval-Augmented Generation (RAG) capabilities remains underutilized. Existing works either focus solely on the text modality or are limited to specific tasks. Moreover, most LVLMs struggle to selectively utilize retrieved information and are sensitive to irrelevant or misleading references. To address these challenges, we propose a self-refinement framework designed to teach LVLMs to Selectively Utilize Retrieved Information (SURf). Specifically, when given questions that are incorrectly answered by the LVLM backbone, we obtain references that help correct the answers (positive references) and those that do not (negative references). We then fine-tune the LVLM backbone using a combination of these positive and negative references. Our experiments across three tasks and seven datasets demonstrate that our framework significantly enhances LVLMs ability to effectively utilize retrieved multimodal references and improves their robustness against irrelevant or misleading information. The source code is available at https://github.com/GasolSun36/SURf. Comment: 19 pages, 9 tables, 11 figures |
| Document Type: | Working Paper |
| Access URL: | http://arxiv.org/abs/2409.14083 |
| Accession Number: | edsarx.2409.14083 |
| Database: | arXiv |
| FullText | Text: Availability: 0 CustomLinks: – Url: http://arxiv.org/abs/2409.14083 Name: EDS - Arxiv Category: fullText Text: View this record from Arxiv MouseOverText: View this record from Arxiv – Url: https://resolver.ebsco.com/c/xy5jbn/result?sid=EBSCO:edsarx&genre=article&issn=&ISBN=&volume=&issue=&date=20240921&spage=&pages=&title=SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information&atitle=SURf%3A%20Teaching%20Large%20Vision-Language%20Models%20to%20Selectively%20Utilize%20Retrieved%20Information&aulast=Sun%2C%20Jiashuo&id=DOI: Name: Full Text Finder (for New FTF UI) (s8985755) Category: fullText Text: Find It @ SCU Libraries MouseOverText: Find It @ SCU Libraries |
|---|---|
| Header | DbId: edsarx DbLabel: arXiv An: edsarx.2409.14083 RelevancyScore: 1112 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 1112.25805664063 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Sun%2C+Jiashuo%22">Sun, Jiashuo</searchLink><br /><searchLink fieldCode="AR" term="%22Zhang%2C+Jihai%22">Zhang, Jihai</searchLink><br /><searchLink fieldCode="AR" term="%22Zhou%2C+Yucheng%22">Zhou, Yucheng</searchLink><br /><searchLink fieldCode="AR" term="%22Su%2C+Zhaochen%22">Su, Zhaochen</searchLink><br /><searchLink fieldCode="AR" term="%22Qu%2C+Xiaoye%22">Qu, Xiaoye</searchLink><br /><searchLink fieldCode="AR" term="%22Cheng%2C+Yu%22">Cheng, Yu</searchLink> – Name: DatePubCY Label: Publication Year Group: Date Data: 2024 – Name: Subset Label: Collection Group: HoldingsInfo Data: Computer Science – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Computer+Science+-+Computer+Vision+and+Pattern+Recognition%22">Computer Science - Computer Vision and Pattern Recognition</searchLink> – Name: Abstract Label: Description Group: Ab Data: Large Vision-Language Models (LVLMs) have become pivotal at the intersection of computer vision and natural language processing. However, the full potential of LVLMs Retrieval-Augmented Generation (RAG) capabilities remains underutilized. Existing works either focus solely on the text modality or are limited to specific tasks. Moreover, most LVLMs struggle to selectively utilize retrieved information and are sensitive to irrelevant or misleading references. To address these challenges, we propose a self-refinement framework designed to teach LVLMs to Selectively Utilize Retrieved Information (SURf). Specifically, when given questions that are incorrectly answered by the LVLM backbone, we obtain references that help correct the answers (positive references) and those that do not (negative references). We then fine-tune the LVLM backbone using a combination of these positive and negative references. Our experiments across three tasks and seven datasets demonstrate that our framework significantly enhances LVLMs ability to effectively utilize retrieved multimodal references and improves their robustness against irrelevant or misleading information. The source code is available at https://github.com/GasolSun36/SURf.<br />Comment: 19 pages, 9 tables, 11 figures – Name: TypeDocument Label: Document Type Group: TypDoc Data: Working Paper – Name: URL Label: Access URL Group: URL Data: <link linkTarget="URL" linkTerm="http://arxiv.org/abs/2409.14083" linkWindow="_blank">http://arxiv.org/abs/2409.14083</link> – Name: AN Label: Accession Number Group: ID Data: edsarx.2409.14083 |
| PLink | https://login.libproxy.scu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&scope=site&db=edsarx&AN=edsarx.2409.14083 |
| RecordInfo | BibRecord: BibEntity: Subjects: – SubjectFull: Computer Science - Computer Vision and Pattern Recognition Type: general Titles: – TitleFull: SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Sun, Jiashuo – PersonEntity: Name: NameFull: Zhang, Jihai – PersonEntity: Name: NameFull: Zhou, Yucheng – PersonEntity: Name: NameFull: Su, Zhaochen – PersonEntity: Name: NameFull: Qu, Xiaoye – PersonEntity: Name: NameFull: Cheng, Yu IsPartOfRelationships: – BibEntity: Dates: – D: 21 M: 09 Type: published Y: 2024 |
| ResultId | 1 |