Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening
| Title: | Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening |
|---|---|
| Authors: | Pereira-Sánchez, Ivan, Sans, Eloi, Navarro, Julia, Duran, Joan |
| Publication Year: | 2024 |
| Collection: | Computer Science |
| Subject Terms: | Electrical Engineering and Systems Science - Image and Video Processing, Computer Science - Computer Vision and Pattern Recognition |
| More Details: | The objective of pansharpening and hypersharpening is to accurately combine a high-resolution panchromatic (PAN) image with a low-resolution multispectral (MS) or hyperspectral (HS) image, respectively. Unfolding fusion methods integrate the powerful representation capabilities of deep learning with the robustness of model-based approaches. These techniques involve unrolling the steps of the optimization scheme derived from the minimization of an energy into a deep learning framework, resulting in efficient and highly interpretable architectures. In this paper, we propose a model-based deep unfolded method for satellite image fusion. Our approach is based on a variational formulation that incorporates the classic observation model for MS/HS data, a high-frequency injection constraint based on the PAN image, and an arbitrary convex prior. For the unfolding stage, we introduce upsampling and downsampling layers that use geometric information encoded in the PAN image through residual networks. The backbone of our method is a multi-head attention residual network (MARNet), which replaces the proximity operator in the optimization scheme and combines multiple head attentions with residual learning to exploit image self-similarities via nonlocal operators defined in terms of patches. Additionally, we incorporate a post-processing module based on the MARNet architecture to further enhance the quality of the fused images. Experimental results on PRISMA, Quickbird, and WorldView2 datasets demonstrate the superior performance of our method and its ability to generalize across different sensor configurations and varying spatial and spectral resolutions. The source code will be available at https://github.com/TAMI-UIB/MARNet. |
| Document Type: | Working Paper |
| Access URL: | http://arxiv.org/abs/2409.02675 |
| Accession Number: | edsarx.2409.02675 |
| Database: | arXiv |
| FullText | Text: Availability: 0 CustomLinks: – Url: http://arxiv.org/abs/2409.02675 Name: EDS - Arxiv Category: fullText Text: View this record from Arxiv MouseOverText: View this record from Arxiv – Url: https://resolver.ebsco.com/c/xy5jbn/result?sid=EBSCO:edsarx&genre=article&issn=&ISBN=&volume=&issue=&date=20240904&spage=&pages=&title=Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening&atitle=Multi-Head%20Attention%20Residual%20Unfolded%20Network%20for%20Model-Based%20Pansharpening&aulast=Pereira-S%C3%A1nchez%2C%20Ivan&id=DOI: Name: Full Text Finder (for New FTF UI) (s8985755) Category: fullText Text: Find It @ SCU Libraries MouseOverText: Find It @ SCU Libraries |
|---|---|
| Header | DbId: edsarx DbLabel: arXiv An: edsarx.2409.02675 RelevancyScore: 1112 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 1112.25573730469 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Pereira-Sánchez%2C+Ivan%22">Pereira-Sánchez, Ivan</searchLink><br /><searchLink fieldCode="AR" term="%22Sans%2C+Eloi%22">Sans, Eloi</searchLink><br /><searchLink fieldCode="AR" term="%22Navarro%2C+Julia%22">Navarro, Julia</searchLink><br /><searchLink fieldCode="AR" term="%22Duran%2C+Joan%22">Duran, Joan</searchLink> – Name: DatePubCY Label: Publication Year Group: Date Data: 2024 – Name: Subset Label: Collection Group: HoldingsInfo Data: Computer Science – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Electrical+Engineering+and+Systems+Science+-+Image+and+Video+Processing%22">Electrical Engineering and Systems Science - Image and Video Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Science+-+Computer+Vision+and+Pattern+Recognition%22">Computer Science - Computer Vision and Pattern Recognition</searchLink> – Name: Abstract Label: Description Group: Ab Data: The objective of pansharpening and hypersharpening is to accurately combine a high-resolution panchromatic (PAN) image with a low-resolution multispectral (MS) or hyperspectral (HS) image, respectively. Unfolding fusion methods integrate the powerful representation capabilities of deep learning with the robustness of model-based approaches. These techniques involve unrolling the steps of the optimization scheme derived from the minimization of an energy into a deep learning framework, resulting in efficient and highly interpretable architectures. In this paper, we propose a model-based deep unfolded method for satellite image fusion. Our approach is based on a variational formulation that incorporates the classic observation model for MS/HS data, a high-frequency injection constraint based on the PAN image, and an arbitrary convex prior. For the unfolding stage, we introduce upsampling and downsampling layers that use geometric information encoded in the PAN image through residual networks. The backbone of our method is a multi-head attention residual network (MARNet), which replaces the proximity operator in the optimization scheme and combines multiple head attentions with residual learning to exploit image self-similarities via nonlocal operators defined in terms of patches. Additionally, we incorporate a post-processing module based on the MARNet architecture to further enhance the quality of the fused images. Experimental results on PRISMA, Quickbird, and WorldView2 datasets demonstrate the superior performance of our method and its ability to generalize across different sensor configurations and varying spatial and spectral resolutions. The source code will be available at https://github.com/TAMI-UIB/MARNet. – Name: TypeDocument Label: Document Type Group: TypDoc Data: Working Paper – Name: URL Label: Access URL Group: URL Data: <link linkTarget="URL" linkTerm="http://arxiv.org/abs/2409.02675" linkWindow="_blank">http://arxiv.org/abs/2409.02675</link> – Name: AN Label: Accession Number Group: ID Data: edsarx.2409.02675 |
| PLink | https://login.libproxy.scu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&scope=site&db=edsarx&AN=edsarx.2409.02675 |
| RecordInfo | BibRecord: BibEntity: Subjects: – SubjectFull: Electrical Engineering and Systems Science - Image and Video Processing Type: general – SubjectFull: Computer Science - Computer Vision and Pattern Recognition Type: general Titles: – TitleFull: Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Pereira-Sánchez, Ivan – PersonEntity: Name: NameFull: Sans, Eloi – PersonEntity: Name: NameFull: Navarro, Julia – PersonEntity: Name: NameFull: Duran, Joan IsPartOfRelationships: – BibEntity: Dates: – D: 04 M: 09 Type: published Y: 2024 |
| ResultId | 1 |