Improving Convergence and Generalization Using Parameter Symmetries
| Title: | Improving Convergence and Generalization Using Parameter Symmetries |
|---|---|
| Authors: | Zhao, Bo, Gower, Robert M., Walters, Robin, Yu, Rose |
| Publication Year: | 2023 |
| Collection: | Computer Science Mathematics |
| Subject Terms: | Computer Science - Machine Learning, Mathematics - Optimization and Control |
| More Details: | In many neural networks, different values of the parameters may result in the same loss value. Parameter space symmetries are loss-invariant transformations that change the model parameters. Teleportation applies such transformations to accelerate optimization. However, the exact mechanism behind this algorithm's success is not well understood. In this paper, we show that teleportation not only speeds up optimization in the short-term, but gives overall faster time to convergence. Additionally, teleporting to minima with different curvatures improves generalization, which suggests a connection between the curvature of the minimum and generalization ability. Finally, we show that integrating teleportation into a wide range of optimization algorithms and optimization-based meta-learning improves convergence. Our results showcase the versatility of teleportation and demonstrate the potential of incorporating symmetry in optimization. Comment: 28 pages, 13 figures, ICLR 2024 |
| Document Type: | Working Paper |
| Access URL: | http://arxiv.org/abs/2305.13404 |
| Accession Number: | edsarx.2305.13404 |
| Database: | arXiv |
| FullText | Text: Availability: 0 CustomLinks: – Url: http://arxiv.org/abs/2305.13404 Name: EDS - Arxiv Category: fullText Text: View this record from Arxiv MouseOverText: View this record from Arxiv – Url: https://resolver.ebsco.com/c/xy5jbn/result?sid=EBSCO:edsarx&genre=article&issn=&ISBN=&volume=&issue=&date=20230522&spage=&pages=&title=Improving Convergence and Generalization Using Parameter Symmetries&atitle=Improving%20Convergence%20and%20Generalization%20Using%20Parameter%20Symmetries&aulast=Zhao%2C%20Bo&id=DOI: Name: Full Text Finder (for New FTF UI) (s8985755) Category: fullText Text: Find It @ SCU Libraries MouseOverText: Find It @ SCU Libraries |
|---|---|
| Header | DbId: edsarx DbLabel: arXiv An: edsarx.2305.13404 RelevancyScore: 1057 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 1057.33166503906 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Improving Convergence and Generalization Using Parameter Symmetries – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Zhao%2C+Bo%22">Zhao, Bo</searchLink><br /><searchLink fieldCode="AR" term="%22Gower%2C+Robert+M%2E%22">Gower, Robert M.</searchLink><br /><searchLink fieldCode="AR" term="%22Walters%2C+Robin%22">Walters, Robin</searchLink><br /><searchLink fieldCode="AR" term="%22Yu%2C+Rose%22">Yu, Rose</searchLink> – Name: DatePubCY Label: Publication Year Group: Date Data: 2023 – Name: Subset Label: Collection Group: HoldingsInfo Data: Computer Science<br />Mathematics – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Computer+Science+-+Machine+Learning%22">Computer Science - Machine Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+-+Optimization+and+Control%22">Mathematics - Optimization and Control</searchLink> – Name: Abstract Label: Description Group: Ab Data: In many neural networks, different values of the parameters may result in the same loss value. Parameter space symmetries are loss-invariant transformations that change the model parameters. Teleportation applies such transformations to accelerate optimization. However, the exact mechanism behind this algorithm's success is not well understood. In this paper, we show that teleportation not only speeds up optimization in the short-term, but gives overall faster time to convergence. Additionally, teleporting to minima with different curvatures improves generalization, which suggests a connection between the curvature of the minimum and generalization ability. Finally, we show that integrating teleportation into a wide range of optimization algorithms and optimization-based meta-learning improves convergence. Our results showcase the versatility of teleportation and demonstrate the potential of incorporating symmetry in optimization.<br />Comment: 28 pages, 13 figures, ICLR 2024 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Working Paper – Name: URL Label: Access URL Group: URL Data: <link linkTarget="URL" linkTerm="http://arxiv.org/abs/2305.13404" linkWindow="_blank">http://arxiv.org/abs/2305.13404</link> – Name: AN Label: Accession Number Group: ID Data: edsarx.2305.13404 |
| PLink | https://login.libproxy.scu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&scope=site&db=edsarx&AN=edsarx.2305.13404 |
| RecordInfo | BibRecord: BibEntity: Subjects: – SubjectFull: Computer Science - Machine Learning Type: general – SubjectFull: Mathematics - Optimization and Control Type: general Titles: – TitleFull: Improving Convergence and Generalization Using Parameter Symmetries Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Zhao, Bo – PersonEntity: Name: NameFull: Gower, Robert M. – PersonEntity: Name: NameFull: Walters, Robin – PersonEntity: Name: NameFull: Yu, Rose IsPartOfRelationships: – BibEntity: Dates: – D: 22 M: 05 Type: published Y: 2023 |
| ResultId | 1 |