Processing of GASKAP-HI pilot survey data using a commercial supercomputer

Bibliographic Details
Title:	Processing of GASKAP-HI pilot survey data using a commercial supercomputer
Authors:	Kemp, Ian P., Pingel, Nickolas M., Worth, Rowan, Wake, Justin, Mitchell, Daniel A., Midgely, Stuart D., Tingay, Steven J., Dempsey, James, Dénes, Helga, Dickey, John M., Gibson, Steven J., Jameson, Kate E., Lynn, Callum, Ma, Yik Ki, Marchal, Antoine, McClure-Griffiths, Naomi M., Stanimirović, Snežana, van Loon, Jacco Th.
Source:	Astronomy and Computing, 2024, 51, 100901
Publication Year:	2024
Collection:	Astrophysics
Subject Terms:	Astrophysics - Instrumentation and Methods for Astrophysics
More Details:	Modern radio telescopes generate large amounts of data, with the next generation Very Large Array (ngVLA) and the Square Kilometre Array (SKA) expected to feed up to 292 GB of visibilities per second to the science data processor (SDP). However, the continued exponential growth in the power of the world's largest supercomputers suggests that for the foreseeable future there will be sufficient capacity available to provide for astronomers' needs in processing 'science ready' products from the new generation of telescopes, with commercial platforms becoming an option for overflow capacity. The purpose of the current work is to trial the use of commercial high performance computing (HPC) for a large scale processing task in astronomy, in this case processing data from the GASKAP-HI pilot surveys. We delineate a four-step process which can be followed by other researchers wishing to port an existing workflow from a public facility to a commercial provider. We used the process to provide reference images for an ongoing upgrade to ASKAPSoft (the ASKAP SDP software), and to provide science images for the GASKAP collaboration, using the joint deconvolution capability of WSClean. We document the approach to optimising the pipeline to minimise cost and elapsed time at the commercial provider, and give a resource estimate for processing future full survey data. Finally we document advantages, disadvantages, and lessons learned from the project, which will aid other researchers aiming to use commercial supercomputing for radio astronomy imaging. We found the key advantage to be immediate access and high availability, and the main disadvantage to be the need for improved HPC knowledge to take best advantage of the facility.
Document Type:	Working Paper
DOI:	10.1016/j.ascom.2024.100901
Access URL:	http://arxiv.org/abs/2411.17118
Accession Number:	edsarx.2411.17118
Database:	arXiv

FullText	Text: Availability: 0 CustomLinks: – Url: http://arxiv.org/abs/2411.17118 Name: EDS - Arxiv Category: fullText Text: View this record from Arxiv MouseOverText: View this record from Arxiv – Url: https://resolver.ebsco.com/c/xy5jbn/result?sid=EBSCO:edsarx&genre=article&issn=&ISBN=&volume=&issue=&date=20241126&spage=&pages=&title=Processing of GASKAP-HI pilot survey data using a commercial supercomputer&atitle=Processing%20of%20GASKAP-HI%20pilot%20survey%20data%20using%20a%20commercial%20supercomputer&aulast=Kemp%2C%20Ian%20P.&id=DOI:10.1016/j.ascom.2024.100901 Name: Full Text Finder (for New FTF UI) (s8985755) Category: fullText Text: Find It @ SCU Libraries MouseOverText: Find It @ SCU Libraries
Header	DbId: edsarx DbLabel: arXiv An: edsarx.2411.17118 RelevancyScore: 1128 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 1128.03332519531
IllustrationInfo
Items	– Name: Title Label: Title Group: Ti Data: Processing of GASKAP-HI pilot survey data using a commercial supercomputer – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Kemp%2C+Ian+P%2E%22">Kemp, Ian P.</searchLink><br /><searchLink fieldCode="AR" term="%22Pingel%2C+Nickolas+M%2E%22">Pingel, Nickolas M.</searchLink><br /><searchLink fieldCode="AR" term="%22Worth%2C+Rowan%22">Worth, Rowan</searchLink><br /><searchLink fieldCode="AR" term="%22Wake%2C+Justin%22">Wake, Justin</searchLink><br /><searchLink fieldCode="AR" term="%22Mitchell%2C+Daniel+A%2E%22">Mitchell, Daniel A.</searchLink><br /><searchLink fieldCode="AR" term="%22Midgely%2C+Stuart+D%2E%22">Midgely, Stuart D.</searchLink><br /><searchLink fieldCode="AR" term="%22Tingay%2C+Steven+J%2E%22">Tingay, Steven J.</searchLink><br /><searchLink fieldCode="AR" term="%22Dempsey%2C+James%22">Dempsey, James</searchLink><br /><searchLink fieldCode="AR" term="%22Dénes%2C+Helga%22">Dénes, Helga</searchLink><br /><searchLink fieldCode="AR" term="%22Dickey%2C+John+M%2E%22">Dickey, John M.</searchLink><br /><searchLink fieldCode="AR" term="%22Gibson%2C+Steven+J%2E%22">Gibson, Steven J.</searchLink><br /><searchLink fieldCode="AR" term="%22Jameson%2C+Kate+E%2E%22">Jameson, Kate E.</searchLink><br /><searchLink fieldCode="AR" term="%22Lynn%2C+Callum%22">Lynn, Callum</searchLink><br /><searchLink fieldCode="AR" term="%22Ma%2C+Yik+Ki%22">Ma, Yik Ki</searchLink><br /><searchLink fieldCode="AR" term="%22Marchal%2C+Antoine%22">Marchal, Antoine</searchLink><br /><searchLink fieldCode="AR" term="%22McClure-Griffiths%2C+Naomi+M%2E%22">McClure-Griffiths, Naomi M.</searchLink><br /><searchLink fieldCode="AR" term="%22Stanimirović%2C+Snežana%22">Stanimirović, Snežana</searchLink><br /><searchLink fieldCode="AR" term="%22van+Loon%2C+Jacco+Th%2E%22">van Loon, Jacco Th.</searchLink> – Name: TitleSource Label: Source Group: Src Data: Astronomy and Computing, 2024, 51, 100901 – Name: DatePubCY Label: Publication Year Group: Date Data: 2024 – Name: Subset Label: Collection Group: HoldingsInfo Data: Astrophysics – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Astrophysics+-+Instrumentation+and+Methods+for+Astrophysics%22">Astrophysics - Instrumentation and Methods for Astrophysics</searchLink> – Name: Abstract Label: Description Group: Ab Data: Modern radio telescopes generate large amounts of data, with the next generation Very Large Array (ngVLA) and the Square Kilometre Array (SKA) expected to feed up to 292 GB of visibilities per second to the science data processor (SDP). However, the continued exponential growth in the power of the world's largest supercomputers suggests that for the foreseeable future there will be sufficient capacity available to provide for astronomers' needs in processing 'science ready' products from the new generation of telescopes, with commercial platforms becoming an option for overflow capacity. The purpose of the current work is to trial the use of commercial high performance computing (HPC) for a large scale processing task in astronomy, in this case processing data from the GASKAP-HI pilot surveys. We delineate a four-step process which can be followed by other researchers wishing to port an existing workflow from a public facility to a commercial provider. We used the process to provide reference images for an ongoing upgrade to ASKAPSoft (the ASKAP SDP software), and to provide science images for the GASKAP collaboration, using the joint deconvolution capability of WSClean. We document the approach to optimising the pipeline to minimise cost and elapsed time at the commercial provider, and give a resource estimate for processing future full survey data. Finally we document advantages, disadvantages, and lessons learned from the project, which will aid other researchers aiming to use commercial supercomputing for radio astronomy imaging. We found the key advantage to be immediate access and high availability, and the main disadvantage to be the need for improved HPC knowledge to take best advantage of the facility. – Name: TypeDocument Label: Document Type Group: TypDoc Data: Working Paper – Name: DOI Label: DOI Group: ID Data: 10.1016/j.ascom.2024.100901 – Name: URL Label: Access URL Group: URL Data: <link linkTarget="URL" linkTerm="http://arxiv.org/abs/2411.17118" linkWindow="_blank">http://arxiv.org/abs/2411.17118</link> – Name: AN Label: Accession Number Group: ID Data: edsarx.2411.17118
PLink	https://login.libproxy.scu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&scope=site&db=edsarx&AN=edsarx.2411.17118
RecordInfo	BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1016/j.ascom.2024.100901 Subjects: – SubjectFull: Astrophysics - Instrumentation and Methods for Astrophysics Type: general Titles: – TitleFull: Processing of GASKAP-HI pilot survey data using a commercial supercomputer Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Kemp, Ian P. – PersonEntity: Name: NameFull: Pingel, Nickolas M. – PersonEntity: Name: NameFull: Worth, Rowan – PersonEntity: Name: NameFull: Wake, Justin – PersonEntity: Name: NameFull: Mitchell, Daniel A. – PersonEntity: Name: NameFull: Midgely, Stuart D. – PersonEntity: Name: NameFull: Tingay, Steven J. – PersonEntity: Name: NameFull: Dempsey, James – PersonEntity: Name: NameFull: Dénes, Helga – PersonEntity: Name: NameFull: Dickey, John M. – PersonEntity: Name: NameFull: Gibson, Steven J. – PersonEntity: Name: NameFull: Jameson, Kate E. – PersonEntity: Name: NameFull: Lynn, Callum – PersonEntity: Name: NameFull: Ma, Yik Ki – PersonEntity: Name: NameFull: Marchal, Antoine – PersonEntity: Name: NameFull: McClure-Griffiths, Naomi M. – PersonEntity: Name: NameFull: Stanimirović, Snežana – PersonEntity: Name: NameFull: van Loon, Jacco Th. IsPartOfRelationships: – BibEntity: Dates: – D: 26 M: 11 Type: published Y: 2024
ResultId	1