Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compression

Authors: Feng, Hao, Zhang, Boyuan, Ye, Fanjiang, Si, Min, Chu, Ching-Hsiang, Tian, Jiannan, Yin, Chunxing, Deng, Summer, Hao, Yuchen, Balaji, Pavan, Geng, Tong, Tao, Dingwen

Source: SC24: International Conference for High Performance Computing, Networking, Storage and Analysis SC High Performance Computing, Networking, Storage and Analysis, SC24: International Conference for. :1-16 Nov, 2024

Relation: SC24: International Conference for High Performance Computing, Networking, Storage and Analysis

The Llama 3 Herd of Models

Authors: Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, Kadian, Abhishek, Al-Dahle, Ahmad, Letman, Aiesha, Mathur, Akhil, Schelten, Alan, Vaughan, Alex, Yang, Amy, Fan, Angela, Goyal, Anirudh, Hartshorn, Anthony, Yang, Aobo, Mitra, Archi, Sravankumar, Archie, Korenev, Artem, Hinsvark, Arthur, Rao, Arun, Zhang, Aston, Rodriguez, Aurelien, Gregerson, Austen, Spataru, Ava, Roziere, Baptiste, Biron, Bethany, Tang, Binh, Chern, Bobbie, Caucheteux, Charlotte, Nayak, Chaya, Bi, Chloe, Marra, Chris, McConnell, Chris, Keller, Christian, Touret, Christophe, Wu, Chunyang, Wong, Corinne, Ferrer, Cristian Canton, Nikolaidis, Cyrus, Allonsius, Damien, Song, Daniel, Pintz, Danielle, Livshits, Danny, Wyatt, Danny, Esiobu, David, Choudhary, Dhruv, Mahajan, Dhruv, Garcia-Olano, Diego, Perino, Diego, Hupkes, Dieuwke, Lakomkin, Egor, AlBadawy, Ehab, Lobanova, Elina, Dinan, Emily, Smith, Eric Michael, Radenovic, Filip, Guzmán, Francisco, Zhang, Frank, Synnaeve, Gabriel, Lee, Gabrielle, Anderson, Georgia Lewis, Thattai, Govind, Nail, Graeme, Mialon, Gregoire, Pang, Guan, Cucurell, Guillem, Nguyen, Hailey, Korevaar, Hannah, Xu, Hu, Touvron, Hugo, Zarov, Iliyan, Ibarra, Imanol Arrieta, Kloumann, Isabel, Misra, Ishan, Evtimov, Ivan, Zhang, Jack, Copet, Jade, Lee, Jaewon, Geffert, Jan, Vranes, Jana, Park, Jason, Mahadeokar, Jay, Shah, Jeet, van der Linde, Jelmer, Billock, Jennifer, Hong, Jenny, Lee, Jenya, Fu, Jeremy, Chi, Jianfeng, Huang, Jianyu, Liu, Jiawen, Wang, Jie, Yu, Jiecao, Bitton, Joanna, Spisak, Joe, Park, Jongsoo, Rocca, Joseph, Johnstun, Joshua, Saxe, Joshua, Jia, Junteng, Alwala, Kalyan Vasuden, Prasad, Karthik, Upasani, Kartikeya, Plawiak, Kate, Li, Ke, Heafield, Kenneth, Stone, Kevin, El-Arini, Khalid, Iyer, Krithika, Malik, Kshitiz, Chiu, Kuenley, Bhalla, Kunal, Lakhotia, Kushal, Rantala-Yeary, Lauren, van der Maaten, Laurens, Chen, Lawrence, Tan, Liang, Jenkins, Liz, Martin, Louis, Madaan, Lovish, Malo, Lubo, Blecher, Lukas, Landzaat, Lukas, de Oliveira, Luke, Muzzi, Madeline, Pasupuleti, Mahesh, Singh, Mannat, Paluri, Manohar, Kardas, Marcin, Tsimpoukelli, Maria, Oldham, Mathew, Rita, Mathieu, Pavlova, Maya, Kambadur, Melanie, Lewis, Mike, Si, Min, Singh, Mitesh Kumar, Hassan, Mona, Goyal, Naman, Torabi, Narjes, Bashlykov, Nikolay, Bogoychev, Nikolay, Chatterji, Niladri, Zhang, Ning, Duchenne, Olivier, Çelebi, Onur, Alrassy, Patrick, Zhang, Pengchuan, Li, Pengwei, Vasic, Petar, Weng, Peter, Bhargava, Prajjwal, Dubal, Pratik, Krishnan, Praveen, Koura, Punit Singh, Xu, Puxin, He, Qing, Dong, Qingxiao, Srinivasan, Ragavan, Ganapathy, Raj, Calderer, Ramon, Cabral, Ricardo Silveira, Stojnic, Robert, Raileanu, Roberta, Maheswari, Rohan, Girdhar, Rohit, Patel, Rohit, Sauvestre, Romain, Polidoro, Ronnie, Sumbaly, Roshan, Taylor, Ross, Silva, Ruan, Hou, Rui, Wang, Rui, Hosseini, Saghar, Chennabasappa, Sahana, Singh, Sanjay, Bell, Sean, Kim, Seohyun Sonia, Edunov, Sergey, Nie, Shaoliang, Narang, Sharan, Raparthy, Sharath, Shen, Sheng, Wan, Shengye, Bhosale, Shruti, Zhang, Shun, Vandenhende, Simon, Batra, Soumya, Whitman, Spencer, Sootla, Sten, Collot, Stephane, Gururangan, Suchin, Borodinsky, Sydney, Herman, Tamar, Fowler, Tara, Sheasha, Tarek, Georgiou, Thomas, Scialom, Thomas, Speckbacher, Tobias, Mihaylov, Todor, Xiao, Tong, Karn, Ujjwal, Goswami, Vedanuj, Gupta, Vibhor, Ramanathan, Vignesh, Kerkez, Viktor, Gonguet, Vincent, Do, Virginie, Vogeti, Vish, Albiero, Vítor, Petrovic, Vladan, Chu, Weiwei, Xiong, Wenhan, Fu, Wenyin, Meers, Whitney, Martinet, Xavier, Wang, Xiaodong, Wang, Xiaofang, Tan, Xiaoqing Ellen, Xia, Xide, Xie, Xinfeng, Jia, Xuchao, Wang, Xuewei, Goldschlag, Yaelle, Gaur, Yashesh, Babaei, Yasmine, Wen, Yi, Song, Yiwen, Zhang, Yuchen, Li, Yue, Mao, Yuning, Coudert, Zacharie Delpierre, Yan, Zheng, Chen, Zhengxing, Papakipos, Zoe, Singh, Aaditya, Srivastava, Aayushi, Jain, Abha, Kelsey, Adam, Shajnfeld, Adam, Gangidi, Adithya, Victoria, Adolfo, Goldstand, Ahuva, Menon, Ajay, Sharma, Ajay, Boesenberg, Alex, Baevski, Alexei, Feinstein, Allie, Kallet, Amanda, Sangani, Amit, Teo, Amos, Yunus, Anam, Lupu, Andrei, Alvarado, Andres, Caples, Andrew, Gu, Andrew, Ho, Andrew, Poulton, Andrew, Ryan, Andrew, Ramchandani, Ankit, Dong, Annie, Franco, Annie, Goyal, Anuj, Saraf, Aparajita, Chowdhury, Arkabandhu, Gabriel, Ashley, Bharambe, Ashwin, Eisenman, Assaf, Yazdan, Azadeh, James, Beau, Maurer, Ben, Leonhardi, Benjamin, Huang, Bernie, Loyd, Beth, De Paola, Beto, Paranjape, Bhargavi, Liu, Bing, Wu, Bo, Ni, Boyu, Hancock, Braden, Wasti, Bram, Spence, Brandon, Stojkovic, Brani, Gamido, Brian, Montalvo, Britt, Parker, Carl, Burton, Carly, Mejia, Catalina, Liu, Ce, Wang, Changhan, Kim, Changkyu, Zhou, Chao, Hu, Chester, Chu, Ching-Hsiang, Cai, Chris, Tindal, Chris, Feichtenhofer, Christoph, Gao, Cynthia, Civin, Damon, Beaty, Dana, Kreymer, Daniel, Li, Daniel, Adkins, David, Xu, David, Testuggine, Davide, David, Delia, Parikh, Devi, Liskovich, Diana, Foss, Didem, Wang, Dingkang, Le, Duc, Holland, Dustin, Dowling, Edward, Jamil, Eissa, Montgomery, Elaine, Presani, Eleonora, Hahn, Emily, Wood, Emily, Le, Eric-Tuan, Brinkman, Erik, Arcaute, Esteban, Dunbar, Evan, Smothers, Evan, Sun, Fei, Kreuk, Felix, Tian, Feng, Kokkinos, Filippos, Ozgenel, Firat, Caggioni, Francesco, Kanayet, Frank, Seide, Frank, Florez, Gabriela Medina, Schwarz, Gabriella, Badeer, Gada, Swee, Georgia, Halpern, Gil, Herman, Grant, Sizov, Grigory, Guangyi, Zhang, Lakshminarayanan, Guna, Inan, Hakan, Shojanazeri, Hamid, Zou, Han, Wang, Hannah, Zha, Hanwen, Habeeb, Haroun, Rudolph, Harrison, Suk, Helen, Aspegren, Henry, Goldman, Hunter, Zhan, Hongyuan, Damlaj, Ibrahim, Molybog, Igor, Tufanov, Igor, Leontiadis, Ilias, Veliche, Irina-Elena, Gat, Itai, Weissman, Jake, Geboski, James, Kohli, James, Lam, Janice, Asher, Japhet, Gaya, Jean-Baptiste, Marcus, Jeff, Tang, Jeff, Chan, Jennifer, Zhen, Jenny, Reizenstein, Jeremy, Teboul, Jeremy, Zhong, Jessica, Jin, Jian, Yang, Jingyi, Cummings, Joe, Carvill, Jon, Shepard, Jon, McPhie, Jonathan, Torres, Jonathan, Ginsburg, Josh, Wang, Junjie, Wu, Kai, U, Kam Hou, Saxena, Karan, Khandelwal, Kartikay, Zand, Katayoun, Matosich, Kathy, Veeraraghavan, Kaushik, Michelena, Kelly, Li, Keqian, Jagadeesh, Kiran, Huang, Kun, Chawla, Kunal, Huang, Kyle, Chen, Lailin, Garg, Lakshya, A, Lavender, Silva, Leandro, Bell, Lee, Zhang, Lei, Guo, Liangpeng, Yu, Licheng, Moshkovich, Liron, Wehrstedt, Luca, Khabsa, Madian, Avalani, Manav, Bhatt, Manish, Mankus, Martynas, Hasson, Matan, Lennie, Matthew, Reso, Matthias, Groshev, Maxim, Naumov, Maxim, Lathi, Maya, Keneally, Meghan, Liu, Miao, Seltzer, Michael L., Valko, Michal, Restrepo, Michelle, Patel, Mihir, Vyatskov, Mik, Samvelyan, Mikayel, Clark, Mike, Macey, Mike, Wang, Mike, Hermoso, Miquel Jubert, Metanat, Mo, Rastegari, Mohammad, Bansal, Munish, Santhanam, Nandhini, Parks, Natascha, White, Natasha, Bawa, Navyata, Singhal, Nayan, Egebo, Nick, Usunier, Nicolas, Mehta, Nikhil, Laptev, Nikolay Pavlovich, Dong, Ning, Cheng, Norman, Chernoguz, Oleg, Hart, Olivia, Salpekar, Omkar, Kalinli, Ozlem, Kent, Parkin, Parekh, Parth, Saab, Paul, Balaji, Pavan, Rittner, Pedro, Bontrager, Philip, Roux, Pierre, Dollar, Piotr, Zvyagina, Polina, Ratanchandani, Prashant, Yuvraj, Pritish, Liang, Qian, Alao, Rachad, Rodriguez, Rachel, Ayub, Rafi, Murthy, Raghotham, Nayani, Raghu, Mitra, Rahul, Parthasarathy, Rangaprabhu, Li, Raymond, Hogan, Rebekkah, Battey, Robin, Wang, Rocky, Howes, Russ, Rinott, Ruty, Mehta, Sachin, Siby, Sachin, Bondu, Sai Jayesh, Datta, Samyak, Chugh, Sara, Hunt, Sara, Dhillon, Sargun, Sidorov, Sasha, Pan, Satadru, Mahajan, Saurabh, Verma, Saurabh, Yamamoto, Seiji, Ramaswamy, Sharadh, Lindsay, Shaun, Feng, Sheng, Lin, Shenghao, Zha, Shengxin Cindy, Patil, Shishir, Shankar, Shiva, Zhang, Shuqiang, Wang, Sinong, Agarwal, Sneha, Sajuyigbe, Soji, Chintala, Soumith, Max, Stephanie, Chen, Stephen, Kehoe, Steve, Satterfield, Steve, Govindaprasad, Sudarshan, Gupta, Sumit, Deng, Summer, Cho, Sungmin, Virk, Sunny, Subramanian, Suraj, Choudhury, Sy, Goldman, Sydney, Remez, Tal, Glaser, Tamar, Best, Tamara, Koehler, Thilo, Robinson, Thomas, Li, Tianhe, Zhang, Tianjun, Matthews, Tim, Chou, Timothy, Shaked, Tzook, Vontimitta, Varun, Ajayi, Victoria, Montanez, Victoria, Mohan, Vijai, Kumar, Vinay Satish, Mangla, Vishal, Ionescu, Vlad, Poenaru, Vlad, Mihailescu, Vlad Tiberiu, Ivanov, Vladimir, Li, Wei, Wang, Wenchen, Jiang, Wenwen, Bouaziz, Wes, Constable, Will, Tang, Xiaocheng, Wu, Xiaojian, Wang, Xiaolan, Wu, Xilun, Gao, Xinbo, Kleinman, Yaniv, Chen, Yanjun, Hu, Ye, Jia, Ye, Qi, Ye, Li, Yenda, Zhang, Yilin, Zhang, Ying, Adi, Yossi, Nam, Youngjin, Yu, Wang, Zhao, Yu, Hao, Yuchen, Qian, Yundi, Li, Yunlu, He, Yuzi, Rait, Zach, DeVito, Zachary, Rosnbrick, Zef, Wen, Zhaoduo, Yang, Zhenyu, Zhao, Zhiwei, Ma, Zhiyu

Subject Terms: Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Computer Vision and Pattern Recognition

Access URL: http://arxiv.org/abs/2407.21783

Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation

Authors: Luo, Liang, Zhang, Buyun, Tsang, Michael, Ma, Yinbin, Chu, Ching-Hsiang, Chen, Yuxin, Li, Shen, Hao, Yuchen, Zhao, Yanli, Lakshminarayanan, Guna, Wen, Ellie Dingqiao, Park, Jongsoo, Mudigere, Dheevatsa, Naumov, Maxim

Subject Terms: Computer Science - Machine Learning, Computer Science - Distributed, Parallel, and Cluster Computing, Computer Science - Information Retrieval

Access URL: http://arxiv.org/abs/2403.00877

Adaptive and Hierarchical Large Message All-to-all Communication Algorithms for Large-scale Dense GPU Systems

Authors: Khorassani, Kawthar Shafie, Chu, Ching-Hsiang, Anthony, Quentin G., Subramoni, Hari, Panda, Dhabaleswar K.

Source: 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid) CCGRID Cluster, Cloud and Internet Computing (CCGrid), 2021 IEEE/ACM 21st International Symposium on. :113-122 May, 2021

Relation: 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid)

Dynamic Kernel Fusion for Bulk Non-contiguous Data Transfer on GPU Clusters

Authors: Chu, Ching-Hsiang, Khorassani, Kawthar Shafie, Zhou, Qinghua, Subramoni, Hari, Panda, Dhabaleswar K.

Source: 2020 IEEE International Conference on Cluster Computing (CLUSTER) CLUSTER Cluster Computing (CLUSTER), 2020 IEEE International Conference on. :130-141 Sep, 2020

Relation: 2020 IEEE International Conference on Cluster Computing (CLUSTER)

Designing a Profiling and Visualization Tool for Scalable and In-depth Analysis of High-Performance GPU Clusters

Authors: Kousha, Pouya, Ramesh, Bharath, Kandadi Suresh, Kaushik, Chu, Ching-Hsiang, Jain, Arpan, Sarkauskas, Nick, Subramoni, Hari, Panda, Dhabaleswar K.

Source: 2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC) HIPC High Performance Computing, Data, and Analytics (HiPC), 2019 IEEE 26th International Conference on. :93-102 Dec, 2019

Relation: 2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC)

High-Performance Adaptive MPI Derived Datatype Communication for Modern Multi-GPU Systems

Authors: Chu, Ching-Hsiang, Hashmi, Jahanzeb Maqbool, Khorassani, Kawthar Shafie, Subramoni, Hari, Panda, Dhabaleswar K.

Source: 2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC) HIPC High Performance Computing, Data, and Analytics (HiPC), 2019 IEEE 26th International Conference on. :267-276 Dec, 2019

Relation: 2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC)

Communication Profiling and Characterization of Deep Learning Workloads on Clusters with High-Performance Interconnects

Authors: Awan, Ammar Ahmad, Jain, Arpan, Chu, Ching-Hsiang, Subramoni, Hari, Panda, Dhableswar K.

Source: 2019 IEEE Symposium on High-Performance Interconnects (HOTI) High-Performance Interconnects (HOTI), 2019 IEEE Symposium on. :49-53 Aug, 2019

Relation: 2019 IEEE Symposium on High-Performance Interconnects (HOTI)

C-GDR: High-Performance Container-Aware GPUDirect MPI Communication Schemes on RDMA Networks

Authors: Zhang, Jie, Lu, Xiaoyi, Chu, Ching-Hsiang, Panda, Dhabaleswar K.

Source: 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) Parallel and Distributed Processing Symposium (IPDPS), 2019 IEEE International. :242-251 May, 2019

Relation: 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation

Authors: Awan, Ammar Ahmad, Bedorf, Jereon, Chu, Ching-Hsiang, Subramoni, Hari, Panda, Dhabaleswar K.

Source: 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID) Cluster, Cloud and Grid Computing (CCGRID), 2019 19th IEEE/ACM International Symposium on. :498-507 May, 2019

Relation: 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

OC-DNN: Exploiting Advanced Unified Memory Capabilities in CUDA 9 and Volta GPUs for Out-of-Core DNN Training

Authors: Awan, Ammar Ahmad, Chu, Ching-Hsiang, Subramoni, Hari, Lu, Xiaoyi, Panda, Dhabaleswar K.

Source: 2018 IEEE 25th International Conference on High Performance Computing (HiPC) HIPC High Performance Computing (HiPC), 2018 IEEE 25th International Conference on. :143-152 Dec, 2018

Relation: 2018 IEEE 25th International Conference on High Performance Computing (HiPC)

Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models

Authors: Mudigere, Dheevatsa, Hao, Yuchen, Huang, Jianyu, Jia, Zhihao, Tulloch, Andrew, Sridharan, Srinivas, Liu, Xing, Ozdal, Mustafa, Nie, Jade, Park, Jongsoo, Luo, Liang, Yang, Jie Amy, Gao, Leon, Ivchenko, Dmytro, Basant, Aarti, Hu, Yuxi, Yang, Jiyan, Ardestani, Ehsan K., Wang, Xiaodong, Komuravelli, Rakesh, Chu, Ching-Hsiang, Yilmaz, Serhat, Li, Huayu, Qian, Jiyuan, Feng, Zhuobo, Ma, Yinbin, Yang, Junjie, Wen, Ellie, Li, Hong, Yang, Lin, Sun, Chonglin, Zhao, Whitney, Melts, Dimitry, Dhulipala, Krishna, Kishore, KR, Graf, Tyler, Eisenman, Assaf, Matam, Kiran Kumar, Gangidi, Adi, Chen, Guoqiang Jerry, Krishnan, Manoj, Nayak, Avinash, Nair, Krishnakumar, Muthiah, Bharath, khorashadi, Mahmoud, Bhattacharya, Pallab, Lapukhov, Petr, Naumov, Maxim, Mathews, Ajit, Qiao, Lin, Smelyanskiy, Mikhail, Jia, Bill, Rao, Vijay

Subject Terms: Computer Science - Distributed, Parallel, and Cluster Computing, Computer Science - Artificial Intelligence, Computer Science - Machine Learning, Computer Science - Performance

Access URL: http://arxiv.org/abs/2104.05158

Efficient and Scalable Multi-Source Streaming Broadcast on GPU Clusters for Deep Learning

Authors: Chu, Ching-Hsiang, Lu, Xiaoyi, Awan, Ammar A., Subramoni, Hari, Hashmi, Jahanzeb, Elton, Bracy, Panda, Dhabaleswar K.

Source: 2017 46th International Conference on Parallel Processing (ICPP) ICPP Parallel Processing (ICPP), 2017 46th International Conference on. :161-170 Aug, 2017

Relation: 2017 46th International Conference on Parallel Processing (ICPP)

MPI-GDS: High Performance MPI Designs with GPUDirect-aSync for CPU-GPU Control Flow Decoupling

Authors: Venkatesh, Akshay, Hamidouche, Khaled, Potluri, Sreeram, Rosetti, Davide, Chu, Ching-Hsiang, Panda, Dhabaleswar K.

Source: 2017 46th International Conference on Parallel Processing (ICPP) ICPP Parallel Processing (ICPP), 2017 46th International Conference on. :151-160 Aug, 2017

Relation: 2017 46th International Conference on Parallel Processing (ICPP)

CUDA Kernel Based Collective Reduction Operations on Large-scale GPU Clusters

Authors: Chu, Ching-Hsiang, Hamidouche, Khaled, Venkatesh, Akshay, Awan, Ammar Ahmad, Panda, Dhabaleswar K.

Source: 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) Cluster, Cloud and Grid Computing (CCGrid), 2016 16th IEEE/ACM International Symposium on. :726-735 May, 2016

Relation: 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid)

Exploiting GPUDirect RDMA in Designing High Performance OpenSHMEM for NVIDIA GPU Clusters

Authors: Hamidouche, Khaled, Venkatesh, Akshay, Awan, Ammar Ahmad, Subramoni, Hari, Chu, Ching-Hsiang, Panda, Dhabaleswar K.

Source: 2015 IEEE International Conference on Cluster Computing Cluster Computing (CLUSTER), 2015 IEEE International Conference on. :78-87 Sep, 2015

Relation: 2015 IEEE International Conference on Cluster Computing (CLUSTER)

Measurement of long-distance Wi-Fi connections: An empirical study

Authors: Chu, Ching-Hsiang, Chen, You-Ming, Huang, Yu-Te, Carvalho, Roberto, Hsu, Chiun-Chieh, Chen, Ling-Jyh

Source: 2014 IEEE International Conference on Communications (ICC) Communications (ICC), 2014 IEEE International Conference on. :2418-2423 Jun, 2014

Relation: ICC 2014 - 2014 IEEE International Conference on Communications

Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?

Authors: Awan, Ammar Ahmad, Chu, Ching-Hsiang, Subramoni, Hari, Panda, Dhabaleswar K.

Subject Terms: Computer Science - Distributed, Parallel, and Cluster Computing

Access URL: http://arxiv.org/abs/1707.09414

The MVAPICH project: Transforming research into high-performance MPI library for HPC community

Authors: Panda, Dhabaleswar Kumar ^*, Subramoni, Hari, Chu, Ching-Hsiang, Bayatpour, Mohammadreza

Source: In Journal of Computational Science May 2021 52

Optimized large-message broadcast for deep learning workloads: MPI, MPI+NCCL, or NCCL2?

Authors: Awan, Ammar Ahmad, Manian, Karthik Vadambacheri, Chu, Ching-Hsiang, Subramoni, Hari, Panda, Dhabaleswar K.

Source: In Parallel Computing July 2019 85:141-152