Publications

You can also find my articles on my Google Scholar profile.

Conference Papers

ISC 25

HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages

Aman Chaturvedi, Daniel Nichols, Siddharth Singh, Abhinav Bhatele
ISC High Performance Conference 2025, 2025

Neurips 24

Loki: Low-Rank Keys for Efficient Sparse Attention

Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele
Advances in Neural Information Processing Systems 37 (NeurIPS), 2024

Neurips 24

Be like a Goldfish, Don’t Memorize! Mitigating Memorization in Generative LLMs

Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, and Tom Goldstein
Advances in Neural Information Processing Systems 37 (NeurIPS), 2024

SC 24

Democratizing AI: Open-Source Scalable LLM Training on GPU-Based Supercomputers

Siddharth Singh, Prajwal Singhania, Aditya Ranjan, John Kirchenbauer, Jonas Geiping, Yuxin Wen, Neel Jain, Abhimanyu Hans, Manli Shu, Aditya Tomar, Tom Goldstein, and Abhinav Bhatele
SC'24: Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, 2024

ICS 23

A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training

Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhinav Bhatele
ICS '23: Proceedings of the 37th International Conference on Supercomputing, 2023

IPDPS 23

Exploiting sparsity in pruned neural networks to optimize large model training

Siddharth Singh and Abhinav Bhatele
2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2023

IPDPS 22

AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning

Siddharth Singh and Abhinav Bhatele
2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2022