Atomic Operations, Warp Shuffle, and the Register Cache: A Deep Dive with Online Softmax

There are multiple posts out there comparing atomic operations and warp shuffle, and for the most part they are right that warp shuffle…