FLASH ATTENTION & PAGED ATTENTION

Picture this.You just rented an NVIDIA A100. It costs three dollars an hour. It has 80 gigabytes of memory and can do nineteen trillion…