Tensor Parallelism by Hand: The Matrix Math Most Explainers Skip

Every explainer says the same sentence: “a tensor is split into N chunks along a dimension so each device holds 1/N of it.” That sentence…