> These methods are typically slow because the number of gradient updates scales linearly with the number of learnable parameters in the network, posing a significant challenge for scaling up.

This is a pretty big problem, though if you use information-bottleneck training you can train each layer simultaneously.