No. Distillation trains on a teach model's logits or output tokens.

How do you have teacher model output anything without it being output tokens (or embeddings / intermediate logits / activations)?