naive weight sharing: training instability, performance degradation; weight transformation to increase diversity; distillation (prediction-logits, self-attention, hidden states) to improve performance

compressed models (Mini-DeiT, Mini-Swin) achieved competitive or better performance than original models DeiT, Swin; good generalization ability for downstream tasks (classification, detection)