平衡自蒸餾（Balanced Self-Distillation, BSD）詳解平衡自蒸餾（BSD）是一種解決長尾分布問題的創(chuàng)新方法，它結(jié)合了自蒸餾技術(shù)和平衡學(xué)習(xí)策略。核心思想是利用模型自身的知識（軟標(biāo)簽）來指導(dǎo)訓(xùn)練，同時通過平衡采樣策略緩解類別不平衡問題。BSD 核心組件1. 平衡采樣器：確保每個批次中各類別樣本比例均衡2. 教師-學(xué)生架構(gòu)：學(xué)生模型從教師模型的軟標(biāo)簽中學(xué)習(xí)3. 軟標(biāo)簽蒸餾：使用教師模型生成的軟標(biāo)簽作為監(jiān)督信號4. 溫度系數(shù)：控制軟標(biāo)簽的"軟化"程度BSD 算法流程importtorchimport torch.nn asnnimport torch.nn.functional asFfrom torch.utils.data import Dataset,DataLoaderimport numpy asnp# 1. 自定義長尾數(shù)據(jù)集class LongTailDataset(Dataset): def __init__(self, num_classes=10, max_samples=1000, imbalance_ratio=100): self.num_classes =num_classes self.samples = [] # 創(chuàng)建長尾分布：樣本數(shù)按指數(shù)衰減 for class_idx in range(num_classes): num_samples= int(max_samples * (imbalance_ratio ** (-class_idx/(num_classes-1))))self.samples.extend([(class_idx, i) for i in range(num_samples)]) def __len__(self): return len(self.samples) def __getitem__(self, idx): class_idx, sample_idx = self.samples[idx] # 生成隨機數(shù)據(jù)作為示例 data= torch.randn(3, 32, 32) * 0.1 + class_idx * 0.3 return data,class_idx# 2. 平衡采樣器class BalancedSampler(torch.utils.data.Sampler): def __init__(self, dataset, batch_size): self.dataset =dataset self.batch_size =batch_size # 按類別組織樣本索引 self.class_indices = {} for idx, (_, label) in enumerate(dataset.samples): if label not in self.class_indices: self.class_indices[label] = [] self.class_indices[label].append(idx)