Nowcasting Future Performance for Population Pruning in Survival-of-the-Fittest Training

概要

Training high-performing neural networks often involves evaluating populations of candidate models with varying hyperparameters, architectures, or random seeds. Training every candidate to completion is computationally expensive, yet stopping seemingly weaker runs early risks discarding models that would ultimately perform better, as learning curves can cross. To address this challenge, we investigate whether intermediate training checkpoints can predict future model accuracy and enable more informed pruning decisions. Our nowcasters predict test accuracy five epochs ahead or at the end of training. They are trained with accuracy and loss histories, weight statistics, and SANE representations as inputs. SANE is a transformer autoencoder that encodes a checkpoint as a sequence of latent tokens. We then use the nowcasters in Survival-of-the-Fittest (SOTF) training. SOTF repeatedly ranks runs during training by predicted accuracy and stops the lowest-ranked ones. Our results show that weight statistics work best for five-epoch forecasts, and SANE representations work best for final accuracy prediction. For evaluation, we compare SOTF with pruning the population by current accuracy. In homogeneous model zoos such as the CNN zoos, pruning by current accuracy already retains a final model only slightly worse than the best. Model zoos with a larger spread in performance, such as the ResNet-18 and ViT zoos, are more difficult to prune by current accuracy alone. In these zoos, SOTF with nowcasted final accuracy almost closes the gap to the best model.

参考リンク

• Paper ↗

リンク先のライセンス・再現性・データ条件をご確認ください。自動分類や実装の目安は保証ではありません。

実装を始めるプロンプト

参考リンクを確認したうえで、次の内容をコピーして利用できます。

# 論文・記事
Nowcasting Future Performance for Population Pruning in Survival-of-the-Fittest Training

# 概要
Training high-performing neural networks often involves evaluating populations of candidate models with varying hyperparameters, architectures, or random seeds. Training every candidate to completion…

# 原典
https://arxiv.org/abs/2611.06512

上記の手法を実装してください。実行前に依存関係とライセンスを確認してください。