open access

https://doi.org/10.14943/doctoral.k16502

https://hdl.handle.net/2115/94994

堅牢な大規模ニューラルネットワーク最適化に関する研究

日本語

2025-03-25

Hokkaido University

xv,149p.

Overfitting in machine learning and deep learning occurs when a model learns not only the underlying patterns in the training data but also the noise, outliers, and specific details that do not generalize to unseen data. This problem often arises when the model is too complex relative to the amount of training data or the variability in the data. When a model has too many parameters or is overly flexible, such as in deep neural networks with many layers and units, it has the capacity to perfectly fit the training data, capturing even the smallest irregularities or random fluctuations. As a result, the model becomes highly specialized to the training set and performs poorly on new data. A primary cause of overfitting is insufficient or unrepresentative training data. If the dataset is too small or lacks diversity, the model encounters only a limited range of examples, which leads it to memorize the data rather than learn generalizable features. In cases of deep learning, where models have vast numbers of parameters, the imbalance between data and model capacity is amplified, making the model prone to overfitting. The lack of variability in the data also contributes to overfitting; if the data is too homogenous or does not capture the full range of real-world variations, the model can latch onto irrelevant patterns specific to the training examples. Additionally, when the data is relatively noise-free, models may become too focused on precise details, learning patterns that are not truly reflective of the broader data distribution. Finally, the tendency of a model to overfit increases if it is trained for too many epochs, leading it to adapt more to the training data than to underlying, generalizable features. In essence, overfitting occurs when the model’s complexity exceeds what the data can support, leading to a failure to generalize beyond the training examples. To combat overfitting, several strategies can be employed. Data augmentation artificially increases the diversity of the training set by applying transformations such as rotations, flips, and color changes to input data, forcing the model to learn more generalized features. Model structure adjustments, such as introducing dropout (randomly deactivating neurons during training) and using batch normalization, help regulate the model’s capacity and stabilize learning. Furthermore, incorporating regularization techniques like L1/L2 penalties constrains the magnitude of model weights, discouraging overly complex solutions. The choice of optimizer also influences overfitting; optimizers like AdamW, which decouples weight decay from gradient updates, provide proper regularization and prevent the model from overfitting by penalizing large weights. Additionally, techniques like learning rate schedules and early stopping halt training at the right moment, before the model begins to overfit. Combining these strategies helps build models that generalize well, achieving a balance between complexity and performance on unseen data. In this thesis, we introduce several novel frameworks from different direction to address the persistent overfitting problem in deep learning models. By incorporating validation loss into the training objective, we enable the model to self-regulate its learning process, a technique we term “meta loss landscape search.” To further enhance generalization, we propose a “meta sample generation/augmentation” method that generates adversarial examples based on the model’s weaknesses, as identified by the testing loss. Finally, we introduce “Adaptive Patching,” a technique tailored for Vision Transformer (ViT) models, which structurally organizes the data within samples to improve generalization. As a summary of the above methods, our research can be summarized as a strategy to solve the problem of model overfitting from three aspects: meta-optimization at the optimizer level, adversarial learning at the data level, and comprehensive optimization of specific models and data.

2025-03-25

甲第16502号

博士(情報科学)

(主査) 教授 棟朝 雅晴, 教授 山本 雅人, 准教授 杉木 章義

情報科学院（情報科学専攻）

博士論文

Version of Record