PhD defense Zhu Liao: Layer Removability in Deep Neural Networks
Télécom Paris, 19 place Marguerite Perey F-91120 Palaiseau [getting there], amphi 3 and in videoconferencing
Full title: Layer Removability in Deep Neural Networks: Inducing, Identifying, and Training for Depth Reduction
Jury
- Van Tâm NGUYEN, Professor, Télécom Paris, Institut Polytechnique de Paris, France (PhD Supervisor)
- Enzo TARTAGLIONE, Professor, Télécom Paris, Institut Polytechnique de Paris, France (PhD Co-Supervisor)
- José Antonio ORAMAS MOGROVEJO, Associate Professor, Universiteit Antwerpen, Belgium (Reviewer)
- Giovanna Paola VARNI, Associate Professor, Università di Trento, Italy (Reviewer)
- Attilio FIANDROTTI, Associate Professor, Università di Torino, Italy (Examiner)
- Anissa MOKRAOUI, Professor, Université Sorbonne Paris Nord, France (Examiner)
Abstract
Modern deep neural networks are becoming increasingly large. These large models have achieved strong performance in many fields. However, their computational cost also grows, much of it arising at inference time. Larger models are typically both wider and deeper, and depth brings a specific cost: since layers are executed one after another, a deeper network needs more sequential steps for every forward pass, and this time cannot be reduced by intra-layer parallelism.
Among different model compression methods, pruning is a popular approach to reducing computational cost. Classical unstructured pruning reduces the number of parameters by setting unimportant weights to zero. However, when individual weights are removed while the neural network’s depth stays unchanged, the model still executes the same sequence of layers, so the inference time may only be reduced moderately. Therefore, this thesis argues that model compression should focus not only on the number of parameters but also on the length of the forward path in the network.
First, we show that pruning can be guided towards having a « collapse » when the pruning budget is allocated according to a suitable layer-level signal. Methods EGP and NEPENTHE push the pruning budget toward layers with low activation entropy (i.e., layers where the activation variation across the dataset is relatively small), producing networks from which entire layers can be removed with little performance loss.
Second, for networks that are already trained, methods EASIER and TLC estimate how much each layer contributes to the task and remove the layers with the smallest contribution while keeping the performance drop within a fixed budget.
Finally, LaCoOT and SHINING make removability part of the training process, so that removable layers already exist when training ends.
Taken together, these three stages show that the depth of a network is not fixed once the architecture is chosen: layer removability can be induced by pruning, identified in trained models, and prepared during training.