This analysis provides practical guidance for choosing pre-trained audio embeddings and highlights the strong robustness of Audio Spectrogram Transformer models when processing shorter audio segments. Building on these observations, we propose MATPAC, a self-supervised method based on Masked Latent Prediction combined with an unsupervised classification objective. This design encourages the model to learn both predictive and semantic structure, leading to representations that outperform previous self-supervised approaches and rival supervised models. We then introduce an improved version, MATPAC++, which addresses the ambiguity of predicting masked content by generating multiple hypotheses and selecting among them with Multiple Choice Learning. This variant achieves state-of-the-art performance on various audio tagging tasks. Finally, we explore how these models can be reused indifferent contexts. We demonstrate that their learned representations can be efficiently compressed into small models without labels, and we show that they can serve as conditioning modules in multimodal systems. In particular, we introduce TinyMU, a compact music language model that leverages MATPAC++ to achieve high-level music reasoning while remaining lightweight and deployable.