Problem
Probability-density estimation asks us to reconstruct an unknown distribution from a finite sample. It is an unsupervised problem: the observations are available, but there is no target density attached to each point. Classical estimators make the statistical assumptions explicit, while neural networks provide flexible function approximation but still need a meaningful training signal.
Gaussian-Mixture Neural Networks explores a simple way to connect these two views. A Gaussian mixture model fitted by maximum likelihood provides an explicit density estimate. A neural network then learns from values produced by that estimator, carrying the statistical target into a model that can generalize across the input space.
The paper was written with Edmondo Trentin and published at ANNPR 2024 in Lecture Notes in Computer Science volume 15154.
Approach
The method begins by fitting a Gaussian mixture to the unlabeled sample. We use the resulting maximum-likelihood estimate to generate target density values for the neural component. In this sense, the GMM acts as a statistical teacher: it supplies a smooth target without requiring labelled examples from the original problem.
The neural estimator is implemented as a multilayer perceptron. Its architecture and training parameters can be varied independently from the mixture used to generate the targets. The repository includes experiments with different probability distributions, sample sizes, numbers of Gaussian components, and network configurations. It also contains comparison code for Parzen windows, nearest-neighbour estimators, standard GMMs, and Parzen Neural Networks.
Hyperparameter searches were carried out with Optuna. Keeping these searches and their stored results in the repository made it possible to compare model families under consistent experimental settings rather than selecting a single favourable run.
Evaluation
The experiments cover both univariate and multivariate synthetic densities, where the true distribution is known and the quality of an estimate can be inspected directly. The proposed model is compared with parametric, non-parametric, and neural alternatives. The reported results show that the learned estimator can improve on the GMM targets used during training and compare favourably with the established neural baseline considered in the study.
What interested me most in this project was the boundary between estimation and supervision. The network is trained with targets, but those targets are constructed from an unlabeled sample by a statistical procedure. This makes the method a useful example of how prior modelling choices can be made visible inside a learning pipeline instead of being hidden behind a large end-to-end architecture.
The repository contains the training scripts, statistical baselines, Optuna studies, and visualizations used throughout the project. The article appears on pages 13-24 of the ANNPR 2024 proceedings.