A Bayesian approach to the maximum likelihood estimation of the parameters \( \omega \) of the linear regression model. More...

#include <bayesian_linear_regression.hpp>

Public Member Functions
	BayesianLinearRegression (const bool centerData=true, const bool scaleData=false, const size_t maxIterations=50, const double tolerance=1e-4)
	Set the parameters of Bayesian Ridge regression object. More...

double	Train (const arma::mat &data, const arma::rowvec &responses)
	Run BayesianLinearRegression. More...

void	Predict (const arma::mat &points, arma::rowvec &predictions) const
	Predict \(y_{i}\) for each data point in the given data matrix using the currently-trained Bayesian Ridge model. More...

void	Predict (const arma::mat &points, arma::rowvec &predictions, arma::rowvec &std) const
	Predict \(y_{i}\) and the standard deviation of the predictive posterior distribution for each data point in the given data matrix, using the currently-trained Bayesian Ridge estimator. More...

double	RMSE (const arma::mat &data, const arma::rowvec &responses) const
	Compute the Root Mean Square Error between the predictions returned by the model and the true responses. More...

const arma::colvec &	Omega () const
	Get the solution vector. More...

double	Alpha () const
	Get the precision (or inverse variance) of the gaussian prior. More...

double	Beta () const
	Get the precision (or inverse variance) beta of the model. More...

double	Variance () const
	Get the estimated variance. More...

const arma::colvec &	DataOffset () const
	Get the mean vector computed on the features over the training points. More...

const arma::colvec &	DataScale () const
	Get the vector of standard deviations computed on the features over the training points. More...

double	ResponsesOffset () const
	Get the mean value of the train responses. More...

bool	CenterData () const
	Get whether the data will be centered during training.

bool &	CenterData ()
	Modify whether the data will be centered during training.

bool	ScaleData () const
	Get whether the data will be scaled by standard deviations during training. More...

bool &	ScaleData ()
	Modify whether the data will be scaled by standard deviations during training. More...

size_t	MaxIterations () const
	Get the maximum number of iterations for training.

size_t &	MaxIterations ()
	Modify the maximum number of iterations for training.

double	Tolerance () const
	Get the tolerance for training to converge.

double &	Tolerance ()
	Modify the tolerance for training to converge.

template<typename Archive >
void	serialize (Archive &ar, const uint32_t version)
	Serialize the BayesianLinearRegression model. More...

Detailed Description

A Bayesian approach to the maximum likelihood estimation of the parameters \( \omega \) of the linear regression model.

The Complexity is governed by the addition of a gaussian isotropic prior of precision \( \alpha \) over \( \omega \):

\[ p(\omega|\alpha) = \mathcal{N}(\omega|0, \alpha^{-1}I) \]

The optimization procedure calculates the posterior distribution of \( \omega \) knowing the data by maximizing an approximation of the log marginal likelihood derived from a type II maximum likelihood approximation. The determination of \( alpha \) and of the noise precision \( beta \) is part of the optimization process, leading to an automatic determination of w. The model being entirely based on probabilty distributions, uncertainties are available and easly computed for both the parameters and the predictions.

The advantage over linear regression and ridge regression is that the regularization is determined from all the training data alone without any require to an hold out method.

The code below is an implementation of the maximization of the evidence function described in the section 3.5.2 of the C.Bishop book, Pattern Recognition and Machine Learning.

@article{MacKay91bayesianinterpolation,
  author = {David J.C. MacKay},
  title = {Bayesian Interpolation},
  journal = {NEURAL COMPUTATION},
  year = {1991},
  volume = {4},
  pages = {415--447}
}

@book{Bishop:2006:PRM:1162264,
  author = {Bishop, Christopher M.},
  title = {Pattern Recognition and Machine Learning (Information Science
           and Statistics)},
  chapter = {3}
  year = {2006},
  isbn = {0387310738},
  publisher = {Springer-Verlag},
  address = {Berlin, Heidelberg},
}

Example of use:

arma::mat xTrain; // Train data matrix. Column-major.
arma::rowvec yTrain; // Train target values.
// Train the model. Regularization strength is optimally tunned with the
// training data alone by applying the Train method.
// Instantiate the estimator with default option.
BayesianLinearRegression estimator;
estimator.Train(xTrain, yTrain);
// Prediction on test points.
arma::mat xTest; // Test data matrix. Column-major.
arma::rowvec predictions;
estimator.Predict(xTest, prediction);
arma::rowvec yTest; // Test target values.
estimator.RMSE(xTest, yTest); // Evaluate using the RMSE score.
// Compute the standard deviations of the predictions.
arma::rowvec stds;
estimator.Predict(xTest, responses, stds)

Constructor & Destructor Documentation

◆ BayesianLinearRegression()

BayesianLinearRegression::BayesianLinearRegression	(	const bool	centerData = `true`,
		const bool	scaleData = `false`,
		const size_t	maxIterations = `50`,
		const double	tolerance = `1e-4`
	)

Set the parameters of Bayesian Ridge regression object.

The regularization parameter is automatically set to its optimal value by maximization of the marginal likelihood.

Parameters

centerData	Whether or not center the data according to the examples.
scaleData	Whether or not scale the data according to the standard deviation of each feature.
maxIterations	Maximum number of iterations for convergency.
tolerance	Level from which the solution is considered sufficientlly stable.

Member Function Documentation

◆ Alpha()

double mlpack::regression::BayesianLinearRegression::Alpha ( ) const

inline

Get the precision (or inverse variance) of the gaussian prior.

Train() must be called before.

Returns: \( \alpha \)

◆ Beta()

double mlpack::regression::BayesianLinearRegression::Beta ( ) const

inline

Get the precision (or inverse variance) beta of the model.

Train() must be called before.

Returns: \( \beta \)

◆ DataOffset()

const arma::colvec& mlpack::regression::BayesianLinearRegression::DataOffset ( ) const

inline

Get the mean vector computed on the features over the training points.

Returns: responsesOffset

◆ DataScale()

const arma::colvec& mlpack::regression::BayesianLinearRegression::DataScale ( ) const

inline

Get the vector of standard deviations computed on the features over the training points.

Returns: dataOffset

◆ Omega()

const arma::colvec& mlpack::regression::BayesianLinearRegression::Omega ( ) const

inline

Get the solution vector.

Returns: omega Solution vector.

◆ Predict() [1/2]

void BayesianLinearRegression::Predict	(	const arma::mat &	points,
		arma::rowvec &	predictions
	)		const

Predict \(y_{i}\) for each data point in the given data matrix using the currently-trained Bayesian Ridge model.

Parameters

points	The data points to apply the model.
predictions	y, Contains the predicted values on completion.

Returns: Root mean squared error computed on the train set.

◆ Predict() [2/2]

void BayesianLinearRegression::Predict	(	const arma::mat &	points,
		arma::rowvec &	predictions,
		arma::rowvec &	std
	)		const

Predict \(y_{i}\) and the standard deviation of the predictive posterior distribution for each data point in the given data matrix, using the currently-trained Bayesian Ridge estimator.

Parameters

points	The data point to apply the model.
predictions	Vector which will contain calculated values on completion.
std	Standard deviations of the predictions.

◆ ResponsesOffset()

double mlpack::regression::BayesianLinearRegression::ResponsesOffset ( ) const

inline

Get the mean value of the train responses.

Returns: responsesOffset

◆ RMSE()

double BayesianLinearRegression::RMSE	(	const arma::mat &	data,
		const arma::rowvec &	responses
	)		const

Compute the Root Mean Square Error between the predictions returned by the model and the true responses.

Parameters

data	Data points to predict
responses	A vector of targets.

Returns: Root mean squared error.

◆ ScaleData() [1/2]

bool mlpack::regression::BayesianLinearRegression::ScaleData ( ) const

inline

Get whether the data will be scaled by standard deviations during training.

◆ ScaleData() [2/2]

bool& mlpack::regression::BayesianLinearRegression::ScaleData ( )

inline

Modify whether the data will be scaled by standard deviations during training.

◆ serialize()

template<typename Archive >

void mlpack::regression::BayesianLinearRegression::serialize	(	Archive &	ar,
		const uint32_t	version
	)

Serialize the BayesianLinearRegression model.

Serialize the Bayesian linear regression model.

◆ Train()

double BayesianLinearRegression::Train	(	const arma::mat &	data,
		const arma::rowvec &	responses
	)

Run BayesianLinearRegression.

The input matrix (like all mlpack matrices) should be column-major – each column is an observation and each row is a dimension.

Parameters

data	Column-major input data, dim(P, N).
responses	A vector of targets, dim(N).

Returns: Root mean squared error.

◆ Variance()

double mlpack::regression::BayesianLinearRegression::Variance ( ) const

inline

Get the estimated variance.

Train() must be called before.

Returns: 1.0 / \( \beta \)

The documentation for this class was generated from the following files:

src/mlpack/methods/bayesian_linear_regression/bayesian_linear_regression.hpp
src/mlpack/methods/bayesian_linear_regression/bayesian_linear_regression.cpp
src/mlpack/methods/bayesian_linear_regression/bayesian_linear_regression_impl.hpp

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ BayesianLinearRegression()

Member Function Documentation

◆ Alpha()

◆ Beta()

◆ DataOffset()

◆ DataScale()

◆ Omega()

◆ Predict() [1/2]

◆ Predict() [2/2]

◆ ResponsesOffset()

◆ RMSE()

◆ ScaleData() [1/2]

◆ ScaleData() [2/2]

◆ serialize()

◆ Train()

◆ Variance()