Spectral mixture kernels for Multi-Output Gaussian processes

Abstract

Multi-Output Gaussian Processes (MOGPs) are the multivariate extension of Gaussian processes (GPs \cite{Rasmussen:2006}), a Bayesian nonparametric method for univariate regression. MOGPs address the multi-channel regression problem by modeling the correlation in time and/or space (as scalar GPs do), but also across channels and thus revealing statistical dependencies among different sources of data. This is crucial in a number of real-world applications such as fault detection, data imputation and financial time-series analysis. Analogously to the univariate case, MOGPs are entirely determined by a multivariate covariance function, which in this case is matrix valued. The design of this matrix-valued covariance function is challenging, since we have to deal with the trade off between (i) choosing a broad class of cross-covariances and auto-covariances, while at the same time (ii) ensuring positive definiteness of the symmetric matrix containing these scalar-valued covariance functions. In the stationary univariate case, these difficulties can be bypassed by virtue of Bochner's theorem, that is, by building the covariance function in the spectral (Fourier) domain to then transform it to the time and/or space domain, thus yielding the (single-output) Spectral Mixture kernel \cite{Wilson:2013}. A classical approach to define multivariate covariance functions for MOGPs is through linear combinations of independent (latent) GPs; this is the case of the Linear Model of Coregionalization (LMC \cite{goo1997}) and the Convolution Model \cite{Alvarez:2008}. In these cases, the resulting multivariate covariance function is a function of both the latent-GP covariances and the linear operator considered, which usually results in symmetric cross-covariances that do not admit lags across channels. Due to their simplicity, these approaches fail to provide interpretability of the dependencies learnt and force the auto-covariances to have similar structure. The main purpose of this work is to extend the spectral mixture concept to MOGPs: We rely on Cram\'er's theorem \cite{Cramer:1940, Yaglom:1987}, the multivariate version of Bochner's theorem, to propose an expressive family of complex-valued square-exponential cross-spectral densities, which, through the Fourier transform yields the Multi-Output Spectral Mixture kernel (MOSM). The proposed MOSM model provides clear interpretation of all the parameters in spectral terms. Besides the theoretical presentation and interpretation of the proposed multi-output covariance kernel based on square-exponential spectral densities, we inquiry the plausibility of complex-valued t-Student cross-spectral densities. We validate our contribution experimentally through an illustrative example using a tri-variate synthetic signal, and then compare it against all the aforementioned methods on two real-world datasets.