A roadmap to research on EP

Tutorials

Video lectures on Approximate Inference

Expectation Propagation: CUED Tutorial slides

A family of algorithms for approximate Bayesian inference
How to construct EP algorithms, illustrated with a variety of different approximations and factor grouping schemes.

From Belief Propagation to Expectation Propagation K. Murphy, 2001
A quick skim of thesis Chapter 3, with some more derivations.

EP Summary E. Sudderth, 2002
An even shorter summary.

EP: A quick reference
A list of equations useful for constructing Gaussian EP algorithms.

Notes on Minka's Expectation Propagation for Gaussian Process classification M. Seeger, 2002

EP in practice
Extending expectation propagation for graphical models Y. Qi, 2004
An overview of techniques used to implement EP in practice.

Expectation Propagation for Exponential Families M. Seeger, 2007

Expectation propagation J. Raymond, A. Manoel, and M. Opper, 2014

Theoretical developments

The impact of different divergence measures

Divergence measures and message passing

Raising factors to powers

Power EP

Kikuchi approximation

Structured Region Graphs: Morphing EP into GBP

Convergence control

Damping and skipping: Expectation-Propagation for the Generative Aspect Model
Double-loop: Expectation propagation for approximate inference in dynamic Bayesian networks T. Heskes and O. Zoeter, UAI'2002

EP within EM

Expectation-Propagation for the Generative Aspect Model
Predictive Automatic Relevance Determination by Expectation Propagation

Sparse approximation

Gaussian Processes - Iterative Sparse Approximations L. Csato, 2002
Sparse-posterior Gaussian Processes for general likelihoods

The objective function

Expectation Propagation for approximate Bayesian inference
The EP energy function and minimization schemes
TAP Gibbs Free Energy, Belief Propagation and Sparsity L. Csato, M. Opper, and O. Winther, NIPS'2001
Expectation propagation for approximate inference in dynamic Bayesian networks T. Heskes and O. Zoeter, UAI'2002
Approximate Inference Techniques with Expectation Constraints T. Heskes, M. Opper, W. Wiegerinck, O. Winther and O. Zoeter, Journal of Statistical Mechanics: Theory and Experiment, 11015 (2005)

Perturbation Corrections

Approximate marginals in latent Gaussian models B. Cseke, T. Heskes, JMLR 2011
Perturbation Corrections in Approximate Inference: Mixture Modelling Applications Ulrich Paquet, Ole Winther, Manfred Opper, JMLR 2009
Perturbative Corrections for Approximate Inference in Gaussian Latent Variable Models M. Opper, U. Paquet, and O. Winther, JMLR 2013

Approximating the messages

Gaussian quadrature based expectation propagation O. Zoeter and T. Heskes, AISTATS 2005
ABC-EP: Expectation Propagation for Likelihood-free Bayesian Computation S. Barthelmé and N. Chopin, ICML 2011
Distributed Bayesian Posterior Sampling via Moment Sharing M. Xu et al, NIPS 2014
Learning to Pass Expectation Propagation Messages N. Heess, D. Tarlow, J. Winn, NIPS 2013
Just-In-Time Learning for Fast and Flexible Inference S. M. Ali Eslami, D. Tarlow, P. Kohli, and J. Winn, NIPS 2014
Kernel-Based Just-In-Time Learning for Passing Expectation Propagation Messages W. Jitkrittum et al, UAI 2015

Uses of EP

Perceptrons, Gaussian process classifiers

Assessing Approximate Inference for Binary Gaussian Process Classification M. Kuss and C. E. Rasmussen, JMLR 2005
Predictive Automatic Relevance Determination by Expectation Propagation
Fast Sparse Gaussian Process Methods: The Informative Vector Machine N. Lawrence, M. Seeger, and R. Herbrich, NIPS'2002
Gaussian Processes - Iterative Sparse Approximations L. Csato, 2002
A family of algorithms for approximate Bayesian inference
Gaussian Processes for Classification: Mean Field Algorithms M. Opper and O. Winther, Neural Computation 12: 2655-2684, 2000
Bayes Machines for Binary Classification D. Hernández-Lobato and J.M. Hernández-Lobato, Pattern Recognition Letters 29(10): 1466-1473, 2008
Expectation Propagation for Microarray Data Classification D. Hernández-Lobato, J.M. Hernández-Lobato, and A. Suárez, Pattern Recognition Letters 31(12): 1618-1626, 2010
Robust Multi-Class Gaussian Process Classification D. Hernández-Lobato, J.M. Hernández-Lobato, and P. Dupont, NIPS 2011
Expectation Propagation for Bayesian Multi-task Feature Selection D. Hernández-Lobato, J.M. Hernández-Lobato, T. Helleppute, and P. Dupont, PKDD 2010
Generalized Spike-and-Slab Priors for Bayesian Group Feature Selection Using Expectation Propagation D. Hernández-Lobato, J.M. Hernández-Lobato, and P. Dupont, JMLR 2013

Ordinal regression

Gaussian processes for ordinal regression W. Chu and Z. Ghahramani, JMLR 2005

Bayesian Conditional Random Fields

Discrete Bayes nets and Markov random fields

Expectation Consistent Approximate Inference M. Opper and O. Winther, JMLR 2005
Tree-structured approximations by expectation propagation

Density estimation with Gaussian processes

Gaussian Processes - Iterative Sparse Approximations L. Csato, 2002

Neural networks, Multilayer perceptrons

Computing with Finite and Infinite Networks O. Winther, NIPS'2001 (ADATAP)
Expectation Propagation for Neural Networks with Sparsity-promoting Priors P. Jylänki, A. Nummenmaa and A. Vehtari, JMLR 2013
Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights D. Soudry, I. Hubara and R. Meir, NIPS 2014
Probabilistic backpropagation for scalable learning of bayesian neural networks J. M. Hernandez-Lobato and R. P. Adams, ICML 2015

Independent Components Analysis (ICA)

TAP Gibbs Free Energy, Belief Propagation and Sparsity L. Csato, M. Opper, and O. Winther, NIPS'2001
Mean Field Approaches to Independent Component Analysis P. A.d.F.R. Højen-Sørensen, O. Winther, and L. K. Hansen, Neural Computation 14: 889-918 (2002)

Text modeling, latent Dirichlet allocation

Expectation-Propagation for the Generative Aspect Model

Hybrid dynamic systems (continuous + discrete state)

Expectation propagation for approximate inference in dynamic Bayesian networks T. Heskes and O. Zoeter, UAI'2002

Window-based expectation propagation for adaptive signal detection in flat-fading channels (Fixed-lag smoothing with EP)

Nonlinear dynamic systems

Gaussian quadrature based expectation propagation O. Zoeter and T. Heskes, AISTATS 2005

Expectation propagation for inference in non-linear dynamical models with Poisson observations B. M. Yu, K. V. Shenoy, M. Sahani, NSSPW'2006

Efficient computation of the maximum a posteriori path and parameter estimation in integrate-and-fire and more general state-space models S. Koyama and L. Paninski, J Comp Neuroscience 2009

Iterated extended Kalman smoothing with Expectation-Propagation A. Ypma and T. Heskes, NNSP'2003

Low-Complexity Iterative Detection for Large-Scale Multiuser MIMO-OFDM Systems Using Approximate Message Passing

S. Wu et al, 2014

Bayesian Inference for Sparse Generalized Linear Models

M. Seeger, S. Gerwinn, M. Bethge, ECML 2007

Bayesian inference for Plackett-Luce ranking models

J. Guiver, E. Snelson, ICML 2009

A Distributed Message Passing Algorithm for Sensor Localization

M. Welling and J. J. Lim, ICANN 2007

Expectation Propagation for Continuous Time Bayesian Networks

U. Nodelman, D. Koller, and C. R. Shelton, UAI 2005

Visualization of time-series data

Hierarchical visualization of time-series data using switching linear dynamical systems O. Zoeter and T. Heskes, PAMI 2003

Expectation Propagation for Infinite Mixtures