Xiran_thesis.pdf (7.54 MB)

Plug-and-Play ADMM for Image Restoration

Download (7.54 MB)
posted on 16.10.2019 by Xiran Wang
The alternating direction method of multiplier (ADMM) is one of the most widely used optimization algorithms in image restoration. Among many features, e.g., provably convergent under mild conditions, its modular structure is particularly appealing to model based image reconstruction problems. In particular, one can separate the log-likelihood and the log-prior in a maximum-a-posteriori formulation using the ADMM algorithm. However, such approach does not allow us to incorporate likelihood or priors that are not expressible as proximal maps (a particular type of optimization involving a convex function regularized by a quadratic penalty). Deep neural network based image denoisers are some of the better known examples. They have demonstrated very promising image restoration results, yet they cannot be expressed as proximal maps. The question to pursue in this thesis is how to integrate non-optimization likelihood or priors using the ADMM algorithm.

The Plug-and-Play (P\&P) ADMM is a generalization of the ADMM algorithm that allows non-optimization based models to be used. Since its introduction in 2013, the method has demonstrated promising performance for various imaging problems, e.g., tomography. Its convergence has also been proven under some restrictive conditions where the denoiser is non-expansive and has a symmetric Jacobian. However, many problems remain unsolved. First, existing work has been focusing on a range of medical imaging problems in tomography while the applications of the Plug-and-Play framework for more general problems have not been explored. Second, even though study on the global convergence analysis has been done, it restricts the compatible denoisers to a class of symmetric smoothing filters that are rarely adopted in state-of-the-art competitive denoisers. Third, due to its ad-hoc nature, the performance of Plug-and-Play ADMM is sensitive to the choice of internal parameters of the algorithm. A more robust version is desirable in order for the framework to be applicable for a wide range of problems. Fourth, the current Plug-and-Play framework still relies on a well-defined forward model of the imaging system, which means it is still an optimization based approach. However, as well-defined models are sometimes unavailable for more complicated imaging problems, a non-optimization based version is desired.

In this thesis, we address the above issues by studying both the theoretical and practical aspects of the algorithm. First, we study the applications of the Plug-and-Play framework for a wide range of general image restoration problems, such as superresolution, deblurring, inpainting, single-photon imaging and even a video segmentation problem used for virtual reality content creation. Efficient implementations of the Plug-and-Play ADMM for these applications are introduced and outperforms state-of-the-art existing algorithms for every task. For superresolution specifically, we derived a closed-form solution for the inversion step that is previously unavailable and potentially applicable for other optimization frameworks. Second, to tackle the Plug-and-Play ADMM's sensitivity on internal parameters, we draw insights from the generalized approximate message passing to design an automatic update scheme for the internal parameters achieving robust performance across different tasks. Third, a new convergence analysis is presented proving a fixed-point convergence for a much wider range of denoisers compared to previous work. With the recent introduction of Multi-agent consensus equilibrium (MACE), a generalized Plug-and-Play framework that can work with an arbitrary number of operators to solve a common problem, this work also introduces the application and design of a MACE algorithm for solving video segmentation which is outside the scope of classical image restoration problems. The proposed MACE algorithm, unlike Plug-and-Play ADMM, does not rely on a well-defined forward model and is capable of including an arbitrary number of operators instead of just two.

The Plug-and-Play ADMM algorithms studied in this thesis have significantly advanced our image restoration capability by allowing non-optimization procedures to be used in the framework. We demonstrated applications in super-resolution, inpainting, deblurring, and single-photon reconstruction, with superior performance than the previous state-of-the-art. The algorithm has also enabled a new line of applications in segmenting foreground masks for virtual reality content creation that is fully automatic and does not require human interactions.





Degree Type

Doctor of Philosophy


Electrical and Computer Engineering

Campus location

West Lafayette

Advisor/Supervisor/Committee Chair

Stanley H. Chan

Additional Committee Member 2

Jan Allebach

Additional Committee Member 3

Charles Bouman

Additional Committee Member 4

Amy Reibman