Lee-Kang (Lester) Liu - Projects

Video Proessing Lab , Department of Electrical and Computer Engineering, of University of California at San Diego

Sparse Reconstruction of Depth Data

A Framework for Depth Video Reconstruction from a Subset of Samples and Its Applications

The growth of depth sensing systems in this decade have facilitated a variety of applications in computer vision. Depending on the systematic configurations, both direct and indirect sensing techniques encounter image processing issues, such as hole filling, depth map super resolution. In this paper, a framework for depth video reconstruction from a subset of samples is proposed. By redefining classical dense depth estimation into two individual problems, sensing and synthesis, we propose a motion compensation assisted sampling (MCAS) scheme and a spatio-temporal depth reconstruction (STDR) algorithm for reconstructing depth video sequences from a subset of samples. Using the 3-dimensional extensible dictionary, 3D-DWT, and applying alternating direction method of multiplier technique, the proposed STDR algorithm possesses scalability for temporal volume and efficiency for processing large scale depth data. Exploiting the temporal information and corresponding RGB images, the proposed MCAS scheme achieves an efficient 1-Stage sampling. Experimental results show that the proposed depth reconstruction framework outperforms the existing methods and is competitive compared to our previous work[1], which requires a pilot signal in the 2-Stage sampling scheme. Finally, to estimate missing reliable depth samples from varying input sources, we present an inference approach using geometrical and color similarities. Applications for depth video super resolution from uniform-grid subsampled data and dense disparity video estimation from a subset of reliable samples are presented.

Supplementary materials: [html]

Matlab Code (*.rar format) [download] (~464 MB)

"*.mat" Data for Matlab Code (*.rar format) [download] (~ 39.8 MB).
Unzip the file and replace the directory ".\Matlab_Code\results_matfiles\"



RGB Image	Input Depth (190x270)	Proposed Method (190x270)	Guided Filter [4]	Hawe [5]	Triangular Interp.

[1] L.-K, Liu, S.H. Chan, and T.Q. Nguyen, "Depth reconstruction from sparse samples: Representation, algorithm, and sampling," IEEE Trans. on Image Process., vol. 24, no. 6, pp. 1983-1996, Jun. 2015.

Depth Reconstruction From Sparse Samples: Representation, Algorithm, and Sampling

The rapid development of 3D technology and computer vision applications has motivated a thrust of methodologies for depth acquisition and estimation. However, existing hardware and software acquisition methods have limited performance due to poor depth precision, low resolution, and high computational cost. In this paper, we present a computationally efficient method to estimate dense depth maps from sparse measurements. There are three main contributions. First, we provide empirical evidence that depth maps can be encoded much more sparsely than natural images using common dictionaries, such as wavelets and contourlets. We also show that a combined wavelet-contourlet dictionary achieves better performance than using either dictionary alone. Second, we propose an alternating direction method of multipliers (ADMM) for depth map reconstruction. A multiscale warm start procedure is proposed to speed up the convergence. Third, we propose a two-stage randomized sampling scheme to optimally choose the sampling locations, thus maximizing the reconstruction performance for a given sampling budget. Experimental results show that the proposed method produces high-quality dense depth estimates, and is robust to noisy measurements. Applications to real data in stereo matching are demonstrated.

The goal of this project is to reconstruct dense depth data from a few depth samples. More specifically, the problem can be interpreted as, "given a sparse amount of samples, how can we reconstruct dense depth data?"

Figure 1: Systematic overview of reconstructing dense Aloe disparity map from 10% samples.

10% samples		Reconstructed disparity map

To achieve our goal, there are two fundamental problems: (1) Given sparse samples, how to efficiently reconstruct dense depth? (2) Given fixed sampling budgets, how to select samples?

We address these fundamental problems in our journal paper [1]. We provide Matlab code for our proposed dense depth reconstruction algorithms, and efficient sampling strategies. We further include demonstration codes for the disparity reconstruction using our proposed algorithms. Readers can download MATLAB toolbox and the user guide from the link as followed in the "Additional Information." The provided MATLAB code is for academic usage. Please contact Lee-Kang Liu (email: l7liu@ucsd.edu) for any questions. Thanks..

Additional Information:

Matlab Toolbox: [Code, ~10MB, zip], [Code, ~10MB, rar]
Matlab Toolbox User Guide: [pdf]
Bibtex:
@Article{Liu_Chan_Nguyen_TIP2015,
Title = {Depth Reconstruction from Sparse Samples: Representation, Algorithm, and Sampling},
Author = {Liu, L.-K, and Chan, S.H., and Nguyen, T.Q.},
Journal = {IEEE Trans. on Image Process.},
Year = {2015},
Month = {Jun.},
Number = {6},
Pages = {1983-1996},
Volume = {24},
doi={10.1109/TIP.2015.2409551},
ISSN={1057-7149},
}

Real-time Disparity Estimation and Skeleton Detection

To achieve real-time human gesture recognition from stereo video is our primary goal of this project, and our system has three features.

We implement disparity estimation algorithm [1] in CUDA, and we achieve output depth video 10 frames per second with 640x480 detph resolution.

We achieve real-time skeleton detection using OpenNI library, and we integrate the stereo plus estimated detph video into the OPENNI environment.

We achieve gesture recongition for further controlling operation.

Left View Image	Right View Image

RGB images extracted from stereo camera.

Disparity Map	Skeleton and Depth Map

Estimated disparity map and skeleton

[1] Z. Lee, J. Juang, and T.Q. Nguyen, "Local disparity estimation with three-method cross census and advance support weight," IEEE Transactions on Multimedia, vol. 15, no. 8, pp. 1855-1864, Dec. 2013