Sampling-based Robust Multi-lateral Filter for Depth Enhancement

Kyoung-Rok Lee, Ramsin Khoshabeh, Truong Q. Nguyen

Video Processing Lab

University of California, San Diego

Abstract

Depth maps are an integral component of 3D video processing. They have a number of uses, including view synthesis for multi-view video, human computer interaction, augmented reality, and 3D scene reconstruction. However, depth maps are often captured at low quality or low resolution due to sensor hardware limitations or estimation errors. In this paper, we propose a new method to enhance noisy or low-resolution depth maps using high-resolution color images. Our method is based on sample selection and refinement in conjunction with multi-lateral filtering, a method derived from joint bilateral filtering using a new weighting metric. Our experimental results verifies that the proposed method performs very well in comparison to existing methods.

 

Results

1. Experimental result of the proposed method

(a) Downsampled by 3x

  Color Image Raw Disparity Map Proposed Method Ground Truth
Moebius
Books
Art

(b) Downsampled by 5x

  Color Image Raw Disparity Map Proposed Method Ground Truth
Moebius
Books
Art

(c) Downsampled by 9x

  Color Image Raw Disparity Map Proposed Method Ground Truth
Moebius
Books
Art

 

2. Visual comparison on the Middlebury datasets. The upsampling methods include: (c) JBU, (d) PWAS, (e) UML, (f) proposed method.

(a) Downsampled by 3x

  (a) Color Image (b) Raw Disparity Map (c) JBU (d) PWAS (e) UML (f) Proposed Method
Moebius
Books
Art

(b) Downsampled by 5x

  (a) Color Image (b) Raw Disparity Map (c) JBU (d) PWAS (e) UML (f) Proposed Method
Moebius
Books
Art

(c) Downsampled by 9x

  (a) Color Image (b) Raw Disparity Map (c) JBU (d) PWAS (e) UML (f) Proposed Method
Moebius
Books
Art

 

3. Quantitative comparisons (average percent of bad pixels)

 

4. Percentage improvement in terms of number of bad pixels after applying the proposed algorithm to all the 109 methods on the Middlebury stereo evaluation

Method Number Tsukuba Venus Teddy Cones
1 0.0812 0.1421 0.0615 0.0516
2 0.1312 0.077 0.1093 0.0839
3 0.1077 0.0342 0.0905 0.0541
4 -0.0399 -0.0025 0.0045 0.0063
5 -0.0598 0.032 0.0153 0.0553
6 0.0737 0.0203 0.0055 0.0059
7 0.0182 0.0794 0.0032 0.0032
8 0.0828 0.5951 0.059 0.0873
9 0.0221 0.0035 -0.0015 0.0149
10 0.1481 0.0393 0.0046 0.0056
11 0.0667 0.0561 0.0514 0.0471
12 0.0259 0.078 0.0033 -0.0062
13 0.0027 0.0298 0.0006 0.0084
14 0.1345 0.0332 0.003 0.0059
16 0.1026 0.0045 0.0045 0.0088
17 -0.1206 0.1787 0.004 0.0066
18 0.0056 0.0418 0.0077 0.0019
19 -0.0021 0.0464 0.011 0.0096
20 -0.0005 0.0132 0.0052 0.0081
21 0.0917 0.0231 0.0028 0.0087
22 0.0231 0.0682 0.0342 0.0258
23 0.0358 0.0829 0.0535 0.0529
24 0.0315 0.0462 0.0595 0.0896
25 0.0774 0.056 0.0019 0.0054
26 -0.0081 0.0146 0.0004 0.0022
27 0.0575 0.1096 0.0071 -0.0028
28 -0.0288 0.0671 0.0167 0.0144
29 -0.0581 0.076 0.0085 0.0122
30 -0.0743 0.035 0.0029 0.0001
31 0.043 0.0819 0.0158 0.0222
32 -0.0549 0 -0.0001 0.0011
33 -0.0296 0.0198 0.0002 0.0016
34 0.0134 0.0016 0.0056 0.0054
35 -0.0496 0.0412 0.0018 -0.0002
36 -0.0844 0.004 0.0311 0.0314
37 0.0144 0.1055 0.0925 0.0097
38 0.1326 0.0238 0.0335 0.0294
39 0.1305 0.1238 0.0152 0.0466
40 0.005 0.1223 0.0435 0.0833
41 -0.1233 0.0485 -0.0006 0.0032
42 0.0128 0.0879 0.0099 0.0024
43 0.0933 0.1203 0.0498 0.0054
44 0.0581 0.1846 0.0849 0.0341
45 0.0707 0.0172 0.0063 0.0075
46 0.0427 0.008 0.0072 0.0143
47 -0.0411 -0.005 0.0424 0.0542
48 0.0241 0.0445 -0.0007 0.0659
49 -0.0078 0.0954 0.008 0.0158
50 0.0537 0.0132 0.0344 0.015
51 -0.036 0.001 0.0504 0.0626
52 0.0315 0.0834 0.008 0.0164
54 0.0717 0.0847 0.0372 0.0145
55 -0.0686 0.0959 0.0012 0.0003
56 0.1384 0.0911 0.0294 0.0039
57 -0.0726 0.0774 -0.0038 -0.04
59 0.0352 0.0997 0.0098 0.0225
60 0.0667 0.098 0.0104 0.0127
61 0.1075 0.0428 0.0088 0.045
62 0.0507 0.0054 0.0242 0.0163
63 0.0334 0.0238 0.0208 0.039
64 -0.0242 0.0391 -0.0004 -0.0012
65 0.0026 0.0821 0.0519 0.022
66 0.0833 0.0055 0.0004 0.0002
67 0.0874 0.0323 0.0192 0.0182
68 0.0868 0.0261 0.0616 0.035
69 0.034 0.0816 0.0863 0.0617
70 0.0907 0.0233 0.0237 0.0169
71 0.097 0.0348 0.014 0.0268
72 0.1771 0.0048 0.0115 0.0274
73 -0.0179 0.0816 0.0739 0.1879
74 0.0192 0.0908 -0.0241 0.0271
75 0.0544 0.0425 0.0576 0.0179
76 0.0354 0.0131 0.0055 0.0064
77 0.0529 0.0216 0.0167 0.0137
78 0.0745 0.0117 0.0052 0.0067
79 -0.0737 0.0215 0.0036 0.0049
80 0.0198 0.0611 0.0581 0.0091
81 0.0648 0.0491 0.0082 0.0103
82 0.0375 0.0221 0.0047 0.0049
83 0.061 0.0146 0.0136 0.015
84 0.1054 0.1085 0.0516 0.0151
85 0.0723 0.0869 0.0355 0.0177
86 -0.0796 0.021 0.0079 0.0168
87 0.0963 0.0107 0.0119 0.0133
88 0.0214 0.0052 0.046 0.0145
89 0.0472 0.2518 0.0151 0.1795
90 0.0453 0.0706 0.0161 0.0321
92 0.1139 0.0386 0.0135 0.0204
93 0.051 0.066 0.0326 0.0489
94 -0.0246 0.0787 0.0202 0.0024
95 0.0086 0.0952 -0.0016 0.0028
97 -0.0067 0.0679 0.0064 0.0099
98 0.0451 0.0221 0.0066 0.006
99 -0.0014 0.1007 0.0065 0.0716
100 0.1276 0.0377 0.0152 0.0369
101 0.0111 0.0058 0.04 0.0115
102 -0.0534 0.0701 0.0077 0.0063
103 0.0233 0.0179 0.0136 0.0106
104 -0.0233 0.0063 0.0026 -0.0019
105 -0.0062 0.0558 0.0045 0.0085
106 0.1546 0.1195 0.0154 0.0397
107 -0.038 0.0931 0.0062 0.001
108 0.1007 0.0041 0.0494 0.0105
109 0.0521 0.0357 0.0107 0.0236
110 0.2228 0.022 0.0174 0.032
111 -0.0842 0.0391 0 0.0048
112 0.0314 0.0703 0.0093 0.0052
113 0.1042 0.1021 0.0059 0.0267
114 0.0705 0.0808 0.0041 0.0122