Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
Rolling Shutter Stereo
Olivier Saurer, Kevin Köser, Jean-Yves Bouguet, Marc Pollefeys
ETH ZürichSwitzerland
GEOMAR KielGermany
Google Inc,Mountain View, CA
ETH ZürichSwitzerland
2
Motivation
Resolution Rolling Shutter Time [ms]
iPhone 4S2
1920 x 1080 26.04
Galaxy S32
1920 x 1080 32.67
• Most CMOS chips have an electronic rolling shutter
Sequential exposure of scanline (rolling shutter)Cheap & Compact Light sensitive
2Oth et al. 2013
Ro
lling Sh
utter D
irection
3
When to Consider Rolling Shutter (RS)?
• RS distortion depends on:• Motion & Depth
• Moving car:
• Human motion:
• Velocity = 25 km / h• RS distortion is measurable
up to: 125m
• Velocity = 5 km / h• RS distortion is measurable
up to: 25m
4
Baker et al. 2010
Related Work
Rolling Shutter Calibration
Oth et al. 2013
Removing Rolling Shutter Wobble
Rolling Shutter Structure from Motion
Klingner et al. 2013
Forssén et al. 2010
Geyer et al. 2005
Hedborg et al. 2010
Dense 3D Reconstruction
?
5
Outline
Rolling Shutter 2-View Geometry
Rolling Shutter Stereo (Proposed Method)
Evaluation
6
Case 1: Valid Epipolar Geometry
RS direction Camera motion direction
Left RScamera
Right RS camera
7
Case 1: Valid Epipolar Geometry
RS direction Camera motion direction
Inter baseline
Intra baselines
Left RScamera
Right RS camera
Important special case: “Streetview”• Intra & inter baseline coincide
8
Case 1: Valid Epipolar Geometry
RS direction Camera motion direction
Inter baseline
Intra baselines
Epipolar line
P
Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry
Left RScamera
Right RS camera
9
Case 1: Valid Epipolar Geometry
RS direction Camera motion direction
Inter baseline
Intra baselines
Epipolar line
P
Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry
Left RScamera
Right RS camera
10
Case 1: Valid Epipolar Geometry
RS direction Camera motion direction
Inter baseline
Intra baselines
Epipolar line
P
Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry
Correct pixel correspondence
Left RScamera
Right RS camera
11
Case 1: Valid Epipolar Geometry
RS direction Camera motion direction
Inter baseline
Intra baselines
Epipolar line
GS
P
Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry
Correct pixel correspondence• Global Shutter triangulated 3D point
Left RScamera
Right RS camera
12
Case 1: Valid Epipolar Geometry
Inter baseline
Intra baselines
Epipolar line
GSRS
P
RS direction Camera motion direction
Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry
Correct pixel correspondenceGS triangulated 3D points havewrong depthRS triangulated 3D points havecorrect depth, considering correct pose at time of exposure
Left RScamera
Right RS camera
13
Standard Stereo vs. RS Stereo
Input Images Standard Stereo
Proposed Method(RS Stereo)
Ground Truth
RS & Camera Motion
RS direction Camera motion direction
14
• Intra & inter baseline do not coincide
Case 2: In General no Epipolar Geometry!1
Proposed MethodStandard Stereo:Only random matches
Left RS camera Right RS camera
1Besides very special configurations [Seitz 2001, Pajdla 2001]
15
Outline
Rolling Shutter 2-View Geometry
Rolling Shutter Stereo (Proposed Method)
Evaluation
16
Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure
Left RS camera
RS direction Camera motion direction
P P’
17
Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure
Left RS camera
RS direction Camera motion direction
P P’
1. Sample 3D planes1
1Plane Sweep: [Collins 1996], [Yang et al. 2003]
18
Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure
Left RS camera
RS direction Camera motion direction
P P’
1. Sample 3D planes1
2. Given:• 3D Point• Camera trajectorySolve for time of exposureusing RS projection model
1Plane Sweep: [Collins 1996], [Yang et al. 2003]
19
Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure
Left RS camera
RS direction Camera motion direction
P P’
1. Sample 3D planes1
2. Given:• 3D Point• Camera trajectorySolve for time of exposureusing RS projection model
3. Texture lookup using time of exposure
1Plane Sweep: [Collins 1996], [Yang et al. 2003]
20
Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure
Left RS camera
RS direction Camera motion direction
P P’P’’
Correlate
1Plane Sweep: [Collins 1996], [Yang et al. 2003]
1. Sample 3D planes1
2. Given:• 3D Point• Camera trajectorySolve for time of exposureusing RS projection model
3. Texture lookup using time of exposure
4. Find best correlation1
21
Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure
Left RS camera
RS direction Camera motion direction
P P’P’’
Correlate
1Plane Sweep: [Collins 1996], [Yang et al. 2003]
1. Sample 3D planes1
2. Given:• 3D Point• Camera trajectorySolve for time of exposureusing RS projection model
3. Texture lookup using time of exposure
4. Find best correlation1
22
Rolling Shutter Projection
• RS projection matrix changes with time :
RS camera
RS direction Camera motion direction
?
23
Rolling Shutter Projection
• RS projection matrix changes with time :
• Find time of exposure when is seen. Solve fix-point function: RS camera
RS direction Camera motion direction
?
RS image sensor
Projection of:Quadratic polynomial1
1Assuming Linear motion [Geyer et al. 2005]
24
Rolling Shutter Projection
• RS projection matrix changes with time :
• Find time of exposure when is seen. Solve fix-point function:
• Valid solution:
RS camera
RS image sensor
Projection of:
RS direction Camera motion direction
?
Quadratic polynomial1
1Assuming Linear motion [Geyer et al. 2005]
25
RS Projection (With Lens Distortion)
• Find time of exposure when is seen. Solve fix-point function:
RS direction Camera motion direction
Polynomial of degree 8
Lens distortion
RS image sensor
Projection of:
26
RS Projection (With Lens Distortion)
• Find time of exposure when is seen. Solve fix-point function:
• Global undistortion does not help
RS direction Camera motion direction
Polynomial of degree 8
Lens distortion
RS image sensor
Projection of:
Polynomial of degree 8
27
Motion Models
• Further motion models are discussed in the paper
Translation Orientation Distortion (# coefficient)
Polynomial Degree
Linear / Orbital / Spiral Const / Linear / Linear 0 2
Linear / Orbital / Spiral Const / Linear / Linear 1 4
Linear Linear 1 5
Linear Const 5 8
Linear Linear 5 9
28
Outline
Rolling Shutter 2 View Geometry
Rolling Shutter Stereo (Proposed Method)
Evaluation
29
RS Warp1.
2. Solve 8 degree polynomial
Rolling Shutter WarpFast Approximation (FA)1. RS warp at grid vertices
2. Interpolate texture coordinates within grid cell
Image size: 976 x 732Speed: 27.7ms / warp (CUDA)
Grid size: 1/10 of image sizeSpeed: 2.2 ms / warp (CUDA)
Left image Right image Interpolate
30
RS Warp1.
2. Solve 8 degree polynomial
Rolling Shutter WarpFast Approximation (FA)1. RS warp at grid vertices
2. Interpolate texture coordinates within grid cell
Image size: 976 x 732Speed: 27.7ms / warp (CUDA)
Grid size: 1/10 of image sizeSpeed: 2.2 ms / warp (CUDA)
Left image Right image Interpolate
~6 Hz with FA, with 50 planes
31
Results – Rendered Ground Truth
Ground Truth(LiDAR)
FARS StereoStandard Stereo
Intra baseline
Intra baseline
RS & Motion
RS & Motion
Castle
Old Town
32
RS
& M
oti
on
Results Standard Stereo vs. RS Stereo
Driving Speed: 25km/hInter baseline: 2mIntra baseline: 0.5m
RS reconstruction & fusion
Standard Stereo reconstruction & fusion
Bird’s eye views
33
Conclusion
• RS hurts 3D reconstruction if ignored!
• Radial distortion is depth dependentand can’t be undone without depth.
• With little additional computational cost, similar quality RS 3D reconstructionis possible as with GS images.
Thank you!
Supported by
Olivier Saurer, Kevin Köser, Jean-Yves Bouguet, Marc Pollefeys
35
Reconstruction from RS Streetview Images