Download pdf - Institut für Elektrische Meßtechnik und Meßsignalverarbeitung … · 2017. 11. 7. · Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung

Professor Horst Cerjak, 19.12.2005

1

WS 2017/18Image-based Measurement 4 Calibration Axel Pinz

Image-based Measurement

• Human visual system

• Neurophysiology

• Color (perception, illumination, calibration, constancy)

• Image formation (CCD, CMOS sensors)

• Image acquisition• Projective geometry

• Camera models, camera calibration

• Fundamental algorithms• Salient point detection and description

• Linear algebra (RQ, SVD), Estimating H, P, F, Practical issues, Camera Pose

• Specific algorithms in detail• Camera calibration, F, PnP

• 3D scene structure (from stereo, motion, …), 4D Multibody structure and motion



2


Calibration Issues

• Linear Models– Homography estimation H

– Epipolar geometry F, E

– Interior camera parameters K

– Exterior camera parameters R,t

– Camera pose R,t

• Algorithms– Interest Point Detection + Description

– Overdetermined systems of linear equations Error Minimization

– Direct Linear Transform – DLT

– Normalization

– Nonlinearities iterative error minimization, Levenberg-Marquardt

– Outliers Robustness, RANSAC

3

2

1

333231

232221

131211

3

2

1

'

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

x

x

H

MtMZ

Y

X

pppp

pppp

pppp

y

x

m

|

1

~

1 34333231

24232221

14131211

RKP



3


Pinhole Camera

• “real” camera

image plane πi (x,y): Zcam = -f

x

y

Zcam

Xcam

Ycam

“principal”

point (x0,y0) “optical axis”

m(x,y)

M(Xcam,Ycam,Zcam)

f

“focal length” f

• 2D projection 3D scene

• m(x,y) ↔ line of sight = viewing direction

M’(Xcam’,Ycam’,Zcam’)

• “Pinhole” C … “center of projection”

Ccam

X Y

Z

M(X,Y,Z,1) M’(X’,Y’,Z’,1)

R,tm(x,y,1)

• “interior” camera parameters

– x0, y0, f, …

• “exterior” parameters

– camera pose

– R, t



4


Pinhole Camera

image plane πi (x,y): Zcam = -f

x

y

Zcam

Xcam

Ycam“principal”

point (x0,y0) “optical axis”

f Ccam

X Y

Z

M(X,Y,Z,1)M’(X’,Y’,Z’,1)

R,tm(x,y,1)

MZ

Y

X

pppp

pppp

pppp

y

x

m

P

1

~

1 34333231

24232221

14131211P : 3 x 4 matrix

“camera projection matrix”

[Pollefeys p.24, eq. (3.8)]

Mm P~



5


The Basic Pinhole Model

Note: Figures taken from, notation according to [Hartley,Zisserman]

xZfYZfXZYXX TT ~)/,/(),,(~

~ … inhomog. coord.

XZ

Y

X

f

f

Z

fY

fX

xZ

Y

X

P

101

0

0

1

… homog. coord.



6


The Basic Pinhole Model

0~

|)1,,diag(

01

01

01

101

0

0

IP fff

f

f

f



7


Principal Point Offset

camXZ

Y

X

yf

xf

Z

ZyfY

ZxfX

xZ

Y

X

P

101

0

0

1

0

0

0

0



8


Principal Point Offset

camcam XXyf

xf

x

0~

|

01

01

01

1

0

0

IK

1

0

0

yf

xf

Kcamera calibration matrix

interior/internal parameters

interior/internal orientation



9


Camera Rotation and Translation

CXX cam

~~~R

XC

X cam

10

~RR XCx

~| IKR

4 x 4

P3 P3

3 x 4

P2 P3



10



C~

| IKRP 3 x 4 projection matrix P

9 degrees of freedom

3 “internal parameters” in K

3 rotation angles in R

3 translations in C~



11



C~

| IKRP

Simplified notation: avoid explicit modeling of C

t|CttXXcam

RKPRR

~ ,

~~



12


• Pinhole– 3 parameters in K

• CCD– 4 parameters

• Finite projective camera– 5 parameters

– “skew” s

From Pinhole Real Cameras: K

1

0

0

yf

xf

K

yyxxy

x

fmfmy

x

, ,

1

0

0

K

mx

my

1

0

0

y

xs

y

x

K

3 x R, 3 x t

9

10

11



13


Projective Camera

• Finite projective camera:– K is an upper triangular matrix

– KR is non-singular

• General projective camera:– P is an arbitrary 3 x 4 matrix of rank 3

– P has also 11 degrees of freedom

4

1|~

| pC MIMIKRP

34333231

24232221

14131211

pppp

pppp

pppp

P

But: We model real cameras

as finite projective cameras

(+ lens distortion)



14


Camera Calibration in Practice (1)

• Take – 1 picture of a 3D calibration target,

– or several pictures of a planar calibration target

(take care so that all parameters can be recovered !)

• Establish point correspondences

• Calculate P– set of linear equations

• Decompose P

niXx ii 1 ,~~

t

|RKP

MZ

Y

X

pppp

pppp

pppp

y

x

m

P

1

~

1 34333231

24232221

14131211



15


3D Targets

[Hartley + Zisserman][Heikkilä]

Photogrammetry [Godding / Jähne]

• Many ways to build …

• Corners vs. circles (center of gravity) …

• Precision of building, attaching, …

• CNC measured points …

• EMT: coordinate measurement machine



16


Simulating a 3D target by actuating a CMM

Camera calibration @ EMT using a CMM (coordinate measurement machine)



17


2D vs. 3D Targets

f = 28mm, z ~ 300mm f = 50mm, z ~ 470mm f = 84mm, z ~ 720mm



18


2D Targets

f = 28mm, z ~ 280mm f = 50mm, z ~ 470mm f = 84mm, z ~ 720mm

• arbitrary scaling !

– z/f ~ const.

– closeup of toy car vs. real car at a distance …

• but: subtle differences in image quality !



19


Image Quality (1)

f = 28mm, z ~ 280mm f = 50mm, z ~ 470mm

lens distortion !



20


Image Quality (2)

f = 28mm, z ~ 280mm f = 50mm, z ~ 470mm

“chromatic aberration”



21


Lens Distortion Model

• Several ways to model

• Most common:– Scene >> lens system C is ok

– Radial lens distortion ki

– Tangential lens distortion tj

– Radial >> tangential

– Polynomial approximation up to varying order

(x0,y0)

,0

xxx ,0

yyy 22 yxr

)1)(2)2((' 2

32

221

64

2

2

31rtyxtrxtrxkrxkrxkxx

)1)(2)2((' 2

32

221

64

2

2

31rtyxtrytrykrykrykyy

r

x

y



22


Camera Calibration in Practice (2)

• Take – 1 picture of a 3D calibration target,

– or several pictures of a planar calibration target

• Establish point correspondences

• Calculate P– set of linear equations

• Decompose P

A first

estim

ate

for

linear

Inte

rior

para

mete

rs (

K)

• Add nonlinear relationships (model ki, tj)

• Perform iterative optimization (w.r.t. some error)

• Enforce constraints (such as structure of K and R)



23


What can be measured with one calibrated camera?

• Directions (lines of sight)

• Angles between directions

• 3D reconstruction (“motion stereo”, SaM):– Camera is moved

– Several images are taken

– Point correspondences between the images are

established

• Special case: planar scene + known extrinsics– Angles, distances, areas



24


More Than One Camera

• Multiple views:– Epipolar geometry

– Uncalibrated stereo: “Fundamental” matrix F

– Calibrated stereo: “Essential” matrix E

– Stereo rig

– Camera motion

many views

tracking

Structure + Motion (S+M)



25


Epipolar Geometry (1)

• Figures from [Hartley + Zisserman]

• C, C’, x, x’, X are co-planar (lie in the “epipolar plane” π)



26



• Assume that only C, C’, and x are known



27



• π projects on “epipolar lines” l and l’

• “baseline”: connects C, C’

• “epipoles”: e, e’

C C’



28



• When 3D position of X varies, π “rotates” about the baseline

• Family of planes – “epipolar pencil” – “Ebenenbüschel”

C C’



29


Epipolar Geometry – Example 1:

Converging Cameras [Hartley+Zisserman]



30


Epipolar Geometry – Example 2:

Forward Translation [Hartley+Zisserman], [Pollefeys]

e

e’



31


The Fundamental Matrix F (1)

We had an example:

Homography H



32



• Transfer xi via Xi in π to xi’

• 2D homography Hπ maps each xi to xi’

''' xel

'''' xexe

0

0

0

' ,''

1

'

2

'

1

'

3

'

2

'

3

'

3

'

2

'

1

ee

ee

ee

e

e

e

e

e

xxelxx

FHH '' :'

“skew-symmetric”

matrix

xl

F'



33



• F relates x in one image with its corresponding

epipolar line l’ in the other image (all X in R3 !):

• The corresponding point x’ must lie on l’:

• This relates to:

• How to estimate F?

xl

F'

0'' ,0'' lxlx T

0' xx T F

Point correspondences



34


Two Calibrated Cameras: Essential Matrix E

• “Essential matrix” E– Similar to F

– Relates calibrated stereo rig

– Internal matrices K and K’ are known

tt T

RRRE R, t

xxxx T 1ˆ ,0ˆ'ˆ KE “normalized coordinates” x̂

FKKEEKKTTT xx ' 0'' 1



35


Calibration Issues

• Linear Models– Homography estimation H

– Epipolar geometry F, E

– Interior camera parameters K

– Exterior camera parameters R,t

– Camera pose R,t

• Algorithms– Interest Point Detection + Description

– Overdetermined systems of linear equations Error Minimization

– Direct Linear Transform – DLT

– Normalization

– Nonlinearities iterative error minimization, Levenberg-Marquardt

– Outliers Robustness, RANSAC

3

2

1

333231

232221

131211

3

2

1

'

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

x

x

H

MtMZ

Y

X

pppp

pppp

pppp

y

x

m

|

1

~

1 34333231

24232221

14131211

RKP