Contacts
- Jean-Michel Morel
morel[AT]cmla.ens-cachan.fr - Guoshen Yu
yu[AT]cmap.polytechnique.fr
References
- J.M. Morel and G.Yu, ASIFT: A New Framework for Fully Affine
Invariant Image Comparison.
SIAM Journal on Imaging Sciences, 2(2):438-469, 2009. preprint - G. Yu and J.M. Morel, A Fully Affine Invariant Image Comparison
Method.
Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Taipei, 2009. preprint - J.M. Morel and G.Yu, On the consistency of the SIFT Method. Preprint, CMLA 2008-26, Sept 2008. preprint
Overview
A fully affine invariant image comparison method, Affine-SIFT (ASIFT) is introduced. While SIFT is fully invariant with respect to only four parameters namely zoom, rotation and translation, the new method treats the two left over parameters : the angles defining the camera axis orientation.
Against any prognosis, simulating all views depending on these two parameters is feasible. The method permits to reliably identify features that have undergone very large affine distortions measured by a new parameter, the transition tilt.
State-of-the-art methods hardly exceed transition tilts of 2 (SIFT), 2.5 (Harris-Affine and Hessian-Affine) and 10 (MSER). ASIFT can handle transition tilts up 36 and higher.
When Does it Work?
The SIFT method works to compare 2D objects or 3D objects with flat enough details, taken from similar view angles but at arbitrary distances.
The typical failure cases are:
- The illumination conditions are different (for instance daylight/nightlight).
- The object has a reflecting surface (typically cars are mirrors ; they change completely aspect under different view angles).
- The object has a strong 3D structure: in that case a change of view angle alters drastically its aspect.
- The object has a self similar or periodic structure: then "true" mismatches occur.
- The view angle is too different.
See our failure case study.
ASIFT corrects the last problem : if the object is under view has similar illumination conditions, has rather flat surface, and is not a mirror, then ASIFT retrieves the object even under extreme changes of angle. In technical terms, ASIFT is more affine invariant than SIFT.
Other popular methods addressing this problem exist: MSER and Hessian-Affine and Harris-Affine. All these methods will be compared below.
Additional Materials
- Slides : one-hour version, 20 minutes version, videos (zip, tar/gz)
- Poster
Online Demo
An online demo that allows you to try ASIFT with your own images is available here. If you are interested in ASIFT and want to try it with your own images, we suggest that you start with the user friendly online demo.
Software
The ASIFT software (version beta) allows you to try ASIFT locally on your machine.
- reference source code (for megawave) : zip tar/gz
- binary
for Linux
?for Windows (Matlab interface included)
the ASIFT software for Windows is developed with contributions from Pierre Moulon.
Dataset
An image dataset for systematic evaluation of robustness to absolute and transition tilt of the image matching algorithms is available : zip tar/gz.
Examples
ASIFT is compared with the four state-of-the-art algorithms the SIFT, Harris-Affine, Hessian-Affine and MSER detectors, all coded with the SIFT descriptor. Various types of images (size 600×450) were used for the experiments. The SIFT software is from D. Lowe. The Harris-Affine, Hessian-Affine and MSER programs are from the web site of K. Mikolajczyk.
- video tracking
- planar objects (paintings, journals, walls, etc.)
- monuments and constructions
- 3D objects (statues, cars, etc)
- complex scenes
- object deformation
In the examples below, the powerful Moisan-Stival procedure is applied to eliminate matches incoherent with epipolar geometry. Matches are connected by white segments.
Video tracking
ASIFT is compared with SIFT on video tracking . In each video, ASIFT and SIFT tracking are shown respectively on the top and bottom. If you prefer high-resolution video, please download the video files by clicking the links below.
| facade (youtube page, high-resolution video) | magazine (youtube page, high-resolution video) |
|---|---|
Planar objects
- Adam short distance (zoom ×1) at frontal view and at 75 degree angle, absolute tilt t = 4 (middle), <4 (left), >4 (right)
Not shown Harris-Affine: 3 matches. Hessian-Affine: 1 match
| ASIFT: 202 matches | SIFT: 15 matches | MSER: 5 matches |
|---|---|---|
|
|
|
- Adam short distance (zoom ×10) at frontal view and at 65 degree angle, absolute tilt t = 2.4
Not shown Harris-Affine: 3 matches. Hessian-Affine: 0 match.
| ASIFT: 341 matches | SIFT: 5 matches | MSER: 4 matches |
|---|---|---|
|
|
|
- Adam short distance (zoom ×10) at frontal view and at 80 degree angle, absolute tilt t = 5.8
Not shown Harris-Affine: 1 match. Hessian-Affine: 0 match.
| ASIFT: 75 matches | SIFT: 1 matches | MSER: 2 matches |
|---|---|---|
|
|
|
- magazine middle distance (zoom ×4) at frontal view and at 80 degree angle, absolute tilt t = 5.8
Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
| ASIFT: 349 matches | SIFT: 0 matches | MSER: 17 matches |
|---|---|---|
|
|
|
- magazine absolute tilt t1= t2= 2, with longitude angles phi 1= 0 deg, phi2= 50 deg, transition tilt t = 3
Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
| ASIFT: 881 matches | SIFT: 3 matches | MSER: 87 matches |
|---|---|---|
|
|
|
- magazine absolute tilt t1= t2= 4, with longitude angles phi1= 0 deg, phi2= 90 deg, transition tilt t = 16
Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
| ASIFT: 88 matches | SIFT: 1 matches | MSER: 9 matches |
|---|---|---|
|
|
|
- facade frontal view and at 75 degree angle, absolute tilt t = 3.8
Not shown SIFT: 0 match. Harris-Affine: 1 match.
| ASIFT: 68 matches |
|
|---|---|
| Hessian-Affine: 1 matches |
|
| MSER: 2 matches |
|
- graffiti no.1 vs no. 6 (images from K. Mikolajczyk), transition tilt t ~ 3.2
Not shown SIFT: 0 match. Hessian-Affine: 1 match.
| ASIFT: 721 matches |
|
|---|---|
| Harris-Affine: 3 matches |
|
| MSER: 70 matches |
|
- direction transition tilt t ~ 2.6
Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
| ASIFT: 50 matches |
|
|---|---|
| SIFT: 0 match |
|
| MSER: 1 match |
|
- parkings transition tilt t ~ 15
Not shown Harris-Affine: 0 match. Hessian-Affine: 0 match.
| ASIFT: 70 matches |
|
|---|---|
| SIFT: 0 match |
|
| MSER: 0 match |
|
- stump transition tilt t ~ 2.6
Not shown Harris-Affine: 2 matches. Hessian-Affine: 1 match.
| ASIFT: 168 matches | SIFT: 1 match | MSER: 6 matches |
|---|---|---|
|
|
|
Monuments and constructions
- Notre Dame transition tilt t ~ 1.4
Not shown SIFT: 6 matches. Harris-Affine: 2 matches.
| ASIFT: 28 matches | Hessian-Affine: 8 matches | MSER: 17 matches |
|---|---|---|
|
|
|
- round building transition tilt t ~ [1.8, inf.)
Not shown Harris-Affine: 5 matches. Hessian-Affine: 7 matches.
| ASIFT: 139 matches | SIFT: 19 matches | MSER: 13 matches |
|---|---|---|
|
|
|
- palace of Versailles transition tilt t ~ 1.8
Not shown Harris-Affine: 2 matches. Hessian-Affine: 1 match.
| ASIFT: 67 matches | SIFT: 26 matches | MSER: 4 matches |
|---|---|---|
|
|
|
- École Polytechnique pictures at frontal view and at 65 degree angle, absolute tilt t = 2.4
Not shown Harris-Affine: 2 matches. Hessian-Affine: 2 matches.
| ASIFT: 101 matches | SIFT: 13 matches | MSER: 4 matches |
|---|---|---|
|
|
|
3D objects
- Statue of Liberty t ~ [1.3, inf.)
Not shown Hessian-Affine: 1 match. MSER: 1 match.
| ASIFT: 32 matches | SIFT: 9 matches | Harris-Affine: 1 match |
|---|---|---|
|
|
|
- statue transition tilt t ~ [1.6, inf.)
Not shown Harris-Affine: 7 matches. Hessian-Affine: 2 matches.
| ASIFT: 67 matches | SIFT: 26 matches | MSER: 4 matches |
|---|---|---|
|
|
|
Not shown Harris-Affine: 0 match. MSER: 1 match.
| ASIFT: 28 matches | SIFT: 1 match | Harris-Affine: 1 match |
|---|---|---|
|
|
|
- can transition tilt t ~ [2.3, inf.)
Not shown SIFT: 0 match. Hessian-Affine: 3 matches.
| ASIFT: 287 matches |
|
|---|---|
| Harris-Affine: 6 matches |
|
| MSER: 22 matches |
|
Complex scenes
- office transition tilt t ~ 3
Not shown SIFT: 0 match. Harris-Affine: 0 match.
| ASIFT: 88 matches |
|
|---|---|
| Hessian-Affine: 1 match |
|
| MSER: 3 matches |
|
- bottles proposed by the authors of MSER, transition tilt t ~ [1.6, 3.0]
Not shown SIFT: 10 matches. Hessian-Affine: 11 matches.
| ASIFT: 254 matches |
|
|---|---|
| Harris-Affine: 23 matches |
|
| MSER: 22 matches |
|
- coffee room transition tilt t ~ [1.5, 3.3]
Not shown Harris-Affine: 0 match. Hessian-Affine: 3 matches.
| ASIFT: 125 matches |
|
|---|---|
| SIFT: 13 matches |
|
| MSER: 5 matches |
|
Object deformation
Images from Ling and Jacobs.
Not shown Harris-Affine: 6 matches. MSER: 4 matches.
| ASIFT: 370 matches | SIFT: 75 matches | Harris-Affine: 8 matches |
|---|---|---|
|
|
|
Not shown Harris-Affine: 25 matches. MSER: 17 matches.
| ASIFT: 528 matches | SIFT: 160 matches | Hessian-Affine: 55 matches |
|---|---|---|
|
|
|
Not shown Hessian-Affine: 10 matches. MSER: 2 matches.
| ASIFT: 141 matches | SIFT: 31 matches | Harris-Affine: 15 matches |
|---|---|---|
|
|
|
Not shown Harris-Affine: 45 matches (1 on the cloth). Hessian-Affine: 22 matches (1 on the cloth).
| ASIFT: 659 matches | SIFT: 304 matches | MSER: 35 matches |
|---|---|---|
|
|
|
Not shown Harris-Affine: 0 match. MSER: 0 match.
| ASIFT: 33 matches |
|
|---|---|
| SIFT: 4 matches |
|
| Hessian-Affine: 1 match |
|
Failure Cases
Day-and-night Illumination Change
All methods fail!
Six images of Notre-Dame under different illumination conditions are compared. The number of matches of ASIFT and SIFT are shown. (Harris-Affine, Hessian-Affine and MSER find less matches than SIFT.) Little view angle change is presented. The red arrows imply recognition failure.
In general, matching succeeds between day images and between night images. However, under day-and-night illumination change, all methods fail.
Description:
- All methods fail under day-and-night illumination change (pairs 1-2, 1-6, 2-3, 2-4, 2-5, 3-6, 4-6, 5-6).
- Matching succeeds between night images, with a substantial scale change (pair 2-6).
- Matching succeeds between day images under same weather condition, with or without a substantial scale change (pairs 1-4, 3-5).
- Matching succeeds between day images under different weather conditions, without a substantial scale change (pairs 1-3, 3-4).
- Matching fails between day images under different weather conditions, with a substantial scale change (pairs 1-5, 4-5).
Strong Relief Effect
All methods fail!
ASIFT, SIFT, Harris-Affine, Hessian-Affine, MSER: 0 match.
The image on the right in close-up shows strong relief effect.
Repetitive shapes
"Good" false matches.
Matches can be arbitrary among repetitive shapes.
ASIFT: 169 matches, many are "good" false matches (for example the matches between the left-most window in the image above and the second window in the image below).
Due to the viewpoint change, SIFT, Harris-Affine, Hessian-Affine and MSER find much less matches (respectively 30, 4, 9 and 4), among which many are "good" false matches as well.






















































































