• algorithm
• demonstration
• archive

Contacts

• Jean-Michel Morel morel[AT]cmla.ens-cachan.fr
• Guoshen Yu yu[AT]cmap.polytechnique.fr

References

1. J.M. Morel and G.Yu, ASIFT: A New Framework for Fully Affine Invariant Image Comparison.
   SIAM Journal on Imaging Sciences, 2(2):438-469, 2009. preprint
2. G. Yu and J.M. Morel, A Fully Affine Invariant Image Comparison Method.
   Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Taipei, 2009. preprint
3. J.M. Morel and G.Yu, On the consistency of the SIFT Method. Preprint, CMLA 2008-26, Sept 2008. preprint

Overview

A fully affine invariant image comparison method, Affine-SIFT (ASIFT) is introduced. While SIFT is fully invariant with respect to only four parameters namely zoom, rotation and translation, the new method treats the two left over parameters : the angles defining the camera axis orientation.

Against any prognosis, simulating all views depending on these two parameters is feasible. The method permits to reliably identify features that have undergone very large affine distortions measured by a new parameter, the transition tilt.

State-of-the-art methods hardly exceed transition tilts of 2 (SIFT), 2.5 (Harris-Affine and Hessian-Affine) and 10 (MSER). ASIFT can handle transition tilts up 36 and higher.

When Does it Work?

The SIFT method works to compare 2D objects or 3D objects with flat enough details, taken from similar view angles but at arbitrary distances.

The typical failure cases are:

• The illumination conditions are different (for instance daylight/nightlight).
• The object has a reflecting surface (typically cars are mirrors ; they change completely aspect under different view angles).
• The object has a strong 3D structure: in that case a change of view angle alters drastically its aspect.
• The object has a self similar or periodic structure: then "true" mismatches occur.
• The view angle is too different.

See our failure case study.

ASIFT corrects the last problem : if the object is under view has similar illumination conditions, has rather flat surface, and is not a mirror, then ASIFT retrieves the object even under extreme changes of angle. In technical terms, ASIFT is more affine invariant than SIFT.

Other popular methods addressing this problem exist: MSER and Hessian-Affine and Harris-Affine. All these methods will be compared below.

Additional Materials

• Slides : one-hour version, 20 minutes version, videos (zip, tar/gz)
• Poster

Online Demo

An online demo that allows you to try ASIFT with your own images is available here. If you are interested in ASIFT and want to try it with your own images, we suggest that you start with the user friendly online demo.

Software

The ASIFT software (version beta) allows you to try ASIFT locally on your machine.

• reference source code (for megawave) : zip tar/gz
• binary
  • for Linux
  • ?for Windows (Matlab interface included)
    the ASIFT software for Windows is developed with contributions from Pierre Moulon.

Dataset

An image dataset for systematic evaluation of robustness to absolute and transition tilt of the image matching algorithms is available : zip tar/gz.

Examples

ASIFT is compared with the four state-of-the-art algorithms the SIFT, Harris-Affine, Hessian-Affine and MSER detectors, all coded with the SIFT descriptor. Various types of images (size 600×450) were used for the experiments. The SIFT software is from D. Lowe. The Harris-Affine, Hessian-Affine and MSER programs are from the web site of K. Mikolajczyk.

• video tracking
• planar objects (paintings, journals, walls, etc.)
• monuments and constructions
• 3D objects (statues, cars, etc)
• complex scenes
• object deformation

In the examples below, the powerful Moisan-Stival procedure is applied to eliminate matches incoherent with epipolar geometry. Matches are connected by white segments.

Video tracking

ASIFT is compared with SIFT on video tracking . In each video, ASIFT and SIFT tracking are shown respectively on the top and bottom. If you prefer high-resolution video, please download the video files by clicking the links below.

facade (youtube page, high-resolution video)	magazine (youtube page, high-resolution video)

Planar objects

• Adam short distance (zoom ×1) at frontal view and at 75 degree angle, absolute tilt t = 4 (middle), <4 (left), >4 (right)

• Adam short distance (zoom ×10) at frontal view and at 65 degree angle, absolute tilt t = 2.4

• Adam short distance (zoom ×10) at frontal view and at 80 degree angle, absolute tilt t = 5.8

• magazine middle distance (zoom ×4) at frontal view and at 80 degree angle, absolute tilt t = 5.8

• magazine absolute tilt t1= t2= 2, with longitude angles phi 1= 0 deg, phi2= 50 deg, transition tilt t = 3

• magazine absolute tilt t1= t2= 4, with longitude angles phi1= 0 deg, phi2= 90 deg, transition tilt t = 16

• facade frontal view and at 75 degree angle, absolute tilt t = 3.8

• graffiti no.1 vs no. 6 (images from K. Mikolajczyk), transition tilt t ~ 3.2

• direction transition tilt t ~ 2.6

• parkings transition tilt t ~ 15

• stump transition tilt t ~ 2.6

Monuments and constructions

• Notre Dame transition tilt t ~ 1.4

• Pentagon

• round building transition tilt t ~ [1.8, inf.)

• palace of Versailles transition tilt t ~ 1.8

• École Polytechnique pictures at frontal view and at 65 degree angle, absolute tilt t = 2.4

3D objects

• Statue of Liberty t ~ [1.3, inf.)

• statue transition tilt t ~ [1.6, inf.)

• car

• can transition tilt t ~ [2.3, inf.)

Complex scenes

• office transition tilt t ~ 3

• bottles proposed by the authors of MSER, transition tilt t ~ [1.6, 3.0]

• coffee room transition tilt t ~ [1.5, 3.3]

Object deformation

Images from Ling and Jacobs.

• Sponge Bob

• CVPR

• flag

• girl

• toy

Failure Cases

Day-and-night Illumination Change

All methods fail!

Six images of Notre-Dame under different illumination conditions are compared. The number of matches of ASIFT and SIFT are shown. (Harris-Affine, Hessian-Affine and MSER find less matches than SIFT.) Little view angle change is presented. The red arrows imply recognition failure.

In general, matching succeeds between day images and between night images. However, under day-and-night illumination change, all methods fail.

Description:

• All methods fail under day-and-night illumination change (pairs 1-2, 1-6, 2-3, 2-4, 2-5, 3-6, 4-6, 5-6).
• Matching succeeds between night images, with a substantial scale change (pair 2-6).
• Matching succeeds between day images under same weather condition, with or without a substantial scale change (pairs 1-4, 3-5).
• Matching succeeds between day images under different weather conditions, without a substantial scale change (pairs 1-3, 3-4).
• Matching fails between day images under different weather conditions, with a substantial scale change (pairs 1-5, 4-5).

Strong Relief Effect

All methods fail!

ASIFT, SIFT, Harris-Affine, Hessian-Affine, MSER: 0 match.

The image on the right in close-up shows strong relief effect.

Repetitive shapes

"Good" false matches.

Matches can be arbitrary among repetitive shapes.

ASIFT: 169 matches, many are "good" false matches (for example the matches between the left-most window in the image above and the second window in the image below).

Due to the viewpoint change, SIFT, Harris-Affine, Hessian-Affine and MSER find much less matches (respectively 30, 4, 9 and 4), among which many are "good" false matches as well.