Abstract

Scale invariant feature detectors often find stable scales in only a few image pixels. Consequently, methods for feature matching typically choose one of two extreme options: matching a sparse set of scale invariant features, or dense matching using arbitrary scales. In this paper we turn our attention to the overwhelming majority of pixels, those where stable scales are not found by standard techniques. We ask, is scale-selection necessary for these pixels, when dense, scale-invariant matching is required and if so, how can it be achieved? We make the following contributions: (i) We show that features computed over different scales, even in low-contrast areas, can be different; selecting a single scale, arbitrarily or otherwise, may lead to poor matches when the images have different scales. (ii) We show that representing each pixel as a set of SIFTs, extracted at multiple scales, allows for far better matches than singlescale descriptors, but at a computational price. Finally, (iii) we demonstrate that each such set may be accurately represented by a low-dimensional, linear subspace. A subspaceto-point mapping may further be used to produce a novel descriptor representation, the Scale-Less SIFT (SLS), as an alternative to single-scale descriptors. These claims are verified by quantitative and qualitative tests, demonstrating significant improvements over existing methods.

Reference (Extended journal version): Tal Hassner, Shay Filosof, Viki Mayzels, and Lihi Zelnik-Manor, SIFTing through Scales, IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), 39(7): 1431-1443 (2017)
PDF
BibTeX

Reference (conference paper): Tal Hassner, Viki Mayzels, and Lihi Zelnik-Manor, On SIFTs and their Scales, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Rhode Island, June 2012
PDF
BibTeX

• (13 June 2016): Extended journal version accepted to IEEE-TPAMI. See above for reference details.

• Followup project on dense correspondences between scenes and scales has been posted, including code and related documents.

Slides from the ICCV’13 tutorial on Dense Image Correspondences for Computer Vision, Sydney Australia, Dec. 2013. (PDF)

Left input image	Right input image
Warp with DSIFT	Warp+crop with our SLS

Dense matches of different objects in different scales. Top: Left and Right input images. Bottom: Left image warped onto Right using the recovered flows: Using DSIFT (bottom left) and our Scale-Less SIFT (SLS) descriptor (bottom right), cropped to the region of confident matches and overlaid on the Right image. DSIFT fails to capture the scale differences and produces an output in the same scale as the input. Our SLS descriptor captures scale changes at each pixel: the output produced by using SLS has the appearance of the Left image in the scale and position of the Right. Please see text for further details.

SLS distribution

MATLAB code for computing dense Scale-Less SIFT descriptors (SLS) is now available for download here. The distribution contains also a demo script for establishing dense correspondences between two images of different scenes in different scales using the SLS descriptors.

Installing and running

1. The code uses the vlfeat distribution to compute SIFT descriptors and the SIFT-Flow release for computing dense correspondences. Please follow these instructions to set these up before running our code.

• Download the following software to your computer and modify the following lines in demo and compute_dense_flow

• The vl_feat code from http://www.vlfeat.org/ In the line “addpath vlfeat-0.9.14/toolbox” - replace “vlfeat-0.9.14” with the name of the folder containing the vl_feat code.

• The Sift-flow code from http://people.csail.mit.edu/celiu/ECCV2008/ In the lines “addpath release” and “addpath release/mex” - replace “release” with the name of the folder containing the SIFT-flow code.

1. To run the algorithm on an example image pair, included in the distribution, simply run the script demo.m. To use your own images or to modify parameters, see their help:
   help demo
   help extract_scalelessdescs

2. Alternatively, to run the demo as a function, you can call: [flo,wrp,clrflo] = compute_dense_flow(Im_Source, Im_Target) wrp provides warped output images such as those shown in the paper. For more information, see help compute_dense_flow

3. Please note that this code was not designed for optimal running times or memory usage. It may therefore require quite a lot of time to produce multiple SIFTs for each pixel and compute their subspaces. The same is true for the memory required for the descriptors at each pixel.

Download

Please cite our paper if you use this code for your research.
11-June-2012, Version 1.1: sls_distribution.1.1.zip (36k)

What’s new

• The code now includes an option to compute SLS descriptors with PCA reduced SIFTs. This produces SLS descriptors which are an order of magnitude smaller than those used in the paper, while still providing high quality dense flows. A more detailed description of this addition will be described in an upcoming paper.

• See the help for extract_scalelessdescs for more information on the new usage options.

• 31-May-2012, Version 1.0: sls_distribution.1.0.zip (36k)

Contact and questions

For questions or concerns, please write talhassner@gmail.com

Copyright and disclaimer
Copyright 2012, Tal Hassner, Viki Mayzels, Shay Filosof, and Lihi Zelnik-Manor On SIFTs and Their Scales