Acceleration methods for image super-resolution


When designing and deploying computer vision systems, engineers are often confronted with the limits of available cameras with respect to the image quality they can provide. Similar to other sensing devices, cameras alter the measured signal and often yield images representing a degraded version of the original scene. Degradation sources include blur induced by such factors as the lens and sensors, frequency aliasing due to subsampling, signal quantization and circuitry reading noise.

The consequence is that the fine details in a scene are often lost during the image acquisition process. On the other hand, the importance of digital imaging technologies in consumer and other markets motivates the need for higher quality images. In particular, high-resolution (HR) images are desired and often required since they can provide details that are critical for subsequent analysis in many applications.

To obtain higher camera resolution, one needs to increase the number of pixel sensors on the camera chip, which can be achieved by either reducing the size of the sensors or increasing the size of the chip. Unfortunately, the application of such solutions is limited due to constraints imposed by physics and image sensor technology. One such constraint is shot noise, which becomes more significant as the size of the pixel sensors decreases. Also, the higher level of capacitance associated with larger chips makes it difficult to accelerate the charge transfer rate in such structures. Finally, although high precision image sensors and optical components are sometimes available, they may also be prohibitively expensive for general purpose commercial applications.

Due to the previous limitations, there is a growing interest in image processing algorithms capable of overcoming the resolution limits of imaging systems. Super-resolution (SR) refers to techniques for synthesizing a HR image or video sequence from a set of degraded and aliased low-resolution (LR) ones. Generally, this is done by exploiting knowledge of the relative subpixel displacements of each LR image with respect to a reference frame. In situations where high quality vision systems cannot be incorporated or are too expensive to deploy, such algorithms may constitute an alternative and allow for the recovery of high quality images from more affordable imaging hardware.

Super-resolution is generally formulated as an optimization problem, in which the HR image sought is computed by iteratively improving an initial solution until the desired level of convergence is achieved. The efficiency of the restoration process is determined by the convergence rate of the iterative method employed, which, in turn, depends on the SR problem formulation.

In this project, we are interested in the development of fast SR algorithms. The motivation comes from the fact that the computational complexity associated with many SR algorithms may hinder their use in time-critical applications. In particular, we developed an efficient method for accelerating computations associated with edge-preserving image SR problems in which a linear shift-invariant (LSI) point spread function (PSF) is employed. Our technique is suitable for SR scenarios where the motion between the observed LR images is translational. It is also possible to accelerate SR algorithms employing rational magnification factors. The use of such factors is motivated in part by the work of Lin and Shum suggesting that, under certain circumstances, optimal magnification factors for SR are non-integers.

Related Work

The super-resolution literature is very rich and good summaries of existing methods are given by Borman and Stevenson and Park et al.. Special issues on super-resolution edited by Kang and Chaudhuri, Ng et al. and Hardie et al. were also published recently. Finally, different topics related to super-resolution, including motionless super-resolution, are discussed in two books by Chaudhuri.

To accelerate SR computations, some methods shift and interpolate all LR frames on a single HR grid, and then apply an iterative deblurring process to this temporary image in order to synthesize the HR image. However, the optimality of the HR images obtained in this manner is difficult to verify, since the interpolation process introduces errors in the input data. Nonetheless, when the motion model employed is translational, Elad and Hel-Or proposed a technique for fusing the LR images in a way that preserves the optimality of the SR reconstruction process in the maximum-likelihood sense. This optimality relies on the assumption that the translations between the LR frames are discretized on the HR grid. These authors assume that rounding the displacement values has a negligible effect on image quality, since it occurs on the HR grid. Farsiu et al. later adapted this method to the robust SR case. Tanaka and Okutomi also exploited such a discretization of the displacements between the observed pixels in order to accelerate computations in SR problems. Their method groups LR pixels whose positions are the same after discretization and computes the average of the pixel values in each group. The computations are then performed on this reduced set of averaged values instead of the original data.

However, such discretizations can produce undesirable artifacts in the restored HR image, particularly when small magnification factors are employed. Unlike these techniques, the proposed method does not sacrifice the optimality of the SR problem formulation when noninteger displacements are present between the LR images. When both the computational speed and the observation model accuracy are deemed important, our method thus constitutes a valuable approach.


We conduct a few restoration experiments using a set of 235 Bayer CFA images captured with a Point Grey Flea2 digital camera. During video acquisition, the camera is slightly rotated manually about its vertical and horizontal axes, which results in small translational motion between the LR frames. The resolution of this set of images is increased using magnification factors of two, three and four, as shown below.

In the plots below, we compare the performance of the PNCG and NCG methods under different conditions by varying the number of LR images used as input. For each experiment, we indicate the number of PNCG and NCG iterations required to solve the problem, along with the computation time. Note that for both measurements, each number corresponds to the sum of the values for all color channels. To facilitate comparisons, the ratio of the number of iterations for the PNCG method to that of the NCG method is indicated for each experiment above the associated pair of bars in the plots on the left. Similarly, the ratio of the computation time for the PNCG method to that of the NCG method is provided in the plots on the right.

Magnification factor of two

Magnification factor of three

Magnification factor of four

We now show that rounding the shift values can have a strong negative impact on the visual quality of the HR image. For this purpose, we repeat the SR experiments presented above; this time, however, we round all displacement values of the LR frames on the HR grid prior to SR restoration. The figure below shows a comparison of the two approaches. For better visualization, we only show a region of interest in the restored images.

By comparing corresponding images in both sets of experiments, one can readily observe that the HR images synthesized after rounding the shift values contain more artifacts than those obtained without altering these displacements. Furthermore, such undesirable effects are more visible when smaller magnification factors are employed. Note that one possibility to reduce the visual impact of such artifacts is to employ a one-norm penalty function in the data consistency term, which is known to be more robust to misregistered LR frames. Although this might be an acceptable approach in the case where the errors in the motion vectors stem from limitations of the image registration algorithm, we argue that this solution is suboptimal when such errors result from a deliberate deterioration of the registration parameter values by enforcing the nearest neighbor displacement paradigm.


Image datasets

Image datasets for performing super-resolution are available here.

Last update: 5 November 2009