Dartmouth CS 89.15/189.5, Fall 2015
Computational Aspects of Digital Photography

Assignment 6: Resampling, Warping & Morphing

Resampling

In this section, we will scale and rotate images, starting with simple transformations and naïve reconstruction. Below we show an example of scaling with various methods, and a small crop of the resulting image to highlight the differences.

original
Original Image
Scaling by 3.5 with nearest neighbor interpolation
Scaling by 3.5 with nearest neighbor interpolation
Scaling by 3.5 with nearest neighbor interpolation using scaleNN
Scaling by 3.5 with bilinear interpolation
Scaling by 3.5 with bilinear interpolation
Scaling by 3.5 with bilinear interpolation using scaleLin
Scaling by 3.5 with bicubic interpolation
Scaling by 3:5 with bicubic interpolation
Scaling by 3:5 with bicubic interpolation using scaleBic
Figure 1: Resampling with nearest neighbor, bilinear and bicubic interpolation.

Basic scaling with nearest-neighbor

We first consider scaling the image by a constant scale factor.

Scaling with bilinear interpolation

Nearest-neighbor reconstruction leads to blocky artifacts and pixelated results. You will address this using a better reconstruction based on bilinear interpolation. For this, consider the four pixels around the computed coordinate location and perform bilinear reconstruction by first performing two linear interpolations along \(x\) for the top and bottom pairs of pixels, and then another interpolation along \(y\) for the two results. In each case, the interpolation weights are given by the fractional \(x\) and \(y\) coordinates.

Rotations (required for graduate credit, or for undergraduates 5% extra credit)

original
rotated image
Original Image
Rotated Image
Figure 2: Example rotation by 3.14159/4 using rotate.

Extra credit (5%): Bicubic or Lanczos

You can obtain better reconstruction by considering more pixels and using smarter weights, such as that given by a bicubic or Lanczos functions.

Warping and morphing

In what follows, you will implement image warping and morphing according to Beier and Neely's method, which was used for special effects such as those of Michael Jackson's Black or White music video. We highly recommend that you read the provided original article, which is well written and includes important references such as Ghost Busters: Beier, Thaddeus, and Shawn Neely. "Feature-based image metamorphosis." ACM SIGGRAPH Computer Graphics. Vol. 26. No. 2. ACM, 1992.

The full method for warping and morphing includes a number of technical components and it is critical that you debug them as you implement each individual one.

UI

We provide you with a simple (to say the least) interface to specify segment location from a web browser. It is based on javascript and the raphael library (http://raphaeljs.com/). Copy click.html and raphael-min.js to a location with your two input images. The two images should have the same size. For the warp part, you can use the same image on both sides. To change the image, modify the beginning of click.html and update imagewidth, imageheight, path1 and path2 according to your images.

You must click on the segment endpoints in the same order on the left and on the right. Unfortunately, you cannot edit the locations after you have clicked.

Yes, this is primitive. Once you are done, simply copy the C++ code below each image into your main function to create the corresponding segments.

segments before
segments after 
vector<Segment> segsBefore;
segsBefore.push_back(Segments(56,11,69,90));
segsBefore.push_back(Segments(97,98,119,104));

vector<Segment> segsAfter;
segsAfter.push_back(Segments(61,103,78,76));
segsAfter.push_back(Segments(93,93,114,101));

Segments

We will start by implementing the Segment class, which is critical to warping. Test these methods thoroughly as they will be used in warping and morphing functions below.

Warping according to one pair of segments

The core component is a method to transform an image according to the displacement of a segment.

segment before
warpBy1(im, Segment(0, 0, 10, 0), Segment(10, 10, 30, 15))

segment after

You can use the javascript UI to specify the segments, using the same image on both side for reference.

Warping according to multiple pairs of segments

In this section, you will extend you warp code to perform transformations according to multiple pairs of segments. For each pixel, transform its 2D coordinates according to each pair of segments and take a weighted average according to the length of each segment and the distance of the pixel to the segments. Specifically, the weight is given according to Beier and Neely: \[ \text{weight} = {\left( \frac{\text{length}^p}{a + \text{dist}} \right)}^b \] where \(a, b, p\) are parameters that control the interpolation. In our test, we have used \(b = p = 1\) and \(a\) equal to roughly 5% of the image.

Use the provided javascript UI to specify segments. The points must be entered in the same order on the left and right image. You can then copy-paste the C++ code generated below the images to create the corresponding segment objects.

Morphing

Given your warping code, you will write a function that generates a morph sequence between two images. Again, make sure you are familiar with morphing from the lecture slides and the article.

You are given the two images and a list of corresponding segments for each image. You must generate N images morphing from the first input image to the second input image.

For each image, compute its corresponding time fraction t. Then linearly interpolate the position of each segment endpoint according to t, where t = 0 corresponds to the position in the first image and t = 1 is the position in the last image. You might want to visualize the results for debugging.

You now need to warp both the first and last image so that their segment locations are moved to the segment location at t, which will align the features of the images. We suggest that you write these two images to disk and verify that the images align and that, as t increases, the images get warp from the configuration of the first image all the way to that of the last one.

Finally, for each t, perform a linear interpolation between the pixel values of the two warped images.

Your function should return a sequence of images. For debugging you can use your main function to write your images to disk using a sequence of names such as morph_1.png, morph_2.png, ...

first image
second image
third image
Visualize Results Aside from just looking at the images, you can explore your results at http://tipix.csail.mit.edu if you wish. Click on load and load your morphed images (make sure to respect the required naming). Then explore your results using the pointer, or click "play" from the top-right information panel.

There are several ways to make a video out of your files (note this is not required). You can install ffmpeg but this involves some number of dependencies. You can use it as follows: 

ffmpeg -r 20 -b 1800 -y -i morph%03d.png out.mp4
A windows-only alternative appears to be http://home.hccnet.nl/s.vd.palen/index.html but we haven't tried it. See also: http://www.videohelp.com/tools/Virtualdub. MATLAB also has some decent tools for this.

Class morph

Create a new function classMorph() in a6_main.cpp that uses your morphing code to morph between your face and that of the next student in the list. Turn in 15 PNG frames of size \(400 \times 400\).

Extra credit

Here are ideas for extensions you could attempt, for 5% each. At most, on the entire assignment, you can get 10% of extra credit:

Submission

Turn in your files using Canvas and make sure all your files are in the a6 directory under the root of the zip file. Include all sources (.cpp and .h) files, any of your images, and the output of your program. Don't include your actual executable (we don't need your _build directory), and remove any superfluous files before submission.

In your readme.txt file, you should also answer the following questions:

Acknowledgments: This assignment is based off of one designed by Frédo Durand, Katherine L. Bouman, Gaurav Chaurasia, Adrian Vasile Dalca, and Neal Wadhwa for MIT's Digital & Computational Photography class.