Testing Structure From Motion Software

So, while we work on improving our implementation of image rectification and beyond, let's see what kind of results we can expect from the data we have.

For that, we're going to use a couple of freely available reconstruction tools that already work.

First one is VisualSFV: http://ccwu.me/vsfm/

A GUI application for 3D reconstruction using structure from motion (SFM) from Changchang Wu.

For that we took our stereo video from the last time. Extracted images from the left camera at 5 images per second (extracting images from both cameras was just too many images, and we still don't have them rectified). Used OpenCV and a calibration board to fix the lens distortion and ran it trough VisualSFM.

The result being sparse and dense reconstruction:

And the output in MeshLab:

Once we had the dense reconstruction, we were able to export it from VisualSFM in .cmp format, which can be used as input for the next tool.

Second one is CMPMVS: http://ptak.felk.cvut.cz/sfmservice/websfm.pl?menu=cmpmvs

A multi-view reconstruction software.

So after letting it do all the work, we ended up with the model:

The results are promising, but there  is a lot of noise in the reconstruction. Looked closely there are differences in disparity where there are houses, but just. The details like cars are lost.

Looking at single video frames, the images still look slightly blurry. Even though the plane vibrates a lot less now, maybe taking images for reconstruction as video is not the best method.

Time to take some real photos and try with that.

Playing with Disparity

Pictures and videos from the plane look cool, but that's all if until we do something with them. And what do we want to do with them? We want 3D reconstruction!

Sounds scary. It is. It takes a lot of math to do 3D reconstruction with epipolar geometry, so we're going to start slow and first play around with disparity map from stereo images.

Nothing extravagant for now, everything done here with python, opencv for python and numpy.

Step 1: take two cameras, take a stereo image pair and take a picture of a checkerboard with each one for calibration.

Step 2: detect the edges (hint: cv2.findChessboardCorners).

Step 3: calibrate the camera and fix the distortion caused by the lens (hint: cv2.calibrateCamera, cv2.getOptimalNewCameraMatrix, cv2.initUndistortRectifyMap).

Step 4: find detectable features on both images, here we're going to use SIFT algorithm  (hint: cv2.SIFT()).

Step 5: find the same matching features in both images, here we're going to use the FLANN algorithm (hint: cv2.FlannBasedMatcher, cv2.FlannBasedMatcher.knnMatch).

Step 6: compute epipolar lines between pictures (hint: cv2.findFundamentalMat, cv2.computeCorrespondEpilines).

Step 7: transform the images so the matching lines will be horizontal with each other between images (hint: cv2.stereoRectifyUncalibrated, cv2.warpPerspective).

Step 8: calculate disparity between two stereo images (hint: cv2.StereoSGBM, cv2.StereoSGBM.compute).

So much for the first try. The disparity map is noisy and not really accurate on the account of uncalibrated stereo rectification.

To improve that we need to perform calibrated stereo rectification, and we'll be able to do that when we figure out the way to get rotation and translation data between stereo sets from epipolar geometry.

Also, a great resource of learning opencv and python: https://github.com/abidrahmank/OpenCV2-Python-Tutorials