Playing with Disparity

Pictures and videos from the plane look cool, but that's all if until we do something with them. And what do we want to do with them? We want 3D reconstruction!

Sounds scary. It is. It takes a lot of math to do 3D reconstruction with epipolar geometry, so we're going to start slow and first play around with disparity map from stereo images.

Nothing extravagant for now, everything done here with python, opencv for python and numpy.

Step 1: take two cameras, take a stereo image pair and take a picture of a checkerboard with each one for calibration.

Step 2: detect the edges (hint: cv2.findChessboardCorners).

Step 3: calibrate the camera and fix the distortion caused by the lens (hint: cv2.calibrateCamera, cv2.getOptimalNewCameraMatrix, cv2.initUndistortRectifyMap).

Step 4: find detectable features on both images, here we're going to use SIFT algorithm  (hint: cv2.SIFT()).

Step 5: find the same matching features in both images, here we're going to use the FLANN algorithm (hint: cv2.FlannBasedMatcher, cv2.FlannBasedMatcher.knnMatch).

Step 6: compute epipolar lines between pictures (hint: cv2.findFundamentalMat, cv2.computeCorrespondEpilines).

Step 7: transform the images so the matching lines will be horizontal with each other between images (hint: cv2.stereoRectifyUncalibrated, cv2.warpPerspective).

Step 8: calculate disparity between two stereo images (hint: cv2.StereoSGBM, cv2.StereoSGBM.compute).

So much for the first try. The disparity map is noisy and not really accurate on the account of uncalibrated stereo rectification.

To improve that we need to perform calibrated stereo rectification, and we'll be able to do that when we figure out the way to get rotation and translation data between stereo sets from epipolar geometry.

Also, a great resource of learning opencv and python: https://github.com/abidrahmank/OpenCV2-Python-Tutorials