Playing with Disparity II

So, learning from previous attempt, let's try again.

This time, with better calibration. Turns out just taking many pictures of the calibration chequerboard is not enough. At least in this case adding two more odd angles improved the lens distortion correction:

 While the individual horizontal and vertical lines on the left image are straight, they are not perpendicular to each other. A property preserved with better calibration images.

Next, improved stereo image rectification. Very important for epipolar geometry is the computation of a good fundamental matrix. Turns out, that just using the best weighted features is not enough. Some manual checking that they are distributed throughout the pictures does wonders:

The bad and good example, the difference a couple of different points used in the fundamental matrix calculation can make.

All this produces a more correct disparity image, where the benches are before the trees and the trees are straight:

And now, that we have a good calibration of intrinsic parameters and a better fundamental matrix, we can calculate the existential matrix and place each camera in to the global coordinate system. And from there we can do 3D reconstruction:

Matched feature points on both images used in the reconstruction. Note the points on the trees, the bench and the table in the background behind the trees.

Camera view of the reconstructed points in 3D space. We can make out the trees, the bench and the background table.

But the reconstruction becomes apparent when we shift the view to the top down perspective. Here we can see the distance and the relative placement between the bench, the trees and the background table.

Testing Structure From Motion Software II

Let's try to get some real data with some real pictures.

On the last flight, lasting 5 minutes 33 seconds a total of 134 pictures were taken (67 with each camera).

Out of those, 38 were matched to a single scene in VisualSFM:

This time the results are much more better. The shape of the house or the vehicles parked outside can be easily distinguished.

And when combined with CMPMVS:

Also showing produced disparity map and generated ortophoto picture of the scene:

The results look good! Houses, roads, bushes, trees and even separate fields can be distinguished now.

But they could be better still. For one, while focus was locked, the exposure and ISO weren't and the cameras "decided" to take pictures at shutter speed 1/100 s with ISO 80. Manually decreasing the shutter speed and increasing ISO would produce even sharper images.

Secondly, decreasing the time between shots would mean more images and more overlapping matching points. Now the shots were taken at 5 second intervals. This could be reduced to 4 seconds for stereo images.

It could be even reduced down to 2 seconds between pictures if the cameras would take turn taking pictures, but then this would become purely structure from motion reconstruction (as it was here if we can't get the stereo registration to work properly).

Seventh Time in the Air

In in the air again to take some pictures.

The plan was simple. Utilize the battery (3 minutes full throttle or 9 minutes half throttle) to the fullest.

Start with 1 minute of full throttle to get some altitude, 66% or 6 minutes half throttle left. Flay a couple of overpasses for 4 minutes, 2 minutes left at half throttle. Land with remaining battery.

Sadly the plan didn't go quite as planned:

First take off attempt was aborted because one of the wheels snagged on the grass and turned off the runway (here were 10 seconds of full throttle lost). Then the plane didn't get as high as intended, so flying by the visible size, the plane was flown farther then intended (here it took longer to fly back). Add some wind higher up, so when trying to hold the plane as still as possible for the photos produced larger turns.

Long story short, 5 minutes and 33 seconds later the plane landed 3 meters from the runway because it ran out of power to run the motor 20 centimetres off the ground approaching the runway. But the new landing gear held. No pivot over this time, just some torn duct tape.

Much was learned today and if the pictures turn out useful, it was worth it.

And here is the scenic view:

AVI@TOR MK4 SP5

We need pictures! And for that the plane needs an upgrade.

First of, a better landing gear:

Made from a spare PCB board from another project as a base (strong and flexible). To which a set of foam and plastic wheels on flat carbon bracket were mounted.

Attached with duct tape, the landing gear work perfectly. The PCB base bends under load and if there is any excessive force applied it goes to the duct tape which breaks and tears but is easy replaceable.


Only the bent rod holding the wheels to the carbon bracket kept braking from the glue. Quick fix, reinforced with zip-ties instead of just glue for extra strength.

Now the plane is finally ready for the extra weight of the camera mount, without the need to fix the landing gear after every heavy landing.

And second, to take pictures we need to synchronise the cameras! Yes, the board to let the phone control the servos and the cameras (to synchronize GPS data with shots taken) is in the works, but for a quick test lets use what we have:

Just change the firmware from our rc2usb board to output a second long pulse every 5 seconds instead of the PWM for the servo tester.

Now cut up and strip down a couple of cheep mini USB cables to get the connectors, wire up the power lines to the pulse output on the board and run the wires trough the camera stand.

We are now ready to take some high resolution pictures!

Testing Structure From Motion Software

So, while we work on improving our implementation of image rectification and beyond, let's see what kind of results we can expect from the data we have.

For that, we're going to use a couple of freely available reconstruction tools that already work.

First one is VisualSFV: http://ccwu.me/vsfm/

A GUI application for 3D reconstruction using structure from motion (SFM) from Changchang Wu.

For that we took our stereo video from the last time. Extracted images from the left camera at 5 images per second (extracting images from both cameras was just too many images, and we still don't have them rectified). Used OpenCV and a calibration board to fix the lens distortion and ran it trough VisualSFM.

The result being sparse and dense reconstruction:

And the output in MeshLab:

Once we had the dense reconstruction, we were able to export it from VisualSFM in .cmp format, which can be used as input for the next tool.

Second one is CMPMVS: http://ptak.felk.cvut.cz/sfmservice/websfm.pl?menu=cmpmvs

A multi-view reconstruction software.

So after letting it do all the work, we ended up with the model:

The results are promising, but there  is a lot of noise in the reconstruction. Looked closely there are differences in disparity where there are houses, but just. The details like cars are lost.

Looking at single video frames, the images still look slightly blurry. Even though the plane vibrates a lot less now, maybe taking images for reconstruction as video is not the best method.

Time to take some real photos and try with that.

Playing with Disparity

Pictures and videos from the plane look cool, but that's all if until we do something with them. And what do we want to do with them? We want 3D reconstruction!

Sounds scary. It is. It takes a lot of math to do 3D reconstruction with epipolar geometry, so we're going to start slow and first play around with disparity map from stereo images.

Nothing extravagant for now, everything done here with python, opencv for python and numpy.

Step 1: take two cameras, take a stereo image pair and take a picture of a checkerboard with each one for calibration.

Step 2: detect the edges (hint: cv2.findChessboardCorners).

Step 3: calibrate the camera and fix the distortion caused by the lens (hint: cv2.calibrateCamera, cv2.getOptimalNewCameraMatrix, cv2.initUndistortRectifyMap).

Step 4: find detectable features on both images, here we're going to use SIFT algorithm  (hint: cv2.SIFT()).

Step 5: find the same matching features in both images, here we're going to use the FLANN algorithm (hint: cv2.FlannBasedMatcher, cv2.FlannBasedMatcher.knnMatch).

Step 6: compute epipolar lines between pictures (hint: cv2.findFundamentalMat, cv2.computeCorrespondEpilines).

Step 7: transform the images so the matching lines will be horizontal with each other between images (hint: cv2.stereoRectifyUncalibrated, cv2.warpPerspective).

Step 8: calculate disparity between two stereo images (hint: cv2.StereoSGBM, cv2.StereoSGBM.compute).

So much for the first try. The disparity map is noisy and not really accurate on the account of uncalibrated stereo rectification.

To improve that we need to perform calibrated stereo rectification, and we'll be able to do that when we figure out the way to get rotation and translation data between stereo sets from epipolar geometry.

Also, a great resource of learning opencv and python: https://github.com/abidrahmank/OpenCV2-Python-Tutorials