Second Time in the Air (MK5)

Now that the plane is finally fully functional (and can stay in the air more then just few minutes) it's time to make some pictures again.

The plane, manually controlled, flew nice and stable, keeping stable speed and altitude:

This time, 8 minute mission for a 8 kilometer flight, with enough battery left over for a few more flights:

And this time (the interesting bits), panorama with just the landings:

And the reconstruction result was good too:

First Time in the Air (MK5)

Maiden flight of the new plane!

This one got off to a rocky start. With all the modifications to the tail, just levelling the control surfaces was not quite enough to get the plane to fly straight. Look out for unintentional barrel rolls in the video :D

Launching the plane for the first time, it tented to pitch the nose down, taking almost half nose up input to fly it straight. This resulted in a "roller coaster" flight, where some altitude was gained, then controls centred and trimmed little by little. Business as usual.

After landing and physically adjusting the trim settings the second flight was a lot smoother.

Combined flight took around 10 minutes with 10.63km travelled at around 50km/h:

And here is the scenic view:

Playing with Disparity II

So, learning from previous attempt, let's try again.

This time, with better calibration. Turns out just taking many pictures of the calibration chequerboard is not enough. At least in this case adding two more odd angles improved the lens distortion correction:

 While the individual horizontal and vertical lines on the left image are straight, they are not perpendicular to each other. A property preserved with better calibration images.

Next, improved stereo image rectification. Very important for epipolar geometry is the computation of a good fundamental matrix. Turns out, that just using the best weighted features is not enough. Some manual checking that they are distributed throughout the pictures does wonders:

The bad and good example, the difference a couple of different points used in the fundamental matrix calculation can make.

All this produces a more correct disparity image, where the benches are before the trees and the trees are straight:

And now, that we have a good calibration of intrinsic parameters and a better fundamental matrix, we can calculate the existential matrix and place each camera in to the global coordinate system. And from there we can do 3D reconstruction:

Matched feature points on both images used in the reconstruction. Note the points on the trees, the bench and the table in the background behind the trees.

Camera view of the reconstructed points in 3D space. We can make out the trees, the bench and the background table.

But the reconstruction becomes apparent when we shift the view to the top down perspective. Here we can see the distance and the relative placement between the bench, the trees and the background table.

Testing Structure From Motion Software II

Let's try to get some real data with some real pictures.

On the last flight, lasting 5 minutes 33 seconds a total of 134 pictures were taken (67 with each camera).

Out of those, 38 were matched to a single scene in VisualSFM:

This time the results are much more better. The shape of the house or the vehicles parked outside can be easily distinguished.

And when combined with CMPMVS:

Also showing produced disparity map and generated ortophoto picture of the scene:

The results look good! Houses, roads, bushes, trees and even separate fields can be distinguished now.

But they could be better still. For one, while focus was locked, the exposure and ISO weren't and the cameras "decided" to take pictures at shutter speed 1/100 s with ISO 80. Manually decreasing the shutter speed and increasing ISO would produce even sharper images.

Secondly, decreasing the time between shots would mean more images and more overlapping matching points. Now the shots were taken at 5 second intervals. This could be reduced to 4 seconds for stereo images.

It could be even reduced down to 2 seconds between pictures if the cameras would take turn taking pictures, but then this would become purely structure from motion reconstruction (as it was here if we can't get the stereo registration to work properly).

Seventh Time in the Air

In in the air again to take some pictures.

The plan was simple. Utilize the battery (3 minutes full throttle or 9 minutes half throttle) to the fullest.

Start with 1 minute of full throttle to get some altitude, 66% or 6 minutes half throttle left. Flay a couple of overpasses for 4 minutes, 2 minutes left at half throttle. Land with remaining battery.

Sadly the plan didn't go quite as planned:

First take off attempt was aborted because one of the wheels snagged on the grass and turned off the runway (here were 10 seconds of full throttle lost). Then the plane didn't get as high as intended, so flying by the visible size, the plane was flown farther then intended (here it took longer to fly back). Add some wind higher up, so when trying to hold the plane as still as possible for the photos produced larger turns.

Long story short, 5 minutes and 33 seconds later the plane landed 3 meters from the runway because it ran out of power to run the motor 20 centimetres off the ground approaching the runway. But the new landing gear held. No pivot over this time, just some torn duct tape.

Much was learned today and if the pictures turn out useful, it was worth it.

And here is the scenic view:

Testing Structure From Motion Software

So, while we work on improving our implementation of image rectification and beyond, let's see what kind of results we can expect from the data we have.

For that, we're going to use a couple of freely available reconstruction tools that already work.

First one is VisualSFV: http://ccwu.me/vsfm/

A GUI application for 3D reconstruction using structure from motion (SFM) from Changchang Wu.

For that we took our stereo video from the last time. Extracted images from the left camera at 5 images per second (extracting images from both cameras was just too many images, and we still don't have them rectified). Used OpenCV and a calibration board to fix the lens distortion and ran it trough VisualSFM.

The result being sparse and dense reconstruction:

And the output in MeshLab:

Once we had the dense reconstruction, we were able to export it from VisualSFM in .cmp format, which can be used as input for the next tool.

Second one is CMPMVS: http://ptak.felk.cvut.cz/sfmservice/websfm.pl?menu=cmpmvs

A multi-view reconstruction software.

So after letting it do all the work, we ended up with the model:

The results are promising, but there  is a lot of noise in the reconstruction. Looked closely there are differences in disparity where there are houses, but just. The details like cars are lost.

Looking at single video frames, the images still look slightly blurry. Even though the plane vibrates a lot less now, maybe taking images for reconstruction as video is not the best method.

Time to take some real photos and try with that.

Playing with Disparity

Pictures and videos from the plane look cool, but that's all if until we do something with them. And what do we want to do with them? We want 3D reconstruction!

Sounds scary. It is. It takes a lot of math to do 3D reconstruction with epipolar geometry, so we're going to start slow and first play around with disparity map from stereo images.

Nothing extravagant for now, everything done here with python, opencv for python and numpy.

Step 1: take two cameras, take a stereo image pair and take a picture of a checkerboard with each one for calibration.

Step 2: detect the edges (hint: cv2.findChessboardCorners).

Step 3: calibrate the camera and fix the distortion caused by the lens (hint: cv2.calibrateCamera, cv2.getOptimalNewCameraMatrix, cv2.initUndistortRectifyMap).

Step 4: find detectable features on both images, here we're going to use SIFT algorithm  (hint: cv2.SIFT()).

Step 5: find the same matching features in both images, here we're going to use the FLANN algorithm (hint: cv2.FlannBasedMatcher, cv2.FlannBasedMatcher.knnMatch).

Step 6: compute epipolar lines between pictures (hint: cv2.findFundamentalMat, cv2.computeCorrespondEpilines).

Step 7: transform the images so the matching lines will be horizontal with each other between images (hint: cv2.stereoRectifyUncalibrated, cv2.warpPerspective).

Step 8: calculate disparity between two stereo images (hint: cv2.StereoSGBM, cv2.StereoSGBM.compute).

So much for the first try. The disparity map is noisy and not really accurate on the account of uncalibrated stereo rectification.

To improve that we need to perform calibrated stereo rectification, and we'll be able to do that when we figure out the way to get rotation and translation data between stereo sets from epipolar geometry.

Also, a great resource of learning opencv and python: https://github.com/abidrahmank/OpenCV2-Python-Tutorials