Testing Structure From Motion Software

So, while we work on improving our implementation of image rectification and beyond, let's see what kind of results we can expect from the data we have.

For that, we're going to use a couple of freely available reconstruction tools that already work.

First one is VisualSFV: http://ccwu.me/vsfm/

A GUI application for 3D reconstruction using structure from motion (SFM) from Changchang Wu.

For that we took our stereo video from the last time. Extracted images from the left camera at 5 images per second (extracting images from both cameras was just too many images, and we still don't have them rectified). Used OpenCV and a calibration board to fix the lens distortion and ran it trough VisualSFM.

The result being sparse and dense reconstruction:

And the output in MeshLab:

Once we had the dense reconstruction, we were able to export it from VisualSFM in .cmp format, which can be used as input for the next tool.

Second one is CMPMVS: http://ptak.felk.cvut.cz/sfmservice/websfm.pl?menu=cmpmvs

A multi-view reconstruction software.

So after letting it do all the work, we ended up with the model:

The results are promising, but there  is a lot of noise in the reconstruction. Looked closely there are differences in disparity where there are houses, but just. The details like cars are lost.

Looking at single video frames, the images still look slightly blurry. Even though the plane vibrates a lot less now, maybe taking images for reconstruction as video is not the best method.

Time to take some real photos and try with that.

Playing with Disparity

Pictures and videos from the plane look cool, but that's all if until we do something with them. And what do we want to do with them? We want 3D reconstruction!

Sounds scary. It is. It takes a lot of math to do 3D reconstruction with epipolar geometry, so we're going to start slow and first play around with disparity map from stereo images.

Nothing extravagant for now, everything done here with python, opencv for python and numpy.

Step 1: take two cameras, take a stereo image pair and take a picture of a checkerboard with each one for calibration.

Step 2: detect the edges (hint: cv2.findChessboardCorners).

Step 3: calibrate the camera and fix the distortion caused by the lens (hint: cv2.calibrateCamera, cv2.getOptimalNewCameraMatrix, cv2.initUndistortRectifyMap).

Step 4: find detectable features on both images, here we're going to use SIFT algorithm  (hint: cv2.SIFT()).

Step 5: find the same matching features in both images, here we're going to use the FLANN algorithm (hint: cv2.FlannBasedMatcher, cv2.FlannBasedMatcher.knnMatch).

Step 6: compute epipolar lines between pictures (hint: cv2.findFundamentalMat, cv2.computeCorrespondEpilines).

Step 7: transform the images so the matching lines will be horizontal with each other between images (hint: cv2.stereoRectifyUncalibrated, cv2.warpPerspective).

Step 8: calculate disparity between two stereo images (hint: cv2.StereoSGBM, cv2.StereoSGBM.compute).

So much for the first try. The disparity map is noisy and not really accurate on the account of uncalibrated stereo rectification.

To improve that we need to perform calibrated stereo rectification, and we'll be able to do that when we figure out the way to get rotation and translation data between stereo sets from epipolar geometry.

Also, a great resource of learning opencv and python: https://github.com/abidrahmank/OpenCV2-Python-Tutorials

Sixth Time in the Air

Fully assembled once more, it's testing time!

With only the control circuit between the autopilot program on the phone and the plane for autonomous flights missing, the weight of the plane is now just below 1.9 Kg. Not bad for all the equipment.

But is it better? Yes!

First, the vibrations are almost gone:

Left picture, previous flight, fully loaded with the stock motor at full throttle.
Right picture, this flight, fully loaded with the new motor at full throttle.

The difference is enormous:

And that's not al. With the new engine, the plane now climbs effortlessly:

In this test, with no one around on the ground or in the air, altitude difference of around 300 m was achieved.

And with no problems with telemetry this time, we learned that the cruising speed at which the altitude can be maintained is around 45 Km/h, which takes about half throttle.

Unfortunately, but not unexpectedly due to all the extra equipment, the gliding profile is somewhat disappointing. With the engine off, the plane glides, but the vertical speed necessary to maintain the cruising speed is around -35 Km/h, peeking at -50 Km/h. In short, it flies but it eats up altitude like crazy as can be seen on the video above at 3:00 mark (yes, the propeller keeps spinning due to resistance).

But, smoother and more powerful motor means better images and video from the cameras, and that's what we want!

Warning, awesome stereo video of the flight:

AVI@TOR MK4 SP4

For better results, use better stuff!

The plane flies, fully loaded even. But! It couldn't climb very high and all the camera recordings were suffering from vibrations coming from the motor.

So, the solution is simple, replace the motor. Implementation, not so much but not on the account of having to turn screws.

Unfortunately the desired one, O.S. 3820-1200 didn't come trough in time, so we got the second best thing, KONTRONIK KORA 15-10.

Cue the montage sequence:

Just add a few more holes since the spacing on this motor a slightly bigger and drill the hole in the center of the prop from 5mm to 10mm because someone goofed up and got the wrong prop mount. But it all balanced out perfectly in the end.

New motor and better regulator. Some day maybe a bigger battery and then a larger plane with newer motor, and so on, but for now this is it.

Few quick tests with a piece of string and a scale, determined the motor with 11x5.5 prop gives out 1.6 Kg of thrust at full throttle. With more aggressive timing setting on the regulator it gave out 1.7 Kg of thrust, but the output was not linear with the throttle.

For comparison, the old motor gave 0.98 Kg of thrust at full throttle.

And another test on the battery capacity gave us: 3 minutes at full throttle and 9 minutes at at half throttle on the new motor. Compared to the 6 minutes at full throttle and 11 minutes at half throttle on the old one. 

Time to test in in the air to see if the trouble was worth it.

Keep Alive Ping

Not dead, just waiting for some parts!

Fifth Time in the Air

Fully loaded and air worthy:

Any yes, it flew! Here's the video to prove it:

And that's not all it did. The camera system works! It successfully takes pictures and video:

Stereo pictures! Might be uncomfortable to look at since the lens distance is not optimized for human viewing, but the depth data is there.

Still taken, synchronized and aligned by hand. There is one more part missing from the plane, and that is the circuit that will relay the data from the autopilot running on the phone to the servos. And part of that will also be automatic synchronised picture taking. As for the alignment, that will be the fun part once we have some real data.

And yes, there is also stereo video. Slightly easier for humans to watch, since distance between lenses is offset by the distance to the object in focus.

But unfortunately the video taken is slightly blurry.  Fault of the stock engine, which is not of the best quality and is causing much vibrations, which are also noticeable on the panoramic video. Not much can be done here but to replace the engine with something better.

And while we're at it, replace it with something more powerful, since while the plane flew, and it flew quite stable, it had horrible climb rate and required full throttle to maintain altitude.

Now with three cameras on board, there is one more flight video:

The landing could be better :P Not used to the extra weight, the runway was missed by 30cm, touching down on high grass and flipping over. So nothing but pride was broken.

And what would a report be without the visualization of the whole flight:

Unfortunately there was problem with the SD card in the phone, so not all the data was captured hence the lack of attitude visualization and force graphs.

All in all a very successful test flight. What works and what doesn't is known, so the next logical step is to work out the kinks and try again!

AVI@TOR MK4 SP3

We know the plane flies and can even do some simple acrobatics. But the real question is, can it fly with all the equipment?

Only one way to find out, which means it's time for some more modifications!

First we determine the location of the camera mount on/in the fuselage:

 

Keep in mind where the center of gravity will be once the cameras are on the plane, then cut the carbon tubing holder size box on one side and appropriate placed holes on the other side of the plane.

Center it just right and it looks like this:

Now that we have the camera holder in place it's time to use the camera holder mounts we prepared and printed.

The mounts will hold the holder to the wing so it wont move about and also to redistribute the weight a bit more evenly:

First locate and mark a good spot, close enough that will hold the stand and leave enough room so the cameras could be put on, but still far enough so the stand can be slide in and out.

Then mark the spot, carve it out (watch out for the servo wires) and protect the servo wires with wax (so once everything is coved with glue and the plane is beyond repair we can remove the servos without it's wires being glued).

Once that is done, drill appropriate holes in to the mounts so the fit around the carbon tube holding the wing, fit them inside the holes and drown them in glue.

After some time when the glue is set, we have out wings with mounts that look like this:

To fully assemble and disassemble the plane now is a whole process, but it's worth it when the end result looks like this:

Will it fly? Only one way to find out :P