More computer vision algorithms.

Another week has passed so let me talk you through the things I have worked on during this time. First of all, we had to write a preliminary project report where we had to reflect on the work we’ve done so far as well as produce readjusted plan for the future. 

In addition to that, I have also spent time optimising computer vision algorithms used in my application. I have switched from using SURF algorithm to using ORB algorithm. ORB is an alternative to SURF and works in the same way: it finds keypoints and descriptors in the image/video. Then we use BFMatcher to match analysed image’s features with live video feed coming from the camera.

I have made this switch because of the studies and tests found online, which suggested that ORB is much faster than SURF when running on iOs devices while providing functionality with a quality very similar that of SURF algorithm.

Image source:

Image above shows time in milliseconds required to run different feature and descriptor extractors algorithms on iOs devices. As we can see SURF algorithm is quite slow compared to ORB. Since that was the case, I have implemented the latter in my app.

Despite implementing ORB algorithm, the app still runs quite slow when detection and matching are happening. My supervisor proposed solution to solve this which was to use another computer vision algorithm which would only track a few points instead of tracking the whole image all the time. To achieve this functionality, I will be using a principle known as optical flow. Optical flow detects patterns in apparent motion and can determine when and how the motion is happening. Application of this algorithm can be found in motion detection, video stabilisation, robotics and other areas.

I will use an implementation of optical flow called Lucas-Kadene. The plan is first to run ORB algorithm to detect and match images. Then once the image is recognised and the boundaries around it are set, I will pass only 4 points to Lucas-Kadene Optical Flow function. This will help me to free up processing power as instead of tracking 400 points it will only be tracking 4 points.

Optical flow Lucas-Kadene is a widely used method for optical flow estimation. It works by detecting and combining information from several nearby pixels and assumes that the flow movement is constant calculates basic optical flow equations for all the pixels in that neighbourhood.*  Essentially, we can pass a vector point location in the pixels coming from video and it will track its movements in relatively close range. And that is exactly what I need.

Source reference:

So far I have managed to implement this functionality in OpenFrameworks for MacOS. The next step is to create appropriate functions and construct a class that I could use with the app on iOs.

Plan of action for the upcoming week:

  • Create functions and a class object for OpticalFlowLucas-Kadane;
  • Implement OpticalFlowLucas-Kadane on iOs;
  • Start with the music player and basic visuals