VisualODF: Visual Obstacle Detection Framework

Author: Rob Russell

Introduction

The Visual Obstacle Detection Framework is a group C++ classes written to make the implementation and integration of obstacle detection algorithms easier to incorporate into a full-blown application by providing an effective framework for creating a test environment. This test environment allows for the easy integration of new kinds of display widgets for displaying the detected obstacles, as well as the necessary feedback for the human user.

What is included with VisualODF?

VisualODF is composed of different types of documents and classes. The most important document is the “VisualODF Handbook” (this document).

This handbook describes the usage of the VisualODF classes and explains how to implement and integrate these items:

• Custom display widgets (TCAS-II type display, for example) to be used in your own applications
• Custom video processing or obstacle detection algorithms integration
• Reusable plugins based on your custom designed widgets to be used with QT Designer

The handbook also explains the details of the following items that are related to experimenting with VisualODF:

• Getting and installing QT4 for Linux and Windows (open source)
  1. Patching and compiling QT4 with MSVC++ (external link)
  2. Running the included examples
• Designing, implementing, and interfacing a custom image processing program using VisualODF
  1. An introduction to signals and slots in QT
  2. Creating widget plugins for QT Designer
  3. Using plugins in QT Designer
  4. Creating and integrating a custom image processing algorithm
  5. Putting it all together
• An introduction to the flight simulator FlightGear
• Capturing videos for processing

Material Covered in the Handbook

Here is a brief introduction to the main topics covered in the handbook.

QT

Perhaps the most fundamental part of VisualODF is the classes upon which it is built and derived from. QT [link] is a cross-platform widget set that is very well designed, easy to use (even for the novice programmer), and freely available for non-commercial projects. It is available for practically every platform and features a complete set of classes for generating interfaces, as well as lower level tasks (such as networking).

QT can be installed on either a Windows computer or a Linux distribution for compiling VisualODF. The QT version must be version 4.1 or greater (version 3.x.x will not work without some re-working – please just use the newer versions of QT). Note: It must be noted that QT is slightly easier to install on a Linux computer than a Windows PC with MSVC++. This is only due to the fact that QT can be compiled as-is on a Linux machine, but must be patched to compile properly using the MSVC++ compiler. QT for Windows (the open-source edition) is configured by default to compile with the MingWG [spelling?] compiler. A simple patch allows it to be compiled without problem if you are using the MSVC++ compiler.

QT Plugins

QT plugins allow custom designed widgets to be installed in form easily and quickly using QT Designer. After a custom widget is created, a plugin based on this widget can be written so that users can quickly create projects with QT Designer, a WYSIWYG (what-you-see-is-what-you-get) form editor.

Signals and Slots

Signals and slots are particularly useful features of QT that simplify message passing between classes. Similar in concept to signals as used in the Linux operating system, signals can be emitted from any class deriving from the QObject class and can be received by any class with an appropriately defined slot. Any number of slots can be connected to any number of signals, and vice-versa. This allows an easy mechanism for our display widgets to receiver notification of a found obstacle or of a newly processed video frame. Signals allow the passing of data directly to classes with slots that are defined accordingly.

Connecting signals and slots between classes is a very simple process, and does not have to be done during initialization. This is useful, for example, if a user wants to switch between several video processing classes that all send signals to display widgets: the signal can be disconnected and re-connected at will depending on which video processing class the user chooses.

All in all, signals and slots allow a very convenient interface for sending and receiving notifications. Signals and slots are not as fast as using other signaling mechanisms (such as mutexes and condition variables) but are perfect for our purposes.

Custom Image Processing Algorithm Implementation

In order to detect an obstacle from a video source, some type of image processing has to be done. This section will describe how to implement and integrate a custom image processing algorithm into VisualODF. While perhaps the most work intensive part of actually creating a working program, the example set forth here should give the user a firm foundation to start on.

FlightGear

FlightGear is an open-source flight simulator with realistic physics, excellent graphics, an extensive scenery database (high resolution over most of the world), and scripting capability. It will be used in this project to simulate aircraft obstacles from the point of view of a UAV.

Capturing Videos

The video output of the FlightGear flight simulator must be recorded. Different methods available are mentioned in this section, as well as the method used by the author.

Installation of VisualODF

The prerequesites of VisualODF are:

• QT version 4.1 or greater. This can be obtained from Trolltech [now called qt.io ]
• Simple DirectMedia Layer Libraries (SDL). These can be obtained from the libsdl website.
• ffmpeg libraries. They can be obtained from the ffmpeg homepage.

These must all be successfully installed before VisualODF can be used. If the example VisualODF program is not going to be compiled and the video playback and capture capability are not needed, then the SDL and ffmpeg libraries do not need to be installed. To install these libraries, please refer to the documentation on the respective websites.

Note: These libraries are cross-platform compatible. Although the sample VisualODF application was only compiled on a Linux platform, it is possible to compile all of the classes on a Windows platform as well.

If you have met the requirements, you must simply navigate to the visualODF directory and type:

and the executable will be built (replace “make” with “nmake” if you are using Windows). To make the plugins, navigate to the src/ directory and then the plugin directory of your choice. Then type:

If you are using Linux, you must have root permissions when you type 

. Again if you are using Windows, use “nmake” instead of “make”.

QT

The following subsections will introduce the QT Toolkit, QT Plugins, Signals and Slots, and all things related to QT.

Getting and Installing QT

QT can be downloaded from the Trolltech [qt.io] website in the downloads section. IMPORTANT: only QT versions 4.1 and greater can be used with Visual ODF. If QT is being installed on a Linux platform, then please follow the method normally used by your distribution to install QT (rpm, apt-get, emerge, urpmi, etc). To install on a Windows platform , please read the special section that follows. Just ensure that you are using QT 4.1 or greater.

Installing QT on a Windows Platform

If the MinGW compiler is being used for development on the Windows platform, then QT should work “out of the box”. If the Visual Studio is to be used instead (as was my case), you must go to this web page and follow the instructions given there [this is probably outdated]. Essentially you will end up applying a patch to the QT sources and then compiling using the Microsoft C++ compiler.

Compiling the Examples

If you have not already done so, please compile the QT examples to ensure that you have the $QTDIR variable set correctly and that the libraries are installed properly. If you cannot get the examples to properly compile, then you will not be able to compile the VisualODF classes.

To compile the examples, open a command prompt and navigate to the $QTDIR directory (the directory you installed QT into). There will be an “examples” directory there with a file called ‘examples.pro’. In order to create the so-called “Makefile”, you must type: 

 in the examples directory. If the executable ‘qmake’ is not found, then likely you didn’t set up the environment variable $QTDIR correctly.

Next, if using Linux, type 

 to compile all the examples. For windows, type   instead. If the examples properly compile (navigate to the various examples and try them out!) then you should have QT properly installed.

Tools Included with QT

The three most important tools that come with QT are :

• QMake (qmake) (normally invoked from the command line) to create Makefiles from .pro files, and
• QT designer (designer) for creating forms with custom-built widgets.
• QT assistant (assistant) for getting help with the QT-based classes. This is an essential reference.

It is also essential to understand the purpose of qmake for creating ‘Makefiles’ to compile your projects, as this should be used when creating QT based projects. The Makefile generated by qmake will ensure that the proper intermediate files are created during compilation of a QT-based class. If the qmake system is not desirable for whatever reason, then please look into the QT MOC (Meta Object Compiler) to determine how to manually compile these intermediate files.

Please read the documentation that comes with QT (by invoking the QT Assistant (“assistant”)) for more information on using qmake.

What is a Custom QT Widget Plugin?

A QT Widget Plugin is a special class that allows a custom widget to work inside of QT Designer. If you design a special widget for displaying obstacles, for example, a Widget Plugin can be created from your custom widget and then inserted in a form using QT Designer. The advantage of creating a Widget Plugin is that form layout is greatly simplified – QT Designer can be used for widget sizing, placement, and connections with other widgets.

Some knowledge of creating a custom QT widget is needed before creating a widget plugin (refer to the examples in the documentation that comes with QT). Once you have created a custom widget, you must simply create a new class that derives from QDesignerCustomWidgetInterface. The class declaration would look like so:

To see an example implemenation of a Widget Plugin, please look at the class “piePlugin” included with VisualODF.

QT Signals and Slots: Implementation

A QT Signal looks like nothing more than a class member function; however, when it is declared in the class definition, it must fall below the 

 keyword.

This slot has an integer as a parameter.

A signal is even easier. It is declared like so:

To connect our signal and slot together, the code looks similar to this:

Whenever we use the keyword 

 with the signal  , the slot   will be notified. Not only that, but the integer parameter   will be passed between the signal and slot.

This simple but powerful mechanism allows us to pass data between classes easily and easy event notification. In the VisualODF example program, signals and slots are used extensively. The following diagram shows how signals and slots work together in the VisualOSF demo.

If you aren’t familiar with the names of the objects in the diagram, that’s ok; the class documentation gives plenty of detail on them. Starting from left to right, the sequence of events that occur that of interest in this example are:

1. The frameServer object has obtained an image (in this case, from a video source). It is ready to pass this data along to any other object that wishes to use it. In this case, we see that there are two slots connected to the frameServers’ 

   1	frameReady

    signal: 

   1	displayImage

   , which belongs to the 

   1

   Video 1

    object, and 

   1	processFrame

    which belongs to the 

   1	Test Algorithm

    object. They both do different things with this data: the 

   1

   Video 1

    object simply displays the data, and the 

   1	Test Algorithm

    object will do some type of processing of the image.
2. The 

   1	Test Algorithm

    in turn has two signals: 

   1	imageReady(QImage *img)

    and 

   1	foundObstacle(...)

    (where the … represent several parameters that are omitted for clarity). These are connected to three other kinds of display widgets (

   1	Video 2, Pie Widget, and TCAS Widget

   ).
3. The three display widgets display the obstacles emitted by the 

   1	foundObstacle(...)

    signal, and the 

   1

   Video 2

    object displays the 

   1

   QImage

    passed to it (presumably the result of the image processing, in this example).

How to Implement a Custom Image Processing Algorithm

To implement an algorithm that can interface with the various signals that are implemented in a display widget (one of our custom made widgets, that is), one must study the example algorithm provided with VisualODF. While it is an extremely simple example, the necessary connections are made that essentially “link” the image processing algorithm with the frameServer and the various visualization widgets. The implementation can be seen in the file TestAlg.cpp.

The important points that need to be noted when creating custom processing algorithm:

1. Your class must derive from the ProcAlg class
2. You must implement the processFrame member function, which is really a virtual slot that is overloaded from the ProcAlg class.
3. This example does not maintain a list of found obstacles, or do any real detection of any kind. In a real implementation, you would have to keep track of found obstacles, decide whether you have detected an obstacle multimple times in your processing, remove obstacles when they are no longer in view, and many other things that add considerably to the complexity of the implementation. This example just shows the minimum needed to connect to the frameServer and the display widgets (TCAS and pie widget).

The image data is stored in a QImage instance. For more information on the QImage class, please open up the QT Assistant and locate the proper reference.

FlightGear: An Open Source, Multi-Platform Flight Simulator

The videos that were captured for the VisualODF demo were acquired while running FlightGear, which is available for free download from here.

FlightGear has many features, among which are (as taken from the FlightGear website):

• Three flight dynamics models
• Extensive and accurate world scenery database
• Accurate and detailed sky model
• A flexible and Open Aircraft Modeling system
• Muliplayer and networking options
• Source code available
• Interface using the NASAL scripting language

FlightGear allows the recording of position and attitude data during flights. This data, combined with a recorded video of the actual flight, could be used as a basis of analysis for obstacle detection. In order to create obstacles in the path of the UAV, one could simply use a second computer and connect to a FlightGear server and position the “obstacle plane” accordingly. Using the autopilot capabilities make scenario creation easier than manual control.

The preceding screen capture shows the view from the cockpit on of the obstacle plane (left windows), and the moving map display showing two red aircraft : one is the obstacle plane, and the second is the UAV (which is flying on a second PC). They are both connected to a FlightGear server.

Video Capture of a Flight Simulator

Two possibilities exist for capturing the video output from FlightGear.

1. Hardware Capture. A hardware device, often a PCI card or USB device, that can interface directly with the video output of a graphics card, can be used to create a video of the FlightGear scenarios. This was the method used for the demo videos that come with VisualODF.
2. Software Capture. While this method may be easier, I had a difficult time finding a free method that worked properly. I located the following candidates for capturing videos:
   1. xvidcap, used for creating mpeg videos of windows in the X environment (Linux)
   2. yukon, which is used for recording OpenGL based programs (Linux)
   3. Fraps (Windows)
   4. GameCam (Windows)

Video Capture in VisualODF

Capturing video output from FlightGear is done external to any VisualODF classes or software. However, the demo program that comes with VisualODF does its own video capture on existing video files (your FlightGear movies, for example).

This is a demonstration of obtaining frames from an existing video source to be displayed and processed with other VisualODF classes included in the demo. The class “TestAlg” (the example image processing algorithm) is able to take these captures frames and invert the RGB values, while the “videoDisplayWidget” simply displays the captured frames.

The class QtFFPlay is responsible for setting up, opening, and acquiring screenshots of a given video stream. It uses the ffmpeg libraries to do this, and should be compatible with any video source that ffmpeg understands. QtFFPlay uses an instance of the class QtFFThread to handle the low-level work related to reading and displaying the video (via the SDL libraries), as well as grabbing video frames.

QtFFPlay is based on the player “ffplay” that is included with the ffmpeg libraries, and shares much code from it. However, ffplay was written in C (not C++), and QtFFPlay can be used within QT (it derives from QWidget). The lack of documentation with the ffmpeg libraries (as well as ffplay) are reflected in the C++ implementations of QtFFPlay and QtFFThread.

Capture Image Format

When the QtFFPlay class captures an image from a video stream, it normally encounters videos using the YUV colorspace. YUV is used in NTSC and PAL broadcasts, and requires less information (and hence smaller files) than the RGB colorspace. YUV has three components: one of luminance and two of crominance. The logical layout of an image using the YUV colorspace in system memory (or rather, as stored in the SDL_Overlay structure, pointed to by the member **pixels) is illustrated by the following tables.

Y-Component Layout (30 pixels, or values)

1	2	3	4	5	6
7	8	9	10	11	12
13	14	15	16	17	18
19	20	21	22	23	24
25	26	27	28	29	30
31	32	33	34	35	36

U-Component layout (9 values)

1,2,7,8	3,4,9,10	5,6,11,12
13,14,19,20	9,10,15,16	11,12,17,18
25,26,31,32	27,28,33,34	29,30,35,36

V-Component layout (9 values)

1,2,7,8	3,4,9,10	5,6,11,12
13,14,19,20	9,10,15,16	11,12,17,18
25,26,31,32	27,28,33,34	29,30,35,36

The gray (top) grid represents 30 pixels in a 6×6 image (6 wide, 6 high). Each U and V component corresponds with four Y values. For example, the Y values (or pixels) 1,2,7, and 8 all use the same U value: the first value in the upper left hand corner of the cyan box (with an index of 1). Likewise, the pixels 3,4,9, and 10 belong to the second U value (with an index of 2). The V component uses the same type of layout as well. It is easy to see that this type of image storage uses less space than a 24-bit RGB value : we would use 90 bytes with RGB and only 48 bytes with YUV. This type of layout of YUV information is called YV12, and is a so-called “planar’ YUV format.

Other types of YUV formats exist as well, but this seems to be a very commonly used one.

For more information on YUV formats, please see this website: http://www.fourcc.org/fccyuv.htm

To do the transformation from YUV to RGB, one must apply the following matrix:

For more information on YUV and its relation to RGB, please see the Wikipedia article on YUV [the image above is from that article].

Putting it All Together

To get an idea of how an application could be built using these various classes, the following outline could be followed:

1. Install QT and the needed libraries, as well as the examples.
2. Compile and install (make && make install) the TCAS widget plugin (or the pieDisplay widget plugin).
3. Compile and install the videoDisplayWidgetPlugin for video display.
4. Create a new C++ project in your favorite IDE or editor (I use eclipse, which is also cross-platform).
5. Create a form using QT Designer and add it to your project. Add at least one TCAS/pieDisplay widget and one videoDisplayWidget to your form. Save it and close.
6. Somewhere in your project include an instance of a frameServer, and use it to open a video file.
7. Also include a custom image processing algorithm (as described earlier in this document) using the ProcAlg class.
8. Connect the signals from the frameServer to your algorithm and possibly a video display, as well as the signals from your algorithm to the TCAS/pieDisplay and possibly video output display.

If your image processing algorithm design is sound and works, then you should have a fully functioning application. The example program that comes with VisualODF demonstrates all of these points and should be used as a starting point (please see the main class obstDet).

After a successful application is built, the next step should be to create a custom QT widget and widget plugin for display. Please see the pieDisplay or tcasdisplay for examples of custom QT widgets, and piePlugin, tcasDispPlugin, or videoDisplayPlugin for example widget plugins. All of the code should be documented well enough to allow you to follow along and build your own widgets and plugins.

Notes and Conclusion

The VisualODF classes should provide a good starting point for an obstacle detection application and HMI interface. Although no real detection is done in the provided example program, the rest of the framework is in place and should aid in the rapid development of such an application.

I did not give detailed information on some of the aspects of using the classes; that is up to the user to read and learn about. Also, some other aspects – such as installing SDL and ffmpeg, and questions like “How do I add a form to my project?” are not covered, as they can be found elsewhere and don’t need to be repeated here.

However, if anyone has questions about anything related to this project, please feel free to contact me.

Other notes:

• This document is contained in the header file obstDet.h and can be generated using the documentation tool doxygen.
• Everything included with VisualODF should compile on a Windows platform; however, this was not extensively tested.
• Some of the videos included (if you have the physical DVD media, that is) were recorded in an AVI format that VisualODF cannot understand, and therefore cannot be used. I do not know the problem with reading them and I didn’t have time to pin down the problem.

---

 1.4.7