Air-Cobot (Aircraft Inspection enhanced by smaRt & Collaborative rOBOT) is a French research and development project of a wheeled collaborative mobile robot able to inspect aircraft during maintenance operations. This multi-partner project involves research laboratories and industry. Research around this prototype was developed in three domains: autonomous navigation, human-robot collaboration and nondestructive testing.
Air-Cobot is presented as the first wheeled robot able to perform visual inspections of aircraft. Inspection robots using other types of sensors have been considered before, such as the European project Robair. Since the launch of the project, other solutions based on image processing began to be developed, such as EasyJet with a drone, the swarm of drones from Toulouse company Donecle and the Aircam project of the aerospace manufacturer Airbus.
Since the beginning of the project in 2013, the Air-Cobot robot is dedicated to inspect the lower parts of an aircraft. In the continuation of the project, there is the prospect of coupling with a drone to inspect an aircraft's upper parts. In October 2016, Airbus Group launched its research project on the hangar of the future in Singapore. The robots from the Air-Cobot and Aircam projects are included in it.
Project description
Objectives Launched in January 2013, the project is part of the Interministerial Fund program of Aerospace Valley, a business cluster in southwestern France. With a budget of over one million euros, Air-Cobot aims to develop an innovative collaborative mobile robot, autonomous in its movements and able to perform the inspection of an aircraft with nondestructive testing sensors during preflight or during maintenance operations in a hangar. Testing has been performed at the premises of Airbus and Air France Industries.
Partners
The project leader is Akka Technologies. There are two academic partners; Akka Technologies and four other companies make up the five commercial partners.
Academic partners Armines and Institut Clément Ader of the École des mines d'Albi-Carmaux are in charge of nondestructive testing. Laboratoire d'analyse et d'architecture des systèmes (LAAS-CNRS) with the Robotics, Action and Perception (RAP) team handles the autonomous navigation.
Industrial partners Akka Technologies, particularly the center for research and development Akka Research Toulouse, leads the project and brings skills in image analysis, navigation and aircraft maintenance. Airbus Innovations is the initiator of the project, providing CAD models of the Airbus A320 and developing operating scenarios. 2MoRO Solutions, a company based in the French Basque Country, is in charge of the maintenance information system. M3 System, a Toulouse-based company, takes care of the outdoor localization solution based on the Global Positioning System (GPS). Sterela, based in the south of Toulouse, provides the 4MOB mobile platform.
Project finance Project finance is provided by banque publique d'investissement, the Aquitaine Regional Council, the Pyrénées-Atlantiques Departemental Council, the Midi-Pyrénées Regional Council and by the European Union.
Expected benefits Aircraft are inspected during maintenance operations either outdoors on an airport between flights, or in a hangar for longer-duration inspections. These inspections are conducted mainly by human operators, visually and sometimes using tools to assess defects. The project aims to improve inspections of aircraft and traceability. A database dedicated to each aircraft type, containing images and three-dimensional scans, will be updated after each maintenance. This allows for example to assess the propagation of a crack.
The human operator's eyes fatigue over time while an automatic solution ensures reliability and repeatability of inspections. The decrease in time taken for inspections is a major objective for aircraft manufacturers and airlines. If maintenance operations are faster, this will optimize the availability of aircraft and reduce maintenance operating costs.
Robot equipment
All electronics equipment is carried by the 4MOB mobile platform manufactured by Sterela. The off-road platform, equipped with four-wheel drive, can move at a speed of 2 metres per second (). Its lithium-ion battery allows an operating time of eight hours. Two bumpers are located at the front and at the rear. These are obstacle detection bumpers. They stop the platform if they are compressed.
The cobot weighs . It has two computers, one running Linux for the autonomous navigation module and the other Windows for the non-destructive testing module. The robot is equipped with several sensors. The pan-tilt-zoom camera manufactured by Axis Communications and Eva 3D scanner manufactured by Artec 3D are dedicated to inspection. The sensors for navigation are an inertial measurement unit; two benches, each equipped with two PointGrey cameras; two Hokuyo laser range finders; and a GPS unit developed by M3 Systems that allows for geofencing tasks in outdoor environments.
Autonomous navigation The autonomous navigation of the Air-Cobot robot is in two phases. The first, navigation in the airport or the factory, allows the robot to move close to the aircraft. The second navigation, around the aircraft, allows the robot to position itself at control points referenced in the aircraft virtual model. In addition, the robot must insert itself in a dynamic environment where humans and vehicles are moving. To address this problem, it has an obstacle avoidance module. Many navigation algorithms are constantly running on the robot with real time constraints. Searches are conducted on optimizing the computing time.
Navigation in the airport or the factory In an outdoor environment, the robot is able to go to the inspection site by localizing through Global Positioning System (GPS) data. The GPS device developed by M3 Systems allows geofencing. At the airport, the robot operates in dedicated navigation corridors respecting speed limits. Alerts are sent to the operator if the robot enters a prohibited area or exceeds a given speed.
Another algorithm based on computer vision provides, in real-time, a lane marking detection. When visible, painted lanes on the ground can provide complementary data to the positioning system to have safer trajectories. If in an indoor environment or an outdoor environment where GPS information is not available, the cobot can be switch to follower mode to move behind the human operator and follow her or him to the aircraft to inspect.
Navigation around the aircraft To perform the inspection, the robot has to navigate around the aircraft and get to the checkpoints called up in the aircraft virtual model. The position of the aircraft in the airport or factory is not known precisely; the cobot needs to detect the aircraft in order to know its position and orientation relative to the aircraft. To do this, the robot is able to locate itself, either with the laser data from its laser range finders, or with image data from its cameras.
Near the aircraft, a point cloud in three dimensions is acquired by changing the orientation of the laser scanning sensors fixed on pan-tilt units. After filtering data to remove floor- or insufficiently large dot clusters, a registration technique with the model of the aircraft is used to estimate the static orientation of the robot. The robot moves and holds this orientation by considering its wheel odometry, its inertial unit and visual odometry.
Laser data are also used horizontally in two dimensions. An algorithm provides a real-time position estimation of the robot when enough elements from the landing gears and engines are visible. A confidence index is calculated based on the number of items collected by lasers. If good data confidence is achieved, the position is updated. This mode is particularly used when the robot moves beneath the aircraft.
For visual localization, the robot estimates its position relative to the aircraft using visual elements (doors, windows, tires, static ports etc.) of the aircraft. During the evolution of the robot, these visual elements are extracted from a three-dimensional virtual model of the aircraft and projected in the image plane of the cameras. The projected shapes are used for pattern recognition to detect those visual elements. The other detection method used is based on the extraction of features with a Speeded Up Robust Features (SURF) approach. A pairing is performed between images of each element to be detected and the actual scene experienced.
By detecting and tracking visual landmarks, in addition to estimating its position relative to the aircraft, the robot can perform a visual servoing. Research in vision is also conducted on simultaneous localization and mapping (SLAM). A merger of information between the two methods of acquisition and laser vision is being considered. Artificial intelligence arbitrating various locations is also under consideration.
Obstacle avoidance In both navigation modes, Air-Cobot is also able to detect, track, identify and avoid obstacles that are in its way. The laser data from laser range sensors and visual data from the cameras can be used for detection, monitoring and identification of the obstacles. The detection and monitoring are better in the two-dimensional laser data, while identification is easier in the images from the cameras; the two methods are complementary. Information from laser data can be used to delimit work areas in the image.
The robot has several possible responses to any obstacles. These will depend on its environment (navigation corridor, tarmac area without many obstacles, cluttered indoor environment etc.) at the time of the encounter with an obstacle. It can stop and wait for a gap in traffic, or avoid an obstacle by using a technique based on a spiral, or perform path planning trajectories.
Computing time optimization Given the number of navigation algorithms calculating simultaneously to provide all the information in real time, research has been conducted to improve the computation time of some numerical methods using field-programmable gate arrays. The research focused on visual perception. The first part was focused on the simultaneous localization and mapping with an extended Kalman filter that estimates the state of a dynamic system from a series of noisy or incomplete measures. The second focused on the location and the detection of obstacles.
Non-destructive testing
Image analysis After having positioned to perform a visual inspection, the robot performs an acquisition with a pan-tilt-zoom camera. Several steps take place: pointing the camera, sensing the element to be inspected, if needed repointing and zooming with the camera, image acquisition and inspection. Image analysis is used on doors to determine whether they are open or closed; on the presence or absence of protection for certain equipment; the state of turbofan blades or the wear of landing gear tires.
The detection uses pattern recognition of regular shapes (rectangles, circles, ellipses). The 3D model of the element to be inspected can be projected in the image plane for more complex shapes. The evaluation is based on indices such as the uniformity of segmented regions, convexity of their forms, or periodicity of the image pixels' intensity.
The feature extraction using speeded up robust features (SURF) is also able to perform the inspection of certain elements having two possible states, such as pitot probes or static ports being covered or not covered. A pairing is performed between images of the element to be inspected in different states and that present on the scene. For these simple items to be inspected, an analysis during navigation is possible and preferable due to its time saving.
Point cloud analysis