Sensor fusion, the best of several worlds. (Technology).

Military leaders have always needed to build a picture of the battlefield and their means of achieving this have grown steadily, especially during the 20th Century. First there was the human eyeball, the range of which was initially extended through prisms in telescopes and binoculars, next with image intensifiers and later by the addition of a night vision capability. Television extended daylight and fair weather vision for many kilometres and seekers operating in the infrared spectrum permitted optical detection at night and in poor weather.

The development of radio frequency sensors extended detection and surveillance ranges far beyond the horizon with only the tools of electronic warfare, electronic support measures (ESM) and electronic countermeasures (ECM) to plague these sensors. Yet ironically, the very profusion of these aids to the detection and surveillance of hostile forces has created as many problems as it has solved by to breeding an environment of monumental headaches on the modern battlefield. One cure for this is the growing evolution of sensor fusion technology.

The purpose of sensor fusion is to exploit the best features of each sensor while minimising their worse features, thereby creating a real-time tactical or operational-level picture of either the immediate combat situation or the over-all battlefield scenario. The initial applications are to enhance battlefield survivability and effectiveness for manned combat aircraft, both fixed-and rotary-wing, but increasingly sensor-fusion is playing a key role in the evolution of network-centric warfare.

One of the main problems with sensor fusion lies not in a lack of sensors, but rather in the fact that each has its advantages and disadvantages, and amongst the latter the most important natural feature is attenuation resulting from the reduction of the passage of energy with respect to distance and environmental factors. In theory, the only handicap to radar ranges is the curvature of the earth but even here weather and atmospheric conditions can affect the passage of radio waves. Radars operate in the 1 to 100 GHz frequency range with wavelengths of 30 cm down to 0.3 cm. The higher the frequency the higher the definition, but unfortunately also, the higher the range reduction due to attenuation.

The lower frequencies with poorer definition are less effected and their ranges are consequently greater but mountainous terrain can cause multiple returns or clutter, while environmental aspects can reduce or increase the passage of waves causing `ducting'. For these reasons, radars in the 1 to 10 GHz (D-I band) range are generally used for surveillance and warning, those in the 8 to 40 GHz (I-K band) for weapon control and the 8 to 94 GHz (I-M band) reserved for weapon seekers, with the 40 to 94 GHz range frequently attributed to millimetric radars.

In the field of electro-optics, television cameras operate in the visible light or 0.9 to 0.4 micron ([micro]m) range but cannot operate very effectively at night (although image intensifier tubes can be installed) or where light waves are attenuated through rain, mist, fog, smoke or dust. Thermal imagers or infrared cameras operating in the 1 to 0.9 [micro]m range can theoretically overcome this problem (there are also ultraviolet light seekers for missiles operating in the 0.4 to 0.2 [micro]m range) but they too suffer attenuation problems from temperature and atmospheric water vapour, and while midwave sensors (3 to 5 [micro]m) tend to exhibit a poorer performance than the long wave (8 to 12 [micro]m) sensors this situation is often reversed where the environment is warmer and moister. For target range finding and designation, lasers operating in the 0.3 to 0.12 [micro]m band offer theoretical performance equivalent to radar but in practice this too is severely restricted by atmospheric water vapour.

The first steps to sensor fusion were undertaken in the late 1960s, when fire control radars in combat aircraft and warships were augmented with television cameras partly for identification purposes and partly as a means of `silent' tracking. In the 1970s, army attack helicopters, such as the Boeing AH-64A Apache, added the Lockheed Martin AN/ASQ-170 TADS (Target Acquisition Designation Sight), which augmented their television-based sensors with thermal imagers, and supplemented the AN/AAQ-11 PNVS (Pilot Night Vision Sensor) with the images from the television camera, the aircraft's forward-looking infrared and Litton's laser rangefinder/designator were projected into the Honeywell HMDSS (Helmet and Display Sighting System). The AH-64D now incorporates the Lockheed Martin AN/APG-78 Longbow millimetric weapon control radar, but the data from these sensors, and the electronic warfare suite, are presented separately and are not fused to produce an instant tactical picture.

High performance combat aircraft have long been using thermal imagers and laser rangefinder/designators in reconnaissance and targeting pods, but the need to free both wing and fuselage pylons to enhance performance and reduce radar signatures has seen a trend toward integrating these sensors within the airframe. However, the profusion of sensors still creates confusion with separate displays for radar, electronic warfare and navigation and, even with the availability of `intelligent' computer-managed fly-by-wire systems that reduce the pilot's role to one of a manager. The simultaneous administration of radar, electro-optical and electronic warfare systems creates an excessive workload and makes decision-making during flight far more difficult.

By digitising sensor data and processing it in the mission management system the new generation of combat aircraft are able to fuse sensor data providing the pilot with an instant picture of the air battle scenario together with both targets and threats. The Lockheed Martin F-22 Raptor is one of the first combat aircraft to enter service with this capability, with the Hughes Common Integrated Processor melding a range of multi-functional sensor suites. The Northrop Grumman/Raytheon AN/APG-77 active array electronically scanned radar provides long-range, multi-target, all-weather detection, tracking and interception capabilities as well as controlling multiple air-to-air missile engagements.

The radar has a range of some 160 kilometres, a weather-mapping capability and options that include air-to-surface engagement. It is complemented by a Sanders AN/ALR-94 electronic warfare suite and the TRW AN/ASQ-220 communications/navigation and identification system, which also includes an inter-flight/intra-flight data link for non-verbal transmission of target data. Although the Raptor does not have an electro-optical system there is provision for the installation of an infrared search and track system. Information is presented to the pilot through helmet- and cockpit-mounted displays although there is no indication of the data's source.

The Eurofighter, for example, takes the concept a stage further. The aircraft will evolve in stages to have a greatly enhanced air-to-surface attack capability including the specialist roles of Suppression of Enemy Air Defences (Sead), reconnaissance and maritime air-to-surface strike. From the very beginning it was decided to adopt a sensor fusion policy both to enhance operational performance by exploiting multiple sensors both in capability and volume coverage. It would also provide more accurate results in terms of target or threat detection, with increased pilot confidence to avoid false warnings.

At the heart of the aircraft is the Mids (Multi-function Information Distribution System) which accepts data (voice, text and imagery via the Link 16 Jtids [Joint Tactical Information Distribution System] data-link) from offboard sensors such as the E-3A Sentry Awacs (Airborne Warning and Control Systems) and other aircraft. Onboard sensors include the Euroradar (BAE Systems Avionics, Eads, Enosa, Fiar) ECR 90 Captor, the Eurofirst (Fiar, Tecnobit, Thales Optronics) Pirate (Passive Infra-Red Airborne Tracking Equipment), the EuroDass (BAE Systems, Elettronica, Indra) defensive aids sub-system (Dass) as well as an Identification Friend or Foe (IFF) system and an attack computer.

The Captor is a multi-mode mechanically scanned radar (it may later become electronically scanned or be replaced by an active element system) which can track and provide some indication of the identity of multiple or single targets in the air as well as detecting and tracking targets on the surface with the aid of an MTI (Moving Target Indication) function. The Pirate is both a flying/landing aid as well as an infrared search and track system with many operational characteristics with the Captor, which it complements. One area of potential growth is for the provision of a missile warning function. The Dass includes an integrated electronic support and countermeasures package, a missile approach warner and a laser warning receiver (British aircraft only) which is integrated with ESM tracks, as well as the counter-measures dispensing system. The Dass will later include a towed ECM system.

The data from various sensors is combined to present a single track with target identity, thus avoiding track duplication. This is projected either in the helmet-mounted system or the cockpit display. In addition to reducing pilot workload and producing a broader, more accurate tactical picture, sensor fusion has other advantages, according to Eads, including a reduction in both data bus and computational loading as well as the creation of sensor redundancy.

Sensor fusion also ensures that a platform can remain effective even in the most severe electronic environments that might jam purely radar-based weapon control systems. The complementary nature of the sensors ensures the fused data is more accurate and it can be transmitted to other platforms via the Mids allowing targets to be prioritised.

The Lockheed Martin F-35 or JSF (Joint Strike Fighter) will have a very similar approach although technologically the sensors will be more sophisticated. Northrop Grumman will be producing the Multi-function Integrated RF System/Multi-Function nose Array which is not only an electronically-scanned active array radar but also will incorporate communications and electronic warfare functions, the last provided by Sanders supported by Litton Amecom. Northrop Grumman will supply a conformal array, multifunction, imaging infrared distributed architecture system with six sensors that will be capable of air-to-air search and track, air-to-surface tracking, target cueing and missile warning. The data will be processed by the Lockheed Martin Tactical Defense Systems' integrated core processor and displayed on the pilot's wide-angle helmet-mounted display overlaid with target and threat data.

Sensor fusion is also being applied on a smaller scale within airborne platforms. ITT Avionics `AN/ALQ-211 Sirfc (Suite of Integrated Radio Frequency Countermeasures) selected for both the Bell Boeing CV-22 Osprey and the US Marine Corps' Bell AH-1Z attack helicopters will act as the sensor fusion processor for both on- and off-board sensors. Elements of the ALQ-211 will be included in the Royal Norwegian Navy's NH Industries NH 90 helicopter. Most of the helicopters ordered by European customers will have the Northrop Grumman/Eads/LFK AN/ AAR-60 Milds (Missile Launch Detection System) which is designed to detect threats and initiate active and passive self-protection, but whose master line replacement unit includes a sensor fusion function that might be used with the aircraft's radar, ESM and electro-optical systems.

Another rotary-wing aircraft scheduled to benefit from data fusion is the Boeing Sikorsky RAH-66 Comanche reconnaissance and attack helicopter. This will feature adaptations of a number of systems in service with other aircraft including the Comanche Integrated Communications, Navigation and Identification System, which is based on the Raptor's AN/ASQ-220, an improved Lockheed Martin AN/APG78 Longbow radar and target acquisition designation sight similar in concept to Tads/PNVS. There will also be an ITT electronic warfare suite and it is planned that from FY10 a third of the total fleet will become `heavy' attack Block 2 versions with sensor fusion.

Sensor fusion is now beginning to assume a greater role in ground warfare where technology has brought about a surge in systems that can be deployed around the Feba (Forward Edge of Battle Area). The problem is perhaps best understood by looking at the current range of British requirements to support ground forces. For long range detection of ground forces there is the airborne Astor (Airborne Stand-Off Radar) ground surveillance radar, which complements traditional aerial reconnaissance assets in both fixed and rotary-wing aircraft. On the ground there is the Mstar (Manportable Surveillance and Acquisition Radar) battlefield surveillance radar and there are armoured reconnaissance vehicles that will become a part of the Fres (Future Rapid Effects System) programme which itself is likely to become associated with the Watchkeeper UAV platform.

Finally there is the Fist (Future Integrated Soldier Technology), which may leave the soldier looking like a cut-price version of a Star Wars Imperial Storm Trooper and includes an on-helmet electro-optical surveillance system. The data from all these sensors, and traditional ones, needs to be integrated to create a comprehensive picture for decision makers, and while it is currently impossible to integrate the data from all these sensors at one site it is possible to fuse sensor data at a number of levels.

Moreover, the profusion of sensors is driving a requirement for sensor-to-shooter capabilities and the demand for a more rapid transfer of all forms of data to the front line, as emphasised during Operation `Enduring Freedom' in Afghanistan. Satellites and unmanned aerial vehicles (UAV) frequently detected valuable targets of opportunity but there were lengthy delays in transferring this information to soldiers or air assets, thereby permitting the enemy to escape or, worse still, leading to attacks on innocent civilians.

At present, information to ground-based headquarters comes from a variety of sources but primarily it is voice radio and this medium can be quickly swamped in a fast-flowing battlefield where multiple crises can materialise simultaneously. The introduction of digitised communications in which open and secure voice transmissions as well as text and visual data is broken down into a numerical format for transmission and reception will not only ease the transfer of data but allow vast amounts of it to be passed around units and headquarters. Fused sensor data will be able to pass around these command and control systems permitting commanders to assemble a more accurate real-time picture of battlefield activity.

There are signs that individual battlefield surveillance systems will begin to employ sensor fusion technology. The Saab Bofors Seos 400 mast-mounted system is based on an eight to twelve-[micro]m thermal imager, TV camera and laser rangefinder with other sensor options and can be used for surveillance and weapon control, fusing the image data to produce a single tactical picture.

The British Ministry of Defence appears to be considering sensor fusion options with its Astor system, which is being supplied by Raytheon. The company has been asked to conduct a study in which the aircraft would be able to control UAVs. And while the initial idea is to complement the airborne radar, the logical conclusion is to fuse the pictures from the UAV electro-optical sensors and the Astor itself.

The Watchkeeper programme calls for two types of UAV, both carrying thermal imager/television payloads operating within 150 kilometres of the Feba. One will be a tactical UAV, which will support battlegroups of infantry battalion size, and the other will support brigades and division/corps headquarters and will also have a Synthetic Aperture Radar with MTI features. There are four competing consortia led by BAE Systems, Northrop Grumman, Lockheed Martin and Thales and a degree of sensor fusion appears to be considered in all the proposals, for the system has to provide an imagery service that includes analysis. A single contractor will be selected during the first half of 2004 and industrial sources suggest the scope of the higher-level Watchkeeper system may be increased to include ESM sensors as a component of the Soothsayer programme and even air-to-surface missiles as an element in the Improved Fire Precision Attack (IFPA) programme.

The Fist is very similar in concept to the US Army Objective Force Warrior (OFW) or the Royal Netherlands Army's Soldier Modernisation Programmes and aims to exploit the new digitised command and control system, the General Dynamics (UK) Bowman, to provide military leaders with realtime images from the sharpest end of the front. OFW includes an infantryman's helmet with thermal and TV cameras, images from which will be fused to provide a comprehensive daynight picture. The system is scheduled to enter service from FY08, a year ahead of the Fist.

At present, neither the British nor US armies appear to be considering sensor fusion to link all levels of command, a pragmatic approach given the complexity of organising such a flow of information even with digitised communications. But for network-centric operations this must be the ultimate goal. Stepping boldly towards this goal are the Swedish Armed Forces, who will conduct an operational demonstration of network-centric warfare in 2005 with demonstrator contracts expected by Saab Net Defence and Saab Systems & Electronics later this year. The objective is to show that new methods of real-time command and control may be conducted with new information-gathering techniques, including sensor fusion in a secure Intranet-style architecture.

Navies seem slower to adopt sensor fusion technology, although the absence of sensor fusion was a contributory cause to the incident when the USS Vincennes shot down an Iranian airliner in 1987. One of the few products on offer is the South African Reutech Systems' RTS-6400 radar/ electro-optical tracker, which will probably be installed in the Meko A-200 frigates ordered by the South African Navy (the first of which has recently been christened SAS Amatola). An I/J-band (eight to twelve-GHz) radar tracker is augmented by a triple field-of-view thermal imager, a dual field-of-view TV camera and a laser rangefinder, which are designed to employ sensor fusion technology for greater accuracy and flexibility.

The Co-operative Engagement Capability (CEC) spearheaded by Raytheon is a form of sensor fusion but is currently confined to radar by creating a uniform track from a variety of on- and off-board sources, somewhat similar to the Eurofighter concept but on a grander scale. The system is to be developed further to conduct surface-to-air missile engagements beyond a ship's radar horizons with a hand-over of the mid-course guidance phase and its datalink input from one ship to another nearer the target.

The US Navy's participation in Operation `Millennium Challenge 2002' conducted off the West Coast of the United States during the summer of 2002 was to demonstrate network-centric warfare based upon sensor fusion. Specifically it was to develop the Joint Fires Initiative for land attack purposes using all available command systems and sensors, including those both of other services and other countries to create a single Total Target Set, which could then be engaged.

The development of sensor fusion to meet land, airborne and seaborne applications will undoubtedly become more established and then be enhanced during the remainder of this century. It is distinctly possible that virtual sensor fusion displays will be introduced, for this is a natural extension of simulation technology which is already widely applied in designing and planning as well as being considered for unmanned combat aerial vehicles.