Exclusive Interview with Top MH370 Search Mathematician Neil Gordon

DSTG report 2

Fig. 1: Probability distribution function or “heat map” of where MH370 might have wound up.

When Australia’s Transport Safety Board (ATSB) was tasked with finding missing Malaysia Airlines flight 370, it tapped another arm of the government, the Defense Science & Technology Group (DSTG), to tell it where to look. There a team led by Dr. Neil Gordon devised a mathematical approach based on Bayesian analysis to weigh all the possible routes that the Boeing 777-200ER could have flown, given the seven Inmarsat “pings,” the plane’s fuel load, environmental conditions, and the different settings available on the autopilot. From this they derived a probabilistic “heat map” of where the plane might have wound up (Fig 1, above). The results showed that the jet most likely flew fast and straight, at high altitude, before running out of fuel and crashing. It was this analysis that allowed the ASTB to define the search area currently being scoured for traces of seabed wreckage. Yet, with less than 10 percent of the area left to be searched and not a trace found, it now appears they looked in the wrong place. Earlier this summer, the three nations responsible for the investigation—Malaysia, China, and Australia—jointly announced that they would not be extending the search after the last portion is completed this fall. Last month Dr. Gordon went on record for the first time to explain what might have gone wrong and where the next place to look for the plane should be. His answers formed the basis of an article for Popular Mechanics; for the readers of this blog I present a less filtered version of what Dr Gordon had to say.

One of the crucial decisions you had to make was how to treat the 18:22 radar return. In your report, you wrote, “The final reported position from radar was very at very long range from the sensor and there was a long time delay between it and the penultimate radar report. The report is at long range and it is likely to have rather poor accuracy because of the angular errors translates into large location errors at that range.” Are you confident that that radar return is not anomalous, it actually comes from the plane?

You’ve got to understand what our job in this investigation is. Our job is to take the data as presented to us by the accident investigators and project a trajectory from that.

Was there any explanation or speculation on why a plane would be detected at that point but not before or after?

I guess it was that they’ve just got snapshots off the radar screen. I’m speculating here but I would imagine they’ve recorded a video of the screen but they don’t necessarily have a digital backup of the measurements.

Any speculation as to why the the satellite data unit was turned off and back on again?

That’s not something that we’ve considered ourselves with. Our job is to process this set of numbers from the Inmarsat system.

At the end of the day, it’s a prioritization exercise, it’s not an exhaustive enumeration of all the possible ways you can do this, because, you know, I can draw trajectories that perfectly match the metadata measurements that fill a humungously large segment of the seventh arc. You can draw an enormous area that you’ve got to look. But the reality is, there’s a finite set of money that’s available, a finite amount of time. You’ve got to prioritize.

A lot of people have looked at the BTO data and said, “You can draw an arbitrary route.” And the answer is, yes, you can draw an arbitrary route, but if you didn’t know that this thing was generating these BTO arcs, the probability that you would fly it just right is essentially zero. 

It prefers simpler explanations rather than highly complex ones. So if doing one turn, staying on the same autopilot mode, fits the data as well as doing 35 turns, all having to be carefully orchestrated with these measurements you didn’t know were going to occur, then yes, it would prefer the single turn explanation, the simple explanation, over the highly convoluted, complex explanation.

Now, as I understand it the idea of Bayesian analysis is that as new data comes in the probability matrix changes. In the case of the seabed search, each time a ship makes a sidescan sonar pass, the probability in that swathe essentially drops.

Absolutely.

And the probability goes up by a commensurate amount somewhere else. 

Another figure you should probably have in your head, I suppose, is that the 120,000 square kilometers that’s currently funded for searching, encompasses approximately 72-ish, 75-ish percent of the probability.

So where is the rest of the probability? 

It’s outside that 120,000—as you increase the area away from that zone, you obviously, you increase the number you’ve searched. If you’d said before they started searching this 120,000 square kilometers, ‘What’s the probability you think you’ll find it in there?” I’d have said, “mid 70s.” Because that’s the probability content of that zone. Conversely, if you’d said, “I’m going to search 120,000 square kilometers, where do I put it to cover the most probability?” Then that’s where I’d have put it.

If it’s not in this 120,000 square kilometers, what are the alternatives?

The analysis that we do defines the probability distribution along the 7th arc. You then have to have a descent scenario. And the one that’s focussed on is the uncontrolled descent, and that’s done for a few reasons. So, if you look at the final electronic communication signaling that goes on in the final reboot phase, as it tried to boot up again at 00:19, that points to very high descent rates. If you look at the simulation results that Boeing have done for uncontrolled descent from that time, they’re consistent with the numbers you get from the final data messages. You also get the probability distribution of how many control mode changes we think have been made in the last five hours of flight. And the probability distribution there is very heavily stacked up on “none.” So if you strongly believe that there was no changes to the autopilot in the last five hours, the final communications messages are pointing to a very rapid descent rate, then clearly that’s going to prefer uncontrolled descent over controlled.

I guess the obvious problem with an uncontrolled descent, though, is that it would imply that you would wind up very close to the seventh arc.

Yes.

And so, would you expect—

Well, I’ve just told you, we’ve only searched just over 75 percent of the probability. When we started I’d have said there was a one in three chance you wouldn’t find it, even if everything was as you’d expect.

heat-map-and-search-box

Fig. 2: The current search box roughly approximates the 90% confidence region. (source: DSTG)

The 00:19 BFO value they believe would have been generated two minutes after the second engine flameout, and so basically within two minutes you’ve got a plane that’s now descending at 15,000 feet per minute, you’re going to quickly run out of altitude. Is the assumption, then, that despite this, things turning pear-shaped very quickly, that the plane nevertheless was able to get 40 nautical miles past the seventh arc? 

Don’t forget the 7th arc isn’t a precise entity. There is a spread of error on that, and how you map it down to the—it’s a range from the satellite, which has got an error on it, and then depending on what altitude you think it happened at, also adds an extra component to the error. And I guess they’ve based their spread beyond that on the results of the Boeing simulation of how the aircraft can come down from that point. Not something I know about, I take their word for it.

The alternative would be that it is very close to the 7th arc, but it’s further north of where they’ve been searching. That of course would require a curved, slower, lower-probability by your analysis route. But it seems like the ATSB is not looking at that right now.

I guess they have got funding in the area to search. You can certainly lay out where you would, if you’re prioritizing where you would go, and what you would do, you can see what you do, but at the end of the day there’s a finite area that’s funded for search.

If they said to you, “Dr Gordon, we’ve got another 50,000, should we go further north or should we go further out?”

Well, if you look at the probability distribution it would say, “Go up north.” [The probability distribution function] spreads out a lot up north. It’s quite constrained and peaked in the southern end, if you look at the distribution, and I guess it’s also interesting in the sense that that falls right into the maximum achievable range area as well. But then as you move north it’s still there but it tails off more slowly, which means you’ve got to search a larger kilometer-squared area to aggregate the same probability.

As you have all this new Bayesian information because of the absence of wreckage in the search zone, are you constantly updating your heat map.

There’s two ways, in a sense, you’d update it. One is you update it because they’ve dragged the sonar over and looked. And that starts fading things out and accentuating other areas. And then there are debris finds which enable you to think about where could they have—what are the plausible places they could have entered the water to get there.

You’re talking about drift analysis. 

Yes. But there are significant uncertainties attached to the drift models that generate those, and the distributions for those are much flatter and wider than the ones that come out of the dynamic analysis based on the satellite data. It takes a lot of aggregation of those to move things. Basically we haven’t had that aggregation yet.

There have been press reports that they are doing revised drift analysis and, perhaps it will rise to the level of significance?

As the data accumulates, the statistics will follow the right zones. As long as you represent the uncertainty properly in the information you’re putting in. So we’ve worked quite hard with David Griffin down from CSIRO to create a scheme to aggregate measurements based on his drift modelling into the process.

After the search has been suspended, will more reports be forthcoming?

I’d certainly imagine we would need to write something that aggregates our thoughts in hindsight, for sure. And update things. I guess it’s a point to say, we’ve put our mathematics out there, we’ve been touring around professional societies here, the international conference of signal processors, gave a talk there, we’ve been telling everyone “Here’s what we’ve done, here’s how the data works, please have a go and tell us where we’ve gone wrong.” So, we’ve made a big effort to ask people. The sort of difficulty we always come up with is, at the end of the day it’s a probability distribution, it’s not an analytic solution with a guarantee. It’s a distribution that represents the prioritized belief as to where you think it is relative to other places.

To get back to this BFO point at 00:19. The thing has just been turned back on. The question is, is it reliable? Is it anomalous, can we trust it as much as we trust some of the other data points?

And I guess that was some of the reticence to use it as an exact indicator, but certainly the manufacturers have done lots of tests for us on the warm-up characteristics are of their devices. So we do have an understanding of what the uncertainty that could be caused by that. I guess what we’ve done is, we’ve said, Let’s imagine that everything was the worst case, pushing the BFO in the opposite direction to disavow a descent, what if it had turned back the other way and it’s going completely the opposite direction—what if, what if, what if—and then try to come up with a bound as to what they were—what’s the minimum descent rate that would now be required to explain this?” And it’s still a big number. And if you look at it across the two of them, between the eight seconds apart, it’s an increasingly big number.

It’s a conundrum, though. Given your analysis and given this BFO value, the plane should have been where they were looking for it.

They’ve searched 70 percent of the probability. It’s not a guarantee, it’s—if you’re going to spend that much money and search that much area, that’s the area that’s going to get you the most probability. But it’s not a probability 1 event.

Maybe part of the problem and the expectations of the public was that there were a lot of very confident pronouncements.

I guess that no statistician would ever have said that.

When the search officially ends at the completion of the 120,000 square kilometers, this is going to be in the fall, our fall your spring, are you going to officially close the books, or are you going to keep working on it? What’s your stance going to be at that point?

I can only speak personally, that we would certainly keep open minds to new analysis we can do. If data comes along we’ll certainly analyse it. If new insights come along, we’re more than happy to analyze them. And go out and seek peer review on the methods, as much as possible throughout the last year and a half or whatever. There’s not only things to learn for this, on the methods, things we can learn as research scientists carry over to lots of domains of activity we do, so yes. We’re not going to be forgetting it.

Are you going to be issuing a report when the search officially ends, or is over? 

That’s a matter for the ATSB. But I can’t imagine they won’t. Certainly the Malaysians have to do a report on the investigation. That’s the Malaysians, not the Australians.

Before the debris was found, and all we had indicating that the plane went south was the BFO data, how did you guys know the BFO hadn’t been tampered with?

All I’ve done is process the data as given to me to produce this distribution.

After we spoke, I sent Dr Gordon two follow up questions, to which he replied via email.

At one time, the ATSB considered the 00:19 BFO values unreliable. What has caused the change of heart?

The BFO at 0019 was always understood to most likely indicate a descent. When the book was being prepared, there was uncertainty about the state of the reference frequency oscillator, namely steady state or start-up. For level flight, the BFO measurements are insensitive to altitude, so the model in the book provides only an indicative altitude estimate.  We did not aim to model altitude rate and the 0019 BFO was initially treated qualitatively. Subsequently, a review was performed by the SATCOM Working Group based on observations of 9M-MRO’s SDU historic transient characteristics and those of other SDUs.  This review resulted in limits on the amount of BFO variation at start-up and these limits allowed a more detailed treatment of the 0019 BFO. This understanding of the limits of the BFO start-up behaviour provides a bounded range of dynamics consistent with the measured BFO rather than a single value.

I would be very curious to see what the output of the filter would be if the starting point was the 18:22 radar return.

The attached figure shows the pdf as a function of longitude for an 1822 initialisation compared with an 1801 initialisation. They are not identical, but they are quite similar.

MATLAB Handle Graphics

Fig. 3: It makes little difference to the DSTG’s probability analysis whether their starting point is the 18:01 or the 18:22 primary radar detection.

If you’d like a truly unfiltered version of our discussion, I’ve put the whole transcript on Dropbox.