Tiny Needle in Big Data
Weak signal emission, detection, retrieval and analysis
We are repeatedly asked about the predictive powers of Ethersource and we need to underline that Ethersource has no “predictive” power per se. The reason Ethersource can estimate – or forecast – the percentages of public votes in a television contest or the outcome of a national election with some accuracy is simply that Ethersource reads and understands massive amounts of data.
This post will focus on something slightly different, namely the ability to find, understand and analyse one or a few tiny pieces of crucial data in massive amounts of data. It is the needle in the haystack dilemma with the only difference that your proverbial haystack is the Internet, and that you have very limited time to detect the relevant blog posts, tweets or chat entries and to analyze them in time to take action. This is what we call weak signal detection.
The Yeonpyeong attack
On November 23, 2010, 1434 local time (0534 GMT), North Korea fired more than 200 artillery shells at the South Korean island of Yeonpyeong, killing at least two soldiers in the heaviest attack since the end of the Korean War in 1953. The attack, which was somewhat of a shock, had a substantial but short-lived market impact. A day after the firings, South Korea’s benchmark KOSPI index opened 2.33 percent lower and the KRW weakened against the USD.
However, the surprise attack was preceded – on November 22 – by an alert signalling a sharp increase in the weak signal violence propensity index (VPI) for Korea as a target, monitored by Ethersource:
Two months later, on January 23, 1332 local Moscow time (1150 CET), the weak signal detection for Putin showed an extremely sharp increase in the violence propensity index (VPI), triggering an automatic alert:
Not even three hours later, 1632 local Moscow time (1432 CET), a suicide bombing at Moscow’s Domodedovo airport kills at least 35 people and injures more than 100. The market impact was short-lived: Russia’s rouble-denominated stock market MICEX fell by nearly two percent following the blast.
Just-in-time weak signal detection and analysis
The above examples are from an Ethersource prototype and quite dated. There have been many unexpected events since and the technology has been refined. Our ongoing analysis confirms that many unexpected events are preceded by leakage of weak signals. Such signals are very difficult – or even impossible – to systematically detect with other technologies. It might appear a big step from anticipating some kind of violent act to actually being able to take counter-measures, as this would require both the availability and the timely detection of more detailed information. Ethersource has addressed this challenge by allowing instantaneous and automated ranking, retrieval and analysis of any Internet post or document contributing to a weak signal alert on any sentiment concept, such as violence propensity, toward any given target concept.
Near real-time weak signal detection and analysis holds great promise for security and financial applications, in our view.