This is the second post working through the Sabisu decision pipeline. In the first post we looked at why we created Sabisu and how the story starts with Data Acquisition.
What follows acquisition?
How long have you spent crafting MS Excel reports, creating complex macros to work out what the data is trying to tell you?
How long have you spent waiting for MS Excel to open because the quantities of data are so large?
What do you do when the datasets are so large there’s no way of processing them?
The answer is the second step on our data pipeline journey: Analytics, which encompasses data transformation, processing, calculations and algorithms.
The Sabisu approach is to build horizontal analytics; algorithms which will apply to many different processes and use-cases, which necessarily means statistical techniques are to the fore.
We integrate them right into the end-user’s line of sight – as soon as they maximise a chart, there’s a full array of statistical process control techniques to hand, e.g., Edge Correlation, which can provide simple prediction of incipient issues:
Or perhaps anomlous signal behaviour detection, used to identify when an asset failure mode initiates or a project characteristic changes:
You’ll find plenty more in the platform including curve fitting and dynamic warping, used to determine whether a situation is recurring, or a repeated process is running optimally:
There are also some specific analytics we run in Spark/Hadoop, or Python, or C++ engines (e.g., alarm analytics).
There are many analytics platforms out there – but beware the ones that merely draw graphs.
We’re always looking for better ways to find the story in the data, learning from data journalism (with the fabulous Guardian people) and other sources.
So we’re here in our journey:
Analytics is not visualisation; many tools skip analytics and draw pretty graphs with data. Visualisation is where we’re going in the next blog post.
We’re always interested in hearing from you with any comments or suggestions, feel free to get in touch.