Machine learning and the holy grail of anomaly detection are getting a lot of attention from investors and businesses at the moment. A subset of artificial intelligence, machine learning explores the study and construction of algorithms that can learn from, and make predictions on, data.
To detect an anomaly, you first have to determine what is normal and this is easier said than done for the majority of businesses. Anomaly detection requires an organisation to define roles and responsibilities and to put in place robust identity and access controls – all of which pose the question: if this was defined and in place, would I need anomaly detection anyway?
Don’t get me wrong. There is huge scope for machine learning to become effective, simply because of the way we can now collate, store and analyse data with new business dynamic SMAC (Social, Media, Analytics and Cloud). Machine learning is now being used for assigning hospital beds to root cause analysis for quality improvements and for advanced marketing activities to personalise the consumer shopping experience. But, in my opinion, there is no doomsday scenario in this.
Humans are still the best at analytics and intuitive reasoning. Where machine learning really helps is in enabling our experts to become more efficient and be able to draw a level of intelligence from the huge amount of “data noise”. The Watson project from IBM, for example, is pushing the boundaries of collaboration around machine learning and has demonstrated some remarkable results, while Darktrace and Cylance are beginning to get traction in specific areas leveraging advanced mathematics to train the machine. At NTT Security, we are leveraging aspects of machine learning to predict Command & Control domains.
But, as I said, experts in their field effectively have to train a machine in a narrow and specific area to make sense of the “data noise” or do further investigations to eliminate false positives, which is the bane of many information security professionals. We have some way to go to really leverage learning machines. Clearly we are at the start of a comprehensive change in data analysis but first we need to define normal and this takes more than a machine.
In summary, policies and procedures have to be defined and adhered to, roles and responsibilities have to be clearly assigned, and identity and access controls have to be enforced. Only then can businesses define normal and detect anomalies.
We will see a greater focus on advanced analytics – looking for the needle in the haystack – and increased awareness that organisations need to reduce complexity and have greater visibility across their estate.