The popularity of AI is soaring day by day. AI has evolved significantly over the past few years and has found its applications in almost every domain, from healthcare to finance to security.
In simple words, Artificial Intelligence (AI) is the ability of our program to think and make intelligent decisions based on past experience without any human interference. Applications are being built using the AI technologies for almost all industries including agriculture, education, and manufacturing etc.
We are constantly getting more and more into the digital era,it suffices to say that today we are more connected than ever. Every aspect of our life is connected to computing devices, ranging from huge data center’s to IoT devices, where a growing number of online entities are collecting vast amounts of personal data. It is getting more important to have a strong security system that can be capable of detection and prevention of illegitimate intrusion.
As we are getting more and more connected, every computing device we use tracks every action there in what is called “logs”. Logs are files that store entries from every software that is run, every network call that is made, every login information and so on. Based on the user that is using the system, patterns emerge in their usage.
The log files contain homogeneous data and any unusual behaviour or an anomaly results in a pattern change in the log files. Looking at the system logs which are easily available, for any unusual pattern can reflect anomalies in the system.
Log files are created in huge amounts in a system and parsing them manually in search of any anomaly might not be possible for a human expert.
A recent publication has leveraged the benefits of AI to detect any kind of anomaly in the system using log files.
The system works in the following manner:-
At first, the unstructured log entries are processed into a structured format by extracting “log key” from the log entries. Then identifiers are then used to group or ungroup log entries produced by concurrent processes to a single sequence. From this parsed data the valuable features are extracted and converted into numerical feature vectors over which deep learning techniques can be used for pattern recognition.
The extracted features are extracted by encoding the logs into numerical feature vectors using a fixed window/kernel to group the log data into log sequences.From the feature vector generated, the objective is to find the anomaly in the pattern of the log messages that raises a flag for suspicious activity or potential threat using a CNN. Convolutional Neural Networks(CNN) have shown state-of-the-art results in the field of pattern recognition. For the pattern recognition process, the weights of the filter of the convolution operation are learned through backpropagation and gradient descent methods, to extract the features from the previous layer by applying the learned receptive field filter. Non-linear activation functions like Sigmoid, ReLU, Tanh and others are used for the non-linear transformation of the feature maps.
Thus, recognising any kind of potential malicious activities using deep learning techniques.
Rohit Sinha, Rittika Sur, Dr.Ruchi Sharma, Dr.Avinash K. Srivastava(2021,Oct) Anomaly Detection Using System Logs: A Deep Learning Approach, International Journal of Information Security and Privacy (IJISP), DOI : 10.4018/IJISP.285584.
Written by Rittika Sur & Rohit Sinha