Anomalies in Logs


Building blocks for complete log anomaly detection solution


Focus on problem and not on data infrastructure

Step 1: select Qubs

We have log-streaming solution, therefore we choose Apache Kafka as our first Qub. For data source, we will use Log Generator Qub.  Then, we select Apache Druid Qub, ClickHouse Qub and OpenSearch Qub as our data store implementations. Select and drag log Anomaly Detection building blocks: DeepLog Qub, LogAnomaly Qub and LogRobust Qub. Drag Kafka topics for communication between components. Finally, drag Grafana Qub to use it as a visualization tools for anomaly recognition validation.

Step 2: connect Qubs

Select from the catalogue which data connectors to use; connect Log Generator to a Kafka topic, say topic-logs. From that topic ingest data into OpenSearch, ClickHouse, and Apache Druid by dragging arrows. Qubinets platform will do the rest. Don’t forget to consume data from topic-logs by all deep learning log anomaly detectors, and direct their output to another topic, say topic-anomalies. Connect Grafana to all three data storage engines.

Step 3: deploy the solution

Your model is ready to be verified and deployed to Kubernetes cluster of your selection. At the same time, Qubinets platform will make all setup for observability of your Qubs so that you can see amount of logs generated, various topic statistics (in, out, lag), ingestion mechanisms performance and most importantly your deep learning models performance.

Step 4: run data and observe

Run the generator, freely change your log data production rates and other parameters. Check your anomalies via instantiated Grafana Qub. Use Qubinets Observability graphs for system performance measurements. Decide what seems to be the best for your needs. Easily remove unneeded components. Voila – your system is ready!


Modular architecture with easy-to-try components

Qubinets - Open source data environments for your AI/Data project | Product Hunt DigitalOcean Referral Badge