Real-time Cybersecurity Threat Detection
To detect cybersecurity threats in real-time, huge volumes of streaming data from servers, network infrastructure, and applications logs must be analyzed in real-time using AI to identify possible threats. Event-driven applications must act quickly and reliably to notify system administrators and update firewall rules to block dangerous traffic.
Benefits that Pravega Offers
- Scalable data ingestion to efficiently handle varying loads over time
- High availability design
- Durable and low latency storage
- Connectors to stream processing engines such as Flink and Spark
- Automatic deletion of older events based on a retention policy
Example Solution Architecture
- Data collectors collect security events from servers and network devices.
- A Flink streaming job aggregates all events from all streams, applies an AI inference model to detects threats, and outputs the following:
- A summary of inferred threats will be updated in the database.
- Threat details will be permanently stored in the Threats stream.
- The database can be PostgreSQL, Elastic Search, Pravega Search, or anything else supported by Flink.
- A web server provides a UI to view a security dashboard. If using Elasticsearch or Pravega Search, this can be Kibana.
- Additional event-driven applications can respond to events in the Threats stream, for instance, by sending a text message to system administrators or by automatically updating firewall rules to block dangerous traffic.