Comparing Popular SIEM Data Pipeline Designs: Elastic , ArcSight, QRadar, Wazuh, and Splunk — Part 2

ArcSight microfocus now open text Logo — Credit microfocus.com

If you found this article interesting, your support by following steps will help me spread the knowledge to others:

👏 Give the article 50 claps

💻 Follow me

Deploy your next app in seconds. Get $200 in cloud credits from @DigitalOcean using my link: https://m.do.co/t/a71522db03a2

We will continue with what we discussed on Part 1. In this article we will provide detailed analysis of the pipeline designs of f ArcSight. The goal or motivation is to guide organizations in understanding how ArcSight solution aligns with their security infrastructure and operational requirements. To provide clarity, and similar to what provides in the first part , a list with descriptions and diagrams will be included, as a picture is worth a thousand words.

Before we delve in to the article , its worth noting that **MicroFocus has been acquired by [OpenText](https://www.opentext.com/)**, and as part of this transition, the ArcSight name has not been independently rebranded but remains under the OpenText branding umbrella.

My assessment is based on my knowledge of the application from a year ago. While many changes are likely, the new YouTube videos posted by OpenText suggest the changes are mainly focused on UI improvements and some additional features.

OpenText/Microfocus ArcSight SIEM suite

According to OpenText website OpenText ArcSight, previously part of Microfocus, is a comprehensive Security Information and Event Management (SIEM) suite designed to enhance threat detection and response capabilities. The suite includes the Enterprise Security Manager (ESM), which offers real-time threat analysis, data collection, and a native threat intelligence feed. ArcSight Logger which used for for high-speed data logging and search, enabling organizations to collect, store, and analyze large volumes of log data from various sources.ArcSight also integrates Security Orchestration, Automation, and Response (SOAR) functionalities, providing automated incident response to reduce threat exposure.Additional components available for purchase as part of the suite include ArcSight Recon, ArcSight Data Platform (ADP), ArcSight Interest, and ArcSight Investigate.

SIEM Deployment Models

ArcSight SIEM can be deployed in three main models which can be listed as follows:

1. Self Managed: Entire infrastructure can be hosted and managed on premises or your data centers. While this model offers control over data it requires resources and expertise for maintenance operation and scalability.
2. Hybrid Based: Organizations can combine on premises infrastructure with cloud based services.This would provide you with the flexibility and the resiliency of the cloud for storage services while maintaining some control over critical data.
3. Cloud Model: This can be achieved by fully utilizing Microsoft Azure , Amazon Aws cloud services or other cloud providers.I couldn’t find any information on the OpenText website indicating that they have a cloud solution. If you have any details, please share them in the comments.

Message Format

Messages in the ArcSight SIEM are in The Common Event Format (CEF). Base CEF Format is listed as follows:

CEF:Version|Device Vendor|Device Product|Device Version|Signature ID|Name|Severity|Extension

Log Entry Example:

CEF:0|Cisco|ASA|9.8.2|722027|Deny TCP (no connection)|6|src=192.168.1.10 dst=10.0.0.5 spt=12345 dpt=80 srcPort=12345 dstPort=80 in=outside out=inside msg=Deny TCP (no connection) cs1Label=ACLName cs1=outside_access_in cs2Label=Interface cs2=GigabitEthernet0/1

Main Components

1. ArcSight Smart Connector(s): Tool that can collect and normalize all the data you need from variety of source devices.
2. ArcSight ESM : ArcSight Enterprise Security Manager (ESM) is a comprehensive threat detection, analysis, triage, and compliance management SIEM platform[1].
3. ArcSight Logger: Log management and storage solution that can forward events to ESM and other destinations.
4. ArcSight Management Center (ArcMC): ArcSight tool that offers unified control of Smart Connectors across your environment.
5. The ArcSight Console: workstation-based interface intended for use by analysts and security staff in a Security Operations Center.
6. Kafka: Apache Kafka is a distributed event store and stream-processing platform.

SIEM Pipeline Design — 1::Basic ArcSight SIEM

The diagram below illustrates a simple and effective log management and analysis system using the ArcSight logger , ESM and Smart Connectors. Logger generally deployed within the perimeter firewall for enhanced physical security to protect collected event data. It independently handles syslog and log file events from various hardware and software network products and can interoperate with ESM utilizing builtin Forwarders and Receivers.

Basic ArcSight SIEM pipeline — Diagram by the author

The design focus on simplicity ease of management and efficiency and it features ArcSight Smart Connectors for data collection . Additionally it utilizes Logger Receivers to receive logs from Syslog Based Appliances where the agent can not be installed. However, it risks potential bottlenecks and intermediate buffering, which may impact data reliability during high data volume periods.

• Project Cost: Small to Moderate — Licensing costs may increased following the acquisition.
• Complexity: Medium
• Time: 5–7 days
• No. of Personnel: 2 -3
• Business Size: Medium
• Detection Scope: ArcSight logger combined with ESM would provide comprehensive network and host-based event monitoring.

SIEM Pipeline Design — 2 ::ArcSight SIEM with Kafka

The second design integrates ArcSight SIEM with Apache Kafka to enhance scalability and real-time data processing capabilities. Kafka acts as a distributed event store and stream-processing platform, enabling the system to handle large volumes of data more efficiently.

ArcSight SIEM with Kafka data pipeline — Image by the author

• Project Cost: Moderate — Licensing costs may increase due to Kafka integration.
• Complexity: Medium to High Time: 7–10 days
• No. of Personnel: 3–5
• Business Size: Medium to Large
• Detection Scope: ArcSight logger with Kafka enhances real-time processing and comprehensive network and host-based event monitoring.

SIEM Pipeline Design — 3 ::ArcSight SIEM with Kafka and ArcMC

This design integrates ArcSight SIEM with Apache Kafka to enhance scalability and real-time data processing capabilities. Kafka acts as a distributed event store and stream-processing platform, enabling the system to handle large volumes of data more efficiently. This design also includes ArcSight ArcMc to provide centralized management and monitoring of the ArcSight smart Connectors and loggers.

ArcSight SIEM with Kafka and ArcMC data pipeline — Image by the author

• Project Cost: Moderate to High — Licensing costs may increase with ArcMC.
• Complexity: High
• Time: 10–14 days
• No. of Personnel: 4–6
• Business Size: Medium to Large
• Detection Scope: ArcSight logger with Kafka and ArcMC provides centralized management and comprehensive network and host-based event monitoring.

SIEM Pipeline Design — 4 ::ArcSight SIEM with Kafka ArcMC & Loggers Peering

The provided diagram illustrates the integration of ArcSight SIEM with Apache Kafka, ArcMC, and Loggers Peering. This setup enhances the system’s scalability, real-time data processing capabilities, and redundancy.

ArcSight SIEM with Kafka ArcMC & Loggers Peering pipeline — Image by the author

• Project Cost: High — Additional costs for loggers peering or clustering.
• Complexity: Very High
• Time: 14–21 days
• No. of Personnel: 5–7
• Business Size: Large
• Detection Scope: ArcSight logger with Kafka, ArcMC, and loggers peering enhances redundancy and provides comprehensive network and host-based event monitoring.

SIEM Pipeline Design — 5 ::Advanced ArcSight SIEM

In addition to the integration of ArcSight SIEM with Apache Kafka, ArcMC, and Loggers Peering. This design leverages a Windows Event Collector as a central log collection point for various data sources, including Linux systems. The setup involves configuring the Windows agent to read logs from the forwarded channel, while a Linux connector listens for syslog messages before sending the data to Kafka. This approach ensures efficient and centralized log collection from both Windows and Linux systems, enabling seamless integration into the Kafka-based SIEM pipeline.

Advanced ArcSight SIEM pipeline — Image by the author

• Project Cost: Very High — Costs include advanced analytics tools and machine learning integration.
• Complexity: Extremely High
• Time: 21–30 days
• No. of Personnel: 6–10
• Business Size: Very Large
• Detection Scope: Advanced ArcSight SIEM with Kafka, ArcMC, loggers peering, ESM Cluster, and additional analytics tools offers state-of-the-art, proactive threat detection and comprehensive network and host-based event monitoring.

References

• ArcSight ESM 7.5 Documentation: ESM 101
• ArcSight SmartConnectors 8.3 Documentation: Installing SmartConnectors

Disclaimer

This article is based solely on research conducted in a controlled home lab environment. All techniques and methodologies discussed are intended for educational and research purposes only. The findings and experiments were performed on a private network and the information and techniques discussed here do not involve or disclose any proprietary or confidential data from any company. While some best practices mentioned here may resemble those used in professional settings, any such resemblance is purely coincidental or attributed to the best practices publicly published on the internet.Any attempt to replicate these methods should be confined to a safe and authorized lab setting, and the author is not responsible for any misuse of this information outside of these contexts.