Information Security Blog

Log Consolidation or SIEM? (BOTH !!!!)

In the “old days” (pre-2009) there was a fundamental decision to make when implementing log management technology. Do I go log consolidation (LogLogic, LogRythm, Splunk) or do I go Security Information Event Management (Arcsight, NetForensics, Sentinel)? It really boiled down to whether or not you needed the increased capabilities of SIEM such as real-time correlation, dynamic real time data normalization, and advanced integration with other core systems including Identity Management, Network Monitoring, Ticketing, CMDB, & Vulnerability/Configuration Management.

The world has changed. We are of the strong opinion that for most F500 class log management implementations it’s no longer SIEM OR Log Consolidator — its now SIEM AND Log Consolidator.

The reason is simple a growingly insane number of events that need to be captured. As compliance requirements relating to PCI, PII, PHI, et al drive the need for a greater and greater number of events to be captured, one of SIEM’s advantages (having a RDBMS as its back-end) can quickly become a disadvantage. When data rates start to approach 5,000 events per second (which is definitely reachable in many environments) the challenges/limitations associated with getting data into and out of an RDBMS can become problematic:

Few DBAs have experience optimizing databases with these characteristics.
Database storage requirements (often on a very expensive SAN) can approach a Terabyte per day.
Recovering from a database issue (e.g., rebuilding a corrupted index) becomes very challenging and time consuming.

Leading vendors such as Arcsight and Novell (Sentinel) recognized this issue and have recently developed Log Consolidators that are intended to work seamlessly with their SIEMs so that you can enjoy a best of breed approach to log management/forensics/compliance.

Leveraging Log Consolidators at the “edge” (e.g., at various business units) simplifies the process of deploying a solution, provides a local searchable event repository, and reduces storage requirements by a factor of 10 (or more). Only those events that require SIEM capabilities (say 20%) are forwarded to the SIEM in real-time to ensure that full SIEM functionality is retained while eliminating the pain associated with a 5,000 EPS RDBMS).

This approach really allows an organization to end up with the “best of both worlds”.