Log data can be a tremendous resource for protecting digital assets against cyber attacks. Trouble is, trying to make sense of all the logs generated by IT networks is like pointing a fire hose at someone dying of thirst. They’re desperate for a drink, but they simply can’t handle that much water all at once!
The deluge of log data comes from software applications and hardware devices, which automatically and continuously deliver time-stamped log files for each event or transaction that takes place. Microsoft systems produce Windows event log files while UNIX servers and networking devices utilize the Syslog standard. Devices that create W3C/IIS log files include application servers (like Apache and IIS) as well as load balancers, firewalls, proxy servers, and content security appliances.
Some networks are big enough to produce billions of log files. So the challenge lies in how to manage the processes and policies for analysis, storage, archiving, and eventually the disposal of the logs. The “fire hose” of information presents a daunting task for IT teams overseeing security and compliance.
But to build strong security postures and to comply with regulations, IT needs to figure out a way to “drink the water” efficiently. In this article, we present an overview of file logs along with a few tips on how to make your log management processes work more efficiently.
Real-Time Log Analysis Exposes Cyber Attacks
Logs create documentation of events that take place on IT systems. The events typically represent some message, token, count, pattern, value, or marker that can be recognized within an ongoing stream of monitored inputs—such as network traffic, error signals, thresholds crossed, and counts accumulated. User-generated events include keystrokes and mouse clicks while system-generated events include program loading and errors, which may pertain to the operating system, an application, or a security event.
The ability to perform real-time network log event collection and analysis plays a critical role in exposing cyberattacks in progress and identifying any damage that’s already been done to digital assets. Each log file contains valuable information, and with proper analysis you can identify intrusion attempts, misconfigured equipment, and other network problems.
In addition to helping you identify network issues, log management also facilitates compliance. Major regulations—such as PCI DSS, HIPAA, and SOX—require you to retain and review logs on a regular basis.
Log analysis is best conducted with security information and event management (SIEM) software, which
tracks and creates audit trails of changes to IT infrastructures. The leading SIEM solutions increase your log management efficiency by automating the collection and consolidation of data and then correlating events from multiple devices across your entire IT environment.
Tips for Making Log Management More Efficient
One of the keys to effective analysis is having the ability to compare current activity to normal activity on your network. This lets you identify anomalies more easily and delve into the root cause of problems.
In addition to monitoring activities across servers, firewalls and other network devices, be sure to monitor workstation and smart device logs. These devices can tell you about events such an end user plugging in a USB. In this example, you can see who connected the USB and whether that user belongs to a group that’s authorized to access that particular device.
Here are some additional tips to improve your log management processes:
- Check devices to make sure the date is correct and that the time syncs across all systems. When correlating security events, the time must be accurate among devices to properly diagnose issues.
- Define the events pertaining to attacks that occurred in the past 12 months to get a sense for what types of activity have recently triggered network breaches.
- Utilize a threat intelligence service to identify the likely attributes of future successful attacks and then determine which event alerts you need in order to detect and stop those attacks; focus on the attack types that represent the greatest threat to your business.
- Rely on the default settings of your SIEM solution for collecting logs from laptops, desktops, servers and network devices; the default settings provide sufficient information for most security events.
- Avoid information overload by adding additional log file details to your settings only when there’s an on-going security event. And don’t forget to turn off the extra detail when the event ends!
- Consider using a third-party application to collect log data from mobile devices; typical SIEM solutions can extract basic log information from mobile devices but with far less detail than what can be gleaned from desktops.
- Generate log details on local systems, but filter and send only critical events to your SIEM; this is another way to avoid information overload. You can hold off on retrieving added details from local systems when performing forensic investigations.
Before selecting a centralized SIEM solution that gives you the ability to apply the tips listed above, test at least three to see which one will work best in your environment. For example, test the speed of queries, which should receive answers in under 10 seconds. If your IT team has to wait longer, they won’t use log analysis as often as you want them to, and your investment will not produce a sufficient return.
Centralized Log Management is Vital to Security and Compliance
Centralized log management should be a key component of your security and compliance initiatives. With centralized logs, you can monitor, audit, and report on file access, unauthorized activity by users, policy changes, and other critical activities performed against files and folders containing proprietary or regulated personal data—such as employee, customer and financial records.
Your centralized strategy should include overseeing Windows event logs, syslogs and W3C logs (as referenced above). That’s because information breaches come equally from internal and external sources. Windows event logs give you visibility into harmful activities by disgruntled employees while syslogs enable control over your network perimeter.
Windows systems have several different event logs that should be monitored consistently. The most important is the security log, which provides key information about who is logged onto the network and what they are doing. Security logs help IT understand if vulnerabilities exist in a security system implementation.
Syslog is a standard message format and log transmission protocol. Networking devices, UNIX and Linux systems, and many software and hardware platforms, implement syslog to transmit and collect those log files in a centralized log management repository. With syslog information, you can capture highly-detailed information about the status of a device or a number of devices. The information can then be parsed to identify atypical behavior through changes in operational or performance patterns.
Storage of syslog log data can also support compliance efforts by providing audit logs to trace any event that may affect network reliability and protection of data. This is important as it proves to auditors that you have control over all information.
W3C comes in handy by providing information on user and server activity. These logs can be monitored for information to identify unauthorized attempts to compromise a system.
For example, your Web server IIS log files are a fixed ASCII format that cannot be customized. These logs record more information than other log file formats, including basic items, such as the IP address of the user, user name, request date and time, service status code, and number of bytes received. In addition, the IIS log file format includes detailed items such as the elapsed time, number of bytes sent, action, and target file.