Overview
The ExLogAnalyzer is a framework that parses logs and triggers events based on the log entries. It can analyze (replay) logs across multiple servers and multiple log types in a single run. ExLogAnalyzer is extensible via per log type extension (which is the layer that maps the log syntax into semantics) as well as extensible via analyzers for such extensions. The goal of ExLogAnalyzer is to make it super easy for developers / customers to write their own analyzers which mine the logs to answer the question at hand. Note that the analyzers do not do any kind of parsing, but they process events raised by the extension and through the data made available to the event arguments, do the appropriate accounting. The ExLogAnalyzer was started as an internal project in Microsoft Exchange and has 3 log type extensions already implemented, but it is designed to work for any product once the appropriate extension is written. The community should be able to developer their own extensions and analyzers and share them or contribute them to the project.
See ExLogAnalyzer.ppt in the download for more details.
What are the goals for the ExLogAnalyzer?
- Simplify analysis of logs.
- Multi-Server support (process log files that span multiple servers).
- Multi-Log Type support (process / cross reference logs of different log types to produce a single report).
- Separation of syntax and semantics.
- Provide an extensibility model to develop and distribute extensions (to support new log types) and analyzers (to encapsulate reporting logic).
- Extensible to support a variety of log formats.
- Ability for the community to developer their own analyzers or even extensions.
What are the Exchange specific goals for ExLogAnalyzer?
- Support for both on prem / data center usage.
- Support for Exchange 2007 / 2010 log types.
What are some of the design guidelines of ExLogAnalyzer?
- Ability for the community to develop extensions / analyzers.
- Have a simple process for tuning analyzers on server boxes without the need for development environment (e.g. Visual Studio).
- Ability for the community to easily share their custom analyzers.
- Take no dependency on products for which the logs are being analyzed.
- Take no hard dependency on tools (e.g. visualization tools).
- Easily support new log types, changes to existing logs.
How to run the tool?
ExLogAnalyzer.exe –SmtpInputPath “d:\....\SmtpReceive\” –OutputPath “d:\LogAnalyzerOutput” –StartTimeUtc 2009-06-16T19:35:00 –EndTimeUtc 2009-06-16T22:00:00
- If OutputPath is not supplied, it will default to current directory. Each extension will output to a sub-directory of the output folder.
- Start/End time are optional, you can specify both, one of them or none.
- StartTime/StartTimeUtc and EndTime/EndTimeUtc provides the flexibility of using Utc or Local time to specify the time range.
- For a complete set of support configuration options see the app config (ExLogAnalyzer.exe.config).
- ExLogAnalyzer will use the configuration values in the app config. Values specified on the command line will override the app config values.
How to analyze the output of ExLogAnalyzer?
The log analyzer outputs its own progress / error log under <OutputPath>\Log\ folder. The content of the other peer folders depend on the extension / log analyzer. Each log analyzer should have its own documentation about the data it generates.
What extensions are currently available for it?
- Connectivity Logs
- Message Tracking Logs
- Smtp Receive Logs
What analyzers are currently available for it?
- There is currently more than 20 Exchange specific analyzers distributed with ExLogAnalyzer.
How to contribute to ExLogAnalyzer?
- Develop your own analyzers or extensions.
- Share them with us / community.
- Spread the word about ExLogAnalyzer.
Coding Guidelines
- All code must have proper Xml documentation.
- All code must pass StyleCop.
ExLogAnalyzer Design Abstractions
- Job - A class that captures the work to do within one run of ExLogAnalyzer. This class triggers events like OnStartup and OnShutdown.
- ExtensionFactory - This is to be implemented by extensions to be able to detect / return the list of analyzers for that log type as well as the list of file groups to be processed.
- ExtensionManager - This is a class that detects all available extensions and uses their ExtensionFactory to detect the corresponding analyzers and LogFileGroups to be processed.
- LogAnalyzer - A base class for all log analyzers. Typically each extension would derive from that to provide {LogType}LogAnalyzer class. Each analyzer for that log type will implement its own concrete MyAnalyzer{LogType}LogAnalyzer. LogAnalyzer events are not tied to a specific session. They include events raised by the GlobalExecutionContext (OnStartup, OnShutdown, OnTimeUpdated, OnLogLine). Extensions can also implement their own log type execution concept which adds more events that are not tied to a specific session. LogAnalyzer is also responsible to create the appropriate SessionLogAnalyzer instance when a new session is started in the log file.
- SessionLogAnalyzer - An instance that is typically created by a specific log analyzer. Each extension would implement its own derived class {LogType}SessionAnalyzer. Such classes will have session related events like OnConnect, OnDisconnect, OnMailFrom, etc. The typical pattern is for analyzers to have a log analyzer that contains the corresponding session log analyzer as a private class.
- LogFileGroup - A folder (mask can be applied) on a single server is a LogFileGroup. The assumption is that if the files in this folder are sorted by name / sequence number, then the events within all the files will be expected to be chronological. All LogFiles belonging to a LogFileGroup are of the same log type.
- LogFile - A file that contains N LogLines.
- LogLine - A log line that contains an event from the log. A log line typically includes a timestamp in addition to some other common fields and possibly a text fields that can contain variable data based on the event type.
- LogSession - This is a base class for all log sessions. Each extension will derive its own {LogType}LogSession. This base class includes basic information like session id, start time, and end time. Session instances are typically passed inside the event arguments for the extension specific events. It is generally recommended to put event type specific data in the event args, rather than in the session. This will make such data only accessible on the event where it is valid and makes sense.
- Execution Context - An execution context is used to raise events on a set of log analyzers associated with the concept. Hence, the core code would raise the event once and execution context will result in creating the appropriate event arguments and triggering the event for all registered analyzers or session analyzers.
- GlobalExecutionContext - That is an execution context that is common across all supported log types. It includes methods to raise events like RaiseOnStartup, RaiseOnShutdown, RaiseOnLogLine, UpdateTime. It also includes the logic that tracks sessions that are not closed during the job run and triggering the OnTearDown for the corresponding analyzers.
- Configuration - A class that is used to retrieve all the configuration parameters. This configuration is defaulted by the app config and values are overriden via the command line.
- Log - This is class that is used to generate the ExLogAnalyzer progress log file.
- Parser - A class that simplifies the parsing of strings.
- StringUtils - Some string utility methods.
License
- See license.doc in the download for details about the license.
|