Windows Azure Diagnostics and log files that are kept open, such as Tomcat logs

Resource Page Description
When transferring custom log files, Windows Azure Diagnostics runtime currently seems to require exclusive locks to the files being transferred. As the result, it doesn't work with log files that are kept open (in shared access mode), such as in case of Tomcat logs.

This project provides a solution for this scenario. The programing model has been designed to be easily pluggable into existing diagnostic configuration code and also to be easily removed when Windows Azure Diagnostics runtime starts supporting log files that are kept open.


Typical usage:
1. Download DiagnosticMonitorConfigurationExtensions.cs (from Downloads tab) and add it to your project. This file defines an extension method named AddDataSourceWithLockedFilesAndStartScheduledTransfer.
2. Call this method when configuring the Azure diagnostics. It is typically done within OnStart() method of a Role class.

For example if the diagnostic configuration code looks something like this:

            DiagnosticMonitorConfiguration config = DiagnosticMonitor.GetDefaultInitialConfiguration();
 
            // ... 
            // configure diagnostics
            // ... 
 
            DiagnosticMonitor.Start("DiagnosticsConnectionString", config);

in order to configure a scheduled transfer of log files that are kept open, the following lines of code need to be added before calling DiagnosticMonitor.Start (example for Tomcat)

            // Configure transfering Tomcat Logs
            config.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
            DirectoryConfiguration tomcatLogs = new DirectoryConfiguration();
            tomcatLogs.Container ="tomcat-logs";
            tomcatLogs.Path = System.IO.Path.Combine(RoleEnvironment.GetLocalResource("TomcatLocation").RootPath, "logs");
 
            Trace.Write("Configuring Tomcat logs at '" + tomcatLogs.Path + "' to be transferred to Windows Azure storage every " + config.Directories.ScheduledTransferPeriod.TotalSeconds + " seconds");
 
            // Windows Azure Diagnostics currently requires exclusive locks to the files being transferred and this doesn't work with Tomcat logs
            // So instead the files will be transferred without using DiagnosticMonitor
            bool useDiagnosticMonitorForTomcatLogs = false;
 
            if (useDiagnosticMonitorForTomcatLogs)
            {
                  Trace.WriteLine("using DiagnosticMonitor");
                  config.Directories.DataSources.Add(tomcatLogs);
            }
            else
            {
                  Trace.WriteLine("using Timer event and custom code");
                  config.Directories.AddDataSourceWithLockedFilesAndStartScheduledTransfer(tomcatLogs);
            }            

With the above sample, assuming that Tomcat is configured to write logs to a subdirectory named "logs", these logs will be copied to a container named "tomcat-logs" in Windows Azure blob and can be retrieved using any of the available tools that allows exploring the content in Windows Azure storage.

As shown above, the steps are similar as in case of regular logs. ScheduledTransferPeriod property of DiagnosticMonitorConfiguration.Directories is set and then DirectoryConfiguration object is created and configured. The main difference is that for log files that are kept open, instead of calling:

              
            config.Directories.DataSources.Add(tomcatLogs);

a method from the provided extension class is called:

            config.Directories.AddDataSourceWithLockedFilesAndStartScheduledTransfer(tomcatLogs); 

You can read more about Transferring Diagnostic Data to Windows Azure Storage here .