The ask:

Ensure that the front end logs from the Azure Application Proxies are flowing into the SEIM via Windows Event Forwarding (WEF).

This post outlines the current challenges with the ask, and provides an approach for converting .ETL logs to .EVTX format as they arrive.

Azure Application Proxy Overview

If you are unfamiliar with the Azure Application Proxy the Microsoft overview is as follows:

Azure Active Directory’s Application Proxy provides secure remote access to on-premises web applications. After a single sign-on to Azure AD, users can access both cloud and on-premises applications through an external URL or an internal application portal. For example, Application Proxy can provide remote access and single sign-on to Remote Desktop, SharePoint, Teams, Tableau, Qlik, and line of business (LOB) applications.

A better source of information for the Azure Application Proxy is: https://learn.microsoft.com/en-us/azure/active-directory/app-proxy/application-proxy

Azure Application Proxy Logging

The Azure Application Proxy has two main logs that are helpful for administrators and security teams:

1. The admin log (standard .EVTX format)
2. The session log (analytic and debug .ETL format)

The second log, “Session”, is very useful for security teams.

The events 24028, 24029, 24030 detail the actual connections being made to the App Proxy connector instance, and are very similar to what you might expect if you were monitoring a web server.

The Challenge

The challenge with the Session log is that it is an .ETL log, rather than .EVTX like the Admin log.

The Admin Log:

The Session Log:

I was aware that “Analytic and Debug” .etl logs were not designed to be forwarded using Windows Event Forwarding, but because we really needed this information in our SEIM we decided to ask Microsoft to be sure:

Question: Can we use Windows Event Forwarding with “Analytic and Debug” logs?

Answer: No, Windows Event Forwarding (WEF) is not designed to work with “Analytic and Debug” logs in the .etl format. WEF is primarily used for forwarding Windows Event Logs in the .evtx format.The “Analytic and Debug” logs, which are in the .etl format, are typically used for advanced troubleshooting and performance analysis purposes. These logs contain low-level system information and are primarily consumed by tools like Windows Performance Analyzer (WPA) and other performance monitoring tools.On the other hand, WEF is specifically designed to collect and forward events from the standard Windows Event Logs, such as the System, Application, Security, and Setup logs, which are stored in the .evtx format.

It is possible to convert .etl logs to .evtx but it is a point in time operation. We considered whether we could do this with a script, but the downside would be that the logs would flow in chunks, rather than being forwarded at the time they arrive and decided to pursue a better fit for our requirement.

I should note that the design Microsoft arrived at here makes sense to me. That is, using the .etl format for the Session logs. In some environments the volume of information flowing through that log would be a challenge for regular event logs, and some of the content in the log might be unnessesary for most customers. Definately not throwing shade at the choice to use .etl “Analytic and Debug” logs for the App Proxy Session log.

The Solution

The solution we arrived at was to hook only the relevant information (etw information that makes up events 24028, 24029 and 24030) from the App Proxy ETW channel. We would do this by writing a Windows Service that would leverage the Microsoft.Diagnostics.Tracing.TraceEvent package.

Said another way: we would subscribe to the relevant ETW channel and pluck out the ETW information that would later become an analytic event of type 24028, 24029 or 24030.

It was possible to learn about the event channels available to us a couple of ways:

• logman query providers
• wevtutil enum-publishers

The second command: wevtutil enum-publishers revealed an ETW source called Microsoft-AadApplicationProxy-Connector that contained the information we needed to work with the Microsoft.Diagnostics.Tracing.TraceEvent package.

This post by Alex Khanin was really helpful to get started with the ETW library: https://medium.com/@alexkhanin/getting-started-with-event-tracing-for-windows-in-c-8d866e8ab5f2

The key point was that we would need to add the following Nuget package to our C# project: Microsoft.Diagnostics.Tracing.TraceEvent

Console App Example

The following code shows how to hook the relevant events to achieve the outcome we would need as a basic console application. But, to do this in a way that will work across multiple servers, survive reboots etc we need to also take this code and implement it as a windows service.

To summarize the code:

• We are defining three event types that we want to monitor for in the ETW channel for the App Proxy: Microsoft-AadApplicationProxy-Connector
• We are writing that event type in the standard .evtx based logfile: The admin log shown earlier in the post. It’ll also still arrive in the analytic and debug log as usual.

Doing this means that our WEF subscription can pick up the 24028, 24029 and 24030 events from the admin log.

using System;
using System.IO;
using System.Threading;
using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.Session;
using System.Diagnostics;
using System.Threading.Tasks;

namespace Session_Logs_Via_ETW
{
    class Program
    {
        private static Logger _logger;
        static void Main(string[] args)
        {
            const string sessionName = "AppProxyLogExporter";
            const string providerName = "Microsoft-AadApplicationProxy-Connector";
            string logFilePath = "application_log.txt";
            _logger = new Logger(logFilePath);
            _logger.Log("Application started.");
            
            using (var session = new TraceEventSession(sessionName))
            {
                Console.CancelKeyPress += (sender, e) =>
                {
                    Console.WriteLine("Stopping the session...");
                    session.Dispose();
                };

                session.Source.Dynamic.All += HandleEvent;
                session.EnableProvider(providerName);
                Console.WriteLine($"Listening to events from provider '{providerName}'. Press Ctrl+C to exit...");
                session.Source.Process();
            }

            _logger.Log("Application ended.");
        }

        public class Logger
        {
            private readonly string _logFilePath;
            public Logger(string logFilePath)
            {
                _logFilePath = logFilePath;
            }
            
            public void Log(string message)
            {
                string logMessage = $"{DateTime.Now}: {message}";
                File.AppendAllText(_logFilePath, logMessage + Environment.NewLine);
            }
        }

        private static async void HandleEvent(TraceEvent traceEvent)
        {
            const int EventId1 = 24028; // Microsoft AAD Application Proxy Connector received a frontend request.
            const int EventId2 = 24029; // Microsoft AAD Application Proxy Connector sent a request to backend application.
            const int EventId3 = 24030; // Microsoft AAD Application Proxy Connector received a response from backend.

            if (traceEvent.ID == (TraceEventID)EventId1)
            {
                Console.WriteLine($"Event {EventId1} received at: {traceEvent.TimeStamp}");
                string source = "AppProxy-ETW-Event-Extractor";
                string log = "Microsoft-AadApplicationProxy-Connector/Admin";
                EventLogEntryType entryType = EventLogEntryType.Information;
                int eventId = (int)traceEvent.ID;
                string input_message = $"{traceEvent}";
                string formated_message = input_message.Replace("&lt;", "<").Replace("&gt;", ">");
                string prefix = "Microsoft AAD Application Proxy Connector received a frontend request." + Environment.NewLine;
                formated_message = prefix + formated_message;
                await WriteToEventLogAsync(source, log, entryType, eventId, formated_message);
            }

            if (traceEvent.ID == (TraceEventID)EventId2)
            {
                Console.WriteLine($"Event {EventId2} received at: {traceEvent.TimeStamp}");
                string source = "AppProxy-ETW-Event-Extractor";
                string log = "Microsoft-AadApplicationProxy-Connector/Admin";
                EventLogEntryType entryType = EventLogEntryType.Information;
                int eventId = (int)traceEvent.ID;
                string input_message = $"{traceEvent}";
                string formated_message = input_message.Replace("&lt;", "<").Replace("&gt;", ">");
                string prefix = "Microsoft AAD Application Proxy Connector sent a request to backend application." + Environment.NewLine;
                formated_message = prefix + formated_message;
                await WriteToEventLogAsync(source, log, entryType, eventId, formated_message);
            }

            if (traceEvent.ID == (TraceEventID)EventId3)
            {
                Console.WriteLine($"Event {EventId3} received at: {traceEvent.TimeStamp}");
                string source = "AppProxy-ETW-Event-Extractor";
                string log = "Microsoft-AadApplicationProxy-Connector/Admin";
                EventLogEntryType entryType = EventLogEntryType.Information;
                int eventId = (int)traceEvent.ID;
                string input_message = $"{traceEvent}";
                string formated_message = input_message.Replace("&lt;", "<").Replace("&gt;", ">");
                string prefix = "Microsoft AAD Application Proxy Connector received a response from backend." + Environment.NewLine;
                formated_message = prefix + formated_message;
                await WriteToEventLogAsync(source, log, entryType, eventId, formated_message);
            }
        }


        public static async Task WriteToEventLogAsync(string source, string log, EventLogEntryType entryType, int eventId, string message)
        {
            // Check if the source exists, if not, create it
            if (!EventLog.SourceExists(source))
            {
                EventLog.CreateEventSource(source, log);
            }

            // Write the entry to the event log
            using (EventLog eventLog = new EventLog(log))
            {
                eventLog.Source = source;
                eventLog.WriteEntry(message, entryType, eventId);
            }
        }

    }
}

Windows Service

Converting the code above to a Windows Service is relatively straightforward. I’d recommend you use this Microsoft article:

https://learn.microsoft.com/en-us/dotnet/framework/windows-services/walkthrough-creating-a-windows-service-application-in-the-component-designer.

Follow the tutorial and replace the code section in the article with the code above and you’ll have a solution that can be tweaked and deployed on the app proxies; installed as a windows service that’ll survive reboots etc.

In case it helps here’s a full basic example with the App Proxy code above in it: https://github.com/chadduffey/etw-to-evtx/blob/main/AppProxy-ETW-to-EVTX/AppProxyLogConverter.cs

Thanks!