Improving Application Telemetry with .NET 8 and OpenTelemetry

Introduction

In today's fast-paced development world, observing and monitoring applications has become a crucial aspect of ensuring performance and reliability. As I continue building and maintaining applications, I've realized that integrating effective telemetry is a powerful way to gain deep insights into my application’s behavior, diagnose issues, and ultimately improve performance. This is where OpenTelemetry comes into play.

In this blog, I will walk through how to improve application telemetry by leveraging .NET 8 and OpenTelemetry, explaining key concepts, sharing some code examples, and illustrating how I have implemented these techniques in my projects. Initially let's look how the new Microsoft's Aspire dashboard and thereafter Prometheus and Grafana to visualize telemetry data, create dashboards, and monitor key metrics. Moreover, I'll discuss the concept of observability, its importance, and how it's closely related to telemetry in microservice-based architectures. Let’s dive in!

What is Observability?

Observability refers to the ability to understand the internal state of a system by analyzing the outputs it generates, such as logs, metrics, and traces. It goes beyond mere monitoring by offering insight into how and why a system behaves the way it does, especially when failures occur.

Observability is essential in complex, distributed systems, such as microservice architectures, where different services are interdependent. By gathering and analyzing various data points, observability provides insights into system performance, helping developers detect and troubleshoot issues early.

OpenTelemetry is a set of standards for generating, collecting, and exporting telemetry data. It enables observability by collecting signals from your services—signals being metrics, traces, and logs—and shipping them to an observability backend such as Prometheus, Grafana, or any other visualization tool. The beauty of OpenTelemetry is that it’s vendor-neutral, meaning I can choose the tools I want to use without being locked into a specific provider.

In essence, telemetry is the "what," and observability is the "why" and "how" in terms of system behavior and performance.

Distributed Tracing

One of the most critical concepts that OpenTelemetry helps with is Distributed Tracing. In distributed systems, tracing allows me to follow the flow of a request across different services. Each request is recorded as a trace, and within that trace, there are multiple spans representing operations or service calls. By tracking traces, I can easily pinpoint where in the system a failure or latency occurred, which is particularly helpful when dealing with microservices architectures.

Instrumentation

Instrumentation in OpenTelemetry refers to adding the necessary hooks in the code to collect telemetry data. It’s the process of configuring applications so that they can emit traces, metrics, and logs automatically. When I instrumented my .NET 8 application, I added libraries and middleware to automatically collect metrics for key components, like the HTTP client and the Entity Framework. Additionally, I configured custom instrumentation for my application-specific needs, like tracking sales in the event booking application.

Integrating OpenTelemetry in .NET 8

One of the exciting updates in .NET 8 is the enhanced support for OpenTelemetry. Unlike other languages that require external APIs, .NET 8 includes built-in telemetry APIs that make it easier to work with logs, metrics, and distributed tracing.

The three main telemetry APIs in .NET 8 are:

1. iLogger for logging
2. Meter for metrics
3. Activity for distributed tracing

These APIs are natively supported in .NET, so I can write my telemetry code using these platform features without needing any third-party packages.

I started by adding the necessary NuGet packages to enable OpenTelemetry in my solution. This included packages for OpenTelemetry extensions, hosting, exporters, and instrumentation for ASP.NET Core and HTTP.

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.OTLP

Thereafter I configured the OpenTelemetry services in Program.cs to instrument metrics, traces, and logs for the application. Below is an example of how I configured OpenTelemetry in the startup file.

// add open telemetry
services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(TicketTelemetryConfig.ApiName))
    .WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddMeter(TicketTelemetryConfig.ApiName)
            .AddOtlpExporter();
    })
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddEntityFrameworkCoreInstrumentation()
            .AddOtlpExporter();
    });

// add logging
builder.Logging.AddOpenTelemetry(logging => logging.AddOtlpExporter());

To configure OpenTelemetry in your application, you start by adding OpenTelemetry to your builder services, which returns an OpenTelemetry builder. First, use the ConfigureResource method to define your application's service, providing it with a descriptive name which in this case is EventBookingApi. This name is important it's the name against which all the metrics, logs and traces will be written down against.

Next, configure metrics by invoking the WithMetrics method, which allows access to the meter builder. You can add intstrumentation for hosting, Kestrel, and HTTP configuration and any ORM specific or database operation related instrumentation. While manual configuration is possible, installing instrumentation packages (like ASP.NET Core, HTTP instrumentation, ORM instrumentation) automates much of this process. In the above example I have added instrumentation for the ASP.NET, HttpClient and the Entity Framework.

I've also added a custom meter and given it a name. Custom meters are a way to define and track specific metrics that are not provided by default in the framework or OpenTelemetry libraries. They allow developers to define their own metrics to capture additional, specific data relevant to the application.

AddOtlpExporter is used to export telemetry data—such as traces and metrics—from your application to a backend system that supports the OTLP protocol. This method is part of the OpenTelemetry SDK and is responsible for sending telemetry data to a monitoring backend (e.g., Jaeger, Prometheus, Zipkin, or any OpenTelemetry-compatible system) for further analysis and visualization. The AddOtlpExporter method allows customization of the endpoint (URL) where telemetry data is sent. You can specify whether the communication is done using gRPC (more efficient) or HTTP (more commonly supported), along with other configuration settings like headers, batch size, timeouts, etc.

For tracing, use the WithTracing method to set up tracing telemetry. Again, instrumentation packages for ASP.NET Core, HTTP, and .NET Framework can be applied, simplifying the process. Additionally, configure a traces exporter like what's done when you configured metrics above.

For logging, reach out to the logging provider and call AddOpenTelemetry, configuring your log exporter via AddOTLPExporter.

Sample Application

The application that we are building is an ASP.NET Core 8 Web API application and is designed for managing ticket management. It uses Entity Framework Core with an in-memory database to handle CRUD operations for tickets. The application includes endpoints for creating, reading, updating, and deleting tickets, each mapped in a streamlined API setup. OpenTelemetry is integrated for logging, metrics, and tracing, providing robust observability.

Here's how my endpoints look like. I'm logging information about my operations.

public static void MapTicketEndpoints(this WebApplication app)
    {
        var logger = app.Logger;

        app.MapGet("/tickets", async (BookingContext db) =>
            await db.Tickets.ToListAsync());

        app.MapGet("/tickets/{id:int}", async (int id, BookingContext db) =>
        {
            logger.LogInformation("Fetching ticket {TicketId}", id);

            return await db.Tickets.FindAsync(id)
                is Ticket ticket
                ? Results.Ok(ticket)
                : Results.NotFound();
        });

        app.MapPost("/tickets", async (Ticket ticket, BookingContext db) =>
        {
            db.Tickets.Add(ticket);

            await db.SaveChangesAsync();

            logger.LogInformation("Created ticket {TicketId}", ticket.Id);

            return Results.Created($"/tickets/{ticket.Id}", ticket);
        });

        app.MapPut("/tickets/{id:int}", async (int id, Ticket updatedTicket, BookingContext db) =>
        {
            var ticket = await db.Tickets.FindAsync(id);
            if (ticket is null) return Results.NotFound();

            ticket.EventName = updatedTicket.EventName;
            ticket.BookingDate = updatedTicket.BookingDate;
            ticket.SeatNumber = updatedTicket.SeatNumber;
            ticket.Price = updatedTicket.Price;
            await db.SaveChangesAsync();

            logger.LogInformation("Updated ticket {TicketId}", id);

            return Results.NoContent();
        });

        app.MapDelete("/tickets/{id:int}", async (int id, BookingContext db) =>
        {
            var ticket = await db.Tickets.FindAsync(id);
            if (ticket is null) return Results.NotFound();

            db.Tickets.Remove(ticket);
            await db.SaveChangesAsync();

            logger.LogInformation("Deleted ticket {TicketId}", id);
            return Results.NoContent();
        });
    }

Run your ASP.NET 8 Web API application just to see if the endpoints are working fine and are returning the right set of results.

Configure Aspire Dashboard.

The Microsoft Aspire Dashboard is a tool designed to help developers visualize telemetry data from applications that use OpenTelemetry. It simplifies the process of monitoring metrics, traces, and logs by providing a user-friendly interface. The dashboard can be used either as part of the .NET Aspire ecosystem or as a standalone tool through a Docker container.

The Dashboard does not store metrics, logs, and traces in a persistent internal database. Instead, it processes and displays telemetry data in-memory, meaning it is designed for short-term diagnostics and development use hence the reason for me to use it for Demo purposes. The telemetry data metrics, logs, and traces—are collected via OpenTelemetry and stored in memory while the dashboard is running. Once the dashboard is stopped or restarted, the telemetry data is lost, as no data is persisted.

Should we require persistent storage, you would need to export the telemetry data to an external system or database using OpenTelemetry-compatible exporters, such as OTLP (OpenTelemetry Protocol), which can send the data to more robust solutions like Azure Monitor, Prometheus, or other external storage systems​.

To continue with our testing of the logs, traces and metrics we need to create containerize the Tickets API service. Hence the need for the Dockerfile like the one below.

FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
USER $APP_UID
WORKDIR /app
EXPOSE 6000

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY ["Tickets/Tickets.csproj", "Tickets/"]
RUN dotnet restore "Tickets/Tickets.csproj"
COPY . .
WORKDIR "/src/Tickets"
RUN dotnet build "Tickets.csproj" -c $BUILD_CONFIGURATION -o /app/build

FROM build AS publish
ARG BUILD_CONFIGURATION=Release
RUN dotnet publish "Tickets.csproj" -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .

ENTRYPOINT ["dotnet", "Tickets.dll"]

Let's run the Aspire Dashboard in a container and send the metrices, logs and traces of the Tickets API service to the same. You'll need to use a Docker Compose to run both the containers. Here's the docker-compose file that I used to spin up both the containers.

services:
  tickets.api:
    image: tickets.api
    container_name: tickets.api
    build:
      context: .
      dockerfile: Tickets/Dockerfile
    ports:
      - "6000:6000"
    environment:
      - ASPNETCORE_ENVIRONMENT=Local
      - ASPNETCORE_HTTP_PORTS=6000
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://tickets.dashboard:18889
    networks:
      - otel
  
  tickets.dashboard:
    image: mcr.microsoft.com/dotnet/nightly/aspire-dashboard:latest
    container_name: dashboard
    ports:
      - 18888:18888
    networks:
      - otel

networks:
  otel:

Once you have the both the containers up and running, you could run the curl commands to hit the Tickets API endpoints.

There are two ways in which you could configure the sink or data source for the AddOtlpExporter. One is where mention the URL of the data source (refer below). However, to simplify code I've used the OTEL_EXPORTER_OTLP_ENDPOINT Environment Variable to configure the endpoint. Refer the environment section in the docker-compose.yml file.

// Give AddOtlpExporter a URL of the sink.
.AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri("http://tickets.dashboard:18889"); // Metrics endpoint
                options.Protocol = OtlpExportProtocol.HttpProtobuf;
            });

Now it's time to view the dashboard to see the logs, traces and the metrics that were sent from the Tickets API.

You can access the dashboard by navigating to http://localhost:18888 in your browser. This port is mapped in the docker-compose.yml file, exposing the dashboard for easy access.

In most cases, the URL to access the dashboard will be http://localhost:18888. This address is based on the port mappings set in the Docker configuration. If authentication is enabled for the Aspire Dashboard, you’ll need a login token to access it. To get this token, simply run docker logs tickets.dashboard. The token will be printed in the logs, and you can use it by appending it to the login URL: http://localhost:18888/login?t=<token>.

With these steps, you’ll be able to quickly and easily view your telemetry data in the Aspire

Conclusion

Implementing OpenTelemetry has significantly improved my ability to observe and monitor my .NET applications in production. With the combination of metrics, traces, and logs, I now have comprehensive visibility into the state and performance of my services. OpenTelemetry's vendor-agnostic approach also means I can choose my preferred observability backend, making it a flexible and powerful tool for any .NET developer looking to gain deep insights into their applications.

The full source code for the application could be found here.

In my next blog I will show how to use Prometheus and Grafana with OpenTelemetry in .NET with some additional services to make it more interesting.