Logging and OpenTelemetry in the Grafbase Gateway

The Grafbase Gateway provides logs, traces, and metrics for monitoring the gateway operation and errors. By default, it outputs logs to the standard output. Additionally, the gateway can send monitoring data to an endpoint that implements the OpenTelemetry protocols.

Level of produced information

You can define the level of information available by setting the log level command line argument:

--log <LOG_LEVEL>
    Set the logging level

    [env: GRAFBASE_LOG=]

    Possible values:
    - off:   Completely disables logging
    - error: Only errors from Grafbase libraries
    - warn:  Warnings and errors from Grafbase libraries
    - info:  Info, warning and error messages from Grafbase libraries
    - debug: Debug, info, warning and error messages from Grafbase libraries
    - trace: Trace, debug, info, warning and error messages from all dependencies

Keep in mind this setting affects both traces and logs.

The default level is info, which logs errors with added span information to the standard output during normal operation. The info level will not include successful requests, which are logged at the debug level. Using the error level will only include error messages without spans in the output. The error level, along with warn and off, will also block traces from being sent.

If you want to silence all logs but still export them along with traces and metrics to an OpenTelemetry endpoint, direct the standard output and standard error to /dev/null.

Logs

By default, the system outputs logs to the standard output. The logs can be in two different formats:

--log-style <LOG_STYLE>
    Set the style of log output

    [env: GRAFBASE_LOG_STYLE=]
    [default: text]

    Possible values:
    - text: Standard text
    - json: JSON objects

The default is text, which is ANSI-colored text for terminal output and ASCII with no coloring for common non-terminal log outputs. The json format provides logs in JSON format and can be useful if the logging platform supports structured data.

Logs can also be sent to an OpenTelemetry endpoint by enabling the OpenTelemetry exporter in the configuration:

[telemetry.exporters.otlp]
enabled = true
endpoint = "http://localhost:1234"

Additionally, logs can be sent to a separate endpoint different from the global OpenTelemetry settings:

[telemetry.logs.exporters.otlp]
enabled = true
endpoint = "http://localhost:1235"

Read more about the options for OpenTelemetry in the configuration section.

Traces

Traces are sent from the info level and provide information on the request lifecycle to the OpenTelemetry endpoint. The traces have a few important settings to define:

[telemetry.tracing]
enabled = true
sampling = 1

The sampling setting defines the percentage (a floating point from 0 to 1) of requests traced. For testing purposes or with a small amount of traffic, this can safely be 1. If the expected traffic is high, sampling every request can be expensive in terms of network, CPU, and storage. The default value is 0.15, which samples 15% of all requests.

The collect options define limits per span:

[telemetry.tracing.collect]
max_events_per_span = 128
max_attributes_per_span = 128
max_links_per_span = 128
max_attributes_per_event = 128
max_attributes_per_link = 128
  • max_events_per_span specifies the maximum number of events to be recorded per span (default: 128)
  • max_attributes_per_span specifies the maximum number of attributes to be recorded per span (default: 128)
  • max_links_per_span specifies the maximum number of links to be recorded per span (default: 128)
  • max_attributes_per_event specifies the maximum number of attributes one event can have (default: 128)
  • max_attributes_per_link specifies the maximum number of attributes one link can have (default: 128)

Traces can be sent to an OpenTelemetry endpoint by enabling the OpenTelemetry exporter in the configuration:

[telemetry.exporters.otlp]
enabled = true
endpoint = "http://localhost:1234"

Additionally, traces can be sent to a separate endpoint different from the global OpenTelemetry settings:

[telemetry.tracing.exporters.otlp]
enabled = true
endpoint = "http://localhost:1235"

Read more about the options for OpenTelemetry in the configuration section.

Spans can also be directly written to standard output by enabling the global stdout exporter, which is mostly useful for evaluation and debugging:

[telemetry.exporters.stdout]
enabled = true

Or enabling it only for tracing:

[telemetry.tracing.exporters.stdout]
enabled = true

Metrics

The Grafbase Gateway can deliver metrics for requests and operations to an OpenTelemetry endpoint. These metrics include request counts, error counts, and histograms with metadata.

Requests are measured from the HTTP server layer and include the following information:

  • http.response.status_code: The HTTP status code.
  • http.response.headers.cache_status: Either HIT or MISS, if available.
  • http.headers.x-grafbase-client-name: The name of the client that triggered this request, if available.
  • http.headers.x-grafbase-client-version: The version of the client that triggered this request, if available.
  • gql.response.status: Whether the underlying GraphQL operation was successful or not, if available.

The request_latency is an exponential histogram of the request latency.

Operations are measured from the gateway layer and include the following information:

  • gql.operation.query: The query of this operation, normalized so all variables are removed. This value cannot contain any private data.
  • gql.operation.type: The type of the operation, either query, mutation, or subscription.
  • gql.operation.name: The name of the operation, if provided.
  • gql.response.cache_status: Either HIT or MISS, if available.
  • gql.response.status: Indicates if the response was successful or not.
  • http.headers.x-grafbase-client-name: The name of the client that triggered this request, if available.
  • http.headers.x-grafbase-client-version: The version of the client that triggered this request, if available.

The gql_operation_latency is an exponential histogram of the GraphQL operation.

Metrics can be sent to an OpenTelemetry endpoint by enabling the OpenTelemetry exporter in the configuration:

[telemetry.exporters.otlp]
enabled = true
endpoint = "http://localhost:1234"

Additionally, metrics can be sent to a separate endpoint different from the global OpenTelemetry settings:

[telemetry.metrics.exporters.otlp]
enabled = true
endpoint = "http://localhost:1235"

Read more about the options for OpenTelemetry in the configuration section.

Likewise, spans can be directly written to standard output by enabling the global stdout exporter, which is mostly useful for evaluation and debugging:

[telemetry.exporters.stdout]
enabled = true

Or enabling it only for metrics:

[telemetry.metrics.exporters.stdout]
enabled = true

OpenTelemetry configuration

The OpenTelemetry settings are defined in the telemetry block of the Gateway configuration:

[telemetry]
service_name = "grafbase-gateway"

Here, the service_name is included in all the traces, metrics and logs, and should be something unique in your system.

Grafbase sends a standard set of resource attributes for every user. You can also define your own set of attributes, available in all the logs, traces and metrics:

[telemetry.resource_attributes]
custom_key = "custom_value"
other_key = "other_value"

Global exporters

The exporter settings can be defined globally for traces, logs, and metrics. If you need different settings for logs, tracing, or metrics, you need to prefix the exporter settings with the appropriate word. For example, the custom OpenTelemetry exporter settings for tracing use the key telemetry.tracing.exporters.otlp.

The traces and metrics can additionally be sent to standard output (logs will always be there):

[telemetry.exporters.stdout]
enabled = true
timeout = 60
  • enabled: Enables the OpenTelemetry exporter (default: false).
  • timeout: The number of seconds data will be kept in memory if the collector does not collect data fast enough (default: 60).

Traces, metrics, and logs can be sent to an external OpenTelemetry collector:

[telemetry.exporters.otlp]
enabled = true
endpoint = "http://localhost:1234"
protocol = "grpc"
timeout = 60
  • enabled: Enables the OpenTelemetry exporter (default: false).
  • endpoint: Defines the URL for the OpenTelemetry collector.
  • protocol: Can either be grpc or http (default: grpc).
  • timeout: The number of seconds data will be kept in memory if the collector does not collect data fast enough (default: 60).

Batch exporting

In general, it is not advisable to trigger a request for every single span, trace, and metrics event in the system. Requests should be batched and data sent in regular intervals. This can be configured from the OpenTelemetry batch settings:

[telemetry.exporters.otlp.batch_export]
scheduled_delay = 5
max_queue_size = 2048
max_export_batch_size = 512
max_concurrent_exports = 1
  • scheduled_delay: The time in seconds between two consecutive requests (default: 5).
  • max_queue_size: The maximum queued items for delayed processing. If the queue gets full, the system drops events (default: 2048).
  • max_export_batch_size The maximum number of events in a single batch. If there are more events before the scheduled delay, it queues the events. (default: 512).
  • max_concurrent_exports: The number of concurrent senders processing the batches (default: 1).

gRPC settings

If the protocol is set to grpc, the Gateway will use these settings.

For collectors using TLS with a custom certificate, define the TLS settings:

[telemetry.exporters.otlp.grpc.tls]
domain_name = "custom_name"
key = "/path/to/key.pem"
cert = "/path/to/cert.pem"
ca = "/path/to/ca.crt"
  • domain_name: The domain name against which to verify the server's TLS certificate.
  • key: Defines the path to the secret key.
  • cert: Defines the path to the X509 certificate file, which must be in PEM format.
  • ca: Defines the path to the X509 CA certificate file, which must be in PEM format.

In some cases, the gRPC collector expects custom headers to be set. Define these in the headers section of the configuration:

[[telemetry.exporters.otlp.grpc.headers]]
authorization = "Bearer {{ env.GRPC_TOKEN }}"

[[telemetry.exporters.otlp.grpc.headers]]
custom = "static value"

HTTP settings

If the protocol is set to http, the Gateway will use these settings. Define custom headers sent in every request:

[[telemetry.exporters.otlp.grpc.headers]]
authorization = "Bearer {{ env.GRPC_TOKEN }}"

[[telemetry.exporters.otlp.grpc.headers]]
custom = "static value"

Currently, the http exporter does not support TLS. If needed, use the grpc exporter.

Was this page helpful?