 .. Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

 ..   http://www.apache.org/licenses/LICENSE-2.0

 .. Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.



Metrics Configuration
=====================

Airflow can be set up to send metrics to `StatsD <https://github.com/etsy/statsd>`__
or `OpenTelemetry <https://opentelemetry.io/>`__.

Setup - StatsD
--------------

To use StatsD you must first install the required packages:

.. code-block:: bash

   pip install 'apache-airflow[statsd]'

then add the following lines to your configuration file e.g. ``airflow.cfg``

.. code-block:: ini

    [metrics]
    statsd_on = True
    statsd_host = localhost
    statsd_port = 8125
    statsd_prefix = airflow

If you want to use a custom StatsD client instead of the default one provided by Airflow,
the following key must be added to the configuration file alongside the module path of your
custom StatsD client. This module must be available on your :envvar:`PYTHONPATH`.

.. code-block:: ini

    [metrics]
    statsd_custom_client_path = x.y.customclient

See :doc:`../modules_management` for details on how Python and Airflow manage modules.


Setup - OpenTelemetry
---------------------

To use OpenTelemetry you must first install the required packages:

.. code-block:: bash

   pip install 'apache-airflow[otel]'

An OpenTelemetry `Collector <https://opentelemetry.io/docs/concepts/components/#collector>`_ (or compatible service) is required for connectivity to a metrics backend.
Add the Collector details to your configuration file e.g. ``airflow.cfg``

.. code-block:: ini

    [metrics]
    otel_on = True
    otel_host = localhost
    otel_port = 8889
    otel_prefix = airflow
    otel_interval_milliseconds = 30000  # The interval between exports, defaults to 60000
    otel_service = Airflow
    otel_ssl_active = False

.. note::

    **The following config keys have been deprecated and will be removed in the future**

        .. code-block:: ini

            [metrics]
            otel_host = localhost
            otel_port = 8889
            otel_interval_milliseconds = 30000
            otel_debugging_on = False
            otel_service = Airflow
            otel_ssl_active = False

    The OpenTelemetry SDK should be configured using standard OpenTelemetry environment variables
    such as ``OTEL_EXPORTER_OTLP_ENDPOINT``, ``OTEL_EXPORTER_OTLP_PROTOCOL``, etc.

    See the OpenTelemetry `exporter protocol specification <https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options>`_  and
    `SDK environment variable documentation <https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_ for more information.


Enable Https
-----------------

To establish an HTTPS connection to the OpenTelemetry collector
You need to configure the SSL certificate and key within the OpenTelemetry collector's ``config.yml`` file.

.. code-block:: yaml

   receivers:
     otlp:
       protocols:
         http:
           endpoint: 0.0.0.0:4318
           tls:
             cert_file: "/path/to/cert/cert.crt"
             key_file: "/path/to/key/key.pem"

Histogram Metrics and Backend Requirements
------------------------------------------

Airflow's timing metrics (``timing()`` / ``timer()``) are emitted as OpenTelemetry
histograms aggregated with
`exponential bucket histograms <https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram>`_,
so bucket boundaries adapt automatically to the observed range and you do not have to
hand-tune explicit buckets for metrics that span very different scales (milliseconds to
hours).

To ingest these correctly end-to-end, the metrics backend you connect to must support
OpenTelemetry exponential histograms and (for Prometheus) their conversion to native
histograms:

* **OpenTelemetry Collector** — use ``opentelemetry-collector-contrib`` version 0.115.0
  or above. Older versions do not translate OTLP exponential histograms into Prometheus
  native histograms.
* **Prometheus** — native histograms must be enabled explicitly, and how you do that
  depends on the Prometheus version:

  * **2.40 to 3.8** — start Prometheus with the ``--enable-feature=native-histograms``
    flag.
  * **3.8 and above** — set ``scrape_native_histograms: true`` in the scrape
    configuration (this option was added in 3.8, and from 3.9 the feature flag is a
    no-op so the config setting is required):

    .. code-block:: yaml

        global:
            scrape_native_histograms: true

If the backend does not support native histograms, exponential-histogram data points may
be dropped or rendered incorrectly. A reference stack (Collector, Prometheus, and Grafana)
wired up for local development is available via ``breeze start-airflow --integration otel``;
see the contributor docs for details.

Allow/Block Lists
-----------------

If you want to avoid sending all the available metrics, you can configure an allow list or block list
of prefixes to send or block only the metrics that start with the elements of the list:

.. code-block:: ini

    [metrics]
    metrics_allow_list = scheduler,executor,dagrun,pool,triggerer,celery

.. code-block:: ini

    [metrics]
    metrics_block_list = scheduler,executor,dagrun,pool,triggerer,celery


Rename Metrics
--------------

If you want to redirect metrics to a different name, you can configure the ``stat_name_handler`` option
in ``[metrics]`` section.  It should point to a function that validates the stat name, applies changes
to the stat name if necessary, and returns the transformed stat name. The function may look as follows:

.. code-block:: python

    def my_custom_stat_name_handler(stat_name: str) -> str:
        return stat_name.lower()[:32]


Other Configuration Options
---------------------------

.. note::

    For a detailed listing of configuration options regarding metrics,
    see the configuration reference documentation - :ref:`config:metrics`.


Metric Descriptions
===================

.. include:: metric_tables.rst
