Form Energy Observability Python Library

OTel tracing and logging tools, emphasizing data pipeline use cases with Dagster.

Using the Library

Install from PyPI

pip install formenergy-observability

Features

See the examples/ directory for examples you can run and screenshots of the traces they produce.

Some key features demonstrated in the examples:

Context-aware tracing, so that spans and events can inherit attributes from parents:

from form_observability import ContextAwareTracer, ctx

_trace = ContextAwareTracer(__name__)

with ctx.set({"my_attribute_name": my_attribute_value}):
    # There could be many intervening function calls and spans.
    # Spans and events in this context automatically have the my_attribute_name
    # attribute set on them.
    _trace.add_event("my_event")

Trace context propagation for Dagster ops:

from form_observability.dagster_otel import otel_op, publish_current_trace_context
from form_observability import ContextAwareTracer

_trace = ContextAwareTracer(__name__)

@op
def root_trace_context_op(context):
    with _trace.start_as_current_span("root"):
        # writes the trace context (as an asset materialization event) for later ops
        publish_current_trace_context(context)

@otel_op()  # picks up the trace context and starts a new span for this op
def my_op(context):
    _trace.add_event("my_event")

@graph
def my_graph()
    # In the graph, publish a root trace using a regular op, and then subsequent
    # @otel_op ops will pick it up and contribute to the same trace.
    setup_done = root_trace_context_op()
    my_op(start=setup_done)

Running the Examples

Get an API key for uploading traces to your observability service (preferably in a development environment), set it in your environment, and run the examples. This should publish traces that you can then view.

export OTEL_EXPORTER_OTLP_HEADERS_JSON="{'x-honeycomb-team': '<API key>'}"
export OTEL_EXPORTER_OTLP_ENDPOINT="your observability service, defaults to Honeycomb"
python examples/basics.py
dagster job execute --python-file examples/dagster_job.py

License

This library is provided under the MIT license, see LICENSE.

Release Notes

• 0.3.2 Update grpcio minimum dependency version to 1.48.1 and dagster to 1.5.0. Thanks @dang-kaiko !
• 0.3.1 Fix type used for otlp_headers to be a dict. Thanks @spenczar !
• 0.3.0 Add ContextAwareTracer.start_new_linked_trace.
• 0.2.0 Switch to context.op_handle and materialization.metadata as dict for Dagster 1.2.
• 0.1.0 Initial external release, including ContextAwareTracer, ctx, and @otel_op.

Development

If you contribute, please ensure you and your employer accept the license for any of your contributions.

Setup

Create and activate a virtual environment with your favorite tool, such as direnv. This project requires Python 3.8.

pip install flit
flit install --deps all --symlink [--python /path/to/venv/python]

By default, flit installs into the current virtual environment. Using the --python argument can let you install into a specific venv other than the currently active one. For example you can activate some other venv, run which python, and then pass the output of that command to flit's --python argument when installing from the form-observability repo.

Initialize pre-commit hooks:

pre-commit install --hook-type pre-commit --hook-type pre-push

Running Tests

From the project root directory:

pytest