Hydrolix Exporter

Status
Stability	beta: metrics, traces, logs
Latest version

This exporter supports sending OpenTelemetry metrics, traces, and logs to Hydrolix.

Hydrolix is a cloud-native data platform designed for high-volume, high-velocity log and event data. It provides efficient data ingestion and querying capabilities optimized for observability data.

Configuration

The following configuration options are supported:

• endpoint (required): The Hydrolix ingest endpoint URL.
• hdx_table (required): The Hydrolix table name where data will be written.
• hdx_transform (required): The Hydrolix transform to apply to incoming data.
• hdx_bearer_token (optional): Bearer token for authentication. Takes precedence if both token and username/password are provided.
• hdx_username (optional): Username for Basic Authentication.
• hdx_password (optional): Password for Basic Authentication.
• timeout (optional): HTTP client timeout. Default is 30s.
• retry_on_failure (optional): Retry configuration for failed exports. Enabled by default with exponential backoff (initial interval 5s, max interval 30s, max elapsed time 5m).
• sending_queue (optional): Queue configuration for buffering telemetry before export. Enabled by default with 10 consumers and a queue size of 1000.

Gzip compression is enabled by default, reducing payload size over the wire.

Note: You must provide either hdx_bearer_token OR both hdx_username and hdx_password for authentication.

Example Configuration

Using Bearer Token (recommended):

hydrolix/metrics:
  endpoint: https://your-hydrolix-cluster.hydrolix.io/ingest/event
  hdx_table: your.table.name
  hdx_transform: your_transform
  hdx_bearer_token: ${env:HYDROLIX_BEARER_TOKEN}

Using Basic Authentication:

hydrolix/metrics:
  endpoint: https://your-hydrolix-cluster.hydrolix.io/ingest/event
  hdx_table: your.table.name
  hdx_transform: your_transform
  hdx_username: ${env:HYDROLIX_USERNAME}
  hdx_password: ${env:HYDROLIX_PASSWORD}

Data Model

Metrics

The exporter sends metrics with the following fields:

• Metric name, description, and unit
• Metric type (gauge, sum, histogram, exponential histogram, summary)
• Timestamp and values
• Service name and resource attributes as flat key-value objects
• Metric-specific attributes as flat key-value objects
• Optional trace context (traceId, spanId) extracted from metric attributes

Traces

The exporter sends traces (spans) with the following fields:

• Trace ID, span ID, and parent span ID
• Span name and kind
• Start time and duration
• Status code and message
• Service name and resource attributes as flat key-value objects
• Span attributes as flat key-value objects
• Span events (logs attached to spans)

Logs

The exporter sends logs with the following fields:

• Timestamp and observed timestamp
• Severity number and text
• Log body
• Service name and resource attributes as flat key-value objects
• Log attributes as flat key-value objects
• Optional trace context (traceId, spanId, traceFlags)
• HTTP-specific attributes (status code, route, method)

Attribute Format

All attributes (tags, serviceTags, scope_attributes) are exported as flat JSON objects for efficient querying:

{
  "tags": {
    "http.method": "GET",
    "http.status_code": "200"
  },
  "serviceTags": {
    "service.name": "api-gateway",
    "service.version": "1.2.3"
  }
}

Example Pipeline Configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 60s
    send_batch_max_size: 500

exporters:
  hydrolix/traces:
    endpoint: https://your-hydrolix-cluster.hydrolix.io/ingest/event
    hdx_table: myorg.spans
    hdx_transform: tf_otel_spans
    hdx_bearer_token: ${env:HYDROLIX_BEARER_TOKEN}
    timeout: 30s

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [hydrolix/traces]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [hydrolix/traces]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [hydrolix/traces]

Authentication

The exporter supports two authentication methods:

Bearer Token Authentication (Recommended)

Uses a bearer token for authentication. If hdx_bearer_token is provided, it takes precedence over username/password.

exporters:
  hydrolix/traces:
    hdx_bearer_token: ${env:HYDROLIX_BEARER_TOKEN}

Basic Authentication

Uses username and password for HTTP Basic Authentication. This is used as a fallback when no bearer token is provided.

exporters:
  hydrolix/traces:
    hdx_username: ${env:HYDROLIX_USERNAME}
    hdx_password: ${env:HYDROLIX_PASSWORD}

Note: It is recommended to use environment variables for sensitive credentials.

Headers

The exporter sets the following HTTP headers on each request:

• Content-Type: application/json
• Content-Encoding: gzip
• x-hdx-table: The configured table name
• x-hdx-transform: The configured transform name
• Authorization: Either Bearer <token> or Basic Authentication credentials

Best Practices

1. Use the batch processor: Always include the batch processor in your pipeline. For high-throughput workloads, set send_batch_max_size to cap individual request sizes and avoid timeouts (e.g. send_batch_max_size: 500).

2. Monitor error responses: The exporter logs detailed error messages including Hydrolix response bodies to help with troubleshooting.

3. Secure credentials: Use environment variables or secret management systems to store Hydrolix credentials.

4. Configure appropriate timeouts: Adjust the timeout setting based on your network conditions and data volume.

5. Test connectivity: Verify your Hydrolix endpoint is accessible and credentials are correct before deploying to production.

Deployment instructions

1. Pull the image from GCP

   # Replace [Your version here] with the desired version, e.g. v1.2.0
   docker pull us-docker.pkg.dev/hdx-art/t/hdx-collector:[Your version here]

2. Create a config file

   Copy the hydrolix-config.yaml from this repo and update it with your url, tables and transforms.

3. Add service account credentials

   Create a service account in your Hydrolix cluster and add the credentials to docker.

4. Deploying the collector

   1. [Option 1] Kubernetes deployment

   2. [Option 2] RPM deployment - For RHEL/CentOS/Fedora systems

   3. [Option 3] Run the image manually

      Export the environment variables

        export HYDROLIX_BEARER_TOKEN=token here
      

      docker run --name otel-collector -p 4317:4317 \
        -p 4318:4318 \
        -e HYDROLIX_BEARER_TOKEN \
        -v [Absolute path to your otel config]:/etc/otelcol/config.yaml \
        us-docker.pkg.dev/hdx-art/t/hdx-collector:[Your version here]

   4. [Option 4] Docker compose

      Docker compose can read environment variables from the system environment variables.

      # docker-compose.yml
      version: '3.8'
      services:
      otel-collector:
      image: otelcol-hydrolix:latest
      container_name: otel-collector
      ports:
      - "4317:4317"
      - "4318:4318"
      environment:
      - HYDROLIX_BEARER_TOKEN=${HYDROLIX_BEARER_TOKEN}
      # or use env_file:
      env_file:
      - .env
      volumes:
      - ./local.yaml:/etc/otelcol/config.yaml