[OpAMP] parse comma separated tags list from non-identifying attributes

[juliaElastic]

What is the problem this PR solves?

Save tags list in the top level tags field in Agent, taking from the comma separated list from non-identifying attributes

How does this PR solve the problem?

Non-identifying attributes are already added to Agent, tags need special handling because they are mapped to the top level tags field, and the comma separated string has to be parsed.

How to test this PR locally

Follow instructions in https://github.com/elastic/fleet-server/blob/main/docs/opamp.md

Download and extract otel collector, e.g.:

curl -L -O https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.144.0/otelcol-contrib_0.144.0_darwin_arm64.tar.gz 

Create otel config to include the new metadata fields, e.g. otel-opamp.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  resourcedetection:
    detectors: ["system","env"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        host.name:
          enabled: true
        host.arch:
          enabled: true
        os.description:
          enabled: true
        os.type:
          enabled: true

exporters:
  debug:
    verbosity: detailed
  elasticsearch/otel:
    endpoints: [ "http://localhost:9200" ]
    api_key: ${env:ES_API_KEY} 
    mapping:
      mode: otel
  otlp:
    endpoint: "http://localhost:4317"
    tls:
      insecure: true

extensions:
  opamp:
    server:
      http:
        endpoint: http://localhost:8220/v1/opamp
        tls:
          insecure: true
        headers:
          Authorization: ApiKey ${env:FLEET_ENROLLMENT_TOKEN}
    instance_uid: ${env:INSTANCE_UID}
    capabilities:
      reports_effective_config: true
    agent_description:
      # Pipelines Lab reads these fields to populate the fleet and configs views.
      non_identifying_attributes:
        # Shown as the collector's name in the fleet list. Defaults to hostname if unset.
        elastic.display.name: ${env:HOSTNAME}
        # Machine-friendly key used to group collectors on the Configs page.
        # Collectors sharing the same slug collapse into one group with a
        # shared health summary and config-hash distribution view.
        elastic.collector.group: "mysql-dev"
        # Human-readable label for the group shown in the UI.
        elastic.collector.group_name: "MySQL Dev"
        # Labels this config on the Configs page and as a header in the effective config view.
        config.description: "The MySQL Stuff"
        # Tags appear in the fleet view and as resource attributes
        # on all self-emitted metrics/logs in Elasticsearch (queryable via ES|QL).
        tags: "dev,west,us-west-1a"
        # OTel semconv deployment tier — e.g. production, staging, dev.
        deployment.environment.name: "dev"

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [resourcedetection]
      exporters: [elasticsearch/otel]
  extensions: [opamp]

  # publish collector internal telemetry
  telemetry:
    metrics:
      level: detailed
      readers:
        - periodic:
            interval: 30000
            exporter:
              otlp:
                protocol: grpc
                endpoint: http://localhost:4317
    resource:
      # semconv: identifies this collector instance in Elasticsearch.
      service.name: mysql-dev
      service.instance.id: ${env:HOSTNAME}
      # Required for Logs and Error Patterns tabs — the backend queries ES by this field.
      service.node.name: ${env:HOSTNAME}
      # Mirrors elastic.display.name above — queryable in ES|QL.
      elastic.display.name: ${env:HOSTNAME}
      # Stamps self-emitted metrics/logs with the host — queryable as host.name in ES|QL.
      host.name: ${env:HOSTNAME}
      # semconv: groups collectors by workload (mirrors elastic.collector.group).
      service.namespace: "mysql-dev"
      # Machine-friendly group key — matches elastic.collector.group in the fleet UI.
      elastic.collector.group: "mysql-dev"
      # Human-readable group label — matches elastic.collector.group_name in the fleet UI.
      elastic.collector.group_name: "MySQL Dev"
      # Mirrors tags above — queryable in ES|QL.
      tags: "dev,west,us-west-1a"
      deployment.environment.name: "dev"

Create API keys and start otel collector:

 cd ~/Downloads/otelcol-contrib_0.144.0_darwin_arm64
 
 export INSTANCE_UID=<uuid> # e.g. "519b8d7a-2da8-7657-b52d-492a9de33313"
 export ES_API_KEY=<api_key> # ES API key from observability onboarding UI
 export FLEET_ENROLLMENT_TOKEN=<enrollment_token> 
 export HOSTNAME=MacBook-Pro.local
 ./otelcol-contrib --config ./otel-opamp.yaml

Check the Agent JSON in Fleet, check that tags are showing up as an array, and non_identifying_attributes show the new metadata fields.

Check the telemetry data in Discover, and find the matching fields in resource.attributes:

Tested ESQL queries on the new fields in telemetry:

Open question:

• The tags field in collector telemetry data is not parsed to a list of strings, I'm not sure how can we fix that. The collector config doesn't accept a list, only string.
  "tags": "dev,west,us-west-1a"

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes https://github.com/elastic/ingest-dev/issues/7062
• Relates [Fleet] Add metadata mappings for OpAMP elasticsearch#145824

[mergify]

This pull request does not have a backport label. Could you fix it @juliaElastic? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[juliaElastic]

using a comma separated list, when tested with an array in the otel config yaml, I was getting an error in the otel collector
tags: ["prod", "west", "us-west-1a"]

'extensions' error reading configuration for "opamp": decoding failed due to the following error(s):

'agent_description.non_identifying_attributes[tags]' expected type 'string', got unconvertible type '[]interface {}'
'service.telemetry.resource[tags]' expected type 'string', got unconvertible type '[]interface {}'

[ycombinator]

I think we should add a test case for duplicate tags, e.g. "west,dev,west". These should probably be deduplicated so the wantTags is []string{"west","dev"}. I can't imagine a use case for storing duplicates?