Add application-scoped YARN resource usage API

[gerashegalov]

Add a typed, packageable Python API for application-scoped YARN resource accounting.

The API accepts injected boto clients, preserves calculator and completeness evidence, returns per-instance seconds plus vcore/memory seconds, and leaves pricing policy to the caller. The existing CLI remains compatible.

Validation:

• python3 -m unittest discover -s yarn-resource-cost -p "test*.py" (39 tests)
• wheel build and isolated import smoke test

[gerashegalov]

Superseded by #267, which adds the required DCO sign-off without rewriting published branch history.

[greptile-apps]

Greptile Summary

The PR adds a packageable, typed API that materializes Spark event logs and archived EMR YARN logs through injected clients, calculates application-scoped resource usage, and exports it alongside the existing CLI.

• Adds request/result dataclasses and application-scoped accounting orchestration.
• Adds Python packaging metadata and public API exports.
• Adds API documentation and focused unit tests.

Confidence Score: 3/5

The PR should not merge until S3 event-log segment grouping and multi-node archived-log identity are preserved.

The new materialization paths can split one multi-segment Spark event log into independently overwriting records and collapse multiple NodeManager logs onto one node, yielding incomplete or incorrectly attributed resource usage.

Files Needing Attention: yarn-resource-cost/yarn_job_cost_api.py

Important Files Changed

Filename	Overview
yarn-resource-cost/yarn_job_cost_api.py	Adds the typed EMR API, but S3 event-segment flattening and archived-node identity loss can produce incorrect usage.
yarn-resource-cost/yarn_resource_cost.py	Re-exports the new API from the package's public module without changing CLI behavior.
yarn-resource-cost/test_yarn_job_cost_api.py	Covers local single-segment input and missing archives but does not exercise multi-segment S3 logs or multi-node NodeManager fallback.
yarn-resource-cost/pyproject.toml	Adds setuptools packaging, the existing CLI entry point, and an optional boto3 dependency.
yarn-resource-cost/README.md	Documents installation and application-scoped API usage with injected boto clients.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Request[EmrApplicationUsageRequest] --> EventS3[Materialize Spark event log]
  Request --> EMR[Describe EMR cluster]
  EMR --> YarnS3[Materialize archived YARN logs]
  EventS3 --> Metadata[Read application metadata]
  YarnS3 --> Evidence[Parse RM and NM evidence]
  Metadata --> Calculate[Calculate application usage]
  Evidence --> Calculate
  Calculate --> Result[YarnApplicationUsageResult]

Loading

_{Reviews (1): Last reviewed commit: "Add application-scoped YARN resource usa..." | Re-trigger Greptile}

[greptile-apps]

Event segments lose grouping

When an S3 event-log directory contains multiple events_* segments, downloading them relative to the directory key places every segment directly under the temporary destination. The event-log reader then treats each segment as a separate application stream, and later segments overwrite earlier metadata for the same application, causing incomplete or incorrect resource usage.

[greptile-apps]

Archived nodes share identity

When archived logs from multiple EMR nodes provide NodeManager fallback evidence, the preserved node/<id>/applications/hadoop-yarn hierarchy makes every log's immediate parent hadoop-yarn. parse_yarn_logs therefore collapses those logs onto one node and overwrites its capacity and instance type, causing recovered containers to contribute incorrect per-instance seconds.