A Generic SDK and CLI for GA4GH API services.
The documentation has this README, the usage, CLI and Python guides, the Rust API reference and the development guides on one site.
Set Configuration.base_path to the API root, including the service's path
prefix. For example, use https://example.org/ga4gh/tes/v1 for TES,
https://example.org/ga4gh/drs/v1 for DRS, or
https://example.org/ga4gh/wes/v1 for WES. A trailing slash is optional.
The SDK resolves service-info, tasks, objects, and runs below that root.
The API root must not contain a query string or fragment.
use ga4gh_sdk::clients::tes::TES;
use ga4gh_sdk::utils::configuration::Configuration;
let config = Configuration::new(
url::Url::parse("https://example.org/ga4gh/tes/v1")?
);
let tes = TES::new(&config).await?;Read a client's state with its accessors: configuration() returns the
configuration of the client's transport, which its requests use;
transport() the transport, for example to share it with a client for an
added endpoint; and discovered_service_info() the service info read at
construction. TES, WES, DRS, and TRS have these accessors (on TRS,
discovered_service_info() returns an Option, as a TRS client can be
built without discovery), and Transport::configuration() returns a
transport's configuration. To change the API root, credentials, or HTTP
client, build a new client, for example with with_transport.
without_discovery(transport, service) builds a TES, WES, or DRS
client from values the caller already has, without a service-info request,
as TRS::without_discovery does.
The service and transport fields of TES, WES, and DRS, the
client and capabilities fields of Discovered, and the config field
of Transport are private, and the config copy of TES, WES, and DRS
is gone. The client state ownership record
lists the replacement of each former public field.
Configs that previously used only the host must include the service path prefix
shown above. Transport::endpoint now accepts only relative paths inside that
root and returns EndpointError for invalid paths or roots; callers should not
pass absolute URLs or . and .. path segments. tes::urlencode and
wes::urlencode now encode a path segment (so a space becomes %20, not +);
use form encoding for query values. Transport now returns HttpStatusError
for non-2xx HTTP responses rather than std::io::Error.
The library returns its own error type, ga4gh_sdk::Error, in place of
Box<dyn std::error::Error> (see Errors). It is an enum
of the kinds of failure (Http, Transport, Endpoint, Parse,
Configuration, UnexpectedService, Other) and is Send + Sync, so
clients and their futures can move between threads. This breaks some code:
- Functions that return
Box<dyn Error>still accept the SDK's errors with?. Butdowncast_ref::<HttpError>()on such a box no longer finds the HTTP error, nor does it on ananyhow::Erroror in asource()chain: downcast toga4gh_sdk::Errorand call itsdowncast_ref, or match on its variants. - An error status in answer to the service-info request of
DRS::newandWES::newis anOthererror (with a hint on the API root for a 404), notHttp; itsdowncast_refreturns the HTTP error.TES::newandTRS::newreturnHttp. Other failures of that request keep their variant, such asTransportfor a failed connection, and no longer have the "Error accessing … service" prefix. - Implementations of
DescriptorFetcherand the attempts passed todrs::resolution::resolve_untilreturnga4gh_sdk::Error;?orError::other(error)converts other errors. - An invalid Service Registry response is now
Error::Parsewith theserde_jsondiagnostic, instead of a string prefixed with the registry endpoint. Match the variant to handle it without parsing the message. The CLI still includes the endpoint in its error output. - The
servicefield ofTES,WESandDRSis now aServiceinstead of aResult; the constructors return the errors. Struct literals of the clients setservice: Service::default()or the service info itself. - The error of
ResolveCause::Requestand of the workflow contracts (ContractError) isga4gh_sdk::Error.
TES, WES and DRS check the service type only in their constructors.
No TES or WES method checks it again (TES::create, create_value,
WES::run_workflow and run_workflow_with_attachments did), so a client
built as a struct literal with another service sends its requests. Use
with_artifact for a service whose service info names another artifact
(see Build on this SDK).
The WES models follow WES 1.1.0 and use the schema names of the spec:
WesRunRequest is now RunRequest, WesRunLog is RunLog, WesState is
State, and so on (the Wes prefix is gone). RunRequest::new takes the
required workflow_type, workflow_type_version and workflow_url, which are
now String; tags and workflow_engine_parameters are string maps. Run list
items are RunListResponseRunsInner and deprecated task log items are
RunLogTaskLogsInner. Run::list_tasks and Run::get_task call the new task
endpoints.
All models now keep what their spec doesn't define (see Models). This breaks some code:
- Model structs have a new public field,
extra. Struct literals of models must set it or end with..Default::default(); prefer the models'newfunctions. - Enums have a new variant,
Unrecognized(String), and are no longerCopy.matchstatements need an arm for it, and code that copies an enum value must.clone()it.TesStateand the WESStateare not affected: they stayCopyand read unknown states asUnknown, which is now their last variant (this changes theirOrdorder). They write such states back asUNKNOWN. - Model enums of a single property, such as the DRS
access_method::Type, now implementDisplay. extrakeys with the name of a field of the model are sent twice in the JSON; only WESrun_workflowrejects them.- Unknown fields are no longer dropped: serializing a model, or
ga4gh-cli registry get, now includes the fields that the server added. - WES
run_workflowsendsRunRequest.extraas further form parts; names of the request's own parts are an error. - TRS methods that take a descriptor type encode it as a path segment and
reject
Unrecognizedvalues that are empty,.or... TesTask::new(executors)now setsexecutors; it used to ignore them.
The SDK uses reqwest 0.12 instead of 0.11:
ga4gh_sdk::reqwestre-exports 0.12, andTransport::with_clientandwith_client_and_configured_authtake a reqwest 0.12Client. Build it withga4gh_sdk::reqwest, or move your ownreqwestdependency to 0.12. Its header, method and status types come fromhttp1.- The Python wheels verify TLS certificates with rustls 0.23. The CLI's S3
downloads in
tes saveuse OpenSSL, like its other requests. - Charset decoding, HTTP/2 and the system proxy settings work as before.
A
Transporterror reports a safe failure category by default; itsreqwest::Errorand source chain remain available for explicit inspection.
A service-info response that isn't valid UTF-8 is now an error of
ServiceInfo::get, get_at, the discovery of TES::new, WES::new and
DRS::new, and capability discovery, as it already was for TRS::new;
such text used to be repaired with replacement characters. See
Service Info.
Each release has an archive of
ga4gh-cli per platform, named ga4gh-cli-<version>-<target>:
| Platform | Target | Archive |
|---|---|---|
| Linux x86-64 | x86_64-unknown-linux-gnu |
.tar.gz |
| Linux aarch64 | aarch64-unknown-linux-gnu |
.tar.gz |
| macOS x86-64 | x86_64-apple-darwin |
.tar.gz |
| macOS arm64 | aarch64-apple-darwin |
.tar.gz |
| Windows x86-64 | x86_64-pc-windows-msvc |
.zip |
| Windows arm64 | aarch64-pc-windows-msvc |
.zip |
The Linux binaries need glibc 2.34 or newer (for example Ubuntu 22.04,
Debian 12, RHEL 9 or newer), the OpenSSL 3 library (libssl3 on Debian and
Ubuntu, openssl-libs on Fedora and RHEL) and CA certificates
(ca-certificates); minimal container images may lack the last two.
Elsewhere, build from source as below. The binaries aren't signed, so macOS
may block them until you remove the quarantine attribute
(xattr -d com.apple.quarantine ga4gh-cli). For example, on Linux x86-64,
check the download against the release's SHA256SUMS and, optionally, its
build provenance, then unpack it (replace the example version with that of
the release):
version=0.2.0
target=x86_64-unknown-linux-gnu
archive="ga4gh-cli-$version-$target.tar.gz"
base="https://github.com/ga4gh/ga4gh-sdk/releases/download/v$version"
curl --fail --location --remote-name "$base/$archive"
curl --fail --location --remote-name "$base/SHA256SUMS"
sha256sum --check --ignore-missing SHA256SUMS
gh attestation verify "$archive" --repo ga4gh/ga4gh-sdk # needs the GitHub CLI
tar -xzf "$archive"
"ga4gh-cli-$version-$target/ga4gh-cli" --helpThese instructions install the SDK from this source repository.
Prerequisites: Rust via
rustup and Git. The repository pins its toolchain in
rust-toolchain.toml; rustup installs that version
on the first build. On Linux, the library and CLI also need the OpenSSL
development files and pkg-config (for example libssl-dev pkg-config on
Debian and Ubuntu).
Install the command-line tool ga4gh-cli into ~/.cargo/bin:
git clone https://github.com/ga4gh/ga4gh-sdk.git
cd ga4gh-sdk
cargo install --path cli --locked
ga4gh-cli --helpTo run the CLI as a container, for example as a Kubernetes Job inside a
cluster, see the container image guide. Releases
publish the image as ghcr.io/ga4gh/ga4gh-cli; the guide says whether a
release includes it yet.
To use the Rust library in your own project, add it as a Git dependency
(package ga4gh-lib, crate ga4gh_sdk):
[dependencies]
ga4gh-lib = { git = "https://github.com/ga4gh/ga4gh-sdk" }For the Python bindings (built with maturin), see
the Python README for installation and the ga4gh_sdk
import name. Maintainers can follow the release guide.
Install Funnel and Docker, then start a local TES server in another terminal. The repository's integration suite uses Funnel v0.11.14; Docker is needed to execute tasks.
funnel server run --Server.HTTPPort=8000Point the CLI at that service by writing ~/.ga4gh-cli/config.json.
base_path is the API root, including the service's path prefix:
mkdir -p ~/.ga4gh-cli
cat > ~/.ga4gh-cli/config.json <<'JSON'
{
"TES": {
"base_path": "http://localhost:8000/ga4gh/tes/v1"
}
}
JSONSubmit the checked-in task file, which runs a greeting in an Alpine container without external storage. Then check its status and list tasks:
task_id=$(ga4gh-cli tes create examples/tes-hello.json)
ga4gh-cli tes status "$task_id"
ga4gh-cli tes listFor scripts that need to preserve arbitrary server-provided IDs, use
ga4gh-cli tes create examples/tes-hello.json --format json and parse its
id field. The text output above is intended for ordinary IDs.
Docker may download the Alpine image on the first run.
The CLI README describes all CLI commands. The
usage guide covers configuration, authentication, and the
Rust library's ServiceInfo, TES, WES, DRS, TRS, and ServiceRegistry
clients. The CLI also has WES commands,
Service Registry commands and a
service-info command.
The provider contract proposals describe possible extensions for leased interactive sessions and workload-bound secrets. They are not implemented SDK features.
Advanced transport callers can use request_response for status and headers
before parsing, then consume reqwest::Response::chunk() incrementally or call
utils::transport::read_bounded for lossless bytes. Dropping the response stops
consumption. The existing text helpers remain available and now bound success
bodies to 8 MiB and error bodies to 16 KiB while reading, including chunked bodies.
Use request_text with explicit BodyLimits for different bounds. A
BodyLimitError retains HTTP status even for an oversized error response.
Transport::with_client and request_builder support custom client budgets and
headers; HTTP types are re-exported as ga4gh_sdk::reqwest.
clients::trs::responses offers tools pages with all pagination headers, follows
the server's next_page URL within the same tools API root, and returns an
opaque files_zip response without extraction or text conversion. These helpers
compose with a configured Transport independently of the basic TRS client.
By default, connecting may take 10 seconds and a request 60 seconds; the
service-info check of the TES, WES, DRS and TRS clients may take 10 seconds and a
TES task or WES run submission, including uploads, 300 seconds. Set them with
Configuration::with_timeouts (or the timeouts object of a configuration file
section); Transport::with_timeouts sets only the first two. The clients'
with_transport constructors take a ready Transport, such as one with an
injected client. The usage guide says which limit
applies to which request.
For a multi-attempt operation, RequestOptions holds one absolute deadline, a local cancellation
token, body bounds and an explicit replay classification. Retries are opt-in,
capped at 32 attempts, and use exponential backoff with jitter; 429/503 honor
delta/date Retry-After within that same deadline. Mark only protocol-defined
reads as ReadOnly (including appropriate lookup POSTs). A non-replayable
operation gets one attempt and reports lost-response ambiguity for reconciliation.
send_once also covers multipart headers/body and permits an explicit larger
upload budget. Dropping a future or canceling its token stops local I/O.
If cancellation or an expired deadline prevents the send future from being
polled, submission returns an inspectable OperationError and makes no attempt.
Once the send future is polled, interruption remains an
AmbiguousSubmissionError: the server may have accepted the request, so reconcile
remote state before resubmitting. Local interruption never cancels a remote job.
DRS get_object_once and get_access_url_once return Resolution::Ready or
Pending with a credential-free resume reference. The immediate convenience
methods return a downcastable PendingResolution error on 202. Opt-in
resolve_object/resolve_access_url honor integral-second Retry-After, one
absolute Operation budget, cancellation and an attempt bound; failures retain
the reference and last pending state. Zero delay becomes one second; missing,
malformed, overflow or greater-than-one-hour delays are protocol errors.
Authentication and malformed 200 responses remain errors, without pending retries.
clients::drs::uri::DrsResolver accepts hostname-based drs://host/encoded-id
references, decodes IDs once, and returns an explicit access plan (URL and
object-store headers). Configure an authority-to-Transport mapping for gateways
and per-service credentials; unmapped authorities get a fresh anonymous HTTPS
transport. Protocol/region preferences and an explicit method index control
selection. Direct URLs and access-ID negotiation share deferred resolution;
pending, unsupported/unavailable methods and local interruption stay distinct.
The host is a name or an IPv4 address, without a port. IPv6 literals and
compact identifier forms are explicitly unsupported in this MVP. It never
downloads object bytes or copies DRS credentials into an object-store request.
DRS get_bulk_objects and get_bulk_access_urls accept optional Passports and
one service/credential context per call. BulkPolicy bounds chunk size,
concurrency, total HTTP attempts and 413 split depth; an advertised
maxBulkRequestLength can lower the caller-selected chunk size. Missing metadata
stays unknown. Replies are correlated by object/access identity rather than
position, with successful siblings, grouped failures, duplicate/omitted identity
diagnostics and pending outcomes retained. Duplicate inputs collapse. The MVP
requires one access ID per object per call, so object-only grouped errors never
receive fabricated per-access attribution. A 202 retains pending keys and delay;
resume through the same bulk API with caller-supplied credentials/Passports.
There is no singular or anonymous fallback. Compatibility is verified against
protocol fixtures, without a claim of broad real-service interoperability.
Bulk requests always use the published DRS 1.4/1.5 shape. For object lookups,
the SDK also reads one documented Gen3 (indexd) reply difference: resolved
objects in resolved_drs_objects instead of resolved_drs_object.
BulkReport::dialects lists the shapes (Standard, Gen3) seen in a call's
replies, merged over all chunks. If a reply has both arrays, the standard one
wins and a diagnostic is added; a null array counts as absent.
Gen3 objects can carry created_time: null, which the typed DrsObject cannot
represent. get_bulk_objects reports such an entry as an error in its outcome
and never invents a timestamp; get_bulk_objects_raw returns each requested
resolved entry as JSON, correlated by id. Single-object get_object has the
same limit for these objects; use get_object_raw. The Gen3 bulk access reply
is not read in another shape, because it was not verified without credentials.
Checked against CanineDC on 2026-11-10; see the opt-in live check.
The GA4GH Starter Kit DRS bulk routes (1.3.0experimental) are not supported.
In its develop branch at 2b76a63
(read on 2026-11-10, not run), they expect selection instead of
bulk_object_ids, report failures in an unresolved_drs_object map, and
return bulk access URLs as a bare array without object or access IDs, which
cannot be correlated safely. The SDK sends only standard requests. In a reply,
it ignores the unresolved_drs_object map, adds a diagnostic and records no
dialect. Because the Starter Kit does not read bulk_object_ids, its resolved
array stays empty and the requested IDs are reported as omitted.
TES::discover and WES::discover return a Discovered<T> wrapper with a
cached CapabilitySnapshot. One service-info request supplies both the client
and the snapshot; typed/raw accessors make no requests and refresh() is
explicit; it replaces the client's discovered service and the snapshot
together. client() and capabilities() read the two parts, into_parts()
splits the wrapper, and Deref still reaches the client's methods. Raw
extensions are preserved. Preflight diagnostics distinguish
supported, unsupported and unknown/stale metadata. create_validated and
run_validated accept an explicit strict flag; ordinary submission stays
advisory. TES backend keys are case insensitive and storage is nonexhaustive.
Task::wait and Run::wait accept WaitOptions with an operation budget, local
cancellation and polling/retry settings. The result distinguishes successful
completion, executor/system failures, preemption, cancellation, polling failure
and observation interruption, retaining the last state and a credential-free
RemoteHandle. Rebuild observation with caller-supplied configuration. Dropping
or canceling observation never cancels remote work; cancel_and_wait is an
explicit separate action. Optional failure logs are fetched once.
TES/WES pages return a lazy Pager with next_page and next_item methods.
It preserves filters/view and opaque tokens, continues past empty intermediate
pages, detects token cycles, and bounds requests/items. Memory holds one page
plus fixed-size token digests bounded by max_requests (default 1000). Checkpoints bind
service, API root and query; a failed request retains the valid continuation.
At a partially consumed or truncated page, a checkpoint replays that page.
Mutable listings provide neither snapshot isolation nor exactly-once processing;
callers reconcile replays. Credentials are supplied separately when resuming.
Query parameters are a JSON object. Transport, Pager and the Python
bindings encode them with one policy, documented in utils::query: a string,
number or boolean gives one parameter, null leaves the parameter out, and a
list of strings, numbers and booleans repeats it once per item, in order
(tag_key=a&tag_key=b; an empty list sends nothing). Keys are sorted. Keys and
values are form encoded, so a space becomes +. Objects, nested lists and
null inside a list are rejected before a request is sent, because TES pairs
tag_key and tag_value items by position.
Behavior change: Pager used to send objects and nested lists as JSON text,
and Pager and the Python Transport left out null list items.
Pager::new and the Python methods now return an error for them instead;
remove such values from the query before the call.
The CLI streams tes list --all --format jsonl and wes list --all --format jsonl through caller-owned writers, with one JSON object per stdout line.
--max-pages and --max-items bound work; limits or later failures preserve
emitted records, report a resume checkpoint on stderr and return a nonzero exit
code. Reusable CLIs use dispatch_to or
dispatch_with_config_to; existing buffered dispatch remains available for
ordinary commands.
WES::run_workflow_streaming accepts StreamingAttachment sources as bytes,
files or owned async readers, with optional known lengths. Multipart content
follows backpressure and shares the normal transport header policy. Paths are
validated before I/O, nested relative filenames stay intact, and one absolute
operation budget covers preparation, upload and response. A consumed source
is never replayed; cancellation or a lost response after transmission reports
that submission may have been accepted. The existing bytes API remains usable.
clients::bundle::PreparedBundle::prepare stages a CWL workflow and its declared
TRS descriptors without submitting it. It validates paths/collisions and the
static run, $import and $include dependency graph, verifies advertised
SHA-256 digests, and computes local digests over original staged bytes. Version
metadata (is_production, meta_version) and retrieved digests are preserved;
an explicit TRS version is not assumed immutable. Unsupported checksums fail by
default; explicit opt-in records them as unverified. The small MVP supports a
primary Workflow in CWL v1.0/v1.1/v1.2; packed/fragment, external and dynamic
dependencies are rejected before submission.
Preparation exposes a reviewable request plan and a credential-free provenance
manifest. Wrapper URLs require an explicit DescriptorFetcher; the provided
anonymous allowlist accepts only bare HTTP(S) origins, never reuses registry
credentials, and never follows redirects.
submit is a separate explicit call and uploads the same immutable staged bytes
that were hashed, returning a run plus manifest/execution linkage. Workflow
parameters, bearer tokens and signed descriptor URLs are excluded from provenance.
See the named Toil integration check for an actual CWL execution.
Enable ga4gh-lib/workflow-contracts for pinned local I/O validation, immutable
WES invocation plans, typed results with selected artifact evidence, and explicit
scientific comparison policies. The CLI exposes workflow prepare, submit,
results, verify and compare. See the workflow guide
for the public sequence-QC example, offline commands and evidence boundaries.
The same optional feature provides exact result reuse
from a trusted receipt and freshly checked artifacts; it does not submit a job.
An SDK for a service with API extensions can build on this SDK without forking it: add clients that share its transport, extend its models and add CLI subcommands. The guide Build on this SDK walks through an example crate that CI builds and tests.
Developer note: If you want to update any of the OpenAPI specifications supported by this tool or extend it to support any other OpenAPI-backed services (e.g., not currently supported GA4GH API-based services or your own services), please refer to our documentation for auto-generating OpenAPI models.
To build the project:
cargo build --workspace --lockedThe library includes a client for the
GA4GH Tool Registry Service (TRS) API, versions 2.0.1 and 2.0.2, for registries such as
WorkflowHub or Dockstore. Set
Configuration.base_path to the TRS API root, including its path prefix,
with or without a trailing slash, e.g. https://workflowhub.eu/ga4gh/trs/v2
or https://dockstore.org/api/ga4gh/trs/v2.
use ga4gh_sdk::clients::trs::models::DescriptorTypeWithPlain;
use ga4gh_sdk::clients::trs::{ToolFilter, TRS};
use ga4gh_sdk::utils::configuration::Configuration;
let config = Configuration::new(url::Url::parse("https://workflowhub.eu/ga4gh/trs/v2")?);
let trs = TRS::new(&config).await?; // fetches and checks service-info
let filter = ToolFilter {
tool_class: Some("Workflow".into()),
limit: Some(10),
..Default::default()
};
let tools = trs.list_tools(&filter).await?;
let descriptor = trs
.get_descriptor("2", "1", DescriptorTypeWithPlain::PlainGalaxy)
.await?;TRS covers service-info, toolClasses, tools with all filters
(including the tags filter of 2.0.2), single tools, versions, descriptors
(also by relative path), test files, file listings, ZIP archives of all files
of a version (get_files_zip, read into memory up to a limit you choose) and
container files. Tool IDs, version IDs and relative paths are
percent-encoded as single path segments, so Dockstore IDs such as
#workflow/github.com/org/repo work as they are. For PLAIN_* descriptor
types, the bare file is returned as a FileWrapper with only content set;
list_tests then returns a list with that one wrapper (or the wrappers, if
the registry sends a JSON array of them). Dockstore sends the
service-info timestamps createdAt and updatedAt as numbers; the client
drops such non-string values instead of failing.
The basic list_tools method returns a page without headers. Use
clients::trs::responses with &trs for server-provided paging URLs, retaining
the selected version profile (pass ToolFilter::query() as parameters).
Use responses::files_zip(trs.transport(), ...) to stream large archives.
The compatibility guide explains version selection,
tag-filter capabilities, and connecting to deployments without usable discovery.
The default transport does not apply tokens from Configuration. For an
authenticated registry, supply a configured reqwest::Client through
Transport::with_client and TRS::with_transport; see the
usage guide. The CLI's trs commands use the default
transport and currently work with public endpoints; see the
CLI README.
Unit tests run by default. Integration suites are opt-in.
To run the workspace's unit and documentation tests, use:
cargo test --workspace --lockedFor opt-in integration suites, see the integration test guide for prerequisites, commands, and service-specific behavior.
Optional tools for running or measuring the default tests:
cargo nextest run
cargo llvm-cov nextestThe project adopts the semver scheme for versioning. Currently the software is in a pre-release stage, so changes to the API, including breaking changes, may occur at any time without further notice.
This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, ideas, fixes, or other code changes. Please read these guidelines if you want to contribute. And please mind the code of conduct for all interactions with the community.
New developers start with the developer guide: the repository layout, setup, tests, documentation and how to add a client. For local checks, coverage, and hooks, see the development guide.
This project is covered by the Apache License 2.0 also shipped with this repository.
The project is maintained by ELIXIR Cloud & AAI, a Driver Project of the Global Alliance for Genomics and Health (GA4GH), under the umbrella of the ELIXIR Compute Platform.
To get in touch with us, please use one of the following routes:
- For filing bug reports, feature requests or other code-related issues, please make use of the project's issue tracker.
- For private/personal issues, more involved communication, or if you would like to join our team as a regular contributor, you can either join our chat board or email the community leaders.