Call a model from your app — and know, for every call, which model served it, why that one, and what it cost.
Llm.Complete, Llm.Extract<T> and Llm.Decide<T> are the calls you write. Behind each of them is one road, written
in Osy#: your router picks the model, room is reserved against every budget the call falls under before anything is
sent, the provider is called, the next model is tried when one fails, and a meter row is written for every attempt —
priced from your own price book, in millionths of a dollar.
use Osysharp.Llm@2;
use Osysharp.Models.Anthropic@1 { egress "api.anthropic.com"; }
app.Secrets = [ new Secret("Anthropic") ];
app.Llm = new LlmSetup {
Models = [
new LlmModel { Name = "Fast", Client = new AnthropicClient { ApiKey = Secret.Anthropic }, Model = "claude-haiku-5-5",
Price = new LlmPrice { Input = 0.10m, Output = 0.50m } },
new LlmModel { Name = "Strong", Client = new AnthropicClient { ApiKey = Secret.Anthropic }, Model = "claude-opus-5-5",
Price = new LlmPrice { Input = 4, Output = 20, CacheRead = 0.20m } },
],
};
var line = await Llm.Complete("Summarise this in one line: " + note); // text in, text out
var invoice = await Llm.Extract<Invoice>(scannedText); // a typed value, by the type's own schema
var urgency = await Llm.Decide<Urgency>(message, "How urgent is this for them?"); // a typed decision, with a confidence
Llm.Complete, Llm.Extract<T> and Llm.Decide<T> each answer a task. await it where you make it for the plain
call; keep the tasks and several calls are in flight at once, each routed, budgeted and metered on its own.
An app that puts a model to work on its own data: reading invoices, sorting an inbox, drafting replies, answering questions over its records. The first call is easy. What is hard arrives a month later — a bill nobody can explain, a provider outage that took a feature down with it, one user who spent the whole month's allowance in an afternoon, a test suite that needs a paid key to run.
This kit is the part of that you would otherwise write yourself, and get subtly wrong:
- One place decides which model serves a call. No call site names a provider. You list the models you have keys
for; a router (
Osysharp.LlmRouteror a class of your own) orders them for each call. - A provider that fails is not an outage. The call moves to the router's next model — and only to a model on the router's list.
- A limit is a limit. A budget's room is reserved before the call leaves, atomically, so twelve calls arriving at once cannot all take the last dollar.
- Every attempt is on the record. Served, failed or refused: the model, the rule that chose it, the facts the rule decided from, the tokens the provider reported, the cost at the price in force.
- Tests need no model. A scripted model stands where a provider would, and everything around it is real.
It has no screens. It is the layer other packages and your own pages build on.
| Providers | A provider is a class implementing ILlmClient (text and tools) or IDecisionModel (typed decisions). Osysharp.Models.Anthropic, Osysharp.Models.OpenAi (and any endpoint that speaks its wire) and Osysharp.Models.Gemini are three; an endpoint nobody ships support for is a class you write. |
| Routing | LlmSetup.Router is an ILlmRouter: a pure function from the facts of a call (its kind, its step, its size, what it needs, what it may cost) to an ordered list of your models, with the reason each other one was dropped. A call site may ask for a model by name; that can only narrow what the router admits. With no router, a call goes to the model it asked for, or to Default. |
| Fallback | The router's order is the fallback order. A provider that fails (an outage, a rate limit, an overload) is recorded and the next model is asked. A stream falls back until its first event; after that the answer is that model's. |
| Budgets | SetBudget("requester", "", LlmBudgetWindow.Day, 5m) — five dollars a day for each person. A budget is for a scope: app, requester, model, or any scope a caller names on its calls (task:8f1c…, agent:Screener). Per day, per month or in total. A call that does not fit is refused before anything is sent (LlmBudgetExceeded) — or, when the budget or the call says StepDown, the router is asked again with what is left as the ceiling and a cheaper model serves. |
| The meter | One ModelCall row per attempt: the model, the provider's id for it, the router's rule, the facts and the rejections as JSON, who asked, the scopes charged, the provider's token counts (cached reads and writes apart), the cost in millionths of a dollar and the price-book row it was priced from. Written once, never changed. SpentOn("task:" + task.Id) totals a scope. |
| Calls made elsewhere | RecordOutsideModelCall(new OutsideModelCall { Model, InputTokens, OutputTokens, CacheReadTokens, Cost, Account, At, Purpose, Scopes }) puts a call a host or another system ran on its own key on the meter, as it was measured there: a ModelCall with Kind == Outside, the account it ran on by name (Account, never the key), and when (At). Cost null: priced from the price book as it stood then. SpentOn and every reader by scope see it with the app's own calls. It counts against every budget an ordinary call with the same scopes falls under, in the period it ran in — after the fact: nothing is held and it is never refused; when it takes a budget past its limit the row's Problem says so, and the next ordinary call against that budget is refused. LlmSpend.OutsideMicros and SpentFromOutsideIn(scope, subject, window) say how much of a budget's use came from outside; charge a key's own calls and the calls recorded from it to one scope ("key:" + name) to give that key one budget. For whoever RecordsModelCalls admits, and the app's own code. |
| The price book | SetModelPrice("Fast", 0.10m, 0.50m) — dollars per million tokens, from a moment on. A call keeps the price it was made at: re-pricing never rewrites what was spent. Until the book has a row, the price on the model's own declaration applies. |
| Typed values | Llm.Extract<T> sends T's own schema (Json.SchemaOf<T>(), with the words you wrote on its members) and reads the answer as a T. An answer that is not the shape is sent back to the model once. |
| Typed decisions | Llm.Decide<T> answers a Decision<T>: the value, a confidence, and whether that confidence is calibrated. A model built to decide (LlmModel.Decider) serves it when you have one; any chat model can stand in, and says its confidence is its own estimate. |
| Concurrency | LlmModel.MaxConcurrent caps the calls on their way to one model; a call past the cap moves to the next candidate. |
| The operator's switch | When the operator has switched generative AI off for the app, every call is refused before anything is sent (LlmSwitchedOff). |
| Events | LlmBudgetNearlySpent once per period when a budget passes its warning share, LlmBudgetSpent when a call is refused. Hear them with [On]. |
Four policies, each with a default you override by declaring it:
policy UsesModels => IsStaff; // who may call a model at all (default: everyone)
policy ReadsModelUsage => IsFinance; // who reads the meter (default: nobody)
policy ManagesModelSpend => IsAdmin; // who sets prices and budgets (default: nobody)
policy RecordsModelCalls => IsBuildHost; // who records a call made elsewhere (default: who manages spend)
// Text.
string answer = await Llm.Complete(prompt);
string answer = await Llm.Complete(prompt, "Strong"); // one of your models, by its name
// Everything said: the whole request, and what the call is.
LlmResult result = await Llm.Complete(request, new LlmCall {
Kind = LlmTaskKind.Reason, Step = LlmStep.Plan, // what the router's floors are declared per
Scopes = ["task:" + task.Id], Purpose = "weekly digest", // what it is charged to, and what for
OnLimit = LlmOnLimit.StepDown, MaxCostMicros = 20000, // what to do when money is short
});
result.Text(); result.Model; result.CostMicros; result.Attempts;
// As it is written, for an answer somebody watches arrive.
stream<LlmStreamEvent> Draft(string prompt) {
foreach (var e in Llm.Stream(prompt)) { yield return e; }
}
// A typed value.
class Invoice { public string Supplier = ""; public decimal Total; public DateOnly? Due; }
Invoice invoice = await Llm.Extract<Invoice>(scannedText, "The supplier is the company that sent it.");
// A typed decision.
enum Urgency { Low, Normal, High }
Decision<Urgency> d = await Llm.Decide<Urgency>(message.Body, "How urgent is this for the customer?");
if (d.Value == Urgency.High && d.IsAtLeast(0.8m)) { Escalate(message); }
A call is refused, before anything is sent, with an exception you can tell apart: LlmSwitchedOff, LlmRouteDenied,
LlmBudgetExceeded (it carries the scope, the subject, the window and what is left), LlmBusy, LlmKeyRefused (a key
the app does not hold, or one the caller may not spend) — all of them LlmRefused. A provider's own failure is an ExternalServiceException.
Calls run from a task too: Parallel.ForEach(notes, note => { Summarise(note); }) meters and budgets each one. The
kit's own writes are committed by themselves, so a call in the middle of your work commits none of your work, and a
call whose surrounding work then fails is still on the meter.
An app that holds more than one key for a provider — one per team, one per customer — lists them, each by its own id, and a call names the one it runs on:
app.Llm = new LlmSetup {
Models = [ new LlmModel { Name = "Fast", Model = "claude-haiku-4-5" } ],
Keys = [
new LlmKey { Id = "team-a", Client = new AnthropicClient { ApiKey = Secret.TeamAKey } },
new LlmKey { Id = "team-b", Client = new AnthropicClient { ApiKey = Secret.TeamBKey }, Models = ["Fast"], MaxConcurrent = 4 },
],
DefaultKey = "team-a", // a call that names no key
};
policy UsesModelKey(string key) => key == "team-a" || IsTeamB; // who may spend which key; everyone, until you say
var answer = await Llm.Complete(request, new LlmCall { Key = "team-b" });
Every model the router orders for a keyed call is sent through that key's client, and the call never moves to another
key or a model's own client: a model the key does not serve (Models) is dropped with that reason, and when every model
on it fails the call fails. The meter records the key's id on every row (ModelCall.Account) and charges the call to
key:<id>, so SetBudget("key", "team-b", …) holds that key's calls — and the calls recorded on it from elsewhere
(RecordOutsideModelCall with Account = "team-b"). The id is plain text the app chooses, never the key: the compiler
refuses an id built from a secret.
SetBudget("app", "", LlmBudgetWindow.Month, 500m); // the whole app, a month
SetBudget("requester", "", LlmBudgetWindow.Day, 5m); // each person, a day — one row for everybody
SetBudget("requester", vip.Id.ToString(), LlmBudgetWindow.Day, 50m); // one person, instead of the row above
SetBudget("task", task.Id.ToString(), LlmBudgetWindow.Total, 2m, // one piece of work, in all —
LlmOnLimit.StepDown); // and step down rather than stop
decimal spent = SpentIn("requester", person.Id.ToString(), LlmBudgetWindow.Day);
Every call is charged to app, to requester (the signed-in person, or the actor the code runs as), to model, and
to whatever scopes its caller names. A reservation holds the most the call could cost — its prompt and the longest
answer it may give, at the model's price — and is settled at what the provider says it used. A call whose process
died gives its room back after LlmSetup.HoldFor (ten minutes).
Code can also hold its own calls to a limit, with no budget set by anybody — "the calls for this job, half a dollar in all":
await Llm.Complete(request, new LlmCall {
Limits = [ new LlmLimit { Scope = "job:" + job.Id, LimitMicros = 500000 } ],
});
decimal used = SpentAgainst("job:" + job.Id);
Every call that carries the limit is counted against it and reserves its room before it leaves, like a budget. It binds only the calls that carry it — code limits its own work, never anybody else's — and budgets apply on top.
Days and months are UTC.
A call made in a workflow's run is charged to the workflow and to that run — workflow:Onboarding and
run.Onboarding:8f1c… — so what a process costs is counted as it happens:
SetBudget("workflow", "Onboarding", LlmBudgetWindow.Month, 200); // every run of it together, a month
SetBudget(RunKind("Onboarding"), "", LlmBudgetWindow.Total, 2); // each run of it, two dollars in all
var cost = SpentOn(RunScope("Onboarding", run)); // what one run has cost
LlmCall.ChargedToItsRun = false is for code that names the run itself.
Put a ScriptedModel where a provider would go, and answer for it from the test:
// the test app's setup
app.Llm = new LlmSetup { Models = [
new LlmModel { Name = "Fast", Client = new ScriptedModel { Name = "Fast" }, Model = "scripted-fast",
Price = new LlmPrice { Input = 1, Output = 5 } },
] };
[Test]
void a_note_is_summarised_and_the_call_is_on_the_meter() {
ScriptedAnswer.Stub((model, request) => Scripted.Text("Early beans, saved in 2024.", 1000, 200));
Assert.Equal("Early beans, saved in 2024.", Summarise(note));
Assert.Equal(2000, ModelCall.Single().CostMicros); // 1000 tokens at $1 a million, 200 at $5
}
Scripted.Text, Scripted.ToolUse and Scripted.Decision build answers; throw an ExternalServiceException from the
script to play a provider that is down; GenerationIsOn.Stub(() => false) plays the operator's switch. Routing,
budgets and the meter are the real ones, so a test asserts what a call cost the same way your app reads it.
public class MyClient : ILlmClient {
public string ApiKey = "";
internal LlmResponse Complete(LlmRequest request) { … } // request.Model is already decided: serve it or throw
internal stream<LlmStreamEvent> Stream(LlmRequest request) { … } // end with MessageComplete, carrying the counts
}
Report the token counts the provider reported — the meter prices from them. Throw ExternalServiceException when the
provider fails; that is what moves the call to the next model. When request.OutputSchemaJson is set, answer with one
text block whose text is the JSON document.
The two methods are internal, as they are on ILlmClient: any package implements them, and only this package calls
them — from the call path, which routes, reserves and meters around each call. Code that holds a client and writes
client.Complete(request) does not compile; it calls Llm.Complete. So there is no model call that is not on the
meter.
The meter, the price book and the budgets are rows in your app's database that only this package's functions write. Each of these is a test in the kit's test app:
| A meter row is made only by the call path, and never changed | a_meter_row_is_written_only_by_the_call_path_and_never_changed |
| A call made elsewhere is recorded only by whom the app names, never on the meter's own scopes | a_call_made_somewhere_else_is_recorded_on_the_meter_as_it_was_measured_there |
| What a budget has counted is changed only by the call path | what_a_budget_has_counted_is_changed_only_by_the_call_path |
Budgets are set only by who manages the spend, through SetBudget |
budgets_are_set_only_by_who_manages_the_spend_and_only_through_the_kit |
Prices are set only by who manages the spend, through SetModelPrice |
prices_are_set_only_by_who_manages_the_spend_and_only_through_the_kit |
| A call is refused before anything is sent when its budget has no room | a_call_that_does_not_fit_its_budget_is_refused_before_anything_is_sent |
| Calls racing for one budget take exactly what fits | a_budget_reserves_atomically_under_parallel_calls |
| A limit a call carries is held the same way, and binds only the calls that carry it | a_limit_a_call_carries_is_reserved_against_before_the_call_leaves, a_limit_binds_only_the_calls_that_carry_it |
| What a limit has counted is changed only by the call path | what_a_limit_has_counted_is_changed_only_by_the_call_path |
| A provider is called only by the kit's call path — an app or another package calling one directly does not compile | AProviderIsCalledOnlyByTheLlmKitTests (the platform's own tests) |
| A call is on the meter though the work around it failed | a_call_is_on_the_meter_though_the_work_it_was_made_for_failed |
| The meter is read only by who the app says | a_call_is_recorded_for_who_it_ran_for_and_read_only_by_who_the_app_says |
A person is taken off the meter only by who manages the spend, through ForgetModelCallRequester, and what their calls cost stays counted |
a_person_is_taken_off_the_meter_and_what_their_calls_cost_stays_counted, only_whoever_manages_spend_takes_a_person_off_the_meter_and_only_through_the_kit |
Anyone who may call a model can read the budgets and what each has counted; the per-call detail is the meter's, and
ReadsModelUsage decides who reads that — with whoever manages the spend, who deletes old rows and takes a person
off them.
The meter keeps no text of a call, and it does say who the call was for. ForgetModelCallRequester(person)
takes a person off every row and deletes what was counted against a budget of their own; the rows and their cost
stay. With Osysharp.Privacy in the app, Osysharp.Privacy.Llm does it when the
person asks and when the period ends.
Two settings of LlmSetup are your organization's to govern (2.12.0). What your code and the kit read is the
organization's value, whatever app.Llm assigns:
AllowedModels(a lock): the provider models a call may be sent to, separated by commas or spaces — an exact id (claude-haiku-4-5) or a prefix ending in*(claude-haiku-*). Empty: any model. A candidate outside it is never sent, whatever your router ordered; the walk goes on to the next one, and a call none inside the list served is refused (PolicyRefused, keyOsysharp.Llm.LlmSetup.AllowedModels).MonthlyTokens(a range): the most tokens your calls may use in a calendar month (UTC) — prompt, answer and cache, as the provider reports them. Each call reserves its prompt (as the kit counts it from the request it sends) and the most its answer may run to (the call'sMaxTokens, else the model'sMaxOutputTokens, else 4096 — and that bound is what the provider is sent). A call whose reservation would take the month past the ceiling, counting the calls still on their way, is refused before anything is sent (PolicyRefused, keyOsysharp.Llm.LlmSetup.MonthlyTokens). An answered call is then counted at the provider's own numbers; a call that sent nothing gives its reservation back; a stream you stop reading keeps it. The month's count isLlmOrgUsage: only the kit's reserve and settle write it, and nothing deletes it.
Each refusal is on the meter too, outcome Refused. Read osy policy before naming a model.
| A model outside the list is refused, whatever the app assigns | a_locked_model_list_refuses_a_model_outside_it_whatever_the_app_assigns |
| The app's router cannot route past the list | the_apps_router_cannot_route_past_the_list |
| A streamed call and a decision are held the same way | stream_and_decide_are_held_like_complete |
| The call that would pass the month's ceiling is refused | the_token_ceiling_refuses_the_call_that_would_pass_it |
| The month's count cannot be deleted or lowered by the app | the_app_cannot_delete_or_lower_the_months_count |
| An app's own value of 0 is held inside the organization's range | an_app_value_of_zero_is_held_inside_the_range |
| A negative or understated output bound cannot pass the ceiling | a_negative_or_understated_output_bound_cannot_pass_the_ceiling |
| Calls started together cannot pass the ceiling between them | calls_started_together_cannot_pass_the_ceiling_between_them |
| A stream you stop reading is counted at its reservation | an_abandoned_stream_is_counted_at_its_reservation |
kits/llm/tests is a seed bank that reads the notes growers send in with their seed: 51 tests over three scripted
models and a small router of its own — the call path, the fallback walk, the price book, budgets under twelve
simultaneous calls, typed extraction with its repair, and decisions from a decision model and from a chat model
standing in.
osy test kits/llm/tests
- Package:
Osysharp.Llm—use Osysharp.Llm@2; - Source:
kits/llm—model/Contract.osy(providers),Routing.osy,Setup.osy,Call.osy(the call path),Streaming.osy,Structured.osy(extract, decide),Budgets.osy,Meter.osy(the meter and the price book),Llm.osy(the calls an app writes),Scripted.osy(the scripted model) - Uses nothing but the language: no host, no secret. The provider packages ask for theirs.
Taken from this package's own tests, which compile and run on every release.
using Osysharp.Llm;
using SeedBank;
[Test(Bank)]
void a_call_whose_provider_failed_gives_its_room_back() {
ScriptedAnswer.Stub((model, request) => {
if (Heard.Count(h => h.What == "fail") > 0) { throw new ExternalServiceException(model + " is down"); }
return Scripted.Text("A line.", 1000, 200);
});
runas (Tove) { SetBudget("app", "", LlmBudgetWindow.Day, 0.2m); UnitOfWork.Commit(); }
runas (Ines) {
new Heard { What = "fail" };
UnitOfWork.Commit();
Assert.Throws<ExternalServiceException>(() => Summarise("beans"));
// Two attempts, both failed: nothing spent, nothing still held.
Assert.Equal(0, LlmSpend.Single().SpentMicros);
Assert.Equal(0, LlmHold.Count());
}
}
From tests/budgets.test.osy.
using Osysharp.Llm;
using SeedBank;
[Test(Bank)]
[runas(Ines)]
void a_call_is_served_by_the_routers_first_model_and_metered_at_its_price() {
ScriptedAnswer.Stub((model, request) => Scripted.Text("Early beans, saved in 2024.", 1000, 200));
Assert.Equal("Early beans, saved in 2024.", Summarise("Early dwarf beans, saved from the 2024 crop, dry stored."));
var row = ModelCall.Single();
Assert.Equal("Small", row.Model);
Assert.Equal("scripted-small", row.ProviderModel);
Assert.Equal(ModelCallKind.Complete, row.Kind);
Assert.Equal(ModelCallOutcome.Served, row.Outcome);
Assert.Equal(1000, row.InputTokens);
Assert.Equal(200, row.OutputTokens);
// 1000 tokens at $1 a million and 200 at $5: 1000 + 1000 millionths of a dollar.
Assert.Equal(2000, row.CostMicros);
Assert.Equal("seed-bank:1", row.RouteRule);
Assert.Equal(Ines.Id, row.Requester);
}
From tests/calls.test.osy.
using Osysharp.Llm;
using SeedBank;
[Test(Bank)]
void only_whoever_manages_spend_takes_a_person_off_the_meter_and_only_through_the_kit() {
ScriptedAnswer.Stub((model, request) => Scripted.Text("A line.", 1000, 200));
runas (Omar) { Summarise("beans"); }
Background.Settle();
var omar = Omar.Id;
// A curator reads the meter and changes nothing on it; Omar himself does not either.
runas (Ines) { Assert.Denied(() => ForgetModelCallRequester(omar)); }
runas (Omar) { Assert.Denied(() => ForgetModelCallRequester(omar)); }
runas (Tove) {
// The treasurer may — through the kit, and no other way: a row rewritten by hand is refused.
Assert.Denied(() => { RewriteWhoAsked(ModelCall.Single(c => c.Requester == omar)); UnitOfWork.Commit(); });
UnitOfWork.Discard();
Assert.Equal(1, ModelCall.Count(c => c.Requester == omar));
// Positive control: the same treasurer, through the kit.
Assert.Equal(1, ForgetModelCallRequester(omar));
UnitOfWork.Commit();
Assert.Equal(0, ModelCall.Count(c => c.Requester == omar));
}
}
From tests/forget.test.osy.
- The Osysharp.Llm reference
- Osysharp.Llm in the package catalogue
- The Osy# reference — the language, the runtime and every kit
llms-full.txt— the whole reference and specification as one Markdown file, for a coding agent
MIT — see LICENSE.