Skip to content
osysharpPublic

About

Osysharp.Llm — an Osy# package. calling a model: the provider and router contracts, the call path (route, reserve, call, fall back, meter), a price book and budgets with hard limits, typed extraction and typed decisions, and a scripted model for tests.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

11 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Osysharp.Llm

Call a model from your app — and know, for every call, which model served it, why that one, and what it cost.

Llm.Complete, Llm.Extract<T> and Llm.Decide<T> are the calls you write. Behind each of them is one road, written in Osy#: your router picks the model, room is reserved against every budget the call falls under before anything is sent, the provider is called, the next model is tried when one fails, and a meter row is written for every attempt — priced from your own price book, in millionths of a dollar.

use Osysharp.Llm@2;
use Osysharp.Models.Anthropic@1 { egress "api.anthropic.com"; }

app.Secrets = [ new Secret("Anthropic") ];

app.Llm = new LlmSetup {
  Models = [
    new LlmModel { Name = "Fast",   Client = new AnthropicClient { ApiKey = Secret.Anthropic }, Model = "claude-haiku-5-5",
                   Price = new LlmPrice { Input = 0.10m, Output = 0.50m } },
    new LlmModel { Name = "Strong", Client = new AnthropicClient { ApiKey = Secret.Anthropic }, Model = "claude-opus-5-5",
                   Price = new LlmPrice { Input = 4, Output = 20, CacheRead = 0.20m } },
  ],
};

var line    = await Llm.Complete("Summarise this in one line: " + note);            // text in, text out
var invoice = await Llm.Extract<Invoice>(scannedText);                               // a typed value, by the type's own schema
var urgency = await Llm.Decide<Urgency>(message, "How urgent is this for them?");    // a typed decision, with a confidence

Llm.Complete, Llm.Extract<T> and Llm.Decide<T> each answer a task. await it where you make it for the plain call; keep the tasks and several calls are in flight at once, each routed, budgeted and metered on its own.

Use case

An app that puts a model to work on its own data: reading invoices, sorting an inbox, drafting replies, answering questions over its records. The first call is easy. What is hard arrives a month later — a bill nobody can explain, a provider outage that took a feature down with it, one user who spent the whole month's allowance in an afternoon, a test suite that needs a paid key to run.

This kit is the part of that you would otherwise write yourself, and get subtly wrong:

  • One place decides which model serves a call. No call site names a provider. You list the models you have keys for; a router (Osysharp.LlmRouter or a class of your own) orders them for each call.
  • A provider that fails is not an outage. The call moves to the router's next model — and only to a model on the router's list.
  • A limit is a limit. A budget's room is reserved before the call leaves, atomically, so twelve calls arriving at once cannot all take the last dollar.
  • Every attempt is on the record. Served, failed or refused: the model, the rule that chose it, the facts the rule decided from, the tokens the provider reported, the cost at the price in force.
  • Tests need no model. A scripted model stands where a provider would, and everything around it is real.

It has no screens. It is the layer other packages and your own pages build on.

What it does

Providers A provider is a class implementing ILlmClient (text and tools) or IDecisionModel (typed decisions). Osysharp.Models.Anthropic, Osysharp.Models.OpenAi (and any endpoint that speaks its wire) and Osysharp.Models.Gemini are three; an endpoint nobody ships support for is a class you write.
Routing LlmSetup.Router is an ILlmRouter: a pure function from the facts of a call (its kind, its step, its size, what it needs, what it may cost) to an ordered list of your models, with the reason each other one was dropped. A call site may ask for a model by name; that can only narrow what the router admits. With no router, a call goes to the model it asked for, or to Default.
Fallback The router's order is the fallback order. A provider that fails (an outage, a rate limit, an overload) is recorded and the next model is asked. A stream falls back until its first event; after that the answer is that model's.
Budgets SetBudget("requester", "", LlmBudgetWindow.Day, 5m) — five dollars a day for each person. A budget is for a scope: app, requester, model, or any scope a caller names on its calls (task:8f1c…, agent:Screener). Per day, per month or in total. A call that does not fit is refused before anything is sent (LlmBudgetExceeded) — or, when the budget or the call says StepDown, the router is asked again with what is left as the ceiling and a cheaper model serves.
The meter One ModelCall row per attempt: the model, the provider's id for it, the router's rule, the facts and the rejections as JSON, who asked, the scopes charged, the provider's token counts (cached reads and writes apart), the cost in millionths of a dollar and the price-book row it was priced from. Written once, never changed. SpentOn("task:" + task.Id) totals a scope.
Calls made elsewhere RecordOutsideModelCall(new OutsideModelCall { Model, InputTokens, OutputTokens, CacheReadTokens, Cost, Account, At, Purpose, Scopes }) puts a call a host or another system ran on its own key on the meter, as it was measured there: a ModelCall with Kind == Outside, the account it ran on by name (Account, never the key), and when (At). Cost null: priced from the price book as it stood then. SpentOn and every reader by scope see it with the app's own calls. It counts against every budget an ordinary call with the same scopes falls under, in the period it ran in — after the fact: nothing is held and it is never refused; when it takes a budget past its limit the row's Problem says so, and the next ordinary call against that budget is refused. LlmSpend.OutsideMicros and SpentFromOutsideIn(scope, subject, window) say how much of a budget's use came from outside; charge a key's own calls and the calls recorded from it to one scope ("key:" + name) to give that key one budget. For whoever RecordsModelCalls admits, and the app's own code.
The price book SetModelPrice("Fast", 0.10m, 0.50m) — dollars per million tokens, from a moment on. A call keeps the price it was made at: re-pricing never rewrites what was spent. Until the book has a row, the price on the model's own declaration applies.
Typed values Llm.Extract<T> sends T's own schema (Json.SchemaOf<T>(), with the words you wrote on its members) and reads the answer as a T. An answer that is not the shape is sent back to the model once.
Typed decisions Llm.Decide<T> answers a Decision<T>: the value, a confidence, and whether that confidence is calibrated. A model built to decide (LlmModel.Decider) serves it when you have one; any chat model can stand in, and says its confidence is its own estimate.
Concurrency LlmModel.MaxConcurrent caps the calls on their way to one model; a call past the cap moves to the next candidate.
The operator's switch When the operator has switched generative AI off for the app, every call is refused before anything is sent (LlmSwitchedOff).
Events LlmBudgetNearlySpent once per period when a budget passes its warning share, LlmBudgetSpent when a call is refused. Hear them with [On].

Who may do what

Four policies, each with a default you override by declaring it:

policy UsesModels        => IsStaff;      // who may call a model at all        (default: everyone)
policy ReadsModelUsage   => IsFinance;    // who reads the meter                (default: nobody)
policy ManagesModelSpend => IsAdmin;      // who sets prices and budgets        (default: nobody)
policy RecordsModelCalls => IsBuildHost;  // who records a call made elsewhere  (default: who manages spend)

Calls

// Text.
string answer = await Llm.Complete(prompt);
string answer = await Llm.Complete(prompt, "Strong");                  // one of your models, by its name

// Everything said: the whole request, and what the call is.
LlmResult result = await Llm.Complete(request, new LlmCall {
  Kind = LlmTaskKind.Reason, Step = LlmStep.Plan,                // what the router's floors are declared per
  Scopes = ["task:" + task.Id], Purpose = "weekly digest",       // what it is charged to, and what for
  OnLimit = LlmOnLimit.StepDown, MaxCostMicros = 20000,          // what to do when money is short
});
result.Text();  result.Model;  result.CostMicros;  result.Attempts;

// As it is written, for an answer somebody watches arrive.
stream<LlmStreamEvent> Draft(string prompt) {
  foreach (var e in Llm.Stream(prompt)) { yield return e; }
}

// A typed value.
class Invoice { public string Supplier = ""; public decimal Total; public DateOnly? Due; }
Invoice invoice = await Llm.Extract<Invoice>(scannedText, "The supplier is the company that sent it.");

// A typed decision.
enum Urgency { Low, Normal, High }
Decision<Urgency> d = await Llm.Decide<Urgency>(message.Body, "How urgent is this for the customer?");
if (d.Value == Urgency.High && d.IsAtLeast(0.8m)) { Escalate(message); }

A call is refused, before anything is sent, with an exception you can tell apart: LlmSwitchedOff, LlmRouteDenied, LlmBudgetExceeded (it carries the scope, the subject, the window and what is left), LlmBusy, LlmKeyRefused (a key the app does not hold, or one the caller may not spend) — all of them LlmRefused. A provider's own failure is an ExternalServiceException.

Calls run from a task too: Parallel.ForEach(notes, note => { Summarise(note); }) meters and budgets each one. The kit's own writes are committed by themselves, so a call in the middle of your work commits none of your work, and a call whose surrounding work then fails is still on the meter.

Keys

An app that holds more than one key for a provider — one per team, one per customer — lists them, each by its own id, and a call names the one it runs on:

app.Llm = new LlmSetup {
  Models = [ new LlmModel { Name = "Fast", Model = "claude-haiku-4-5" } ],
  Keys = [
    new LlmKey { Id = "team-a", Client = new AnthropicClient { ApiKey = Secret.TeamAKey } },
    new LlmKey { Id = "team-b", Client = new AnthropicClient { ApiKey = Secret.TeamBKey }, Models = ["Fast"], MaxConcurrent = 4 },
  ],
  DefaultKey = "team-a",                                          // a call that names no key
};
policy UsesModelKey(string key) => key == "team-a" || IsTeamB;    // who may spend which key; everyone, until you say

var answer = await Llm.Complete(request, new LlmCall { Key = "team-b" });

Every model the router orders for a keyed call is sent through that key's client, and the call never moves to another key or a model's own client: a model the key does not serve (Models) is dropped with that reason, and when every model on it fails the call fails. The meter records the key's id on every row (ModelCall.Account) and charges the call to key:<id>, so SetBudget("key", "team-b", …) holds that key's calls — and the calls recorded on it from elsewhere (RecordOutsideModelCall with Account = "team-b"). The id is plain text the app chooses, never the key: the compiler refuses an id built from a secret.

Budgets

SetBudget("app", "", LlmBudgetWindow.Month, 500m);                          // the whole app, a month
SetBudget("requester", "", LlmBudgetWindow.Day, 5m);                        // each person, a day — one row for everybody
SetBudget("requester", vip.Id.ToString(), LlmBudgetWindow.Day, 50m);        // one person, instead of the row above
SetBudget("task", task.Id.ToString(), LlmBudgetWindow.Total, 2m,            // one piece of work, in all —
          LlmOnLimit.StepDown);                                             //   and step down rather than stop

decimal spent = SpentIn("requester", person.Id.ToString(), LlmBudgetWindow.Day);

Every call is charged to app, to requester (the signed-in person, or the actor the code runs as), to model, and to whatever scopes its caller names. A reservation holds the most the call could cost — its prompt and the longest answer it may give, at the model's price — and is settled at what the provider says it used. A call whose process died gives its room back after LlmSetup.HoldFor (ten minutes).

Code can also hold its own calls to a limit, with no budget set by anybody — "the calls for this job, half a dollar in all":

await Llm.Complete(request, new LlmCall {
  Limits = [ new LlmLimit { Scope = "job:" + job.Id, LimitMicros = 500000 } ],
});
decimal used = SpentAgainst("job:" + job.Id);

Every call that carries the limit is counted against it and reserves its room before it leaves, like a budget. It binds only the calls that carry it — code limits its own work, never anybody else's — and budgets apply on top.

Days and months are UTC.

A workflow's run is a scope, with nothing said

A call made in a workflow's run is charged to the workflow and to that run — workflow:Onboarding and run.Onboarding:8f1c… — so what a process costs is counted as it happens:

SetBudget("workflow", "Onboarding", LlmBudgetWindow.Month, 200);       // every run of it together, a month
SetBudget(RunKind("Onboarding"), "", LlmBudgetWindow.Total, 2);        // each run of it, two dollars in all
var cost = SpentOn(RunScope("Onboarding", run));                       // what one run has cost

LlmCall.ChargedToItsRun = false is for code that names the run itself.

Testing

Put a ScriptedModel where a provider would go, and answer for it from the test:

// the test app's setup
app.Llm = new LlmSetup { Models = [
  new LlmModel { Name = "Fast", Client = new ScriptedModel { Name = "Fast" }, Model = "scripted-fast",
                 Price = new LlmPrice { Input = 1, Output = 5 } },
] };

[Test]
void a_note_is_summarised_and_the_call_is_on_the_meter() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Early beans, saved in 2024.", 1000, 200));
  Assert.Equal("Early beans, saved in 2024.", Summarise(note));
  Assert.Equal(2000, ModelCall.Single().CostMicros);          // 1000 tokens at $1 a million, 200 at $5
}

Scripted.Text, Scripted.ToolUse and Scripted.Decision build answers; throw an ExternalServiceException from the script to play a provider that is down; GenerationIsOn.Stub(() => false) plays the operator's switch. Routing, budgets and the meter are the real ones, so a test asserts what a call cost the same way your app reads it.

Writing a provider

public class MyClient : ILlmClient {
  public string ApiKey = "";
  internal LlmResponse Complete(LlmRequest request) { … }           // request.Model is already decided: serve it or throw
  internal stream<LlmStreamEvent> Stream(LlmRequest request) { … }  // end with MessageComplete, carrying the counts
}

Report the token counts the provider reported — the meter prices from them. Throw ExternalServiceException when the provider fails; that is what moves the call to the next model. When request.OutputSchemaJson is set, answer with one text block whose text is the JSON document.

The two methods are internal, as they are on ILlmClient: any package implements them, and only this package calls them — from the call path, which routes, reserves and meters around each call. Code that holds a client and writes client.Complete(request) does not compile; it calls Llm.Complete. So there is no model call that is not on the meter.

What only the kit writes

The meter, the price book and the budgets are rows in your app's database that only this package's functions write. Each of these is a test in the kit's test app:

A meter row is made only by the call path, and never changed a_meter_row_is_written_only_by_the_call_path_and_never_changed
A call made elsewhere is recorded only by whom the app names, never on the meter's own scopes a_call_made_somewhere_else_is_recorded_on_the_meter_as_it_was_measured_there
What a budget has counted is changed only by the call path what_a_budget_has_counted_is_changed_only_by_the_call_path
Budgets are set only by who manages the spend, through SetBudget budgets_are_set_only_by_who_manages_the_spend_and_only_through_the_kit
Prices are set only by who manages the spend, through SetModelPrice prices_are_set_only_by_who_manages_the_spend_and_only_through_the_kit
A call is refused before anything is sent when its budget has no room a_call_that_does_not_fit_its_budget_is_refused_before_anything_is_sent
Calls racing for one budget take exactly what fits a_budget_reserves_atomically_under_parallel_calls
A limit a call carries is held the same way, and binds only the calls that carry it a_limit_a_call_carries_is_reserved_against_before_the_call_leaves, a_limit_binds_only_the_calls_that_carry_it
What a limit has counted is changed only by the call path what_a_limit_has_counted_is_changed_only_by_the_call_path
A provider is called only by the kit's call path — an app or another package calling one directly does not compile AProviderIsCalledOnlyByTheLlmKitTests (the platform's own tests)
A call is on the meter though the work around it failed a_call_is_on_the_meter_though_the_work_it_was_made_for_failed
The meter is read only by who the app says a_call_is_recorded_for_who_it_ran_for_and_read_only_by_who_the_app_says
A person is taken off the meter only by who manages the spend, through ForgetModelCallRequester, and what their calls cost stays counted a_person_is_taken_off_the_meter_and_what_their_calls_cost_stays_counted, only_whoever_manages_spend_takes_a_person_off_the_meter_and_only_through_the_kit

Anyone who may call a model can read the budgets and what each has counted; the per-call detail is the meter's, and ReadsModelUsage decides who reads that — with whoever manages the spend, who deletes old rows and takes a person off them.

The meter keeps no text of a call, and it does say who the call was for. ForgetModelCallRequester(person) takes a person off every row and deletes what was counted against a budget of their own; the rows and their cost stay. With Osysharp.Privacy in the app, Osysharp.Privacy.Llm does it when the person asks and when the period ends.

Your organization's floor

Two settings of LlmSetup are your organization's to govern (2.12.0). What your code and the kit read is the organization's value, whatever app.Llm assigns:

  • AllowedModels (a lock): the provider models a call may be sent to, separated by commas or spaces — an exact id (claude-haiku-4-5) or a prefix ending in * (claude-haiku-*). Empty: any model. A candidate outside it is never sent, whatever your router ordered; the walk goes on to the next one, and a call none inside the list served is refused (PolicyRefused, key Osysharp.Llm.LlmSetup.AllowedModels).
  • MonthlyTokens (a range): the most tokens your calls may use in a calendar month (UTC) — prompt, answer and cache, as the provider reports them. Each call reserves its prompt (as the kit counts it from the request it sends) and the most its answer may run to (the call's MaxTokens, else the model's MaxOutputTokens, else 4096 — and that bound is what the provider is sent). A call whose reservation would take the month past the ceiling, counting the calls still on their way, is refused before anything is sent (PolicyRefused, key Osysharp.Llm.LlmSetup.MonthlyTokens). An answered call is then counted at the provider's own numbers; a call that sent nothing gives its reservation back; a stream you stop reading keeps it. The month's count is LlmOrgUsage: only the kit's reserve and settle write it, and nothing deletes it.

Each refusal is on the meter too, outcome Refused. Read osy policy before naming a model.

A model outside the list is refused, whatever the app assigns a_locked_model_list_refuses_a_model_outside_it_whatever_the_app_assigns
The app's router cannot route past the list the_apps_router_cannot_route_past_the_list
A streamed call and a decision are held the same way stream_and_decide_are_held_like_complete
The call that would pass the month's ceiling is refused the_token_ceiling_refuses_the_call_that_would_pass_it
The month's count cannot be deleted or lowered by the app the_app_cannot_delete_or_lower_the_months_count
An app's own value of 0 is held inside the organization's range an_app_value_of_zero_is_held_inside_the_range
A negative or understated output bound cannot pass the ceiling a_negative_or_understated_output_bound_cannot_pass_the_ceiling
Calls started together cannot pass the ceiling between them calls_started_together_cannot_pass_the_ceiling_between_them
A stream you stop reading is counted at its reservation an_abandoned_stream_is_counted_at_its_reservation

Its tests

kits/llm/tests is a seed bank that reads the notes growers send in with their seed: 51 tests over three scripted models and a small router of its own — the call path, the fallback walk, the price book, budgets under twelve simultaneous calls, typed extraction with its repair, and decisions from a decision model and from a chat model standing in.

osy test kits/llm/tests

Source and package

  • Package: Osysharp.Llm — use Osysharp.Llm@2;
  • Source: kits/llm — model/Contract.osy (providers), Routing.osy, Setup.osy, Call.osy (the call path), Streaming.osy, Structured.osy (extract, decide), Budgets.osy, Meter.osy (the meter and the price book), Llm.osy (the calls an app writes), Scripted.osy (the scripted model)
  • Uses nothing but the language: no host, no secret. The provider packages ask for theirs.

Examples

Taken from this package's own tests, which compile and run on every release.

using Osysharp.Llm;
using SeedBank;

[Test(Bank)]
void a_call_whose_provider_failed_gives_its_room_back() {
  ScriptedAnswer.Stub((model, request) => {
    if (Heard.Count(h => h.What == "fail") > 0) { throw new ExternalServiceException(model + " is down"); }
    return Scripted.Text("A line.", 1000, 200);
  });
  runas (Tove) { SetBudget("app", "", LlmBudgetWindow.Day, 0.2m); UnitOfWork.Commit(); }
  runas (Ines) {
    new Heard { What = "fail" };
    UnitOfWork.Commit();
    Assert.Throws<ExternalServiceException>(() => Summarise("beans"));
    // Two attempts, both failed: nothing spent, nothing still held.
    Assert.Equal(0, LlmSpend.Single().SpentMicros);
    Assert.Equal(0, LlmHold.Count());
  }
}

From tests/budgets.test.osy.

using Osysharp.Llm;
using SeedBank;

[Test(Bank)]
[runas(Ines)]
void a_call_is_served_by_the_routers_first_model_and_metered_at_its_price() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Early beans, saved in 2024.", 1000, 200));
  Assert.Equal("Early beans, saved in 2024.", Summarise("Early dwarf beans, saved from the 2024 crop, dry stored."));

  var row = ModelCall.Single();
  Assert.Equal("Small", row.Model);
  Assert.Equal("scripted-small", row.ProviderModel);
  Assert.Equal(ModelCallKind.Complete, row.Kind);
  Assert.Equal(ModelCallOutcome.Served, row.Outcome);
  Assert.Equal(1000, row.InputTokens);
  Assert.Equal(200, row.OutputTokens);
  // 1000 tokens at $1 a million and 200 at $5: 1000 + 1000 millionths of a dollar.
  Assert.Equal(2000, row.CostMicros);
  Assert.Equal("seed-bank:1", row.RouteRule);
  Assert.Equal(Ines.Id, row.Requester);
}

From tests/calls.test.osy.

using Osysharp.Llm;
using SeedBank;

[Test(Bank)]
void only_whoever_manages_spend_takes_a_person_off_the_meter_and_only_through_the_kit() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("A line.", 1000, 200));
  runas (Omar) { Summarise("beans"); }
  Background.Settle();
  var omar = Omar.Id;
  // A curator reads the meter and changes nothing on it; Omar himself does not either.
  runas (Ines) { Assert.Denied(() => ForgetModelCallRequester(omar)); }
  runas (Omar) { Assert.Denied(() => ForgetModelCallRequester(omar)); }
  runas (Tove) {
    // The treasurer may — through the kit, and no other way: a row rewritten by hand is refused.
    Assert.Denied(() => { RewriteWhoAsked(ModelCall.Single(c => c.Requester == omar)); UnitOfWork.Commit(); });
    UnitOfWork.Discard();
    Assert.Equal(1, ModelCall.Count(c => c.Requester == omar));
    // Positive control: the same treasurer, through the kit.
    Assert.Equal(1, ForgetModelCallRequester(omar));
    UnitOfWork.Commit();
    Assert.Equal(0, ModelCall.Count(c => c.Requester == omar));
  }
}

From tests/forget.test.osy.

Learn more

Licence

MIT — see LICENSE.

About

Osysharp.Llm — an Osy# package. calling a model: the provider and router contracts, the call path (route, reserve, call, fall back, meter), a price book and budgets with hard limits, typed extraction and typed decisions, and a scripted model for tests.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors