Skip to content

inference_api_details.md

Latest commit

 

History

History
259 lines (204 loc) · 7 KB

inference_api_details.md

File metadata and controls

259 lines (204 loc) · 7 KB
Raw

LLM Inference API: create_chat_completion

This document describes how to call the create_chat_completion endpoint implemented by extensions/business/edge_inference_api/llm_inference_api.py, with special focus on response_format for structured synthetic data generation.

Endpoint

  • Method: POST
  • Path: /create_chat_completion
  • Behavior: Synchronous alias for /predict. Internally it registers a request, dispatches to the loopback LLM engine, then blocks until the request completes or times out.

Notes:

  • The FastAPI layer is generated by naeural_core and returns wrapped responses by default (unless RESPONSE_FORMAT is set to RAW in the plugin config).
  • The plugin defaults to loopback local-only mode with tunneling disabled, which allows anonymous access. If tunneling is enabled or auth is required, you must pass a token.

Request Body

The request body is JSON with these fields (all are optional unless noted):

  • messages (required): list of { role, content } dicts.
    • role must be one of: system, user, assistant, tool.
    • content must be a non-empty string. Leading/trailing whitespace is trimmed.
  • temperature (default 0.7): float in [0.0, 1.5].
  • max_tokens (default 512): int in [16, 4096].
  • top_p (default 1.0): float in (0, 1].
  • repeat_penalty (default 1.0): float; forwarded to the LLM backend.
  • response_format (default null): structured output control (see below).
  • metadata (default {}): stored with the request; returned only by /request_status?return_full=true.
  • authorization (optional): bearer token string. You can pass either:
    • "Bearer <token>", or
    • "<token>" (no prefix).
  • extras (optional): a dict of extra params. If provided, it is merged into the request parameters via **kwargs (advanced/experimental).

Validation failures return an error payload and the HTTP layer raises a 500 by default (because the FastAPI template treats any error key as a server error).

response_format (focus)

response_format is validated and normalized in LLMInferenceApiPlugin.check_and_normalize_response_format. It accepts null, a dict, or a JSON string representing a dict.

Accepted forms

A) llama-cpp-python form (preferred by this API)

{
  "type": "json_object"
}

or

{
  "type": "json_object",
  "schema": { /* JSON Schema */ }
}

B) llama.cpp server alternative form (normalized to json_object)

{
  "type": "json_schema",
  "json_schema": {
    "schema": { /* JSON Schema */ }
  }
}

or

{
  "type": "json_schema",
  "schema": { /* JSON Schema */ }
}

Normalization behavior

  • If response_format is a string, it is parsed as JSON. Empty string is treated as null.
  • type is required and must be a string.
  • Supported types: json_object, json_schema.
  • For json_schema, the API rewrites it to:
{
  "type": "json_object",
  "schema": { /* extracted schema */ }
}

This is because llama-cpp uses json_object to drive grammar-based JSON constraints.

Schema rules

  • schema (or json_schema.schema) must be a JSON object (dict) and JSON-serializable.
  • If type is json_schema and no schema is provided, the request is rejected.

What this does in the backend

The response_format is passed through to llama_cpp.Llama.create_chat_completion(...). When type == "json_object", llama-cpp builds a JSON grammar and constrains output to valid JSON (and to your schema if provided). This is the key mechanism for synthetic data generation.

Example Requests

1) Basic chat completion

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a short greeting."}
  ]
}

2) JSON-only output (no schema)

{
  "messages": [
    {"role": "system", "content": "Return a JSON object only."},
    {"role": "user", "content": "Generate a simple profile."}
  ],
  "response_format": {"type": "json_object"}
}

3) JSON output with schema (recommended for synthetic data)

{
  "messages": [
    {"role": "system", "content": "Output must match the schema."},
    {"role": "user", "content": "Create a synthetic customer record."}
  ],
  "response_format": {
    "type": "json_object",
    "schema": {
      "type": "object",
      "properties": {
        "customer_id": {"type": "string"},
        "age": {"type": "integer"},
        "segment": {"type": "string"}
      },
      "required": ["customer_id", "age", "segment"],
      "additionalProperties": false
    }
  }
}

4) json_schema alternative form (auto-normalized)

{
  "messages": [
    {"role": "system", "content": "Return only schema-valid JSON."},
    {"role": "user", "content": "Generate a synthetic record."}
  ],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "schema": {
        "type": "object",
        "properties": {
          "id": {"type": "string"},
          "label": {"type": "string"}
        },
        "required": ["id", "label"],
        "additionalProperties": false
      }
    }
  }
}

Response Shape (sync)

Default (wrapped) response

By default, FastAPI responses are wrapped with extra node metadata:

{
  "result": {
    "REQUEST_ID": "<uuid>",
    "MODEL_NAME": "<model_name>",
    "TEXT_RESPONSE": ["..."],
    "FULL_OUTPUT": [ { /* raw llama_cpp response */ } ]
  },
  "server_node_addr": "...",
  "evm_network": "...",
  "ee_node_alias": "...",
  "ee_node_address": "...",
  "ee_node_eth_address": "...",
  "ee_node_network": "...",
  "ee_node_ver": "..."
}

Important notes:

  • TEXT_RESPONSE and FULL_OUTPUT are typically lists, even for a single request.
  • FULL_OUTPUT contains the raw llama-cpp response object(s), which includes the structured JSON text in choices[0].message.content.
  • metadata is not included in this result; it is available only via /request_status?request_id=<id>&return_full=true.

Raw response

If RESPONSE_FORMAT is set to RAW in config, the response body is just:

{
  "REQUEST_ID": "<uuid>",
  "MODEL_NAME": "<model_name>",
  "TEXT_RESPONSE": ["..."],
  "FULL_OUTPUT": [ { /* raw llama_cpp response */ } ]
}

Synthetic Data Guidance

  • Use response_format with a strict JSON schema and additionalProperties: false to prevent drift and keep outputs parseable.
  • Consider low temperature (e.g., 0.0 to 0.2) to reduce randomness in schema-constrained outputs.
  • Validate outputs downstream even with schema constraints, since generation can still fail in edge cases (timeouts, model errors, etc.).

Related Implementation Files

  • extensions/business/edge_inference_api/llm_inference_api.py
  • extensions/business/edge_inference_api/base_inference_api.py
  • extensions/serving/default_inference/nlp/llama_cpp_base.py
  • /usr/local/lib/python3.10/dist-packages/naeural_core/business/default/web_app/fast_api_web_app.py
  • /usr/local/lib/python3.10/dist-packages/llama_cpp/llama_chat_format.py