Daily rebase#3
Merged
Merged
Conversation
…830) Add FeedbackTargetInfo struct and three new trait methods to InferenceQueries: - get_feedback_target_info: retrieves function info by target_id (inference or episode) - get_chat_inference_tool_params: gets tool params for demonstration validation - get_json_inference_output_schema: gets output schema for demonstration validation This removes inline SQL queries from feedback/mod.rs in favor of trait methods, enabling future Postgres support for these operations. Remove serde alias from FunctionInfo.function_name
…#5692) * Add migration * Step 1 (partial): Add chat/json inference tables and InferenceCountQueries - Add migration 20260116194140_inference_tables.sql creating partitioned chat_inferences and json_inferences tables with indexes - Add migration 20260116195939_inference_default_partitions.sql creating default partitions for backfilling historical data - Implement InferenceCountQueries trait for PostgresConnectionInfo - Uses static SQL queries to satisfy sqlx 0.9 SqlSafeStr requirements - Feedback count methods stubbed until step-2 (feedback tables) - Add scripts/reset-dev-postgres.sh for dev database reset Address code review feedback for inference_count.rs - Log warning instead of silently falling back to epoch on RFC3339 parse error - Remove misleading test that tested ClickHouse table naming in Postgres module Refactor Postgres inference_count to use sqlx QueryBuilder - Replace format! SQL construction with QueryBuilder for type safety - Use push() for trusted SQL fragments, push_bind() for parameters - Remove local time_window_units helper, use TimeWindow::to_postgres_time_unit() - Extract reusable helper functions for count queries Use ENABLE_POSTGRES_READ flag to dispatch inference count queries - Update handlers to check ENABLE_POSTGRES_READ flag and use either Postgres or ClickHouse connection - Affected handlers: - get_inference_count_handler - get_inference_with_feedback_count_handler - get_function_throughput_by_variant_handler - list_functions_with_inference_count_handler - get_episode_inference_count_handler - Add tests verifying Postgres path returns error when disabled - Business logic functions are database-agnostic (take &impl InferenceCountQueries) Fix nested dollar quoting in pg_cron setup Use $cron_setup$ tag for outer DO block to avoid conflict with inner $$ used in cron.schedule command strings. Make pg_trgm extension and trigram indexes conditional - Only create pg_trgm extension if available (check pg_available_extensions) - Only create trigram indexes if pg_trgm is enabled - Graceful degradation: substring search falls back to sequential scans Add pg_cron partition management and inference retention config - Update tensorzero_drop_old_partitions to accept retention key parameter - Add pg_cron jobs to inference_tables migration (conditional on pg_cron availability): - Daily partition creation at 00:05 UTC - Daily old partition cleanup at 00:30 UTC (reads inference_retention_days) - Add inference_retention_days config option to PostgresConfig - Gateway writes retention config to tensorzero_retention_config table on startup - If retention not configured, partitions are retained indefinitely Add TODO for inference_retention_days documentation (#5764) Warn that enabling retention will immediately start dropping old partitions, and clients must manage their own backups. Add ErrorDetails::NotImplemented for stubbed Postgres features - Add NotImplemented variant with message field - Returns HTTP 501 Not Implemented status code - Replace PostgresConnection errors in inference_count.rs stubs - Reference TODO(#5691) for implementing feedback table queries Add load_fixtures_postgres.sh for loading small fixtures Loads chat_inferences and json_inferences from JSONL fixtures. Only small fixtures for now (Step 1 tables). Usage: ./ui/fixtures/load_fixtures_postgres.sh Env vars: TENSORZERO_POSTGRES_URL, TENSORZERO_SKIP_TRUNCATE Add load_fixtures_postgres.sh for loading small fixtures Loads chat_inferences and json_inferences from JSONL fixtures. Derives created_at from UUIDv7 timestamp (first 48 bits = ms since epoch). Usage: ./ui/fixtures/load_fixtures_postgres.sh Env vars: TENSORZERO_POSTGRES_URL, TENSORZERO_SKIP_TRUNCATE * e2e tests * PR comments created_at should not have a default since we compute it Test postgres functions Update fixtures * I don't think we should sleep
* Add timeouts (so we get retries) to pytest * Fix
* Improve download-large-fixtures.py * Improve download-large-fixtures.py * Retry ETag verification during R2 sync Co-authored-by: gabriel <gabriel@tensorzero.com> * Allow cursoragent in CLA workflow Co-authored-by: gabriel <gabriel@tensorzero.com> * Fix * Fix
* temporarily disable python client build job * remove the build-windows job too * fixed general job stuff
…err (#5878) * Remove spurious inference request in check_test_streaming_cache_with_err We were performing two streaming requests, and ignoring the first one. The first one could have populated the cache in the background, causing an unexpected cache hit for the real request * Fix lint
* filter * Retry CH checks * update system template * Use R2 sync in CI * Faster cleanup * Faster cleanup * Fix * made dev images use debug builds * Speed up CI * Make CH calls more resilient in CI * Bump image tests timeout This should fix some of the flakes we're seeing on CI (as some of these tests can take over 30 seconds, even when running locally) * Fix * disabled clickhouse cloud fast channel tests --------- Co-authored-by: Andrew Jesson <andrew@tensorzero.com> Co-authored-by: Gabriel Bianconi <1275491+GabrielBianconi@users.noreply.github.com> Co-authored-by: Aaron Hill <aaron@tensorzero.com>
* Namespace valkey keys to be tensorzero_... * Version functions * Graceful migration
* added a durable-tool for rejecting tool calls * cleanup * cleanup * consolidated the tool list to one place * made filtering consistent for all unavailable tools * set up types that enforce that the UI shouldnt see missing tool calls * added conversions between event and input message types * directly register internal publishing task * make autoreject a tasktool * fix bug in filtering logic * cleaned up PR comments * added a check to avoid duplicate rejections
* Use hyphens for workflow-evaluations UI route
Changes `/workflow_evaluations` to `/workflow-evaluations` in the UI to follow
web URL conventions (hyphens for user-facing URLs).
- Rename routes directory and update route definitions
- Update URL helper functions and tests
- Update sidebar and dashboard navigation links
- Update e2e test URLs and filenames
- Document URL conventions in ui/AGENTS.md
Backend API routes (e.g., `/internal/workflow_evaluations`) retain underscores
for consistency with gateway API conventions (like Stripe/OpenAI).
Closes #5381
* Fix import path for WorkflowEvalRunSelector
* Remove unnecessary Stripe/OpenAI reference from docs
* Address PR feedback: fix breadcrumb hrefs and move URL docs
- Fix breadcrumb hrefs to use hyphenated URL (/workflow-evaluations)
- Move URL conventions documentation from AGENTS.md to ui/app/ROUTES.md
for progressive discovery (sibling to routes.ts)
- Remove bold formatting from documentation per reviewer feedback
* Clarify that both RR7 and gateway API routes use underscores
* Move URL conventions to ui/app/routes/README.md
* Add React Router streaming to api-keys page Implements the Suspense/Await pattern for non-blocking page loads: - ApiKeysPageHeader - shared header component - ApiKeysContentSkeleton - skeleton fallback for loading state - ApiKeysErrorState - error state using useAsyncError() - ApiKeysContent - main content component - ApiKeysPage - orchestrates Suspense/Await pattern Naming follows the pattern established in #5651. * Move modal state outside Suspense boundary and add location.key
* bumped napi to v3 * also bumped node version * reverted changes to docker compose files
* add roundtrip tests for get datapoints / inferences for tool calls * add datapoint tool result validation * Rename test_type_roundtrip.py to test_roundtrip_tool_types.py * Add text content type round-trip tests following inference pattern * Add template content type round-trip tests * Add thought content type round-trip tests using dummy reasoner model * Add None assertions to fix pyright type errors - Add assertions for result.content, input_messages, message.content - Apply pattern across all 4 test files (tool, text, template, thought) - Reduces pyright errors from 81 to 40 * Add type narrowing to fix remaining pyright errors - Add isinstance checks for ChatInferenceResponse/JsonInferenceResponse before accessing inference_id - Add output is not None assertions for stored inferences - Import JsonInferenceResponse in template types - Move isinstance checks before inference_id access for proper type narrowing - Reduces pyright errors from 40 to 0 * Add sleep delays for batch writes in roundtrip tests When batch_writes is enabled, ClickHouse writes are async and not immediately available for retrieval. Add 1-second sleep delays after inference calls to wait for writes to complete before querying. Fixes CI failures where get_inferences returned empty results. * align sync tests * wait after follow-up * wrap all asserts in try
* Change rate limiting backend connection errors from 500 to 503 (#5736) Change PostgresConnection and ValkeyConnection errors to return HTTP 503 (Service Unavailable) instead of 500 (Internal Server Error). This is more semantically correct for transient backend failures, signals to clients that retrying might succeed, and distinguishes infrastructure issues from application bugs. * Change ClickHouseConnection error from 500 to 503 For consistency with PostgresConnection and ValkeyConnection, change ClickHouseConnection errors to return HTTP 503 (Service Unavailable) instead of 500 (Internal Server Error). This is more semantically correct for transient backend failures and signals to clients that retrying might succeed. --------- Co-authored-by: Claude <noreply@anthropic.com>
Move the clone datapoints SQL queries from the endpoint handler to the DatasetQueries trait. This centralizes the database logic and enables future Postgres support. The clone operation: 1. Generates new UUIDs for each source datapoint 2. Inserts cloned rows into Chat and Json datapoint tables in parallel 3. Verifies which clones succeeded (source might not exist) 4. Returns mapping of source IDs to new IDs (None if source missing)
* Run pre-commit check in CI * Fix
* Wrap inference ClickHouse write in a new 'write_inference' span This will let us track how clickhouse write times contribute to request latency * Adjust test to account for write_inference span
* Add streaming primitives for detail pages - BasicInfoLayoutSkeleton: reusable skeleton for BasicInfoLayout sections, accepts configurable rows prop (default 5) - SectionErrorContainer, SectionErrorNotice: containers for section-level error states - SectionAsyncErrorState: error component for React Router <Await> boundaries, extracts error message from async errors * Fix SectionAsyncErrorState to handle non-string error data * Add React Router streaming to dataset datapoint detail page Implements streaming pattern for instant navigation: - Loader returns promise for deferred loading - Suspense/Await wrapper for skeleton and error states - DatapointPageHeader shared across skeleton, error, and content - Uses BasicInfoLayoutSkeleton and SectionAsyncErrorState primitives
* Reorganize UI sidebar navigation - Rename "Dashboard" to "Overview" with LayoutGrid icon - Promote "Evaluations" to top-level category (Inference Evaluations, Workflow Evaluations) - Remove confusing "Workflows" category - Move API Keys and Docs to sidebar footer - Rebuild sidebar collapse button using SidebarMenuButton for consistent hover states - Add subheading prop to PageHeader component - Add subheading to API Keys page explaining their purpose - Rename "Documentation" to "Docs" in sidebar * Remove subheading prop from PageHeader * Update Dashboard references to Overview * Improve sidebar footer: narrower collapse button, show gateway status when collapsed * Add text fade transitions, fix accessibility and remove dead code * Improve sidebar transitions: height on labels, no truncate, fade on status text * Add fade transition to TensorZero logo text * Update Overview page structure to match sidebar * Add Autopilot tile to Overview page when available * Restructure Overview: Autopilot/Playground as top row, 4-column grid below * Restore 3-column grid for main sections * Finalize sidebar and Overview page reorganization - Move Playground to Tools section in sidebar - Update Overview page layout: 3-col grid for main sections, separate row for Data/Tools - Update tile hover styles: orange-200 border, orange-600 title - Clean up unused imports * Move external link arrow closer to Docs text * Polish Overview page footer links: smaller text, orange hover, rename Docs * Show version mismatch warning in collapsed sidebar tooltip * Restore version mismatch tooltip with explanation * Fix nested tooltips when collapsed with version mismatch * Fix version mismatch layout: show below status text, simplify tooltip * Reorganize sidebar and overview grid - Merge Data and Tools sections into Resources (Playground, Datasets, API Keys) - Remove Docs and API Keys from sidebar footer - Stack Evaluations + Optimization in column 2, Resources as column 3 - Add API Keys to Resources in overview grid - Align Optimization to bottom of column 2 - Use consistent gap spacing on mobile * Remove placeholder API Keys description * Add hover highlight to overview directory cards * Use card-highlight-icon-bg for hover border * Add card-highlight-border semantic token * Clean up unused numEvaluationRunsDesc and comments * Add orange-500 hover color to directory card subtitle
* Fix OpenAI error message format * Fix OpenAI error message format * Fix * Fix OpenAI auth format with base path Co-authored-by: gabriel <gabriel@tensorzero.com> * Fix tests * Fix * Fix --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The ui will automatically reconnect. This stops the gateway shutdown from hanging indefinitely when browser tabs are open
* add list datasets tool * generate python schemas * no defaults, optional * robust param parsing * comment
… to InferenceQueries (#5837) The get_output() function in human_feedback.rs used format!() with direct string interpolation for inference_id, episode_id, function_name, variant_name, and table_name. This is a potential SQL injection vulnerability. Added InferenceQueries::get_inference_output() trait method with a proper parameterized query implementation in ClickHouse. Added e2e tests: - test_get_inference_output_chat_inference - test_get_inference_output_json_inference - test_get_inference_output_not_found
* E2e tests for endpoints setup for postgres * Fix tests
This unblocks dependabot (it required a manual change)
* Use Docker in merge queue CI * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix
* Improve download-large-fixtures.py * Improve download-large-fixtures.py * Update small fixtures and organize downloads * Fix * Fix * Fix * Fix * Use R2 sync in CI * Fix * Add creds * Add creds * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix * Fix
These consistently take 4 or 5 retries to pass in ci, always due to timeouts
* Streaming Responses API include_encrypted_reasoning * Allow cursoragent in CLA workflow Co-authored-by: gabriel <gabriel@tensorzero.com> * Improve reasoning for streaming OpenAI Responses API * Fix * Fix * Fix * Fix * Fix --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.