feat: YAML/code ingestion pipeline + search_kubeflow_code MCP tool

[kmr-rohit]

Summary

• Add YAML/code ingestion KFP pipeline (code-pipeline.py) with YAML-aware chunking (splits at --- boundaries, extracts K8s metadata: kind, name, namespace) and Python AST-aware chunking (splits at function/class boundaries)
• Add search_kubeflow_code MCP tool with optional resource_kind filter to disambiguate similar resource types (e.g., Service vs ServiceAccount)
• Update Agent CRD with multi-tool strategy: call both search_kubeflow_docs and search_kubeflow_code when query spans docs and code, or when confidence is low

New files

• pipelines/code-pipeline.py — 3-step KFP pipeline: download → chunk+embed → store in Milvus code_rag collection
• pipelines/code_utils.py — YAML/Python/JSON parsing utilities with RecursiveCharacterTextSplitter fallback for oversized chunks

Modified files

• kagent-feast-mcp/mcp-server/server.py — Added search_kubeflow_code tool, extracted _search_collection() helper, added resource_kind filter support
• kagent-feast-mcp/manifests/kagent/setup.yaml — Updated Agent CRD with 3-tool routing and multi-tool strategy
• pipelines/requirements.txt — Added pyyaml

Milvus code_rag schema

Same as docs_rag plus: resource_kind (VARCHAR 128), resource_name (VARCHAR 256), resource_namespace (VARCHAR 256), file_type (VARCHAR 64)

Tested on OCI cluster

• Ingested kubeflow/manifests → applications/pipeline/upstream: 279 files → 477 chunks
• Semantic search verified: "metadata grpc service" → Service/metadata-grpc-service at 0.73 cosine similarity
• E2E tested through kagent UI with code, docs, and cross-tool queries

Pipeline bugs fixed during testing

• CRI-O requires fully qualified Docker image names (docker.io/...)
• Pinned sentence-transformers==3.3.1 + transformers==4.44.2 for PyTorch 2.3 compatibility
• Replaced langchain with langchain-text-splitters (module moved in newer versions)

Test plan

• Compile pipeline: cd pipelines && python code-pipeline.py — produces code_rag_pipeline.yaml
• Run pipeline on KFP — ingested 279 files from kubeflow/manifests → applications/pipeline/upstream
• Verify code_rag collection — 477 chunks with correct resource_kind, resource_name, resource_namespace metadata
• Test search_kubeflow_code with resource_kind filter — filtering by Service returns only Service resources, not ServiceAccounts
• Test agent routing — code queries route to search_kubeflow_code, docs queries route to search_kubeflow_docs
• Test multi-tool strategy — ambiguous queries (e.g., "How does Prometheus scrape pipeline metrics?") invoke both tools

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign franciscojavierarceo for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[SanthoshToorpu]

Hey @kmr-rohit what does this PR do exactly is it like a repo code searcher or something else?

[kmr-rohit]

Hey @kmr-rohit what does this PR do exactly is it like a repo code searcher or something else?

Hi @SanthoshToorpu , this pr adds initial draft of code/yaml file ingestion pipeline. It accepts repo base url to get all .yaml , .yml , .py , .json files ( Deployments, Service, ConfigMaps etc. ). Introduced a new mcp tool which can query for manifest file chunks with Resource kind filter. So agent can query for Manifiest files , vars used in pipelines , from ingested code chunks. This also introduces multi tool call in case of low confidence for better understanding.

So a query like : How does Prometheus scrape pipeline metrics? will probably invoke both code search tool and documentation search tool.

There are still some improvements i have on my list : Adding more code file type chunk support. More robust tool calling with react loop. Also this will help in building Developer Integeration support as mentioned here :
https://github.com/kubeflow/docs-agent/blob/main/gsoc2026_agentic_rag.md#6-direct-ide-integration-the-byo-agent-experience

[kmr-rohit]

Closing in favor of #205, which consolidates this PR along with #140 into a single clean PR.

#205 includes all the work from here (code pipeline, search_kubeflow_code tool) plus:

• Shared _search_collection helper
• Issues pipeline and search_github_issues with repo/state filters (from feat: add search_github_issues MCP tool and issues ingestion pipeline #140)
• Extracted code_utils.py for testable YAML/AST parsing
• Updated agent CRD with all 3 tools

Test coverage for all of this is in #206.