Agent Search System

A comprehensive Python-based search system for discovering, indexing, and searching agents with multiple backend engines.

Features

1. Agent Registry (agent_registry.py)

• Register and manage agent metadata
• Search agents by name, description, or tags
• Filter by capability
• Track performance and reliability metrics
• Usage tracking and statistics

2. Semantic Search Engine (semantic_search.py)

• TF-IDF vectorization for semantic understanding
• Cosine similarity matching
• Keyword extraction
• Document clustering
• Context-aware search results

3. Web Search Integration (web_search.py)

• DuckDuckGo search (privacy-friendly)
• Google Custom Search (with API key)
• GitHub repository search
• Stack Overflow question search
• Web page content fetching and parsing

4. Document Indexer (document_indexer.py)

• Index local documents (txt, md, pdf, json, code files)
• Full-text search across documents
• Document metadata extraction
• Batch indexing with recursion support
• Index statistics and management

5. Unified Search Engine (search_engine.py)

• Composite search across all backends
• Task-based agent recommendations
• Multi-format export (JSON, CSV)
• Search statistics and insights

Installation

pip install -r requirements.txt

Requirements:

• Python 3.8+
• requests
• numpy
• scikit-learn
• beautifulsoup4
• lxml

Quick Start

Basic Usage

from search_engine import AgentSearchSystem

# Initialize
search_system = AgentSearchSystem()

# Register an agent
agent = {
    'id': 'agent_001',
    'name': 'Data Analyzer',
    'description': 'Analyzes data and generates insights',
    'capabilities': ['data_processing', 'analysis'],
    'tags': ['analytics', 'data'],
    'version': '1.0.0',
    'author': 'DataTeam'
}
search_system.register_and_index_agent(agent)

# Search agents
results = search_system.search_agents("data analysis")

# Get recommendations for a task
recommendations = search_system.get_agent_recommendations(
    "I need to process and analyze customer data"
)

# Composite search (all sources)
results = search_system.composite_search(
    query="machine learning",
    include_agents=True,
    include_web=True,
    include_documents=True,
    document_dir="./docs"
)

Module Documentation

AgentRegistry

registry = AgentRegistry("agents.db")

# Register agent
registry.register_agent(agent_data)

# Search agents
results = registry.search_agents("query")

# Find by capability
agents = registry.get_agent_by_capability("data_processing")

# Update metrics
registry.update_agent_metrics(agent_id, performance=0.95, reliability=0.99)

# Get all agents
all_agents = registry.get_all_agents()

SemanticSearchEngine

semantic = SemanticSearchEngine()

# Index documents
semantic.index_documents(documents)

# Search with relevance scores
results = semantic.search(query, top_k=5)

# Get detailed results
detailed = semantic.search_with_details(query)

# Calculate similarity
score = semantic.similarity_score(text1, text2)

# Extract keywords
keywords = semantic.get_keywords(text, top_n=10)

# Cluster documents
clusters = semantic.cluster_documents(n_clusters=3)

WebSearchEngine

web = WebSearchEngine()

# DuckDuckGo search
results = web.search_duckduckgo("query")

# GitHub search
repos = web.search_github("machine learning", language="python")

# Stack Overflow search
questions = web.search_stackoverflow("python error handling")

# Fetch page content
content = web.fetch_page_content("https://example.com")

DocumentIndexer

indexer = DocumentIndexer("documents.db")

# Index directory
count = indexer.index_directory("./docs", recursive=True)

# Index single document
indexer.index_document("file.txt", tags=["important"])

# Search documents
results = indexer.search_documents("query", file_types=['.md', '.txt'])

# Get full document
doc = indexer.get_document(doc_id)

# Get statistics
stats = indexer.get_index_stats()

AgentSearchSystem

search = AgentSearchSystem()

# Search agents (registry, semantic, or capability)
results = search.search_agents("query", search_type="all")

# Web search
web_results = search.search_web("query", service="duckduckgo")

# Document search
doc_results = search.search_documents("query", directory="./docs")

# Get recommendations
recommendations = search.get_agent_recommendations(task_description)

# Composite search
results = search.composite_search(
    query="query",
    include_agents=True,
    include_web=True,
    include_documents=True
)

# Export results
json_export = search.export_search_results(results, format="json")
csv_export = search.export_search_results(results, format="csv")

Examples

Run the example script to see all features in action:

python example_usage.py

This demonstrates:

1. Agent registration
2. Registry-based search
3. Semantic search
4. Capability search
5. Task-based recommendations
6. Composite search
7. System statistics
8. Keyword extraction
9. Web search
10. Result export

Database Schema

Agents Table

• id: Unique agent identifier
• name: Agent name
• description: Agent description
• capabilities: JSON array of capabilities
• tags: JSON array of tags
• version: Agent version
• author: Creator/maintainer
• created_at: Registration timestamp
• updated_at: Last update timestamp

Documents Table

• id: Unique document identifier
• path: File path
• title: Document title
• content: Full text content
• file_type: File extension
• indexed_at: Index timestamp
• modified_at: File modification time
• file_size: File size in bytes
• tags: JSON array of tags

Advanced Features

Performance Optimization

• TF-IDF vectorization with configurable parameters
• Cosine similarity for fast distance calculation
• Indexed document search with SQL queries
• Batch indexing for multiple documents

Extensibility

• Pluggable search backends
• Customizable document parsing
• Configurable semantic parameters
• Support for multiple file formats

Search Quality

• Keyword extraction from documents
• Document clustering for categorization
• Relevance scoring with threshold
• Capability-based agent discovery

Configuration

Create a config.json file for custom settings:

{
  "semantic_search": {
    "max_features": 500,
    "min_df": 1,
    "max_df": 0.95,
    "ngram_range": [1, 2]
  },
  "web_search": {
    "timeout": 10,
    "max_retries": 3
  },
  "indexing": {
    "batch_size": 100,
    "supported_formats": [".txt", ".md", ".pdf", ".json"]
  }
}

Performance Metrics

• Semantic Search: O(n) with pre-indexed documents
• Registry Search: O(n) full-text search via SQL
• Web Search: API-dependent (typically 0.5-2 seconds)
• Document Indexing: O(n) where n = number of documents
• Capability Matching: O(n) with fuzzy matching

Security Considerations

• Web search uses privacy-respecting backends (DuckDuckGo)
• Document indexing respects file permissions
• Database queries use parameterized statements (SQL injection safe)
• No API keys stored in code (use environment variables)

Future Enhancements

• ML-based relevance ranking
• Distributed agent network discovery
• Real-time agent health monitoring
• Advanced NLP with transformers
• Caching layer for frequently searched items
• API endpoints (Flask/FastAPI)
• Web UI dashboard
• Rate limiting and throttling

License

MIT License

Support

For issues or feature requests, please create an issue in the repository.