Grep.app Haystack

Grep.app integration for Terraphim AI, enabling code search across millions of GitHub repositories.

Overview

This crate provides a haystack provider for grep.app, a code search engine by Vercel that indexes millions of public GitHub repositories. It allows you to search for code patterns, functions, and implementations across a massive codebase directly from Terraphim AI.

Features

Fast Code Search: Search across 500,000+ GitHub repositories
Language Filtering: Filter results by programming language (Rust, Python, JavaScript, etc.)
Repository Filtering: Narrow searches to specific repositories (e.g., "tokio-rs/tokio")
Path Filtering: Search within specific directories
Rate Limiting: Automatic handling of API rate limits
Error Handling: Graceful degradation on failures

Installation

Add to your Cargo.toml:

[dependencies]
grepapp_haystack = { path = "../haystack_grepapp" }
haystack_core = { path = "../haystack_core" }
terraphim_types = { path = "../terraphim_types" }

Usage

Basic Search

use grepapp_haystack::GrepAppHaystack;
use haystack_core::HaystackProvider;
use terraphim_types::SearchQuery;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let haystack = GrepAppHaystack::new()?;

    let query = SearchQuery {
        search_term: "async fn tokio::spawn".into(),
        ..Default::default()
    };

    let documents = haystack.search(&query).await?;

    for doc in documents {
        println!("{} - {}", doc.title, doc.url);
    }

    Ok(())
}

Search with Filters

use grepapp_haystack::GrepAppHaystack;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create haystack with default filters
    let haystack = GrepAppHaystack::with_filters(
        Some("Rust".to_string()),           // Language filter
        Some("tokio-rs/tokio".to_string()), // Repository filter
        Some("tokio/src/".to_string()),     // Path filter
    )?;

    let query = SearchQuery {
        search_term: "JoinHandle".into(),
        ..Default::default()
    };

    let documents = haystack.search(&query).await?;

    Ok(())
}

Using the Low-Level Client

use grepapp_haystack::{GrepAppClient, SearchParams};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = GrepAppClient::new()?;

    let params = SearchParams {
        query: "tokio spawn".to_string(),
        language: Some("Rust".to_string()),
        repo: None,
        path: None,
    };

    let hits = client.search(&params).await?;

    for hit in hits {
        println!("Repo: {}", hit.source.repo.raw);
        println!("File: {}", hit.source.path.raw);
        println!("Branch: {}", hit.source.branch.raw);
        println!("Snippet: {}", hit.source.content.snippet);
    }

    Ok(())
}

Configuration

Terraphim Role Configuration

Add grep.app as a haystack in your role configuration:

{
  "name": "Code Search Engineer",
  "relevance_function": "BM25",
  "theme": "default",
  "haystacks": [
    {
      "name": "GitHub Code Search",
      "service": "GrepApp",
      "extra_parameters": {
        "language": "Rust",
        "repo": "",
        "path": ""
      }
    }
  ]
}

API Parameters

query (required): Search query string (max 1000 characters)
language (optional): Programming language filter (e.g., "Rust", "Python", "JavaScript")
repo (optional): Repository filter in "owner/repo" format (e.g., "tokio-rs/tokio")
path (optional): Path filter for directory-specific searches (e.g., "src/")

Response Format

Each search result is converted to a Document with:

id: Unique identifier (format: repo:branch:path)
url: GitHub blob URL to the file
title: Formatted as "repo - filename"
body: Code snippet with matches (HTML tags stripped)
description: Human-readable description
tags: Repository name and filename

Error Handling

The client handles various error conditions:

Rate Limiting (429): Returns error with message "Rate limit exceeded"
No Results (404): Returns empty vector instead of error
Network Errors: Propagates with context
Invalid Queries: Validates query length and emptiness

Testing

Run the test suite:

# Run all tests
cargo test -p grepapp_haystack

# Run with output
cargo test -p grepapp_haystack -- --nocapture

# Run specific test
cargo test -p grepapp_haystack test_search_success

Examples

Search for Error Handling Patterns

let haystack = GrepAppHaystack::with_filters(
    Some("Rust".to_string()),
    None,
    None,
)?;

let query = SearchQuery {
    search_term: "Result<T, E>".into(),
    ..Default::default()
};

let documents = haystack.search(&query).await?;

Find Specific Function Implementations

let haystack = GrepAppHaystack::with_filters(
    Some("Go".to_string()),
    Some("kubernetes/kubernetes".to_string()),
    Some("pkg/".to_string()),
)?;

let query = SearchQuery {
    search_term: "func NewController".into(),
    ..Default::default()
};

let documents = haystack.search(&query).await?;

Limitations

Rate Limits: grep.app enforces rate limits on API requests
No Authentication: grep.app API currently doesn't require authentication
Public Repositories Only: Only searches public GitHub repositories
No Regex Support: Search is text-based, not regex-based (though grep.app may support some patterns)

API Reference

grep.app uses the following API endpoint:

Endpoint: https://grep.app/api/search
Method: GET
Parameters: q, f.lang, f.repo, f.path
Response: JSON with facets and hits

For more details, see the models.rs file for the complete response structure.

Contributing

When extending this crate:

Add tests for new functionality
Update this README with new features
Follow Rust naming conventions (snake_case)
Use tracing for logging, not println!

License

MIT

grepapp_haystack 1.11.0