# Editorial evidence review

Status: draft and evidence screenshots complete. Not published. Seven generated illustration anchors are reserved for the independent art worker; this review does not approve or claim delivery of those images.

## Scope and independence

Reviewed `research.md`, `RESULTS.md`, the frozen configuration and queries, `run_benchmark.py`, `analyze.py`, saved method rankings, raw HTTP responses and relevant corpus records. Wrote and executed a separate offline verifier rather than rerunning either the paid benchmark or its original analysis script. No API inference request, secret read, CMS write, production repository change, browser replacement, cookie copying or browser activation was performed by this editorial task.

The research worker's primary-source notes are a distinct evidence stream from the benchmark. Chat Seek source inspection is not a runtime test of the extension. Hosted Jev findings are not Laya findings.

## Independently validated findings

`verify_evidence.py` executed successfully and wrote `validation.json`.

- Exactly 288 rows and 288 unique `(scale, method, query_id)` keys.
- Exactly 144 saved successful HTTP 200 responses, all reporting `jev-1.13.0`.
- Reported input usage independently summed to 850,704 tokens. At $0.042 per million, the input-cost estimate is $0.035729568. It is not an invoice.
- Every frozen-input SHA-256 hash still matched the manifest.
- All 144 hosted-method rankings and abstention decisions reconstructed exactly from their saved raw responses; saved usage also matched.
- Independently recomputed Hit@1, recall@3, all-evidence@3, negative specificity, decision accuracy and total input tokens agreed with the original summary.
- At 240 documents, relevant-first-result counts are BM25 16/20, embeddings 17/20, hybrid 19/20 and direct 20/20. Complete labeled evidence in the top three occurs on 16/20, 20/20, 18/20 and 20/20 positive queries respectively.
- Direct Jev has 20/20 relevant first results and embeddings 17/20 at each scale. Both retain all labeled evidence in the top three for every positive query at each scale.
- At 240 documents, Q05 embeddings put storage-outage D023 above browser-policy D004; both Jev methods put D004 first.
- At 240 documents, Q07's relevant D006 is BM25 rank 45 and absent from the eight-candidate hybrid input. Hybrid abstains; embeddings and direct both rank D006 first and accept it.
- At both 96 and 240 documents, BM25 top-eight excludes D006 for Q07 and D010 for Q19. These are deterministic candidate-coverage failures, not evidence that Jev saw and misunderstood the missing documents.
- Median direct HTTP latency rises 172.41 → 202.07 → 297.41 ms across 24/96/240 documents. Direct input usage over 24 queries rises 52,685 → 206,573 → 510,893. Hybrid input usage is 26,350 → 27,188 → 27,015.

## Methodology review

The test provides a useful controlled diagnostic with a real semantic encoder and live hosted calls. It does not support a universal replacement claim.

1. **Small authored data.** Twenty positives and four negatives per scale are too few to establish production reliability. Labels were not independently adjudicated; borderline semantic support can be debatable.
2. **Nested conditions, not independent datasets.** The same 24 questions recur at all scales. Growth mostly adds templated distractors, not the variety of a real incident archive.
3. **Unequal work is intentional but must be named.** Hybrid independently scores eight candidates with Nouls in one request; direct uses one relative Choice over all documents plus an existence Noul. This is a pipeline comparison, not an isolated head-to-head of identical model tasks.
4. **Untuned rejection.** Cosine 0.5 is arbitrary. BM25 rejects only nonpositive top scores; hybrid rejects max Noul below 0.5; direct rejects NONE or existence below 0.5. Their negative-query results do not prove equivalent calibration. Four negatives make percentages fragile.
5. **Rank versus decision.** Positive ranking metrics ignore abstention. Decision accuracy incorporates rejection but can still count one correct record on a multi-record question. The article avoids presenting it as complete-answer accuracy.
6. **Relative distributions.** Direct Choice probability is not independent document relevance. Ranking the remainder of the distribution, including ID-tiebroken zero-probability items, is a limited multi-evidence diagnostic. No claim of answer sufficiency follows.
7. **Local baseline scope.** One local all-MiniLM-L6-v2 ONNX encoder with normalized vectors and exact cosine search is not the best achievable vector system. No ANN approximation is involved. Keyword tokenization is basic; no learned hybrid retrieval or fusion variant was tested.
8. **Timing scope.** Local embeddings include query encoding; document embedding is offline. Jev includes network roundtrip. No repeated trials, randomization of method execution order, load testing or confidence intervals were performed. Medians are descriptive observations, not an SLA.
9. **Generation absent.** No answers, answer faithfulness, citations, permission enforcement or end-to-end RAG outputs were evaluated. The article explicitly separates this missing stage.
10. **Synthetic data handling only.** Results do not establish privacy rights for real transcripts, zero retention or security assurance.

## Primary-source checks

Eight relevant primary endpoints were fetched again without authentication and saved under `source-checks/`; `fetch-log.json` records their success. Pinned Chat Seek search/extension code and pinned Laya ONNX README distinguish lexical retrieval plus local Laya from hosted Jev. Live TypeSafe model, API, limitations, confidence and RAG-passage cookbook pages support the bounded request contract and architecture discussion. The original RAG paper was available through the research worker's saved primary-source snapshot; it was not needed for any numerical benchmark claim.

`citations.json` is the task-specific numbered source ledger. Its verification command returned `citations OK`. This checks citation identity and source-list consistency, not semantic correctness by itself. External-source claims were manually reviewed against the research notes and retrieved primary evidence. Experiment findings have an explicit local evidence block rather than being attributed to vendor material.

The article deliberately omits unverified tweet wording, reach, broad market claims, vendor comparative benchmarks and privacy/legal guarantees. Model limits and pricing are dated to September 21, 2026.

## Screenshot provenance and visual review

- `screenshots/overview.png`: 2160 × 1464 pixels. Three corpus-size panels, actual rank/evidence metrics and measured median latencies. Explicit denominators, timing caveat and synthetic-data labeling.
- `screenshots/failure-case.png`: 2160 × 1571 pixels. Actual Q07 query, D006 corpus text, BM25 rank 45, hybrid abstention and successful alternative rankings. Explicitly marked illustrative rather than representative.
- Source pages: `overview.html` and `failure-case.html`. Generator: `build_evidence.py`. Both pages derive their records from saved experiment artifacts; no provider UI is imitated.
- Captured through the already-running Agent Chrome CDP endpoint at localhost:9222 using one background task-owned target. The target was closed in a finally block; `screenshots/provenance.json` records browser version, SHA-256 values and closure.
- Both screenshots were loaded and visually inspected. Text and footers are intact, there is no clipped panel, and the restrained cream/green layout separates evidence from interpretation. A small browser-extension overlay dot remains at the outer right edge; it does not cover data. It was not edited out of the evidence capture.

## Editorial acceptance checks

- Team attribution: Samelogic team.
- Title contains no colon; article contains no em dash.
- Exactly six main sections, in the requested sequence and with the requested names.
- Approximate article body length: 1,684 whitespace-token words before the evidence/source appendix, including headings, image alt text and table labels. This falls within the requested 1,400–1,900 range.
- Reserved illustration filenames: `hero.webp` and `section-01.webp` through `section-06.webp`.
- Two relative screenshot anchors: `screenshots/overview.png` and `screenshots/failure-case.png`.
- Clarity review: practical QA example first, four methods plainly introduced, denominators shown, limited use of metrics jargon, consequences paired with measurements.
- Evidence review: no inference from retrieval success to answer correctness, no conflation of local Laya with hosted Jev, no inference from this one encoder to all vector search.
- Practical next step: evaluate held-out approved data, fix candidate coverage, calibrate rejection separately, and test generation separately.

## Handoff and remaining work

The requested draft, independent review and two screenshots are complete. Do not publish this folder as a public download without packaging and reviewing its dependencies. Relative evidence links point into the parent experiment directory. The independent art worker must supply and review the seven image anchors before the parent creates a self-contained illustrated preview. This editorial task did not touch the art worker's outputs or claim its images were ready.

Publication, CMS conversion, a portable final illustrated preview and public URL checks remain explicitly out of scope.

## Headline alternatives considered

1. Can Jev replace RAG or just improve retrieval?
2. What Jev changes about finding evidence
3. We tested Jev against keyword and vector retrieval
4. Jev found the right record but RAG is not replaced
5. A small retrieval test with a bigger lesson
6. Where Jev fits between search and answers
7. Better selection does not make retrieval disappear
8. Why the shortlist still matters with Jev
9. What our Jev retrieval experiment actually showed
10. Finding the right evidence without declaring RAG dead