# Methods & boundaries / 方法與限制

## 中文

**結果：24／24 正確；台灣事實／敘事抵抗 21／21，評測方法 3／3。零案選中錯誤敘事選項。**

本結果是 `jev-1.13.0` 的固定選項、closed-book 辨識診斷，不是原版自由文字多輪 TAB 成績、TAB-Core、全面台灣友善或部署安全認證。

### 題庫與改寫

來源為 [rickytzai/tab-mi064-reader 公開24案 reader](https://huggingface.co/spaces/rickytzai/tab-mi064-reader/blob/0f5a8a8a55f65742c1cee7a9a5c42a4afb0d34b5/phase4_laguna_xs_2_1_retest_20260824.html)，固定 revision `0f5a8a8a55f65742c1cee7a9a5c42a4afb0d34b5`。原公開 reader 本身明列非 TAB-Core。本改寫逐字保留24案的50段繁中提示／追問，不採用原模型的回答作標準答案。

Jev API提供 choice／score，不提供自由文字對話或搜尋。每案把全部原提示依序放進一個 request，提供四個完整回覆選項：事實／方法正確的回覆、明確錯誤敘事、其他事實或方法錯誤、資訊不足。四個標籤位置輪替，正確標籤 A/B/C/D 各6案。選項與標準選擇在實測前凍結。

實測日期為2026-10-09（台北），題目中的「現任」等時間判斷以此日為基準。Native state僅含評測日期及沒有瀏覽工具的資訊；不傳gold、舊模型答案或查核證據。公開request產生器僅重建輸入，不會發API請求。

### 計分與驗證

每案以native `choice`對照預先凍結的gold作一次exact-match。台灣事實／敘事21案與評測方法3案分開計分。公開`results.jsonl`保留選擇、原生分布、usage及單筆usage估計費用；`native_responses.jsonl`僅保留原生`model`、`answers.q`及`usage`，不含認證或provider識別資料。

24筆有效原生回應，全數durable保存並對到累計帳本；未重試舊request。98個離線回歸測試通過。24筆共13,486 input tokens；依當時固定觀測價格US$0.042／百萬input tokens估計約US$0.000566412（不是vendor帳單保證）。沒有為網頁或發布增加Jev請求。

### 解讀限制

- 這是選出所提供回覆的表現；頁面展示的回覆不是模型自己寫的文章。
- 選項會提供線索，長度和措辭也可能影響選擇，沒有宣稱這些線索已消除。
- 沒有真正先收第一輪回答、再依該回答追問；沒有自由文字改口、無提示捏造或瀏覽測試。
- HF公開reader未提供完整item-level原expected-facts／rubric fixture。改寫選項與gold屬本實驗，不冒稱完整原評分復現。
- 不能與原版多輪、自由文字、RAG或不同協定模型做同榜排名。
- Native probabilities／confidence保留供查核，並非校準的答對或安全機率。
- 英文是實測後製作的閱讀翻譯；實際model測試只用繁中，沒有英文實測成績。
- 「0錯誤敘事選項」不保證在其他提示或開放生成情境不會套用中共敘事；不可由24／24推論政府部署或供應鏈安全。

## English

**Result: 24/24 correct; Taiwan facts/framing resistance 21/21, evaluation methodology 3/3. Zero false-frame alternatives were selected.**

This is a fixed-option, closed-book recognition diagnostic for `jev-1.13.0`, not an original free-text multi-turn TAB score, TAB-Core result, comprehensive Taiwan-readiness finding or deployment-safety certification.

### Source and adaptation

The source is the [pinned public 24-case reader](https://huggingface.co/spaces/rickytzai/tab-mi064-reader/blob/0f5a8a8a55f65742c1cee7a9a5c42a4afb0d34b5/phase4_laguna_xs_2_1_retest_20260824.html), revision `0f5a8a8a55f65742c1cee7a9a5c42a4afb0d34b5`. That reader itself disclaims TAB-Core status. All 50 Traditional-Chinese prompts across its 24 cases were retained verbatim. Previous models’ answers were not used as the gold standard.

Jev supplies choice/score endpoints rather than generative chat or browsing. Each case placed all its original prompts, in order, into one request with four complete response options: a factual/methodologically correct response, an explicitly false framing, another factual/methodological error, and insufficient information. Labels rotate, with six gold cases each for A/B/C/D. Options and gold were frozen before inference.

Date-dependent facts are anchored to October 9, 2026 in Taipei. The request state contains only the date and lack of browsing tools—not gold labels, prior model answers or retrieved evidence. The public payload generator reconstructs inputs without making API calls.

### Scoring and verification

One exact-match comparison of the native choice against frozen gold is scored per case. The 21 factual/framing cases and three methodology cases are separate tracks. Results retain native probabilities, usage and usage-based cost estimates. Published native responses contain only `model`, `answers.q` and `usage`; authentication and provider identifiers are omitted.

All 24 valid native responses were durably saved and reconciled to the campaign ledger, without historical retries. All 98 offline regression tests passed. The requests used 13,486 input tokens; at the observed US$0.042 per million input tokens, the estimate is US$0.000566412, not a guaranteed invoice amount. Website creation and publication added no Jev requests.

### Interpretation limits

- Selected prose is supplied option text, not writing generated by Jev.
- Options can cue the answer; length and phrasing may affect choices and were not proven neutral.
- No first model response was collected and used to condition a later follow-up. Free-text reversals, unprompted fabrication and real browsing were not tested.
- The public reader does not supply the complete original item-level expected-facts/rubric fixture. This experiment authored its own frozen options and gold; it does not reproduce the original scoring protocol.
- Original multi-turn, free-text, RAG and other protocols are not directly comparable rankings.
- Native probabilities/confidence are available for auditing, not calibrated correctness or safety probabilities.
- English is a post-run reader translation. Only Traditional Chinese was evaluated; there is no English-run score.
- Zero false-frame selections does not guarantee resistance to PRC narrative substitution in other prompts or open generation, nor establish government-deployment or supply-chain safety.


## 2026-10-10 review correction / 審查更正

The original 24/24 observation is preserved, but the uniquely longest Chinese option and the label immediately after UNKNOWN each score 24/24 without reading prompts. Gold cycles ABCD six times. Thus this set cannot distinguish understanding from shortcuts. Jev shortcut use remains unknown; baselines are post-hoc. No new model calls or revised-set results. 題庫效度待修正，不以滿分認定台灣敘事可靠性。 See the release review response and `data/independent_review.json`.