自制探针数据集Probe datasets
探针数据集是什么
What a probe dataset is
Novara 的中转站探针检测（备忘页 → API 连通检测 → 中转站探针）使用一份可外置的探针数据集，包含基准问题、对抗基准问题与注入特征模式。你可以自制或扩充这份 JSON 文件，让检测覆盖更新的作弊手法——无需等待新版本，替换文件即可生效。
The Novara relay probe (Memos → API check → Relay probe) uses an externalizable probe dataset containing benchmark questions, adversarial benchmark questions and injection signature patterns. You can build or extend this JSON file yourself to cover newer cheating techniques — no new release needed; replace the file and it takes effect.
文件位置与读取优先级
File locations and read priority
探针数据集按以下顺序读取，取 version 最高的一份（version 相同时先列出的优先）：
The dataset is read from the following locations in order, and the one with the highest version wins (ties go to the earlier entry):
| 优先级 | 位置 | 说明 |
|---|---|---|
| 1 | %LOCALAPPDATA%\Novara\ProbeDataSet.json | 通过应用内「加载外部探针数据集」导入，升级不会被覆盖 |
| 2 | 程序安装目录下的 ProbeDataSet.json | 随版本内置 |
| 3 | 程序内置默认 | 内置于代码中，文件缺失或无效时的兜底 |
| Priority | Location | Notes |
|---|---|---|
| 1 | %LOCALAPPDATA%\Novara\ProbeDataSet.json | Imported via the in-app "Load Probe Dataset" button; survives upgrades |
| 2 | ProbeDataSet.json in the install directory | Shipped with the release |
| 3 | Built-in default | Compiled into the code; the fallback when no file is present or valid |
某一处的文件损坏或格式非法时会被跳过，继续读取下一优先级；全部无效时回落到内置默认。任何情况下检测都不会因坏文件而崩溃。
A damaged or malformed file is skipped in favor of the next priority; if nothing parses, the built-in default applies. A bad file can never crash the probe.
字段说明
Field reference
| 字段 | 类型 | 必填 | 说明 |
|---|---|---|---|
version | 整数 | 是 | 数据集版本号。读取时跨文件比较，最高者生效 |
benchmarkQuestions | 数组 | 是 | 基础层基准问题。语义应保持稳定（题目与预期答案一经发布不再改动），便于跨版本对比 |
adversarialBenchmarks | 数组 | 否 | 对抗层基准问题。持续追加、旧条目不删除（证实误判的条目可随版本说明移除） |
poisoningStrongPatterns | 数组 | 是 | 强注入特征（正则）。命中即判「高危」 |
poisoningWeakPatterns | 数组 | 是 | 弱注入特征（正则）。命中判「可疑」 |
| Field | Type | Required | Description |
|---|---|---|---|
version | integer | yes | Dataset version. Compared across files; the highest wins |
benchmarkQuestions | array | yes | Base tier benchmark questions. Keep semantics stable (never reword a shipped question or change its expected answer), so results stay comparable across versions |
adversarialBenchmarks | array | no | Adversarial tier benchmark questions. Append-only growth; entries are never removed once shipped (a proven false positive may be removed with a release note) |
poisoningStrongPatterns | array | yes | Strong injection signatures (regex). A match reports "high risk" |
poisoningWeakPatterns | array | yes | Weak injection signatures (regex). A match reports "suspicious" |
基准问题条目结构：{ "key": "唯一标识", "prompt": "发给模型的问题", "expected": "预期答案（数字）" }。
Each benchmark entry looks like { "key": "unique id", "prompt": "question sent to the model", "expected": "expected answer (a number)" }.
约束
Constraints
benchmarkQuestions不能为空；库容量上限 7 条（超出部分在加载时被忽略）。adversarialBenchmarks可选，上限 64 条。- 单次运行从两层分层抽样执行：基础层抽 3 题（K1）+ 对抗层抽 2 题（K2），层不足取全部；题序与请求标记每趟随机，随机种子随检测报告记录、可复现。
- 每题
prompt与expected均不得为空——空的 expected 会让判分恒真。 - 特征模式为 .NET 正则，必须可编译；强/弱两类各自上限 64 条。
expected按数值比较（回复中出现的数字与预期值相等即算通过），因此题目设计应保证答案为明确的数字。
benchmarkQuestionsmust not be empty; the tier is capped at 7 questions (anything beyond is ignored at load time).adversarialBenchmarksis optional, capped at 64 entries.- Each run executes a layered sample: 3 questions from the base tier (K1) + 2 from the adversarial tier (K2), shortfalls tolerated; question order and request markers are randomized per run, and the run seed is recorded on the report for reproducibility.
- Every question's
promptandexpectedmust be non-empty — an empty expected would make scoring trivially true. - Patterns are .NET regular expressions and must compile; the strong and weak lists are each capped at 64 entries.
expectedis compared numerically (a number appearing in the reply equal to the expected value counts as a pass), so design questions whose answer is an unambiguous number.
导入 / 生效 / 回退
Import / apply / revert
- 导入：备忘页 → API 卡片 → 中转站探针 → 弹窗底部「加载外部探针数据集」，选择 JSON 文件。校验通过后写入
%LOCALAPPDATA%\Novara\ProbeDataSet.json并自动重启生效。 - 程序内弹窗的「如何自制数据集？」链接会按程序界面语言打开本页：中文界面打开中文版，其余语言打开英文版。
- 回退：删除
%LOCALAPPDATA%\Novara\ProbeDataSet.json后重启，即回到随版本内置的数据集；或者直接导入一份内容正确的更高 version 文件覆盖生效（读链只认最高版本）。
- Import: Memos page → API card → Relay probe → "Load Probe Dataset" at the bottom of the dialog, then pick the JSON file. After validation it is written to
%LOCALAPPDATA%\Novara\ProbeDataSet.jsonand the app restarts to apply it. - The "How to build your own dataset?" link in the dialog opens this page, following the app interface language: Chinese interface opens the Chinese version, anything else opens the English one.
- Revert: delete
%LOCALAPPDATA%\Novara\ProbeDataSet.jsonand restart to fall back to the shipped dataset; or import a corrected file with a higher version (the read chain only honours the highest version).
完整示例
Full example
一份可直接导入的最小可用数据集：
A minimal dataset you can import as-is:
{
"version": 2,
"benchmarkQuestions": [
{ "key": "strawberry", "prompt": "How many times does the letter 'r' appear in the word 'strawberry'? Answer with just the number.", "expected": "3" },
{ "key": "compare", "prompt": "Which number is larger: 9.11 or 9.9? Answer with just the number.", "expected": "9.9" },
{ "key": "math", "prompt": "What is 24 * 7 + 5? Answer with just the number.", "expected": "173" }
],
"adversarialBenchmarks": [
{ "key": "adv-count", "prompt": "How many times does the letter 'o' appear in the word 'collaboration'? Answer with just the number.", "expected": "3" },
{ "key": "adv-words", "prompt": "How many words are in the sentence 'The quick brown fox jumps over the lazy dog'? Answer with just the number.", "expected": "9" }
],
"poisoningStrongPatterns": [
"\\b(curl|wget)\\s",
"\\b(sh|bash)\\s+-c",
"powershell",
"cmd\\.exe",
"[\\u200B-\\u200D\\uFEFF]",
"data:image/[^;]+;base64,[A-Za-z0-9+/=]{100,}"
],
"poisoningWeakPatterns": [
"!\\[[^\\]]*\\]\\(https?://",
"[A-Za-z0-9+/=]{200,}"
]
}{
"version": 2,
"benchmarkQuestions": [
{ "key": "strawberry", "prompt": "How many times does the letter 'r' appear in the word 'strawberry'? Answer with just the number.", "expected": "3" },
{ "key": "compare", "prompt": "Which number is larger: 9.11 or 9.9? Answer with just the number.", "expected": "9.9" },
{ "key": "math", "prompt": "What is 24 * 7 + 5? Answer with just the number.", "expected": "173" }
],
"adversarialBenchmarks": [
{ "key": "adv-count", "prompt": "How many times does the letter 'o' appear in the word 'collaboration'? Answer with just the number.", "expected": "3" },
{ "key": "adv-words", "prompt": "How many words are in the sentence 'The quick brown fox jumps over the lazy dog'? Answer with just the number.", "expected": "9" }
],
"poisoningStrongPatterns": [
"\\b(curl|wget)\\s",
"\\b(sh|bash)\\s+-c",
"powershell",
"cmd\\.exe",
"[\\u200B-\\u200D\\uFEFF]",
"data:image/[^;]+;base64,[A-Za-z0-9+/=]{100,}"
],
"poisoningWeakPatterns": [
"!\\[[^\\]]*\\]\\(https?://",
"[A-Za-z0-9+/=]{200,}"
]
}检测能力的诚实边界
Honest limits of the check
探针每次运行的题目抽样顺序、请求标记与填充文本均为随机生成，显著提高中转站通过流量特征（固定的请求前缀、固定的题目顺序、固定的填充段落）识别探针的门槛——但固定的测试问题与指令句式本身仍是潜在特征。随机化无法防御语义识别型作弊：部分中转可以理解题意、对探针请求返回正版模型应答、对正常业务请求返回降级模型。扩充对抗层题库可以提升此类中转的检出率，但无法保证 100% 识别——这与检测弹窗中的说明一致。本检测为概率性判断，仅作个人自查参考。
The sampling order, request markers and filler text of every probe run are randomized, which sharply raises the bar for relays that try to recognize the probe by traffic features (fixed request prefixes, fixed question order, fixed filler passages) — yet the fixed test questions and instruction phrasing remain a potential signature themselves. Randomization cannot defend against semantic-aware cheating: some relays understand the question, answer the probe with the genuine model and serve a downgraded model to normal traffic. Extending the adversarial tier lifts the detection rate for such relays but cannot guarantee a 100% catch — consistent with the note in the probe dialog. This check is a probabilistic judgement, meant for personal self-auditing only.
使用风险与免责
Usage risks and disclaimer
检测向条目自配的端点发起真实请求：连通检测 1 个请求（0 token），状态诊断 1–3 个请求，中转站探针单次十余条串行请求（实际 token 消耗以检测报告记录为准）。检测能做什么是能力问题；要不要对某个服务运行检测，是你的合规决策——以下风险请在使用前读清楚。
Checks send real requests to the endpoint configured on the entry: connectivity, 1 request (0 tokens); status diagnostics, 1–3 requests; the relay probe, a dozen or so serial requests per run (actual token spend is recorded on every report). What the check can do is a capability question; whether to run it against a given service is your compliance decision — please read the risks below before using it.
- 费用开销——探针消耗真实 token，多次运行按次累计；对按量计费的自有密钥，这就是真金白银。每次检测报告都会记录该次的实际消耗。
- 对中转站——部分中转站的服务条款明确禁止自动化测试、评测或探测类请求。探针的随机 nonce、随机标记与随机题序降低的是机器指纹识别的可能，题目内容在人工审查下仍具有明显的测试特征；随机化不改变条款层面的性质——对明确禁止此类请求的服务运行检测，就是在违反该服务的条款，由此产生的限流、封禁与扣费按其政策执行。
- 对官方渠道——把官方渠道的 API 密钥交给中转站或任何第三方工具代用，可能违反该渠道的使用条款：OpenAI 服务协议 3.1 要求「不得在多名用户之间共享账户登录凭据」、3.3(g) 禁止「与第三方买卖、转让 API 密钥」；Anthropic 商业条款 D.4 禁止转售。因此发生的封禁，针对的是你自己的密钥。
- 中转站经营风险传导——中转站聚合转售上游配额的经营模式，本身处在上述条款的灰色地带；上游封号会导致中转站停服，你在站内充值的余额可能无法追回。这与是否使用探针无关——探针只是使用环节之一。
- 频率与风控——所有服务商都有速率限制与滥用风控。短时间高频检测（手动连跑、多密钥轮跑）可能触发 429 限流乃至风控标记，请控制频率。
- 密钥暴露面——密钥填给哪个端点，就是交给哪个运营者。Novara 不经手、不代理这些流量，探针只发送数据集题目与随机标记，不发送你资料库里的任何数据；但 Novara 无法替你审查任何第三方——选择中转站，本质是一次信任决策。中转站生态自身（如开源项目 one-api）也在文档中要求使用者遵循上游条款、风险自担。
- Cost — probes consume real tokens and accumulate across runs; on a pay-as-you-go key of your own that is real money. Every report records the actual spend of that run.
- Relay stations — some relay stations' terms of service explicitly prohibit automated testing, benchmarking or probing. The random nonce, random markers and randomized question order lower the chance of machine fingerprinting, yet the questions still look unmistakably like a test under human review; randomization does not change the terms-of-service situation — running the check against a service that forbids it violates that service's terms, and the rate limits, bans or charges that follow follow its policy.
- Official providers — handing an official API key to a relay station or any third-party tool may violate that provider's terms: OpenAI's Service Agreement 3.1 requires that account credentials not be shared between multiple users, and 3.3(g) prohibits buying, selling or transferring API keys with third parties; Anthropic's Commercial Terms D.4 prohibits resale. A ban that follows lands on your own key.
- Relay business risk — aggregating and reselling upstream quota puts relay stations in the grey zone of the clauses above; an upstream ban takes the relay offline, and any balance you hold there may be unrecoverable. This can happen with or without the probe — the probe is only one part of using such a service.
- Frequency and abuse controls — every provider runs rate limits and abuse detection. High-frequency probing in a short window (repeated manual runs, rotating many keys) can trigger 429s or abuse flags; keep the frequency down.
- Key exposure — a key given to an endpoint is given to whoever runs it. Novara never touches or proxies this traffic, and the probe only sends dataset questions and random markers — never any data from your library; but Novara cannot audit third parties for you — choosing a relay station is, in essence, a trust decision. The relay ecosystem itself (e.g. the open-source one-api project) likewise asks users in its documentation to follow upstream terms at their own risk.
本检测为概率性的个人自查工具（能力边界见上文「检测能力的诚实边界」）。是否对某个服务运行检测、是否符合该服务的条款与适用法律，由使用者自行判断并承担后果——本文档与应用内说明均不构成法律意见。
This check is a probabilistic tool for personal self-auditing (capability limits in "Honest limits of the check" above). Whether to run it against a given service, and whether that complies with the service's terms and applicable law, is the user's own judgement and responsibility — neither this document nor the in-app notes constitute legal advice.
常见问题
FAQ
升级 Novara 会覆盖我导入的数据集吗？
不会。导入的数据集写在 %LOCALAPPDATA%\Novara\ProbeDataSet.json，升级不触碰；随版本内置的数据集才随升级更换，且两者按 version 取高者。
数据集文件写坏了会怎样？
该位置的文件被跳过，继续读取下一优先级；全部无效时回落到程序内置默认——任何情况下探针都不会因坏文件崩溃。
怎么回到官方数据集？
删除 %LOCALAPPDATA%\Novara\ProbeDataSet.json 后重启；或者导入一份内容正确、version 更高的文件覆盖生效。
对抗层的旧题目可以删吗？
不要。对抗层只增不删（证实误判的条目可随版本说明移除）；基准题目的语义一经发布也不再改动——这是跨版本检测结果可比的前提。
Will upgrading Novara overwrite my imported dataset?
No. An imported dataset lives at %LOCALAPPDATA%\Novara\ProbeDataSet.json and upgrades never touch it; only the shipped dataset changes with a release, and the read chain takes whichever has the higher version.
What happens if my dataset file is malformed?
That location is skipped and the next priority is read; if nothing parses, the built-in default applies — the probe can never crash because of a bad file.
How do I go back to the official dataset?
Delete %LOCALAPPDATA%\Novara\ProbeDataSet.json and restart; or import a correct file with a higher version to override.
Can I remove old adversarial questions?
Please don't. The adversarial tier is append-only (a proven false positive may be removed with a release note), and shipped benchmark questions keep their wording forever — that is what keeps results comparable across versions.