Paired A/B evaluation infrastructure for DeepSeek Harness (dsh) components: interleaved repeated runs through the real runtime, verifier self-checks, one-variable arms, regression gating, cache- and calendar-aware cost, CLI + web UI
Install verification: Ecosystem plugin: installs via dsh plugin add but declares no dsh.bundle and mounts no plugin capability — the L1-L5 install ladder does not rate it; the security scan still runs.Install verification details →
What it doesAI
Paired A/B evaluation infrastructure for DeepSeek Harness components with verifier self-checks and regression gating.
- Interleaved repeated runs through real runtime
- Verifier self-checks and one-variable arms
- CLI and web UI with cache-aware cost
Capability tags are AI-inferred or rule-extracted from the README (see badge) — vocabulary-limited, not install-verified. AI = inferred; README = literal keyword from the readme.
Installation
dsh plugin --profile web add github:TT-Wang/dsh-eval-infrasecurity scan passed · default branch source
Install method: GitHub · runtime test inconclusive (L5 unknown)
Verification & Compatibility○ Ecosystem plugin
Grouped by plugin version: each card lists the dsh versions actually tested against that artifact version and their install verdicts (L5 wins). Untested combinations are omitted — absence means untested, never assumed compatible. L1–L3 are a plugin-level static archive (independent of the dsh version — the same verdict on every dsh row): they sit in each row's level grid next to L4/L5, marked "static", while their evidence is stored once per plugin inside that row's "checks & evidence" instead of being duplicated per version.
External credentials (plugin-level): none detected
Based on 2 requirement(s) across 3 @deepseek-ai/dsh-* packages. raw ranges
^0.0.1-rc.1 || ^0.1.0-rc.1 ×2 ^0.1.0-rc.1 || ^0.1.2-rc.1 ×1
✗ Running dsh 0.2.1-rc.2 is outside the declared range — the official install gate rejects it (incompatible-version).
To force-install, grant an exact-version exemption: dsh plugin allow-version dsh-eval-infra@0.1.0 0.2.1-rc.2 (the exemption lives in the local profile; this site does not assume one)
Whether it actually runs is a separate question → see the measured matrix below (L1–L5).
| plugin ＼ dsh | dsh 0.2.1-rc.2 current | dsh 0.2.1-alpha.4 | dsh 0.2.1-alpha.2 | dsh 0.2.0-rc.2 | dsh 0.1.7-rc.2 |
|---|---|---|---|---|---|
| b5b639 | · | · | · | · | · |
| 936f39 | ○ | ○ | ○ | ○ | ○ |
b5b639local2026-09-05
Ecosystem plugin: installs via dsh plugin add but declares no dsh.bundle and mounts no plugin capability — its level reads L5 · Ecosystem plugin.
936f39github○ 13 ecosystem plugin2026-10-18
https://github.com/TT-Wang/dsh-eval-infra
○dsh 0.2.1-rc.2current2026-10-18Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.2.1-rc.2
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.2.1-rc.2 · profile: l4-sandbox
Issued: 2026-10-18 · expires: 2026-10-25
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.2.1-alpha.42026-10-15Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.2.1-alpha.4
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.2.1-alpha.4 · profile: l4-sandbox
Issued: 2026-10-15 · expires: 2026-10-22
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.2.1-alpha.22026-10-11Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.2.1-alpha.2
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.2.1-alpha.2 · profile: l4-sandbox
Issued: 2026-10-11 · expires: 2026-10-18
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.2.0-rc.22026-10-09Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.2.0-rc.2
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.2.0-rc.2 · profile: l4-sandbox
Issued: 2026-10-09 · expires: 2026-10-16
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.1.7-rc.22026-09-25Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.1.7-rc.2
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.1.7-rc.2 · profile: l4-sandbox
Issued: 2026-09-25 · expires: 2026-10-02
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.1.7-rc.12026-09-23Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.1.7-rc.1
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.1.7-rc.1 · profile: l4-sandbox
Issued: 2026-09-23 · expires: 2026-09-30
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
This level was not executed in this scan.
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.1.7-alpha.22026-09-23Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.1.7-alpha.2
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.1.7-alpha.2 · profile: l4-sandbox
Issued: 2026-09-23 · expires: 2026-09-30
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.1.6-alpha.22026-09-19Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.1.6-alpha.2
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.1.6-alpha.2 · profile: l4-sandbox
Issued: 2026-09-19 · expires: 2026-09-26
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.1.6-alpha.12026-09-17Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.1.6-alpha.1
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.1.6-alpha.1 · profile: l4-sandbox
Issued: 2026-09-17 · expires: 2026-09-24
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.1.5-rc.22026-09-11Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.1.5-rc.2
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.1.5-rc.2 · profile: l4-sandbox
Issued: 2026-09-11 · expires: 2026-09-18
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.1.3-alpha.22026-09-08Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.1.3-alpha.2
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.1.3-alpha.2 · profile: l4-sandbox
Issued: 2026-09-08 · expires: 2026-09-15
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○dsh 0.1.3-alpha.12026-09-07Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: 0.1.3-alpha.1
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dsh0.1.3-alpha.1 · profile: l4-sandbox
Issued: 2026-09-07 · expires: 2026-09-14
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
○prehistory2026-10-05Ecosystem plugin
- • Node.js: v24.14.0
- • DSH: unknown
- • Plugin artifact: github · https://github.com/TT-Wang/dsh-eval-infra (version unstated)
ID: dsh-eval-infra@936f39@dshunknown · profile: l4-sandbox
Issued: 2026-10-05 · expires: 2026-10-12
View checks & evidence
not in scope for web smoke; see prior standard-profile records
not in scope for web smoke; see prior standard-profile records
not present in web-profile loader inventory
L1–L3 are plugin-level static conclusions (baseline, identical across dsh versions): evidence is stored once per plugin.
Author badges
Embed the official badges in your README to showcase security & install-test status:
`[](https://www.dsh.so/artifact/dsh-eval-infra/)`Security Report
Two automated layers, neither a manual review: a static scan (per-plugin rule checks) and a model-driven dynamic security audit that covers only the subset static-flagged at warn or above and never executes plugin code.
Static low, dynamic no conclusion — stricter wins → overall low.
v0.1.0dsh manifest0.1.0git2026-10-210.1.0same versionclean · gitVet is an AST static scan (npm artifact or git source).
not in the dynamic queue (queue only covers plugins static-flagged at warn or above) — this does not mean the dynamic audit found it safe.
Rule: the dynamic audit can only raise risk, never lower it (raise-only, never an exemption); a wrong raise is withdrawn by human review and the tier falls back to the static verdict. Overall verdict = the stricter of the two layers.
Disclaimer: the static scan and the dynamic security audit are both automated analysis, not a security guarantee. Always review what you install.
Activity
Last commit 2026-09-09 · activity: Stable
• Repo created: 2026-09-03
• Stars: ★ 1 · Forks: 0
• Health: Stable — committed within 90 days
Source
Author README
View on GitHub ↗The author's original README from the repository — fetched live, unedited (the English page loads the default README).