Analyzer Zoo & Model Registry

Reference tables for what's implemented. Discover the same information at runtime via evalvitals.list_specs() and evalvitals.registry.analyzers (see Discovery).

Analyzers

Analyzer	Key	Capability	Modality	Paper	Status
Attention summary	attention	ATTENTION	text + image	—	✓
Attention rollout	rollout	ATTENTION	text + image	Abnar & Zuidema, 2020	✓
Attention sink	attention_sink	ATTENTION	text + image	Gu et al. 2023	✓
Relative attention	relative_attention	ATTENTION	image (VLM)	arXiv:2502.17422	✓
RISE	rise	GENERATE	text	Petsiuk et al. 2018	✓
MM-SHAP	mm_shap	LOGPROBS	image (VLM)	arXiv:2212.08158	✓
VL-SHAP	vl_shap	LOGPROBS	image (VLM)	arXiv:2212.08158	✓
Token entropy	token_entropy	LOGITS	text + image	—	✓
Logprob entropy	logprob_entropy	LOGPROBS	text + image	Kadavath et al. 2022	✓
Self-consistency	self_consistency	GENERATE	text + image	Wang et al. 2023	✓
Verbalized confidence	verbalized_confidence	GENERATE	text + image	—	✓
POPE	pope	GENERATE	image (VLM)	arXiv:2305.10355	✓
CHAIR	chair	GENERATE	image (VLM)	arXiv:1809.02156	✓
OPERA	opera	ATTENTION	image (VLM)	arXiv:2311.17911	stub
VCD	vcd	LOGITS	image (VLM)	arXiv:2311.16922	stub
Logit lens	logit_lens	HIDDEN_STATES	text + image	nostalgebraist 2020	✓
Tuned lens	tuned_lens	HIDDEN_STATES	text + image	Belrose et al. 2023	stub
Grad-CAM	gradcam	GRADIENTS	image (VLM)	Selvaraju et al. 2017	stub
Generic attn explain	generic_attention	ATTENTION + GRADIENTS	text + image	Chefer et al. 2021	stub
Linear CKA	cka	HIDDEN_STATES	text + image	Kornblith et al. 2019	✓
Linear probe	linear_probe	HIDDEN_STATES	text + image	—	stub
Causal trace	causal_trace	HIDDEN_STATES	text + image	Meng et al. 2022	stub
Loop detect	loop_detect	Trajectory	agent	—	✓
Ignored obs	ignored_obs	Trajectory	agent	—	✓
First-error judge	first_error_judge	Trajectory	agent	Zhang et al. 2024	✓
Counterfactual	counterfactual	Trajectory	agent	Pearl 2000	✓

Model Registry

From list_specs():

Key	Family	Type	Notes
qwen2.5-7b-instruct	Qwen2	LLM	reference checkpoint
qwen3-4b	Qwen3	LLM	reasoning, small smoke-test
qwen3-8b	Qwen3	LLM	reasoning
qwen3-30b-a3b	Qwen3-MoE	LLM	MoE
deepseek-v3	DeepSeek-V3	LLM	MoE + MLA
llama-3.1-8b-instruct	Llama	LLM	—
gemma-3-1b-it	Gemma3	LLM	text-only
qwen3-vl-4b-instruct	Qwen3-VL	VLM	small smoke-test
qwen2.5-vl-7b-instruct	Qwen2.5-VL	VLM	reference for relative-attention
qwen2-vl-7b-instruct	Qwen2-VL	VLM	—
qwen3-vl-8b-instruct	Qwen3-VL	VLM	—
glm-4.5v	GLM-MoE	VLM	MoE + reasoning, 106B
glm-4.1v-9b-thinking	GLM	VLM	reasoning
kimi-vl-a3b-thinking	Kimi-VL	VLM	MoE + MLA + reasoning
llama-4-scout	Llama4	VLM	MoE
step-1o-vision	Step	VLM	API-only