BullshitBench v2
Pushing Back on Bullshit by Model
1
Claude Sonnet 4.6 (High)
91.0%
6%
3%
2
Claude Opus 4.5 (High)
90.0%
8%
3
Claude Sonnet 4.6
89.0%
9%
4
Claude Opus 4.6 (High)
87.0%
10.0%
3%
5
Claude Opus 4.6
83.0%
14.0%
3%
6
Claude Sonnet 4.5 (High)
79.0%
13.0%
8%
7
Claude Opus 4.5
79.0%
10.0%
11.0%
8
Qwen3.5 397b A17b (High)
78.0%
17.0%
5%
9
Claude Haiku 4.5 (High)
77.0%
12.0%
11.0%
10
Claude Sonnet 4.5
74.0%
13.0%
13.0%
11
Claude Haiku 4.5
71.0%
21.0%
8%
12
Qwen3.5 397b A17b
69.0%
16.0%
15.0%
13
Grok 4.20 Multi Agent (Low)
67.0%
14.0%
19.0%
14
Grok 4.20 Multi Agent (Xhigh)
64.0%
16.0%
20.0%
15
Grok 4.20 (Low)
56.0%
18.0%
26.0%
16
Grok 4.20 (Xhigh)
54.0%
21.0%
25.0%
17
Kimi K2.5
52.0%
20.0%
28.0%
18
Claude 3.5 Haiku
50.0%
15.0%
35.0%
19
Claude 3.7 Sonnet:thinking
49.0%
17.0%
34.0%
20
GPT-5.4
48.0%
36.0%
16.0%
21
Gemini 3 Pro Preview (Low)
48.0%
15.0%
37.0%
22
GPT-5.2 Codex (Low)
45.0%
26.0%
29.0%
23
Claude 3.5 Sonnet
45.0%
19.0%
36.0%
24
GPT-5.1 Chat
45.0%
19.0%
36.0%
25
Claude Opus 4.1
43.0%
28.0%
29.0%
26
Claude 3.7 Sonnet
43.0%
19.0%
38.0%
27
Hunter Alpha
43.0%
19.0%
38.0%
28
GPT-5.4 (Xhigh)
42.0%
38.0%
20.0%
29
Claude Opus 4.1 (High)
42.0%
29.0%
29.0%
30
GPT-5.3 Chat
40.0%
29.0%
31.0%
31
GPT-5.2 Codex (Xhigh)
39.0%
28.0%
33.0%
32
GPT-5 Codex
39.0%
26.0%
35.0%
33
GPT-5.2
38.0%
39.0%
23.0%
34
Gemini 3.1 Pro Preview (Low)
37.0%
32.0%
31.0%
35
GPT-5.2 Codex (High)
37.0%
27.0%
36.0%
36
Healer Alpha
37.0%
20.0%
43.0%
37
Gemini 3 Pro Preview (High)
36.0%
23.0%
41.0%
38
Hunter Alpha (Xhigh)
35.0%
20.0%
45.0%
39
Claude Opus 4
34.0%
36.0%
30.0%
40
GPT-5.4 Mini (High)
32.0%
37.0%
31.0%
41
GPT-5.4 Mini
32.0%
36.0%
32.0%
42
GPT-5.1 Codex
32.0%
25.0%
43.0%
43
GPT-5.4 Mini (Xhigh)
31.0%
47.0%
22.0%
44
Kimi K2.5 (High)
31.0%
35.0%
34.0%
45
Gemini 3.1 Pro Preview (High)
31.0%
23.0%
46.0%
46
Claude Sonnet 4 (High)
30.0%
27.0%
43.0%
47
Claude Sonnet 4
29.0%
29.0%
42.0%
48
GPT-5.2 (High)
28.0%
50.0%
22.0%
49
Llama 4 Maverick
28.0%
30.0%
42.0%
50
GLM 5 (High)
28.0%
19.0%
53.0%
51
GPT-5.2 Chat
27.0%
33.0%
40.0%
52
Healer Alpha (Xhigh)
26.0%
17.0%
57.0%
53
o3
26.0%
16.0%
58.0%
54
GPT-5.1
25.0%
44.0%
31.0%
55
GPT-5.3 Codex (Low)
24.0%
39.0%
37.0%
56
GPT-5
21.0%
42.0%
37.0%
57
GPT-5.3 Codex (High)
20.0%
34.0%
46.0%
58
Gemini 2.5 Pro
20.0%
22.0%
58.0%
59
GLM 5
20.0%
22.0%
58.0%
60
GPT-5.3 Codex (Xhigh)
19.0%
31.0%
50.0%
61
Grok 4.1 Fast (High)
19.0%
22.0%
59.0%
62
Llama 4 Scout
19.0%
20.0%
61.0%
63
Gemini 2.5 Flash
19.0%
14.0%
67.0%
64
GPT-5 Chat
18.0%
30.0%
52.0%
65
Mimo V2 Flash
16.0%
28.0%
56.0%
66
Gemini 2.0 Flash 001
15.0%
19.0%
66.0%
67
GPT-5.4 Nano (High)
14.0%
30.0%
56.0%
68
Llama 3.1 8b Instruct
14.0%
25.0%
61.0%
69
GPT-4.1
14.0%
19.0%
67.0%
70
GPT-5.4 Nano
13.0%
26.0%
61.0%
71
Deepseek V3.2 (High)
13.0%
23.0%
64.0%
72
Mimo V2 Flash (High)
13.0%
15.0%
72.0%
73
GPT-4o 2024 08 06
12.0%
27.0%
61.0%
74
GPT OSS 120b (Low)
11.0%
22.0%
67.0%
75
Seed 1.6
11.0%
20.0%
69.0%
76
Gemini 3.1 Flash Lite Preview
11.0%
15.0%
74.0%
77
Deepseek V3.2
10.0%
21.0%
69.0%
78
GPT-5.4 Nano (Xhigh)
10.0%
20.0%
70.0%
79
Gemini 3 Flash Preview (High)
10.0%
19.0%
71.0%
80
Gemini 3 Flash Preview
10.0%
16.0%
74.0%
81
Claude 3 Haiku
10.0%
15.0%
75.0%
82
Grok 4.1 Fast
10.0%
10.0%
80.0%
83
Minimax M2.5 (Low)
9%
29.0%
62.0%
84
Minimax M2.5 (High)
8%
26.0%
66.0%
85
o4-mini (Low)
8%
13.0%
79.0%
86
Seed 1.6 (High)
7%
32.0%
61.0%
87
Intellect 3 (Low)
7%
11.0%
82.0%
88
GPT OSS 120b (High)
5%
23.0%
72.0%
89
Intellect 3 (High)
5%
10.0%
85.0%
90
o4-mini (High)
4%
25.0%
71.0%
91
Ernie 4.5 300b A47b
4%
22.0%
74.0%
92
Gemma 3 27b IT
3%
9%
88.0%
93
Mistral Large 2512
13.0%
85.0%
94
GPT-4o Mini 2024 07 18
12.0%
86.0%
Detection Rate by Domain
| Model | Overall | Legal | Medical | Software | Finance | Physics |
|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 (High) | 91.0% | 80.0% | 86.7% | 92.5% | 93.3% | 100.0% |
| Claude Opus 4.5 (High) | 90.0% | 80.0% | 93.3% | 90.0% | 93.3% | 93.3% |
| Claude Sonnet 4.6 | 89.0% | 73.3% | 86.7% | 92.5% | 86.7% | 100.0% |
| Claude Opus 4.6 (High) | 87.0% | 86.7% | 86.7% | 85.0% | 86.7% | 93.3% |
| Claude Opus 4.6 | 83.0% | 73.3% | 80.0% | 85.0% | 80.0% | 93.3% |
| Claude Opus 4.5 | 79.0% | 80.0% | 80.0% | 77.5% | 73.3% | 86.7% |
| Claude Sonnet 4.5 (High) | 79.0% | 73.3% | 80.0% | 77.5% | 73.3% | 93.3% |
| Qwen3.5 397b A17b (High) | 78.0% | 80.0% | 93.3% | 67.5% | 73.3% | 93.3% |
| Claude Haiku 4.5 (High) | 77.0% | 80.0% | 80.0% | 75.0% | 66.7% | 86.7% |
| Claude Sonnet 4.5 | 74.0% | 73.3% | 86.7% | 62.5% | 73.3% | 93.3% |
| Claude Haiku 4.5 | 71.0% | 66.7% | 73.3% | 67.5% | 66.7% | 86.7% |
| Qwen3.5 397b A17b | 69.0% | 73.3% | 86.7% | 60.0% | 60.0% | 80.0% |
| Grok 4.20 Multi Agent (Low) | 67.0% | 73.3% | 93.3% | 57.5% | 46.7% | 80.0% |
| Grok 4.20 Multi Agent (Xhigh) | 64.0% | 80.0% | 93.3% | 47.5% | 40.0% | 86.7% |
| Grok 4.20 (Low) | 56.0% | 60.0% | 66.7% | 45.0% | 40.0% | 86.7% |
| Grok 4.20 (Xhigh) | 54.0% | 66.7% | 66.7% | 40.0% | 40.0% | 80.0% |
| Kimi K2.5 | 52.0% | 53.3% | 60.0% | 45.0% | 40.0% | 73.3% |
| Claude 3.5 Haiku | 50.0% | 40.0% | 60.0% | 50.0% | 40.0% | 60.0% |
| Claude 3.7 Sonnet:thinking | 49.0% | 60.0% | 60.0% | 30.0% | 46.7% | 80.0% |
| Gemini 3 Pro Preview (Low) | 48.0% | 46.7% | 46.7% | 37.5% | 53.3% | 73.3% |
| GPT-5.4 | 48.0% | 46.7% | 53.3% | 37.5% | 46.7% | 73.3% |
| Claude 3.5 Sonnet | 45.0% | 40.0% | 53.3% | 40.0% | 46.7% | 53.3% |
| GPT-5.1 Chat | 45.0% | 53.3% | 53.3% | 32.5% | 46.7% | 60.0% |
| GPT-5.2 Codex (Low) | 45.0% | 40.0% | 60.0% | 37.5% | 40.0% | 60.0% |
| Claude 3.7 Sonnet | 43.0% | 40.0% | 66.7% | 32.5% | 40.0% | 53.3% |
| Claude Opus 4.1 | 43.0% | 40.0% | 60.0% | 30.0% | 53.3% | 53.3% |
| Hunter Alpha | 43.0% | 33.3% | 46.7% | 37.5% | 40.0% | 66.7% |
| Claude Opus 4.1 (High) | 42.0% | 53.3% | 60.0% | 20.0% | 46.7% | 66.7% |
| GPT-5.4 (Xhigh) | 42.0% | 26.7% | 53.3% | 32.5% | 33.3% | 80.0% |
| GPT-5.3 Chat | 40.0% | 33.3% | 46.7% | 22.5% | 46.7% | 80.0% |
| GPT-5 Codex | 39.0% | 26.7% | 53.3% | 30.0% | 40.0% | 60.0% |
| GPT-5.2 Codex (Xhigh) | 39.0% | 46.7% | 46.7% | 27.5% | 33.3% | 60.0% |
| GPT-5.2 | 38.0% | 33.3% | 46.7% | 32.5% | 33.3% | 53.3% |
| Gemini 3.1 Pro Preview (Low) | 37.0% | 20.0% | 46.7% | 30.0% | 40.0% | 60.0% |
| GPT-5.2 Codex (High) | 37.0% | 40.0% | 40.0% | 27.5% | 33.3% | 60.0% |
| Healer Alpha | 37.0% | 26.7% | 53.3% | 27.5% | 33.3% | 60.0% |
| Gemini 3 Pro Preview (High) | 36.0% | 33.3% | 33.3% | 27.5% | 33.3% | 66.7% |
| Hunter Alpha (Xhigh) | 35.0% | 33.3% | 26.7% | 35.0% | 40.0% | 40.0% |
| Claude Opus 4 | 34.0% | 26.7% | 46.7% | 25.0% | 33.3% | 53.3% |
| GPT-5.1 Codex | 32.0% | 26.7% | 33.3% | 32.5% | 20.0% | 46.7% |
| GPT-5.4 Mini | 32.0% | 40.0% | 46.7% | 20.0% | 33.3% | 40.0% |
| GPT-5.4 Mini (High) | 32.0% | 33.3% | 40.0% | 25.0% | 33.3% | 40.0% |
| Gemini 3.1 Pro Preview (High) | 31.0% | 20.0% | 33.3% | 25.0% | 40.0% | 46.7% |
| GPT-5.4 Mini (Xhigh) | 31.0% | 20.0% | 60.0% | 20.0% | 20.0% | 53.3% |
| Kimi K2.5 (High) | 31.0% | 20.0% | 40.0% | 27.5% | 33.3% | 40.0% |
| Claude Sonnet 4 (High) | 30.0% | 6.7% | 66.7% | 12.5% | 46.7% | 46.7% |
| Claude Sonnet 4 | 29.0% | 26.7% | 33.3% | 12.5% | 33.3% | 66.7% |
| GLM 5 (High) | 28.0% | 13.3% | 33.3% | 15.0% | 33.3% | 66.7% |
| GPT-5.2 (High) | 28.0% | 13.3% | 33.3% | 25.0% | 26.7% | 46.7% |
| Llama 4 Maverick | 28.0% | 33.3% | 40.0% | 15.0% | 26.7% | 46.7% |
| GPT-5.2 Chat | 27.0% | 6.7% | 33.3% | 20.0% | 26.7% | 60.0% |
| Healer Alpha (Xhigh) | 26.0% | 6.7% | 46.7% | 17.5% | 26.7% | 46.7% |
| o3 | 26.0% | 6.7% | 33.3% | 17.5% | 33.3% | 53.3% |
| GPT-5.1 | 25.0% | 0.0% | 33.3% | 17.5% | 33.3% | 53.3% |
| GPT-5.3 Codex (Low) | 24.0% | 13.3% | 40.0% | 22.5% | 6.7% | 40.0% |
| GPT-5 | 21.0% | 0.0% | 40.0% | 12.5% | 20.0% | 46.7% |
| Gemini 2.5 Pro | 20.0% | 13.3% | 26.7% | 15.0% | 20.0% | 33.3% |
| GLM 5 | 20.0% | 6.7% | 26.7% | 12.5% | 26.7% | 40.0% |
| GPT-5.3 Codex (High) | 20.0% | 6.7% | 33.3% | 15.0% | 13.3% | 40.0% |
| Gemini 2.5 Flash | 19.0% | 13.3% | 26.7% | 2.5% | 33.3% | 46.7% |
| GPT-5.3 Codex (Xhigh) | 19.0% | 6.7% | 26.7% | 12.5% | 13.3% | 46.7% |
| Grok 4.1 Fast (High) | 19.0% | 20.0% | 33.3% | 7.5% | 26.7% | 26.7% |
| Llama 4 Scout | 19.0% | 20.0% | 20.0% | 17.5% | 20.0% | 20.0% |
| GPT-5 Chat | 18.0% | 13.3% | 33.3% | 5.0% | 20.0% | 40.0% |
| Mimo V2 Flash | 16.0% | 13.3% | 20.0% | 12.5% | 20.0% | 20.0% |
| Gemini 2.0 Flash 001 | 15.0% | 0.0% | 20.0% | 10.0% | 33.3% | 20.0% |
| GPT-4.1 | 14.0% | 0.0% | 26.7% | 10.0% | 13.3% | 26.7% |
| GPT-5.4 Nano (High) | 14.0% | 20.0% | 13.3% | 10.0% | 13.3% | 20.0% |
| Llama 3.1 8b Instruct | 14.0% | 20.0% | 20.0% | 12.5% | 6.7% | 13.3% |
| Deepseek V3.2 (High) | 13.0% | 6.7% | 13.3% | 10.0% | 13.3% | 26.7% |
| GPT-5.4 Nano | 13.0% | 20.0% | 20.0% | 10.0% | 6.7% | 13.3% |
| Mimo V2 Flash (High) | 13.0% | 13.3% | 20.0% | 7.5% | 6.7% | 26.7% |
| GPT-4o 2024 08 06 | 12.0% | 20.0% | 13.3% | 5.0% | 6.7% | 26.7% |
| Gemini 3.1 Flash Lite Preview | 11.0% | 0.0% | 26.7% | 7.5% | 13.3% | 13.3% |
| GPT OSS 120b (Low) | 11.0% | 0.0% | 20.0% | 5.0% | 13.3% | 26.7% |
| Seed 1.6 | 11.0% | 6.7% | 26.7% | 5.0% | 6.7% | 20.0% |
| Claude 3 Haiku | 10.0% | 0.0% | 13.3% | 10.0% | 13.3% | 13.3% |
| Deepseek V3.2 | 10.0% | 13.3% | 6.7% | 5.0% | 13.3% | 20.0% |
| Gemini 3 Flash Preview | 10.0% | 0.0% | 20.0% | 5.0% | 13.3% | 20.0% |
| Gemini 3 Flash Preview (High) | 10.0% | 0.0% | 26.7% | 7.5% | 13.3% | 6.7% |
| GPT-5.4 Nano (Xhigh) | 10.0% | 20.0% | 6.7% | 10.0% | 6.7% | 6.7% |
| Grok 4.1 Fast | 10.0% | 13.3% | 6.7% | 7.5% | 6.7% | 20.0% |
| Minimax M2.5 (Low) | 9.0% | 13.3% | 13.3% | 2.5% | 13.3% | 13.3% |
| Minimax M2.5 (High) | 8.0% | 13.3% | 6.7% | 0.0% | 0.0% | 33.3% |
| o4-mini (Low) | 8.0% | 0.0% | 6.7% | 5.0% | 13.3% | 20.0% |
| Intellect 3 (Low) | 7.0% | 0.0% | 26.7% | 0.0% | 20.0% | 0.0% |
| Seed 1.6 (High) | 7.0% | 6.7% | 20.0% | 2.5% | 6.7% | 6.7% |
| GPT OSS 120b (High) | 5.0% | 0.0% | 0.0% | 2.5% | 6.7% | 20.0% |
| Intellect 3 (High) | 5.0% | 6.7% | 6.7% | 0.0% | 20.0% | 0.0% |
| Ernie 4.5 300b A47b | 4.0% | 0.0% | 0.0% | 2.5% | 6.7% | 13.3% |
| o4-mini (High) | 4.0% | 0.0% | 6.7% | 0.0% | 6.7% | 13.3% |
| Gemma 3 27b IT | 3.0% | 0.0% | 6.7% | 0.0% | 6.7% | 6.7% |
| GPT-4o Mini 2024 07 18 | 2.0% | 0.0% | 6.7% | 2.5% | 0.0% | 0.0% |
| Mistral Large 2512 | 2.0% | 0.0% | 0.0% | 0.0% | 0.0% | 13.3% |