-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
460 lines (419 loc) · 20.8 KB
/
Copy pathindex.html
File metadata and controls
460 lines (419 loc) · 20.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="description"
content="NINJA: A simple yet effective jailbreaking method that exploits long contexts to bypass LLM safety filters by embedding harmful goals in benign text.">
<meta name="keywords" content="LLM Safety, Jailbreaking, Long Context, NINJA Attack">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Jailbreaking in the Haystack: NINJA Attack</title>
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">
<link rel="stylesheet" href="./static/css/bulma.min.css">
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="./static/css/bulma-slider.min.css">
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="./static/css/index.css">
<link rel="icon" href="./static/images/favicon.svg">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script defer src="./static/js/fontawesome.all.min.js"></script>
<script src="./static/js/bulma-carousel.min.js"></script>
<script src="./static/js/bulma-slider.min.js"></script>
<script src="./static/js/index.js"></script>
</head>
<body>
<nav class="navbar" role="navigation" aria-label="main navigation">
<div class="navbar-brand">
<a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false">
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
</a>
</div>
<div class="navbar-menu">
<div class="navbar-start" style="flex-grow: 1; justify-content: center;">
<a class="navbar-item" href="https://ar-forum.github.io/">
<span class="icon">
<i class="fas fa-home"></i>
</span>
</a>
<div class="navbar-item has-dropdown is-hoverable">
<a class="navbar-link">
More Research
</a>
<div class="navbar-dropdown">
<a class="navbar-item" href="https://ar-forum.github.io/">
AR Lab @ CMU
</a>
</div>
</div>
</div>
</div>
</nav>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">Jailbreaking in the Haystack</h1>
<div class="is-size-5 publication-authors">
<span class="author-block">
<a href="https://rishi1001.github.io/">Rishi Shah</a>,</span>
<span class="author-block">
<a href="https://chenwu.io/">Chen Wu</a>,</span>
<span class="author-block">
Shashwat Saxena,
</span>
<span class="author-block">
<a href="https://fjzzq2002.github.io/">Ziqian Zhong</a>,
</span>
<span class="author-block">
<a href="https://arobey1.github.io/">Alexander Robey</a>,
</span>
<span class="author-block">
<a href="https://www.cs.cmu.edu/~aditirag">Aditi Raghunathan</a>
</span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block">Carnegie Mellon University</span>
</div>
<div class="column has-text-centered">
<div class="publication-links">
<!-- PDF Link. -->
<span class="link-block">
<a href="./15865_Jailbreaking_in_the_Hays.pdf"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Paper</span>
</a>
</span>
<span class="link-block">
<a href="#"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv (Coming Soon)</span>
</a>
</span>
<!-- Code Link. -->
<span class="link-block">
<a href="https://github.com/AR-FORUM/NINJA_Attack"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code</span>
</a>
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<img src="./static/images/Ninja_cover_image.png" alt="NINJA Attack Overview" style="width: 100%; max-width: 800px; margin: 0 auto; display: block;">
<h2 class="subtitle has-text-centered" style="margin-top: 20px;">
<span class="dnerf">NINJA</span> (Needle-in-Haystack Jailbreak Attack) embeds harmful goals in benign long contexts to bypass LLM safety filters.
</h2>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<!-- Abstract. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
Recent advances in long-context language models (LMs) have enabled million-token inputs, expanding their capabilities across complex tasks like computer-use agents. Yet, the safety implications of these extended contexts remain unclear.
</p>
<p>
To bridge this gap, we introduce <strong>NINJA</strong> (short for <em>Needle-in-haystack jailbreak attack</em>), a method that jailbreaks aligned LMs by appending benign, model-generated content to harmful user goals. Critical to our method is the observation that the position of harmful goals play an important role in safety.
</p>
<p>
Experiments on standard safety benchmark, HarmBench, show that NINJA significantly increases attack success rates across state-of-the-art open and proprietary models, including LLaMA, Qwen, and Gemini. Unlike prior jailbreaking methods, our approach is low-resource, transferable, and less detectable. Moreover, we show that NINJA is compute-optimal -- under a fixed compute budget, increasing context length can outperform increasing the number of trials in best-of-N jailbreak.
</p>
<p>
These findings reveal that even benign long contexts -- when crafted with careful goal positioning -- introduce fundamental vulnerabilities in modern LMs.
</p>
</div>
</div>
</div>
<!--/ Abstract. -->
<!-- Key Insights Box. -->
<div class="columns is-centered">
<div class="column is-four-fifths">
<div style="background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 30px; border-radius: 10px; margin: 30px 0;">
<h3 style="color: white; font-size: 1.4em; margin-bottom: 15px; text-align: center;">🥷 Key Insights</h3>
<div style="display: grid; grid-template-columns: 1fr 1fr 1fr; gap: 20px; text-align: center;">
<div>
<div style="font-size: 2em; font-weight: bold;">2.5x</div>
<div style="font-size: 0.9em;">ASR improvement on Llama-3.1</div>
</div>
<div>
<div style="font-size: 2em; font-weight: bold;">100%</div>
<div style="font-size: 0.9em;">Benign context (undetectable)</div>
</div>
<div>
<div style="font-size: 2em; font-weight: bold;">↑ Length</div>
<div style="font-size: 0.9em;">More effective than ↑ trials</div>
</div>
</div>
</div>
</div>
</div>
<!--/ Key Insights Box. -->
<!-- Method Overview. -->
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Method Overview</h2>
<div class="content has-text-justified">
<p>
The NINJA attack operates in three simple stages:
</p>
<ol style="text-align: left; max-width: 600px; margin: 0 auto;">
<li><strong>Keyword Extraction:</strong> Extract key nouns, adjectives, and verbs from the harmful goal to capture its core semantics.</li>
<li><strong>Context Generation:</strong> Iteratively prompt the LLM to generate benign, educational passages around these keywords until reaching the target context length.</li>
<li><strong>Goal Positioning:</strong> Position the harmful goal at the beginning of the long context for maximum attack success.</li>
</ol>
<br>
<img src="./static/images/ninja_anim.png" alt="NINJA Method Visualization" style="width: 100%; max-width: 700px;">
</div>
</div>
</div>
<!--/ Method Overview. -->
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered">
<!-- Goal Positioning. -->
<div class="column">
<div class="content">
<h2 class="title is-3">Goal Positioning Matters</h2>
<p>
We find that placing the harmful goal at the <strong>beginning</strong> of the context yields the highest attack success rate, likely due to increased model attention and limited opportunity for safety filters to override early generation.
</p>
<img src="./static/images/goal_position_20000_Llama3.1-8B-Instruct.png" alt="Goal Positioning Results" style="width: 100%;">
<p style="margin-top: 10px; font-size: 0.9em; color: #666;">
Positioning the harmful goal at the start (0% position) maximizes attack success compared to the end (100% position).
</p>
</div>
</div>
<!--/ Goal Positioning. -->
<!-- Compute Optimal. -->
<div class="column">
<h2 class="title is-3">Compute-Optimal Attack</h2>
<div class="columns is-centered">
<div class="column content">
<p>
Under a fixed compute budget, extending benign context length is more effective than scaling the number of trials (best-of-N). Longer contexts yield higher attack success with fewer attempts.
</p>
<img src="./static/images/compute_optimal_plot_PER_EXAMPLE.png" alt="Compute Optimal Results" style="width: 100%;">
<p style="margin-top: 10px; font-size: 0.9em; color: #666;">
Compute-optimal trade-off: longer contexts achieve higher ASR with fewer samples.
</p>
</div>
</div>
</div>
</div>
<!--/ Compute Optimal. -->
</div>
</section>
<!-- <section class="hero is-light is-small">
<div class="hero-body">
<div class="container">
<h2 class="title is-3 has-text-centered">Key Results</h2>
<div id="results-carousel" class="carousel results-carousel">
<div class="item">
<img src="./static/images/goal_position_20000_Llama3.1-8B-Instruct.png" alt="Goal Positioning Results Llama 3.1" style="width: 100%;">
<p class="has-text-centered" style="padding: 10px;">Goal positioning significantly affects attack success on Llama 3.1-8B-Instruct</p>
</div>
<div class="item">
<img src="./static/images/goal_position_20000_qwen2.5-7b-instruct.png" alt="Goal Positioning Results Qwen" style="width: 100%;">
<p class="has-text-centered" style="padding: 10px;">Goal positioning effects on Qwen2.5-7B-Instruct</p>
</div>
<div class="item">
<img src="./static/images/mistral_7b_v3_with_variance_plot.png" alt="Mistral 7B Results with Variance" style="width: 100%;">
<p class="has-text-centered" style="padding: 10px;">Attack success rate vs context length on Mistral-7B-v3 with variance bands</p>
</div>
<div class="item">
<img src="./static/images/llama3_8b_relevant_vs_irrelevant_with_variance.png" alt="Relevant vs Irrelevant Context" style="width: 100%;">
<p class="has-text-centered" style="padding: 10px;">Relevant long contexts dramatically increase attack success compared to irrelevant contexts</p>
</div>
<div class="item">
<img src="./static/images/experiment_results_workshop.png" alt="Main Results" style="width: 100%;">
<p class="has-text-centered" style="padding: 10px;">Attack success rates across different models and context lengths</p>
</div>
</div>
</div>
</div>
</section> -->
<section class="section">
<div class="container is-max-desktop">
<!-- Key Contributions. -->
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-3">Key Contributions</h2>
<!-- Contribution 1. -->
<h3 class="title is-4">1. Simple Yet Highly Effective Attack</h3>
<div class="content has-text-justified">
<p>
We introduce the NINJA (Needle-in-Haystack Jailbreak) Attack, a simple yet highly effective method for jailbreaking aligned language models. By appending benign, model-generated content to a harmful user goal, our approach significantly boosts the attack success rate (ASR) across various models:
</p>
<ul>
<li><strong>Llama-3.1-8B-Instruct:</strong> 23.7% → 58.8% ASR</li>
<li><strong>Qwen2.5-7B-Instruct:</strong> 23.7% → 42.5% ASR</li>
<li><strong>Gemini Flash:</strong> 23% → 29% ASR</li>
</ul>
<p>The NINJA attack is low-resource and less detectable than prior jailbreaking methods.</p>
</div>
<br/>
<!--/ Contribution 1. -->
<!-- Contribution 2. -->
<h3 class="title is-4">2. Goal Positioning Analysis</h3>
<div class="content has-text-justified">
<p>
We provide a detailed empirical analysis of goal positioning, revealing that placing the harmful request at the beginning of the context is the most effective strategy for maximizing the attack success rate. This finding highlights a key vulnerability in how long-context models process and prioritize information.
</p>
</div>
<br/>
<!--/ Contribution 2. -->
<!-- Contribution 3. -->
<h3 class="title is-4">3. Compute-Aware Scaling Law</h3>
<div class="content has-text-justified">
<p>
We propose a compute-aware scaling law for optimizing jailbreak attacks, demonstrating how to select the optimal context length to maximize the ASR within a given best-of-N compute budget. Our findings show that under a larger compute budget, using a longer context is more effective than increasing the number of attack attempts.
</p>
</div>
<!--/ Contribution 3. -->
</div>
</div>
<!--/ Key Contributions. -->
<!-- Comparison Table. -->
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-3">Performance Comparison</h2>
<div class="content has-text-justified">
<p>
We compare NINJA against established jailbreaking methods including PAIR (optimization-based) and Many-shot jailbreaking (requires harmful demonstrations). NINJA achieves the highest ASR on Llama-3.1 and Qwen2.5, while using entirely benign context.
</p>
</div>
<div class="content has-text-centered">
<table style="margin: 0 auto; border-collapse: collapse; width: 85%;">
<thead>
<tr style="background-color: #f5f5f5;">
<th style="padding: 12px; border: 1px solid #ddd;">Method</th>
<th style="padding: 12px; border: 1px solid #ddd;">Llama-3.1-8B</th>
<th style="padding: 12px; border: 1px solid #ddd;">Qwen2.5-7B</th>
<th style="padding: 12px; border: 1px solid #ddd;">Mistral-7B-v3</th>
<th style="padding: 12px; border: 1px solid #ddd;">Gemini Flash</th>
</tr>
</thead>
<tbody>
<tr>
<td style="padding: 12px; border: 1px solid #ddd;">PAIR</td>
<td style="padding: 12px; border: 1px solid #ddd;">22.0%</td>
<td style="padding: 12px; border: 1px solid #ddd;">34.6%</td>
<td style="padding: 12px; border: 1px solid #ddd;">41.3%</td>
<td style="padding: 12px; border: 1px solid #ddd;">15.3%</td>
</tr>
<tr>
<td style="padding: 12px; border: 1px solid #ddd;">Many-shot</td>
<td style="padding: 12px; border: 1px solid #ddd;">45.0%</td>
<td style="padding: 12px; border: 1px solid #ddd;">22.5%</td>
<td style="padding: 12px; border: 1px solid #ddd;">12.5%</td>
<td style="padding: 12px; border: 1px solid #ddd;">50.0%</td>
</tr>
<tr style="background-color: #e8f4f8; font-weight: bold;">
<td style="padding: 12px; border: 1px solid #ddd;">NINJA (Ours)</td>
<td style="padding: 12px; border: 1px solid #ddd;">58.8%</td>
<td style="padding: 12px; border: 1px solid #ddd;">42.5%</td>
<td style="padding: 12px; border: 1px solid #ddd;">54.5%</td>
<td style="padding: 12px; border: 1px solid #ddd;">28.8%</td>
</tr>
</tbody>
</table>
<p style="margin-top: 15px; font-size: 0.9em; color: #666;">
Attack Success Rate (ASR) comparison on HarmBench. NINJA achieves highest performance on Llama, Qwen, and Mistral while using only benign context.
</p>
</div>
</div>
</div>
<!--/ Comparison Table. -->
<!-- Related Work. -->
<div class="columns is-centered">
<div class="column is-full-width">
<h2 class="title is-3">Related Work</h2>
<div class="content has-text-justified">
<p>
Our work builds upon and differs from several lines of research in LLM safety:
</p>
<p>
<strong>Many-Shot Jailbreaking:</strong> Works like <a href="https://www.anthropic.com/research/many-shot-jailbreaking">Anthropic's Many-Shot Jailbreaking</a> show that long contexts with harmful demonstrations can compromise safety. Unlike these methods, NINJA uses entirely benign contexts.
</p>
<p>
<strong>Agent-Based Vulnerabilities:</strong> <a href="https://arxiv.org/abs/2311.09433">Recent work</a> has shown that LLMs are more vulnerable when used as agents. We isolate the effect of context length itself, independent of agentic scaffolding.
</p>
<p>
<strong>Long-Context Safety:</strong> Our work contributes to the emerging understanding that long-context capabilities introduce new safety challenges that require dedicated mitigation strategies beyond standard alignment techniques.
</p>
<p>
<strong>Best-of-N Attacks:</strong> We demonstrate that context-length scaling provides a more compute-efficient alternative to traditional best-of-N jailbreaking approaches.
</p>
</div>
</div>
</div>
<!--/ Related Work. -->
</div>
</section>
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>@article{shah2025jailbreaking,
author = {Shah, Rishi and Wu, Chen and Saxena, Shashwat and Zhong, Ziqian and Robey, Alexander and Raghunathan, Aditi},
title = {Jailbreaking in the Haystack: NINJA Attack on Long-Context LLMs},
journal = {arXiv preprint},
year = {2025},
}</code></pre>
</div>
</section>
<footer class="footer">
<div class="container">
<div class="content has-text-centered">
<a class="icon-link"
href="./15865_Jailbreaking_in_the_Hays.pdf">
<i class="fas fa-file-pdf"></i>
</a>
<a class="icon-link" href="https://github.com/AR-FORUM/NINJA_Attack" class="external-link">
<i class="fab fa-github"></i>
</a>
</div>
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This website is licensed under a <a rel="license"
href="http://creativecommons.org/licenses/by-sa/4.0/">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
</p>
<p>
Website template borrowed from <a href="https://github.com/nerfies/nerfies.github.io">Nerfies</a>.
</p>
</div>
</div>
</div>
</div>
</footer>
</body>
</html>