-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathgenerate_docs.py
More file actions
231 lines (156 loc) · 6.82 KB
/
Copy pathgenerate_docs.py
File metadata and controls
231 lines (156 loc) · 6.82 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
from pathlib import Path
ROOT = Path(__file__).parent
LICENSE = """\
MIT License
Copyright (c) 2025 Joao Felipe De Souza
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
DESIGN = """\
# Design Document -- speculative-decoding-real-v2
## 1. Motivation
speculative-decoding-validation (project 10) attempted speculative decoding
with manual KV cache management but was limited by DynamicCache incompatibility.
This project uses the official HuggingFace assisted generation API:
verifier.generate(input_ids, assistant_model=draft, num_assistant_tokens=K)
This handles KV cache correctly internally and matches production usage.
---
## 2. Method
Baseline:
verifier.generate(input_ids, max_new_tokens=N, do_sample=False)
Speculative:
verifier.generate(input_ids, assistant_model=draft,
max_new_tokens=N, do_sample=False,
num_assistant_tokens=K)
Both: WARMUP=2, REPEATS=5, mean of times.
---
## 3. Model Pairs
Pair Draft Verifier Ratio
distilgpt2 -> gpt2-medium 82M 355M 0.23 <- BEST
gpt2 -> gpt2-medium 124M 355M 0.35
gpt2-medium -> gpt2-large 355M 774M 0.46
gpt2 -> gpt2 (self) 124M 124M 1.00 <- NEVER WINS
---
## 4. Key Results
### Best: distilgpt2 -> gpt2-medium, ratio=0.23
K=1 (optimal for this pair):
n=16: 0.914x n=128: 1.439x
n=64: 1.125x n=256: 1.590x
n=512: 1.747x <- BEST
### gpt2 -> gpt2-medium, ratio=0.35
K=5 (optimal):
n=64: 1.101x n=256: 1.467x
n=128: 1.224x n=512: 1.490x
### gpt2-medium -> gpt2-large, ratio=0.46
K=7 (optimal): n=256: 1.170x
### Self-speculation (ratio=1.00): never exceeds 0.821x
---
## 5. Key Findings
1. Draft/verifier cost ratio is the dominant parameter.
ratio=0.23 -> 1.747x ratio=0.35 -> 1.490x
ratio=0.46 -> 1.170x ratio=1.00 -> 0.821x
2. Speedup grows with output length in all pairs.
Short sequences pay more overhead; long sequences amortize it.
3. Optimal K depends on draft quality.
ratio=0.23: K=1 (low alignment, more speculation wastes)
ratio=0.35: K=5 (moderate alignment)
ratio=0.46: K=7 (better alignment, worth speculating more)
4. For very cheap drafts (ratio=0.23, K=1), speculation acts like
assisted decoding: one cheap candidate per verifier step.
5. 19/36 configs exceed 1.0x speedup.
---
## 6. Comparison with Project 10
Project 10 (manual KV): best 1.14x (median cycle)
This project (HF API): best 1.747x
Difference: HF API manages KV cache correctly,
avoids the DynamicCache reuse bug.
---
## 7. Connections
speculative-decoding-sim cost_ratio and output_length validated
speculative-decoding-validation KV bug resolved via official API
real-model-profiler decode cost confirms memory-bound model
"""
README = """\
# speculative-decoding-real-v2
**Measures real speculative decoding speedup using the official HuggingFace
`assistant_model` API across 4 model pairs and multiple output lengths.**
Validates [speculative-decoding-sim](https://github.com/JohnScheuer/speculative-decoding-sim) (project 6)
and resolves the KV cache bug from [speculative-decoding-validation](https://github.com/JohnScheuer/speculative-decoding-validation) (project 10).
> For design details see [DESIGN.md](DESIGN.md).
---
## Best Result: 1.747x speedup
distilgpt2 (82M) -> gpt2-medium (355M)
K=1, n=512 output tokens
Baseline: 8.19s Speculative: 4.69s Speedup: 1.747x
---
## Key Findings
### 1. Draft/verifier cost ratio is the dominant parameter
ratio=0.23 (distilgpt2->gpt2-medium): 1.747x at n=512
ratio=0.35 (gpt2->gpt2-medium): 1.490x at n=512
ratio=0.46 (gpt2-medium->gpt2-large): 1.170x at n=256
ratio=1.00 (gpt2->gpt2, self): 0.821x (never wins)
Validates speculative-decoding-sim cost_ratio breakeven prediction.
### 2. Speedup grows with output length
distilgpt2->gpt2-medium (K=1):
n=16: 0.914x
n=64: 1.125x
n=128: 1.439x
n=256: 1.590x
n=512: 1.747x <- speedup keeps growing
gpt2->gpt2-medium (K=5):
n=64: 1.101x n=256: 1.467x n=512: 1.490x
### 3. Optimal K depends on draft quality
ratio=0.23: K=1 optimal (low alignment)
ratio=0.35: K=5 optimal (moderate alignment)
ratio=0.46: K=7 optimal (better alignment)
At K=1 with ratio=0.23, speculation acts like assisted decoding:
one cheap candidate per verifier step. Trying K>1 adds overhead
that outweighs the benefit at this alignment level.
### 4. 19/36 configs exceed 1.0x speedup
---
## Quick Start
python3 -m venv venv
source venv/bin/activate
pip install torch transformers
python3 profile_speculative.py # basic pairs
python3 profile_speculative_v2.py # all 4 pairs + long outputs
---
## vs Project 10
| Aspect | Project 10 | This project |
|--------|-----------|--------------|
| API | Manual KV cache | HF assistant_model |
| KV issue | DynamicCache bug | Handled by HF |
| Best speedup | 1.14x | 1.747x |
| Output lengths | up to 64 tok | up to 512 tok |
---
## Results
results/sweep_k.csv K sweep for original pairs
results/output_length.csv Output length for original pairs
results/all_results_v2.csv All v2 results (37 rows)
---
## Portfolio Context
Project 20 in a series on LLM inference infrastructure.
Full series: https://github.com/JohnScheuer
"""
(ROOT / "LICENSE").write_text(LICENSE)
(ROOT / "DESIGN.md").write_text(DESIGN)
(ROOT / "README.md").write_text(README)
print("Wrote LICENSE, DESIGN.md, README.md")