forked from omnihuman-lab/omnihuman-lab.github.io
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
222 lines (191 loc) · 12.1 KB
/
Copy pathindex.html
File metadata and controls
222 lines (191 loc) · 12.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<meta name="theme-color" content="#000000" />
<link rel="stylesheet" href="./static/css/bulma.min.css">
<link rel="stylesheet" href="Open-Sans.css">
<link rel="stylesheet" href="index.css">
<title></title>
<script defer="defer" src="./static/js/main.cb41f6a5.js"></script>
<link href="./static/css/main.4017e162.css" rel="stylesheet">
<meta name="description"
content="OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models">
<title>OmniHuman-1 Project</title>
</head>
<body>
<nav class="navbar" role="navigation" aria-label="main navigation">
<div class="navbar-brand">
<a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false">
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
<span aria-hidden="true"></span>
</a>
</div>
<div class="navbar-menu">
<div class="navbar-start" style="flex-grow: 1; justify-content: center;">
<div class="navbar-item has-dropdown is-hoverable">
<a class="navbar-link">
More Research
</a>
<div class="navbar-dropdown">
<a class="navbar-item" href="https://loopyavatar.github.io/">
Loopy
</a>
<a class="navbar-item" href="https://cyberhost.github.io/">
CyberHost
</a>
<a class="navbar-item" href="https://omnihuman-lab.github.io/">
OmniHuman
</a>
</div>
</div>
</div>
</div>
</nav>
<div id="root" class="column-flex">
<div id="title-flex" class="column-flex">
<h1> OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models </h1>
<span>
<a target="_blank" href="" onclick="return false;">Gaojie Lin</a><sup>*</sup>,
<a href="https://jianwenjiang.github.io">Jianwen Jiang</a><sup>*†</sup>,
<a target="_blank" href="" onclick="return false;">Jiaqi Yang</a><sup>*</sup>,
<a href="https://zhengzerong.github.io/">Zerong Zheng</a><sup>*</sup>,
<a target="_blank" href="" onclick="return false;">Chao Liang</a><sup></sup>
<br />
</span>
<span>Bytedance</span>
<span><sup>*</sup>Equal contribution,<sup>†</sup>Project lead
<!-- ,<sup>‡</sup>Internship at Bytedance -->
</span>
<div class="flex flex-gap" style="margin-bottom:0.5em;">
<a target="_blank" href="http://arxiv.org/abs/2502.01061" ><button>Paper</button></a>
<!-- <a target="_blank" href="" onclick="alert('Coming Soon!');return false;"><button>Paper</button></a> -->
<a target="_blank" href="https://omnihuman-lab.github.io"><button>Page</button></a>
</div>
<small><span><b>TL;DR</b>: We propose an end-to-end multimodality-conditioned human video generation framework named OmniHuman, which can generate human videos based on a single human image and motion signals (e.g., audio only, video only, or a combination of audio and video). In OmniHuman, we introduce a multimodality motion conditioning mixed training strategy, allowing the model to benefit from data scaling up of mixed conditioning. This overcomes the issue that previous end-to-end approaches faced due to the scarcity of high-quality data. OmniHuman significantly outperforms existing methods, generating extremely realistic human videos based on weak signal inputs, especially audio. It supports image inputs of any aspect ratio, whether they are portraits, half-body, or full-body images, delivering more lifelike and high-quality results across various scenarios.</span></small>
<small><span><b>Currently, we do not offer services/downloads anywhere, nor do we have any SNS accounts for the project.</b></span></small>
<small><span><b>Please be cautious of fraudulent information. We will provide timely updates on future developments.</b></span></small>
<div class='responsive-image-container'>
<img src='image/overall.png' alt='' />
</div>
</div>
<div id="sections" class="column-flex">
<h3>Generated Videos</h3>
<p>
OmniHuman supports various visual and audio styles. It can generate realistic human videos <strong>at any aspect ratio and body proportion (portrait, half-body, full-body all in one)</strong>, with realism stemming from comprehensive aspects including motion, lighting, and texture details.<br/>
</p>
<p class="styled-text">
<b>*</b> Note that to generate all results on this page, <strong>only any single image and audio are required</strong>, except for the demo showcasing video and combined driving signals. For the sake of a clean layout, we have omitted the display of reference images, which are the first frame of the generated video in most cases. If you need comparisons or further information, please do not hesitate to contact us.
</p>
<div class="video-slider">
<video src="video/main1.mp4"></video>
<video src="video/main2.mp4"></video>
<video src="video/main3.mp4"></video>
</div>
<h3>Talking</h3>
<p>OmniHuman can support input of any aspect ratio in terms of speech. It significantly improves the handling of gestures, which is a challenge for existing methods, and produces highly realistic results. The audio and images for some of the test cases are sourced from <a href="https://www.youtube.com/watch?v=5Jk8qITsqdM&t=127s&ab_channel=TEDxTalks">link1</a>, <a href="https://www.youtube.com/watch?v=ITxWUu6UcWQ&t=251s&ab_channel=TEDxTalks">link2</a>, <a href="https://www.youtube.com/watch?v=oO8w6XcXJUs&ab_channel=RealTimewithBillMaher">link3</a>, <a href="https://www.youtube.com/watch?v=oO8w6XcXJUs&ab_channel=RealTimewithBillMaher">link4</a>.</p>
<div class="video-slider">
<video src="video/talk1.mp4"></video>
<video src="video/talk2.mp4"></video>
<video src="video/talk3.mp4"></video>
</div>
<h3>Diversity</h3>
<p>In terms of input diversity, OmniHuman supports cartoons, artificial objects, animals, and challenging poses, ensuring motion characteristics match each style's unique features.</p>
<div class="video-slider">
<video src="video/div1.mp4"></video>
<video src="video/div2.mp4"></video>
<video src="video/div3.mp4"></video>
</div>
<div class="video-slider">
<video src="video/div4.mp4"></video>
<video src="video/div5.mp4"></video>
<video src="video/div6.mp4"></video>
<video src="video/div7.mp4"></video>
</div>
<h3>More Halfbody Cases with Hands</h3>
<p>Here, we also provide additional examples specifically showcasing gesture movements. Some input images and audio come from TED, Pexels and AIGC.</p>
<div class="video-slider">
<video src="video/hands1.mp4"></video>
<video src="video/hands2.mp4"></video>
<video src="video/hands3.mp4"></video>
</div>
<div class="video-slider">
<!-- <video src="video/hands4.mp4"></video> -->
<video src="video/hands5.mp4"></video>
<video src="video/hands6.mp4"></video>
</div>
<h3>More Portrait Cases</h3>
<p>Here, we also include a section dedicated to portrait aspect ratio results, which are derived from test samples in the <a href="https://celebv-hq.github.io/">CelebV-HQ</a>datasets.</p>
<div class="video-slider">
<video src="video/head1.mp4"></video>
<video src="video/head2.mp4"></video>
<video src="video/head3.mp4"></video>
</div>
<div class="video-slider">
<video src="video/head4.mp4"></video>
<video src="video/head5.mp4"></video>
<video src="video/head6.mp4"></video>
</div>
<h3>Singing</h3>
<p>OmniHuman can support various music styles and accommodate multiple body poses and singing forms. It can handle high-pitched songs and display different motion styles for different types of music. Please remember to select the highest video quality. The generated video quality also highly depends on the quality of the reference image.
<div class="iframe-container">
<iframe src="https://www.youtube.com/embed/XF5vOR7Bpzs?si=8xdu-gtd9AzUd615" allowfullscreen></iframe>
<iframe src="https://www.youtube.com/embed/0cwvT-J7PcQ?si=tfnipoBtxDU0gcNH" allowfullscreen></iframe>
<iframe src="https://www.youtube.com/embed/4w2AeGHy31Q?si=DQaeCWRj0V9SRm52" allowfullscreen></iframe>
<iframe src="https://www.youtube.com/embed/1NU8NzvAxEg?si=tu3zNPGjUtdL1ag9" allowfullscreen></iframe>
</div>
<div class="iframe-container">
<iframe src="https://www.youtube.com/embed/ns8kn5NGL44?si=5-uMWr431xZamSlY" allowfullscreen></iframe>
<iframe src="https://www.youtube.com/embed/GGTcVOb2S9k?si=tLMOPd_1rOXXeO5x" allowfullscreen></iframe>
<iframe src="https://www.youtube.com/embed/CVVNK06LVac?si=VVJTt1nHcKq_xq7n" allowfullscreen></iframe>
<iframe src="https://www.youtube.com/embed/XV4E0-TVsvg?si=Q_936m0uHLW86cQk" allowfullscreen></iframe>
</div>
<h3>Compatibility with Video Driving</h3>
<p>Due to OmniHuman's mixed condition training characteristics, it can support not only audio driving but also video driving to mimic specific video actions, as well as combined audio and video driving (case is from <a href="https://www.youtube.com/watch?v=AbpNlshqtJc&ab_channel=TED">link</a>) to control specific body parts like recent methods. Below, we demonstrate these capabilities.</p>
<div class="video-slider">
<video src="video/comp.mp4"></video>
</div>
<h3>Ethics Concerns</h3>
<p>
The images and audios used in these demos are from public sources or generated by models, and are solely used to demonstrate the capabilities of this research work. If there are any concerns, please contact us (jianwen.alan@gmail.com) and we will delete it in time. The template of this webpage is based on the one from <a href="https://www.microsoft.com/en-us/research/project/vasa-1/">VASA-1</a>, and some test audios are from <a href="https://www.microsoft.com/en-us/research/project/vasa-1">VASA-1</a>,<a href="https://loopyavatar.github.io">Loopy</a>,<a href="https://cyberhost.github.io/">CyberHost</a>.
</p>
<h3>BibTeX</h3>
<p>If you find this project useful for your research, you can cite us and check out our other related works:</p>
<pre><code>
@article{lin2025omnihuman1,
title={OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models},
author={Gaojie Lin and Jianwen Jiang and Jiaqi Yang and Zerong Zheng and Chao Liang},
journal={arXiv preprint arXiv:2502.01061},
year={2025}
}
@article{jiang2024loopy,
title={Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency},
author={Jiang, Jianwen and Liang, Chao and Yang, Jiaqi and Lin, Gaojie and Zhong, Tianyun and Zheng, Yanbo},
journal={arXiv preprint arXiv:2409.02634},
year={2024}
}
@article{lin2024cyberhost,
title={CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention},
author={Lin, Gaojie and Jiang, Jianwen and Liang, Chao and Zhong, Tianyun and Yang, Jiaqi and Zheng, Yanbo},
journal={arXiv preprint arXiv:2409.01876},
year={2024}
}
</code></pre>
<br/>
<br/>
<br/>
</div>
</div>
<script src="index.js"></script>
<script>
function comming_soon_click() {
alert('Comming soon!');
}
function TBD_click() {
alert('TBD');
}
</script>
</body>
</html>