Skip to content

Add TPU backend for general_perf and llm_perf#119

Open
Yi-sir wants to merge 7 commits into
bytedance:mainfrom
Yi-sir:main
Open

Add TPU backend for general_perf and llm_perf#119
Yi-sir wants to merge 7 commits into
bytedance:mainfrom
Yi-sir:main

Conversation

@Yi-sir
Copy link
Copy Markdown

@Yi-sir Yi-sir commented Nov 6, 2024

  1. 提交general_perf TPU后端,包含编译、运行时及README。除TPU后端外,还包括如下改动:
    1.1 general_perf/backends/CPU/calculate_cpu_diff.sh 判断后端是否为TPU,若是则跳过venv
    1.2 general_perf/launch.py 调用上述sh时增加传参hardware_type
    1.3 general_perf/launch.py 当compile_only为true时,修复传参错误导致解析workload失败的问题
    1.4 general_perf/core/perf_engine.py 修改了更安全的获取cpu型号代码。旧代码在某些情况下获取失败时会导致程序崩溃
    1.5 general_perf/backends/CPU/ 后端文件将import torch提到import tensorflow之前,因为在sophgo/tpuc_dev中按照旧顺序import时进程会卡死
    1.6 general_perf/backends/runtime_backend.py 修复了load()方法的注释
    目前只测试了fp32 1b的resnet50和yolov5,后续会补充测试内容及完善README。
  2. 提交了llm_perf的TPU后端,包含调度部分,暂未包含modeling_xxx.py。模型部分近期将提交。

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Nov 6, 2024

CLA assistant check
All committers have signed the CLA.

@Yi-sir
Copy link
Copy Markdown
Author

Yi-sir commented Nov 13, 2024

  1. 支持了resnet50-torch-fp32 int8量化,需要交互设置量化参数。支持双芯异步perf

@Yi-sir
Copy link
Copy Markdown
Author

Yi-sir commented Mar 10, 2025

75eb268 这笔增加了kv cache,但需要修改transformers里的一些代码才能运行。
image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants