Physicalai/genesis

README.md

@@ -27,8 +27,8 @@ to embodied intelligence and deployment:
     <tr>
       <td rowspan="3"><strong>Physical Simulation</strong></td>
       <td><a href="projects/Physical-AI/Physical-Simulation/Genesis-Simulation/">Genesis Simulation</a></td>
-      <td>Load and control a Franka Panda, tune PD controllers, solve inverse kinematics, execute pick-and-place, and scale to parallel GPU environments.</td>
-      <td><img src="assets/readme/genesis.gif" alt="Genesis parallel robot simulation" width="240"></td>
+      <td>Progress from Franka control, inverse kinematics, and parallel GPU simulation to ROCm vision and tactile perception, then build a guarded language-guided agent with an interactive live HUD and reproducible scene layouts.</td>
+      <td><img src="assets/readme/gs06.gif" alt="Genesis language-guided Physical AI agent" width="240"></td>
     </tr>
     <tr>
       <td><a href="projects/Physical-AI/Physical-Simulation/Mujoco-Simulation/mujoco-torch/">MuJoCo + PyTorch</a></td>
@@ -88,9 +88,12 @@ teaching content was made possible through the joint efforts of these partners.
 | Nanjing University | [Prof. Jingwei Xu](https://njudeepengine.github.io/jingweixu/), [NJUDeepEngine](https://github.com/NJUDeepEngine) | LLM |
 | National Yang Ming Chiao Tung University | [Prof. Ping-Chun Hsieh](https://pinghsieh.github.io/), [Reinforcement Learning and Bandits Lab](https://pinghsieh.github.io/group.html) | Physical AI, Reinforcement Learning on MuJoCo |
 
-We also thank the open-source projects that make these labs possible,
-including [Genesis](https://github.com/Genesis-Embodied-AI/Genesis) and
-[MuJoCo](https://github.com/google-deepmind/mujoco).
+We also thank the AMD AECG team for contributing portions of the Physical AI
+teaching materials, along with the open-source projects that make these labs
+possible, including [Genesis](https://github.com/Genesis-Embodied-AI/Genesis)
+and [MuJoCo](https://github.com/google-deepmind/mujoco). Detailed source
+attributions and links to the original repositories are provided in each
+relevant notebook.
 
 ## License
 

README.zh-TW.md

@@ -23,8 +23,8 @@
     <tr>
       <td rowspan="3"><strong>物理模擬</strong></td>
       <td><a href="projects/Physical-AI/Physical-Simulation/Genesis-Simulation/">Genesis Simulation</a></td>
-      <td>載入並控制 Franka Panda、調整 PD 控制器、求解逆向運動學、執行夾取與放置，並擴展至 GPU 平行模擬環境。</td>
-      <td><img src="assets/readme/genesis.gif" alt="Genesis 平行機器人模擬" width="240"></td>
+      <td>從 Franka 控制、逆向運動學與 GPU 平行模擬，進階至 ROCm 視覺與觸覺感知，最後建立具備互動式即時 HUD、可重現場景配置與安全驗證流程的語言引導代理。</td>
+      <td><img src="assets/readme/gs06.gif" alt="Genesis 語言引導實體 AI 代理" width="240"></td>
     </tr>
     <tr>
       <td><a href="projects/Physical-AI/Physical-Simulation/Mujoco-Simulation/mujoco-torch/">MuJoCo + PyTorch</a></td>
@@ -77,7 +77,7 @@ AUP 感謝以下大學、教授與實驗室的共同投入，讓這些教學內
 | 南京大學 | [徐经纬教授](https://njudeepengine.github.io/jingweixu/)、[NJUDeepEngine](https://github.com/NJUDeepEngine) | LLM |
 | 國立陽明交通大學 | [謝秉均教授](https://pinghsieh.github.io/)、[Reinforcement Learning and Bandits Lab](https://pinghsieh.github.io/group.html) | 實體 AI、MuJoCo 強化學習 |
 
-我們也感謝支援這些課程的開源專案，包括 [Genesis](https://github.com/Genesis-Embodied-AI/Genesis) 與 [MuJoCo](https://github.com/google-deepmind/mujoco)。
+我們也感謝 AMD AECG 團隊提供部分實體 AI 教學素材，以及支援這些課程的開源專案，包括 [Genesis](https://github.com/Genesis-Embodied-AI/Genesis) 與 [MuJoCo](https://github.com/google-deepmind/mujoco)。各項素材的詳細來源說明與原始 repository 連結，皆附於相關 notebook 中。
 
 ## 授權
 

projects/Physical-AI/Physical-Simulation/Genesis-Simulation/.dockerignore

@@ -0,0 +1,8 @@
+.git
+.ipynb_checkpoints
+Artifacts
+Videos
+models
+**/__pycache__
+*.pyc
+*.gguf

projects/Physical-AI/Physical-Simulation/Genesis-Simulation/.gitignore

@@ -0,0 +1,7 @@
+.ipynb_checkpoints/
+Artifacts/
+Videos/
+models/
+helpers/__pycache__/
+*.pyc
+*.gguf

projects/Physical-AI/Physical-Simulation/Genesis-Simulation/Dockerfile

@@ -18,12 +18,31 @@
 # SOFTWARE.
 
 ARG BASE_IMAGE=ghcr.io/amdresearch/auplc-base:latest
+ARG LLAMA_CPP_REF=8e7f22b67ef4667b4ddd50230771287f328cfb3f
+FROM ubuntu:24.04 AS llama-builder
+
+ARG LLAMA_CPP_REF
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential ca-certificates cmake git glslc libvulkan-dev spirv-headers \
+    && rm -rf /var/lib/apt/lists/*
+RUN git clone --filter=blob:none https://github.com/ggml-org/llama.cpp.git /src/llama.cpp && \
+    git -C /src/llama.cpp checkout "${LLAMA_CPP_REF}" && \
+    cmake -S /src/llama.cpp -B /src/llama.cpp/build \
+      -DCMAKE_BUILD_TYPE=Release \
+      -DGGML_VULKAN=ON \
+      -DGGML_NATIVE=OFF \
+      -DLLAMA_CURL=OFF \
+      -DBUILD_SHARED_LIBS=OFF && \
+    cmake --build /src/llama.cpp/build --target llama-server --parallel
+
 FROM ${BASE_IMAGE}
 
 ARG GENESIS_WORLD_VERSION=1.3.1
 
 ENV ROCM_PYTHON_LIB=/opt/rocm-python/lib
 ENV LD_LIBRARY_PATH="${ROCM_PYTHON_LIB}:${LD_LIBRARY_PATH}"
+ENV LLAMA_SERVER_BIN=/opt/llama/bin/llama-server
+ENV LLAMA_MODEL_PATH=/opt/workspace/PhySim/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf
 
 USER root
 RUN SDK_LIB=$(python3 -c "import _rocm_sdk_core, os; print(os.path.join(os.path.dirname(_rocm_sdk_core.__file__), 'lib'))") && \
@@ -35,13 +54,21 @@ RUN SDK_LIB=$(python3 -c "import _rocm_sdk_core, os; print(os.path.join(os.path.
 RUN apt-get update && apt-get install -y --no-install-recommends \
     ffmpeg \
     libgl1 libglx-mesa0 libgl1-mesa-dri libegl1 libgbm1 libglib2.0-0 \
+    libvulkan1 mesa-vulkan-drivers \
     && rm -rf /var/lib/apt/lists/*
 
 RUN pip3 install --no-cache-dir \
+    "huggingface-hub>=0.34,<2" \
+    "ipywidgets>=8,<9" \
     "numpy>=1.26.4,<2.4" \
+    "opencv-python>=4.10,<6" \
+    "requests>=2.31" \
+    "tqdm>=4.66" \
     "genesis-world==${GENESIS_WORLD_VERSION}"
 
-COPY --chown=jovyan:1000 PhySim*.ipynb /opt/workspace/PhySim/
+COPY --from=llama-builder /src/llama.cpp/build/bin/llama-server /opt/llama/bin/llama-server
+COPY --chown=jovyan:1000 GS*.ipynb /opt/workspace/PhySim/
+COPY --chown=jovyan:1000 helpers /opt/workspace/PhySim/helpers
 COPY --chown=jovyan:1000 xml /opt/workspace/PhySim/xml
 
 USER jovyan