An AI-powered application that transcribes voice commands in multiple languages (like Hindi) and automatically translates them into English to generate high-quality images.
This project bridges the gap between multilingual speech and text-to-image AI. It uses OpenAI Whisper for robust speech recognition and translation, and Stable Diffusion for visual generation, all served through a clean Streamlit web interface.
- Multilingual Support: Transcribe and translate non-English speech (e.g., Hindi) into English prompts.
- CPU Optimized: Configured to run on standard hardware using the
tiny-sdmodel for faster inference. - Interactive UI: Simple web-based interface for uploading audio and viewing results in real-time.
- Auto-Translation: Seamlessly converts local languages into optimized English prompts for better AI image quality.
- Language: Python
- AI Models: OpenAI Whisper, Stable Diffusion (v1.5/Tiny)
- Libraries: PyTorch, Diffusers, Transformers, Streamlit, Scipy
- System Tools: FFmpeg (for audio processing)
- Clone the Repository
git clone [https://github.com/jagadeesh0604/voice-to-image-stable-diffusion.git](https://github.com/jagadeesh0604/voice-to-image-stable-diffusion.git) cd voice-to-image-stable-diffusion - Set up Virtual Environment
python -m venv venv .\venv\Scripts\Activate.ps1 # Windows
- Install Dependencies
pip install openai-whisper diffusers transformers torch accelerate streamlit
- Install FFmpeg Ensure FFmpeg is installed and added to your system's PATH.
📖 Usage
- Run the Streamlit application:
streamlit run src/app.py
- Upload a .wav or .mp3 audio file.
- The system will transcribe the audio, translate it if necessary, and generate a corresponding image
🖼️ Sample Result Input (Hindi Audio) Translation (English) Generated Output "Ek purana shehar jo pahadon ke beech mein hai.." "An old city, which is between the mountains, in the time of the evening...." [https://drive.google.com/file/d/1_QJvBESXn1Vbp1aESgYG_L6-QE7_3WuS/view?usp=drive_link]