Image to HTML

Turn a screenshot of a simple web layout into Bootstrap HTML with a CNN + LSTM model — a personal deep-learning experiment built on top of pix2code.

What it does

1. A convolutional encoder reads the screenshot.
2. An LSTM decoder emits tokens of a small domain-specific language (DSL) that describes the layout (header, row, btn-active, …).
3. A compiler maps the DSL to Bootstrap HTML using compiler/assets/web-dsl-mapping.json.
4. A Gradio interface wraps the whole pipeline so you can drop in an image and copy the generated code.

screenshot ──► CNN encoder ──► LSTM decoder ──► DSL tokens ──► compiler ──► HTML

The model is the pix2code architecture with the visual encoder replaced by a pre-trained convolutional autoencoder (following Taneem Jan's variant), which trains faster and generalises a little better on the web dataset.

Quick start

git clone https://github.com/jalakhras/image-to-html.git
cd image-to-html
pip install -r requirements.txt

# pre-trained weights (~570 MB) are published as a GitHub release, not in git
python scripts/download_weights.py

python app.py          # opens the Gradio UI at http://127.0.0.1:7860

Try the screenshots in examples/ first — they come from the pix2code web dataset the model was trained on.

Training your own model

python scripts/download_datasets.py                       # pix2code datasets (web / iOS / Android)
python training/build_datasets.py training/datasets/web/all_data
python training/convert_imgs_to_arrays.py training/datasets/web/training_set training/datasets/web/training_features
python training/train.py training/datasets/web/training_features bin 1

training/ is the upstream pix2code training code with two small changes made for this experiment: integer-safe train/eval splitting in build_datasets.py (Python 3) and lighter defaults in Config.py (IMAGE_SIZE = 64, EPOCHS = 5) so a run fits on a laptop GPU.

Project layout

app.py                     Gradio interface
main_program.py            inference pipeline: image -> DSL -> HTML
classes/                   model definition, sampler, vocabulary
classes/model/bin/         vocabulary + model JSON (weights downloaded separately)
compiler/                  DSL -> HTML compiler and Bootstrap mapping
training/                  pix2code training pipeline (dataset build, features, train, sample)
scripts/                   download helpers for weights and datasets
examples/                  sample screenshots

Limitations

• Trained only on the synthetic pix2code web dataset — it recognises the handful of Bootstrap components in that DSL, not arbitrary designs.
• Output is structure, not content: text is filled with placeholders.
• Greedy decoding only; beam search from upstream is available in training/classes/BeamSearch.py if you want to experiment.

Credits

• pix2code — Tony Beltramelli, paper · code (Apache-2.0). The DSL, compiler, datasets and training pipeline come from there.
• Autoencoder variant and Gradio demo — Taneem Jan. This repository adapts that demo; original author headers are kept in the files concerned.
• Everything else (repo consolidation, English UI, download scripts, training tweaks) is my own experimentation.

License

Apache-2.0, following pix2code. See LICENSE.