UI-Agent Btganation

This repo will contain our dataset, finetuning and any btganation experiments.

How to use

Dataset Prep

• make data dirctory which contains 4 main dirs

  • raw: the raw TFRecords downloaded from https://console.cloud.google.com/storage/browser/gresearch/android_control
  • episodes: each episodes cosists of multiple steps to achieve some goal.
  • images: contains all the screenshots of all the episodes steps.
  • samples: the flattened final steps

• in data_prep/extract_episodes.py, change TFRECORD_PATH to the one you've

• in data/flatten_episodes.py, change output name to the required one

• run verify_dataset.py for verification

Note: if we're using another dataset, we'll map it to this format

Datasets Utils

• now there's only android_control.py
• it's a torch dataset implementation to convert raw samples -> model tensors

models_raw

• create this dirctory to include any local model you've

models_wrappers

• this contains an abstract class base_model.py that we should implement for every model we try

output

• this contains the PEFT adapters outputs

internvl (later there will be a dir for every model we try)

• this contains the training/fine-tuning process of the model

config.py

• contains some important paths, remember to set them

How to install requirements

• pip install -r requirements.txt

How to run

• python -m internvl.train
• python -m internvl.test
• running the apk planner: python run_planner.py --apk apk_src/clock.apk --dry-run