diff --git a/.env b/.env new file mode 100644 index 0000000..aa8bfdc --- /dev/null +++ b/.env @@ -0,0 +1,3 @@ +SCANNER_HOST_OUTPUT_DIR=/opt/EntoScan +LABEL_MAKER_URL=https://darsa.info/EntoScan-labels/ +USE_MOCK_SCANNER=1 \ No newline at end of file diff --git a/README.md b/README.md deleted file mode 100644 index 3ecd7ff..0000000 --- a/README.md +++ /dev/null @@ -1,209 +0,0 @@ - -# EntoScan Documentation - - - -**Table of Contents** - -- [Hardware](#hardware) - - [Scanner](#scanner) - - [3D printed files](#3d-printed-files) - - [`Well-plate-holder`:](#well-plate-holder) - - [`Barcode-stemp`](#barcode-stemp) - - [`Custom-lid-white`](#custom-lid-white) - - [`Drying-rack`](#drying-rack) -- [Software](#software) - - [Data structure](#data-structure) -- [Usage](#usage) - - [Materials:](#materials) - - [Setup](#setup) - - [Software modifications](#software-modifications) - - [Prepare Datasheets](#prepare-datasheets) - - [Use the scanner](#use-the-scanner) - - [Cleaning](#cleaning) -- [Additionals](#additionals) - - [Create labels](#create-labels) - - [Drying and weighing the specimens](#drying-and-weighing-the-specimens) - - [Image petridishes](#image-petridishes) - - [Imaging datasheets and plates](#imaging-datasheets-and-plates) - - [Digitalization of the datasheets](#digitalization-of-the-datasheets) - - - -# Hardware - -## Scanner -A flatbed scanner was modified in a way to have a cutout in the scanning area in order to have a fixed position for the well plate and to lower the scanned objects to have them in focus. -**All modifications was done with the scanner unplugged from power source and USB Cable and in dust free conditions.** -- `Epson Perfection V850 Pro`: The 4 main screws on the scanning surface where removed to open the scanner. The sticked glass plate was gently removed with a spatula. Remaining glue was removed as much as possible -- `Aluminum plate with cutout`; Dimensions of the original glass plate (250x378x3.3mm W/H/D) and a cutout (135x95mm W/H) in the scanning area replaced the glass plate (see image). **IMPORTANT** Art the top of the aluminum plate must be attached a stripe of white paper as it is probably a calibration reference for the scanner (see image). -- `Double sided tape` to attach the aluminum plate. -## 3D printed files -Materials: -- PLA Filament -- 3D Printer -- 1mm glass plate (134mmx94mm) -- Two-Part Epoxy Adhesives - -Optional for drying rack: -- ASA Filament - - -### `Well-plate-holder`: -- File: [well_plate_holder.stl](printing_files/well_plate_holder.stl) -1. Printed with PLA Filament (Dimensions: 145mm x 105mm) -2. glue 1mm glass plate with size of 134mm x 94mm to the bottom of the holder - - -### `Barcode-stemp` - -- File: [barcode_stamp.stl](printing_files/barcode_stamp.stl) Onshape: [Onshape: barcode_stamp.stl](https://cad.onshape.com/documents/c7decab2f4ab9ea7045bb338/w/21b260deb17a1b2bc026efe8/e/9caeb74d3272a08aeae81f82?renderMode=0&uiState=68653414abc77a5fb9d01293) -- Print with PLA Filament (Dimensions: Ø14mm) - -### `Custom-lid-white` -- File: [custom_lid.stl](printing_files/custom_lid.stl) -- Print with `white` PLA Filament (Dimensions: 127mm x 85mm) - - -### `Drying-rack` - -- File: [drying_rack.stl](printing_files/drying_rack.stl)` -- printed with ASA Filament to prevent bending during drying process - - -# Software - -## Data structure - -In the defined data directory the following folders and files will be created: -- `FOLDER:LABEL` for every scanned plate in the format of the label (on the barcode). usually in the format "YEAR-MONTH-DAY_N" where N is a counter of the different plates for the same day -- `FOLDER:drymass` for every scanned plate a blank CSV file will be created where the biomasses per insects and comments can be filled in. -- `FOLDER:jpeg`: a flat folder with all scanned wells in the format `YEAR-MONTH-DAY_N_WELL_TIMESTAMP.jpeg` e.g. 2025-06-13_004_B6_13-41-56.jpeg -- `FOLDER:pdfs`: (created manually) save imaged datasheets and plates see [imaging-datasheets-and-plates](#imaging-datasheets-and-plates) -- `FILE:plates.csv/json`: Store the metadata of the performed scans -- `FILE:metadata.csv/_bz.csv`: Will be crated by running the command `python3 create_metadata.py` and list the drymass of every file in the jpeg folder. The drymasses must be entered before in the CSV files in the `drymass`-folder. - - -# Usage - -This protocol covers the workflow of imaging and weighting insects individually and store the data to use it later on for the prediciton of biomass based on the images of the insects. - -## Materials: -- Modified [flatbed scanner](#scanner) with [well-plate-holder](#well-plate-holder) -- Well plates 6x4 (24 wells) approx 10 pcs. -- [Barcode stemps](#barcode-stemp) (4 pcs.) -- Insect specimens in ethanol -- Tweezers -- Label paper (see [labels](#create-labels)) -- [white custom lid](#custom-lid-white) (2-3 pcs.) - -## Setup -### Software modifications -**Adjust exact position of wellplate** -The cutout in the aluminum plate will be probably not on the exact same position, so the scanned area must be adjusted in the software. To do so the following two variables of [entoscan.py](entoscan.py) can be adjusted: -``` -PLATE_OFFSET_L = 54 #Top right X axis in mm -PLATE_OFFSET_T = 12 #Top right Y axis in mm -``` -- To check if the location is right, place the wellplate on the wellplate holder and check the scanning area with the following command: -``` -python3 entoscan.py --check-scanning-area -``` - -**Define data directory** - -All the output will be stored in the same data directory. -The data directory is defined globally in the variable `DATA_DIR_PATH` in the file [data_handler.py](data_handler.py): -``` -DATA_DIR_PATH = os.path.join("/home/entoscan/data") -``` - -## Prepare Datasheets -Files (located in `tools/datasheets`): -- [weights.pdf](tools/datasheets/weights.pdf) -- [drying_oven.pdf](tools/datasheets/drying_oven.pdf) - -`weights.pdf` -- For each scanned plate fill out the header (Label and stick the barcode to it) -- Note down comments during the scanning / weighting process for each plate - -`drying_oven.pdf` -- For noting down when at which time which plates was put in the drying oven - - -## Use the scanner -**Insect Handling** -1. `Give the wellplate an identity and orientation`: Add one label to the `barcode-stemp` and put it in the bottom right well of the wellplate (well C6) -2. Pick 23 insect specimen and place them in the center of the remaining wells each. - -**Use Entoscan** -1. Close the prepared wellplate with the [`white custom lid`](#custom-lid-white) -2. Place the wellplate on the wellplate holder -3. Run `python3 entoscan.py` in the terminal - - **duplicates:**:In order to scan each specimen multiple times the command under 3. can be re runned with the same plate. (Already taken scans will be not overwritten as the files are saved with the time as a suffix) - - Take the plate out, move the specimen with the tweezer a bit, and putt the wellplate back at the wellplate holder - -## Cleaning - -# Additionals -## Create labels -This Guide applies if Label paper from avery-zweckform.eu are used. -Round labels Ø10mm are used with the code **"L6019REV-25"** https://www.amazon.de/Zweckform-L6019REV-25-Removable-Numbering-Inscribable/dp/B0002S4998?language=en_GB¤cy=EUR&th=1 -If other labes are used the file can be changed accordingly - -File: [label.typ](tools/label.typ) -- Add one line to variable `labels` in the format: (DATE, UMBER_OF_PLATES, REPLICATIONS, FORCE_PRINT (false)), where -- `DATE` is the current data in the format YYYY-MM-DD and it will be checked during scanning. -- `NUMBER_OF_PLATES` is the number different plates you will scan on this day. A number of 3 will produces labels with the suffix 001, 002 and 003 -- `REPLICATIONS` is the number of replications of each label. -- `FORCE_PRINT`: Boolean true or false (default). By default only the last entry of the `labels` array will be printed. If this option is set to true, this item will be showed as well. - - -Run locally: -- Install [typst](https://github.com/typst/typst?tab=readme-ov-file#installation) -- Run "typst compile tools/label.typ" -- Or install Tinymist extension for your code editor and run it from there. https://github.com/Myriad-Dreamin/tinymist?tab=readme-ov-file#installation - -Run it in the Official Typst Web App: -- Open https://typst.app/, create an account, Start a new project with an empty document. -- Copy the code from [label.typ](tools/label.typ) to the Webeditor -- Click on the "Quick export PDF" button - -Printing: -1. Make a test print (The vertical and horizontal offset can vary to a big extend depending on the printer, and the printing settings) -2. Print it on the label paper - -## Drying and weighing the specimens - -## Image petridishes -In order to image petridishes the label must either have a "petri" prefix or the the argument `--petridish` must be set. -Examples: -``` -python3 entoscan.py --petridish -``` -``` -python3 entoscan.py --bypassed-label="petri_2025-07-" -``` - -## Imaging datasheets and plates - -To image the **datasheets** and **plates** I used an app called Adobe Scan, which is available for Android and IOS https://www.adobe.com/acrobat/mobile/scanner-app.html. -**Settings for the datasheets:** -- Filters: Mark "Apply to all pages" and then choose "Original color" -- Resize "A4 portrait" - -**Settings for plates:** -- Filters: Mark "Apply to all pages" and then choose "Original color" - -All scans can be saved in the data direcetory in a folder called `pdfs` - - -## Digitalization of the datasheets - -For Digitalization of the datasheets I used an online tool [https://www.pen-to-print.com/App/tables/] which convert the imaged table to a digital table. - -In most cases it dedects everything correctly but its very important to check for the following things: -- **Double Check if it dedected the numbers correctly!!** -- Spaces after the dot (0. 48 instead of 0.48) -- Commas instead of dots as decimal seperator (0,48 instead of 0.48) - diff --git a/README.md b/README.md new file mode 120000 index 0000000..d36017b --- /dev/null +++ b/README.md @@ -0,0 +1 @@ +services/webapp/app/www/about.md \ No newline at end of file diff --git a/build_and_push_docker_services.sh b/build_and_push_docker_services.sh new file mode 100644 index 0000000..2cec490 --- /dev/null +++ b/build_and_push_docker_services.sh @@ -0,0 +1,27 @@ +#docker buildx create --use + +INSECT_SCANNER_API_VERSION=0.1.0 +INSECT_SCANNER_WEBAPP_VERSION=0.1.0 +DOCKERHUB_USER=qgeissmann + +# API +docker buildx build \ + --platform linux/amd64 \ + -f services/scanner-api/Dockerfile \ + -t ${DOCKERHUB_USER}/insect-scanner-api:${INSECT_SCANNER_API_VERSION} \ + -t ${DOCKERHUB_USER}/insect-scanner-api:latest \ + services/scanner-api \ + --push + +# Webapp +docker buildx build \ + --platform linux/amd64 \ + -f services/webapp/Dockerfile \ + -t ${DOCKERHUB_USER}/insect-scanner-webapp:${INSECT_SCANNER_WEBAPP_VERSION} \ + -t ${DOCKERHUB_USER}/insect-scanner-webapp:latest \ + services/webapp \ + --push + +## Run local containers to prototype +# sudo docker compose -f docker-compose-local.yaml up --force-recreate --build + diff --git a/create_metadata.py b/create_metadata.py deleted file mode 100644 index ecc1558..0000000 --- a/create_metadata.py +++ /dev/null @@ -1,91 +0,0 @@ -#%% -import pandas as pd -import os -import data_handler as dh - -""" -This script combines all the drymass metadata to one single file. -It can be modified so that it for example skips all entrys with comments. -Currently it filters only drymasses of wells "D6". - -""" -BASE_DIR = dh.DATA_DIR_PATH - -IMAGE_DIR = os.path.join(BASE_DIR, "jpeg") - -EXPORT_CSV = os.path.join(BASE_DIR, "metadata_drymass.csv") -EXPORT_CSV_BZ = os.path.join(BASE_DIR, "metadata_drymass_bz.csv") - - - -def list_all_files(directory_path): - """ - Lists all files in a given directory. - - Args: - directory_path (str): The path to the directory. - - Returns: - list: A list of filenames in the directory. - Returns an empty list if the directory doesn't exist - or if an error occurs. - """ - try: - if not os.path.isdir(directory_path): - print(f"Error: '{directory_path}' is not a valid directory.") - return [] - for f in os.listdir(directory_path): # f means "file" - if not f.startswith('.'): # ignore hidden files - yield f - except FileNotFoundError: - print(f"Error: Directory '{directory_path}' not found.") - return [] - except Exception as e: - print(f"An error occurred: {e}") - return [] - -if __name__ == "__main__": - image_files = sorted(list(list_all_files(IMAGE_DIR))) - current_plate = "" - drymass_df = pd.DataFrame() - METADATA_DF = pd.DataFrame() - - for index, file in enumerate(image_files): - # for testing the following two lines can be out commented - # if index >= 30: - # break - name_part, extension = os.path.splitext(file) - parts = name_part.split('_') - if len(parts) != 4 and extension != ".jpeg": - continue - - plate_id = parts[0] + "_" + parts[1] - well = parts[2] - timestamp = parts[3] - insect_id = plate_id + "_" + well - - # skip D6 walls - if well == "D6": - continue - - if current_plate is not plate_id: - drymass_csv = dh.get_drymass_csv_path(plate_id) - drymass_df = pd.read_csv(drymass_csv, index_col='INSECT_ID') - current_plate = plate_id - - drymass = drymass_df.loc[insect_id, 'DRYMASS_MG'] - general_comments = drymass_df.loc[insect_id, 'COMMENTS'] - - insect_metadata = pd.DataFrame({ - 'IMAGE_FILENAME': [file], - 'DRYMASS_MG': [drymass], - 'COMMENTS': [general_comments]}) - - METADATA_DF = pd.concat([METADATA_DF, insect_metadata], ignore_index=True) - - - METADATA_DF.to_csv(EXPORT_CSV, index=False) - METADATA_DF.to_csv(EXPORT_CSV_BZ, index=False, compression='bz2') - - -# %% diff --git a/data_handler.py b/data_handler.py deleted file mode 100644 index 9ee2e0e..0000000 --- a/data_handler.py +++ /dev/null @@ -1,145 +0,0 @@ -#%% -import pandas as pd -import os - -# DATA DIRECTORY -DATA_DIR_PATH = os.path.join("/media/robert/Extreme SSD/entoscan/data") -USER_SCANNING = "Robert Thalheim" -SILICA_GEL_USED = True - -PLATES_CSV_PATH = os.path.join(DATA_DIR_PATH, "plates.csv") -PLATES_JSON_PATH = os.path.join(DATA_DIR_PATH, "plates.json") - - -#%% -def add_plate_to_database(plate_id, timestamp, plate_type, scan_resolution, scan_depth, scan_brightness, scan_date, scan_repetition) -> None: - # Read the csv file - try: - PLATE_DF = pd.read_csv(PLATES_CSV_PATH) - except FileNotFoundError: - PLATE_DF = pd.DataFrame() - - # Create a new row to add - new_plate = pd.DataFrame({ - 'PLATE-ID': [plate_id], - 'TIMESTAMP': [timestamp], - 'PLATE_TYPE': [plate_type], - 'SCAN_RESOLUTION': [scan_resolution], - 'SCAN_DEPTH': [scan_depth], - 'SCAN_BRIGHTNESS': [scan_brightness], - 'SCAN_DATE': [scan_date], - 'SCAN_REPETITION': [scan_repetition], - 'SILICA_GEL_USED': [SILICA_GEL_USED], - 'USER_SCANNING': [USER_SCANNING], - 'USER_WEIGHTING': '', - 'SAMPLE_ID': '', - 'SAMPLE_NAME': '', - 'COMMENTS': '' - }) - - # Append to existing DataFrame - PLATE_DF = pd.concat([PLATE_DF, new_plate], ignore_index=True) - - # Save back to CSV and JSON - PLATE_DF.to_csv(PLATES_CSV_PATH, index=False) - PLATE_DF.to_json(PLATES_JSON_PATH, orient='records', indent=4) - - print(f'Plate {plate_id} added in the repetition {scan_repetition}.') - -def get_drymass_csv_path(plate_id) -> str: - return os.path.join(DATA_DIR_PATH, "drymass", f'{plate_id}.csv') - -def create_blank_drymass_csv(plate_id: str) -> None: - """ - Create a blank CSV file for a given plate_id with predefined columns for biomass and comments. - The CSV file is saved in the 'insects' subdirectory of the DATA_DIR_PATH. - """ - csv_path = get_drymass_csv_path(plate_id) - - # create directory - os.makedirs(os.path.dirname(csv_path), exist_ok=True) - - columns = [ - 'INSECT_ID', - 'DRYMASS_MG', - 'COMMENTS' - ] - - data = [] - plate_rows = ['A', 'B', 'C', 'D'] - plate_cols = range(1, 7) # 1 bis 6 - - for row_char in plate_rows: - for col_num in plate_cols: - well_id = f"{row_char}{col_num}" - insect_id = f"{plate_id}_{well_id}" - data.append({col: (insect_id if col == 'INSECT_ID' else None) for col in columns}) - - df = pd.DataFrame(data, columns=columns) - - df.to_csv(csv_path, index=False) - print(f"Create blank csv-file {plate_id}.csv in: {csv_path}") - -def get_image_path(plate_label, timestamp, well, type) -> str: - - IMAGE_NAME = f'{plate_label}_{well}_{timestamp}' - match type: - case "thumbnail" : - image_save_path = os.path.join(DATA_DIR_PATH, - plate_label, - well, - IMAGE_NAME + ".thumbnail.jpeg[Q=95]",) - - case "tiff" : - image_save_path = os.path.join(DATA_DIR_PATH, - plate_label, - well, - IMAGE_NAME + ".tiff") - case "jpeg" : - image_save_path = os.path.join(DATA_DIR_PATH, - "jpeg", - IMAGE_NAME + ".jpeg") - - return image_save_path - -def get_data_directory() -> str: - return DATA_DIR_PATH - -def get_plate_csv_path() -> str: - return PLATES_CSV_PATH - -def get_root_plate_path(plate_label) -> str: - - return os.path.join(DATA_DIR_PATH, plate_label) - - -#%% - -# def resize(filein, fileout, maxw, maxh) -> None: -# """function for resizing image without loading it in the RAM, -# copied from: https://github.com/libvips/pyvips/wiki/example-resize """ -# out = pyvips.Image.thumbnail(filein, maxw, height=maxh) -# out.write_to_file(fileout) - - -# def create_jpeg_files(label): - -# destination_path = os.path.join(DATA_DIR_PATH, "jpeg") - -# plate_rows = ['A', 'B', 'C', 'D'] -# plate_cols = range(1, 7) # 1 to 6 - -# for row_char in plate_rows: -# for col_num in plate_cols: -# well_id = f"{row_char}{col_num}" -# data_dir = f'{DATA_DIR_PATH}/{label}/{well_id}' -# all_files = os.listdir(data_dir) - -# for file in all_files: - -# name_part, extension = os.path.splitext(file) -# if extension == ".tiff": -# resize(data_dir + '/' + file, destination_path + '/' + name_part + ".jpeg[Q=100]", maxw = 2048, maxh = 2048) -# create_jpeg_files("2025-05-07_001") - -# %% diff --git a/docker-compose-local.yaml b/docker-compose-local.yaml new file mode 100644 index 0000000..54e93ce --- /dev/null +++ b/docker-compose-local.yaml @@ -0,0 +1,78 @@ +version: "3.9" + +services: + api: + build: + context: ./services/scanner-api + dockerfile: Dockerfile + image: insect-scanner-api:latest + container_name: insect-scanner + environment: + - SCANNER_OUTPUT_DIR=/data + - PG_DSN=postgresql://postgres:postgres@db:5432/postgres?connect_timeout=5 + env_file: + - .env + ports: ["8000:8000"] + volumes: + - ${SCANNER_HOST_OUTPUT_DIR}:/data + - /run/udev:/run/udev:ro + - /etc/machine-id:/etc/host_machine_id:ro + devices: + - /dev/bus/usb:/dev/bus/usb + user: "0:0" + privileged: true + restart: unless-stopped + depends_on: + db: + condition: service_healthy + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8000/healthz || exit 1"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 15s + + webapp-shiny: + build: + context: ./services/webapp + dockerfile: Dockerfile + image: insect-scanner-webapp:latest + container_name: insect-scanner-webapp + environment: + - API_BASE=http://api:8000 + - DB_HOST=db + - DB_PORT=5432 + - DB_NAME=postgres + - DB_USER=postgres + - DB_PASS=postgres + - PUBLIC_API_BASE=http://localhost:8000 + - LABEL_MAKER_URL=${LABEL_MAKER_URL} + env_file: + - .env + ports: ["8080:8080"] + depends_on: + api: + condition: service_healthy + restart: unless-stopped + db: + image: postgres:16 + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: postgres + networks: + default: + aliases: + - postgres + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres -d postgres"] + interval: 2s + timeout: 2s + retries: 30 + volumes: +# - ${SCANNER_HOST_OUTPUT_DIR}/db:/var/lib/postgresql/data + - pgdata:/var/lib/postgresql/data + ports: + - "5432:5432" +volumes: + pgdata: diff --git a/docker-compose.yaml b/docker-compose.yaml new file mode 100644 index 0000000..8f0f1c4 --- /dev/null +++ b/docker-compose.yaml @@ -0,0 +1,73 @@ +version: "3.9" + +services: + api: + image: qgeissmann/insect-scanner-api:0.1.0 # or :latest + container_name: insect-scanner + environment: + - SCANNER_OUTPUT_DIR=/data + - PG_DSN=postgresql://postgres:postgres@db:5432/postgres?connect_timeout=5 + env_file: + - .env + ports: ["8000:8000"] + volumes: + - ${SCANNER_HOST_OUTPUT_DIR}:/data + - /run/udev:/run/udev:ro + - /etc/machine-id:/etc/host_machine_id:ro + devices: + - /dev/bus/usb:/dev/bus/usb + user: "0:0" + privileged: true + restart: unless-stopped + depends_on: + db: + condition: service_healthy + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8000/healthz || exit 1"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 15s + # Optional: always pull newest tag at 'up' + pull_policy: always + + webapp-shiny: + image: qgeissmann/insect-scanner-webapp:0.1.0 # or :latest + container_name: insect-scanner-webapp + environment: + - API_BASE=http://api:8000 + - DB_HOST=db + - DB_PORT=5432 + - DB_NAME=postgres + - DB_USER=postgres + - DB_PASS=postgres + - PUBLIC_API_BASE=http://localhost:8000 + - LABEL_MAKER_URL=${LABEL_MAKER_URL} + ports: ["8080:8080"] + depends_on: + api: + condition: service_healthy + restart: unless-stopped + + db: + image: postgres:16 + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: postgres + networks: + default: + aliases: + - postgres + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres -d postgres"] + interval: 2s + timeout: 2s + retries: 30 + volumes: + - pgdata:/var/lib/postgresql/data + ports: + - "5432:5432" + +volumes: + pgdata: diff --git a/entoscan.py b/entoscan.py deleted file mode 100644 index ed954f8..0000000 --- a/entoscan.py +++ /dev/null @@ -1,532 +0,0 @@ -import argparse -import cv2 -import datetime as dt -import glob #Find paths/files with a specific pattern (in this case all *.tiff files) -import io -import logging #in this script the levels warning, error and info will be used -import os -import subprocess #to ececute terminal line commands -import pyvips #Manipulate images efficiantly without loading it in the RAM -import pandas as pd -import json - - -import data_handler as dh # custom data handler (data_handler.py) -from PIL import Image #for showing images -#from qreader import QReader if switching back to qr codes this library can be used -from pylibdmtx.pylibdmtx import decode #to decode data matrices - -log = logging.getLogger(__name__) - -""" Default scan variables """ - -SCAN_DEVICE_NAME = "epkowa" - -SCAN_FORMAT_PREVIEW = "png" -SCAN_DEPTH_PREVIEW = 8 - -SCAN_FORMAT_HIGH_RESOLUTION = "tiff" -SCAN_DEPTH_HIGH_RESOLUTION = 16 - -SCAN_FORMAT_PETRIDISH = "jpeg" -SCAN_DEPTH_PETRIDISH = 8 - -SCAN_RESOLUTION_BARCODE = 300 -SCAN_RESOLUTION_PETRIDISH = 6400 -SCAN_RESOLUTION_PLATE = 6400 - -""" -Constant variables of plate location and expected barcode location -All Dimensions an Measurements in mm. -""" -PLATE_OFFSET_L = 54 #Top right X axis -PLATE_OFFSET_T = 12 #Top right Y axis -PLATE_WIDTH = 116 #real width: 127.38 -PLATE_HEIGHT = 77.33333333 #real height: 85.11 -WELL_WIDTH = 20 #simplified with and hight of the wells to fit in a raster -WELL_HEIGHT = 20 - -CUTOUT_OFFSET_L = 45 #Top right X axis -CUTOUT_OFFSET_T = 6 #Top right Y axis -CUTOUT_WIDTH = 135 -CUTOUT_HEIGHT = 90 - - -# calculate the starting location of the barcode -BARCODE_SCAN_OFFSET_T = PLATE_OFFSET_T + PLATE_HEIGHT - WELL_HEIGHT -BARVODE_SCAN_OFFSET_L = PLATE_OFFSET_L - -DATA_DIR = dh.get_data_directory() - -# Translate numbers to row letters -DICT_PLATE_X_TO_LETTER={ - 0: "A", - 1: "B", - 2: "C", - 3: "D", -} - -# Translate numbers to column numbers -DICT_PLATE_Y_TO_FLIPPED_Y={ - 0: 6, - 1: 5, - 2: 4, - 3: 3, - 4: 2, - 5: 1, -} - -# Add optional parameter for debugging -parser = argparse.ArgumentParser(description='Image scanner for SCALE-WP2.') -parser.add_argument("--test-run", action='store_true', default=False, - help="set to test run") -parser.add_argument("--bypass-label-validation", action="store_true", default=False, - help="set if the label validation should be bypassed") -parser.add_argument("--bypassed-label", default=None, - help="set the bypassed label") -parser.add_argument("--lower-scan-resolution", default=0, - help="change the scan resolution of the final scan") -parser.add_argument("--petridish", action="store_true", default=False, - help="when this option is set or when the label have a 'petri' suffix a scan of a petri dish will be done") -ARGS = parser.parse_args() - -class ScanConfig: - """ This class can hold arguments for the scanimage command. - It is parsed with the get_scan_command() function""" - def __init__(self, resolution, depth , format, l_offset, t_offset, width, height): - self.device_name = SCAN_DEVICE_NAME - self.resolution = resolution - self.depth = depth - self.format = format - self.l_offset = l_offset - self.t_offset = t_offset - self.width = width - self.height = height - -PREVIEW_SCAN_CONFIG = ScanConfig( - resolution = SCAN_RESOLUTION_BARCODE, - depth = SCAN_DEPTH_PREVIEW, - format = SCAN_FORMAT_PREVIEW, - l_offset = BARVODE_SCAN_OFFSET_L, - t_offset = BARCODE_SCAN_OFFSET_T, - width = WELL_HEIGHT, - height = WELL_WIDTH, - ) - -HIGH_RESOLUTION_SCAN_CONFIG = ScanConfig( - resolution = SCAN_RESOLUTION_PLATE, - depth = SCAN_DEPTH_HIGH_RESOLUTION, - format = SCAN_FORMAT_HIGH_RESOLUTION, - l_offset = PLATE_OFFSET_L, - t_offset = PLATE_OFFSET_T, - width = PLATE_WIDTH, - height = PLATE_HEIGHT, - ) - -PETRIDISH_SCAN_CONFIG = ScanConfig( - resolution = SCAN_RESOLUTION_PETRIDISH, - depth = SCAN_DEPTH_PETRIDISH, - format = SCAN_FORMAT_PETRIDISH, - l_offset = CUTOUT_OFFSET_L + 45, - t_offset = CUTOUT_OFFSET_T, - width = CUTOUT_WIDTH - 45, - height = CUTOUT_HEIGHT, - ) - -class Messages: - err_no_or_multiple_barcodes_found = "No or multiple barcodes found. Have you rotated the plate correctly and placed only one barcode at it?" - war_cannot_preview_image = "Cannot diplay preview image" - inf_check_barcode = "Search barcode" - inf_low_res_scan = "Low resolution scan is in progress with the following arguments:" - inf_high_res_scan = "High resolution scan is in progress with the following arguments:" - inf_check_label_vality = "Checking vality of label" - err_scanimage = "Error during scanimage:" - -M = Messages() - - -def get_scan_command(sc = ScanConfig, **kwargs) -> list: - """ - Create the scan command as a list. - It need an Object of type ScanConfig - """ - command = ["scanimage", - f"--device-name={SCAN_DEVICE_NAME}", - f"--resolution={sc.resolution}", - f"--depth={sc.depth}", - f"--format={sc.format}", - f"-l {sc.l_offset}", - f"-t {sc.t_offset}", - f"-x {sc.width}", - f"-y {sc.height}", - ] - """ Handles further arguments, possible arguments need to be added manually""" - for key,value in kwargs.items(): - match key: - case "brightness" : command.append(f"--brightness={value}") - case "out" : command.append(f"-o{value}") - return command - - -def scan_barcode(): - """ - 1. create the command for the preview barcode scan - 2. execute the command, show the image and return - """ - log.info( M.inf_check_barcode ) - command = get_scan_command(PREVIEW_SCAN_CONFIG) - process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - log.info( M.inf_low_res_scan ) - log.info( command ) - - output, error = process.communicate() # type of output is "bytes" in the format: b'\x89PNG\ ... - - if error: - log.error(M.err_scanimage + error.decode()) - return - - """convert the output to an image (class: 'PIL.PngImagePlugin.PngImageFile')""" - png_img_file = Image.open(io.BytesIO(output)) - - """ Try to display the scanned image to the user using the default image viwer. - Might not working in other operating systems """ - - try: - png_img_file.show() - except Exception as e: - """ Im not sure if this raise is right implemented here """ - log.warning(M.war_cannot_preview_image + e) - - return png_img_file - -def decode_data_matrix(img) -> str: - """ - 1. With pylibdmtx.decode() it reads the data matrix and return an array with all data matrix found - 2. If only one data matrix was found it returns the value of the data matrix - """ - - decoded = decode(img) - - """ Check if there was multiple data matrixes found and decoded is not emty """ - assert decoded and not len(decoded) > 1 , "No or multiple barcodes found." - - log.info("One barcode dedected: " + decoded[0].data.decode("utf-8")) - - return decoded[0].data.decode("utf-8") #takes the only and first entry, choose the .data content and convert it to a string - -def check_label_vality(label) -> bool: - """ - Check if if the label start with the current date - """ - log.info(M.inf_check_label_vality) - str_date_today = dt.datetime.today().strftime('%Y-%m-%d') - petri = False - valid = False - - if str_date_today in label: - valid = True - log.info(f"The label is valid: {label}") - else: - log.error(f"The label doesn't match the current date: {str_date_today}. The label is {label} instead.") - - if label.startswith("petri"): - petri = True - - return valid, petri - - -def resize(filein, fileout, maxw, maxh) -> None: - """function for resizing image without loading it in the RAM, - copied from: https://github.com/libvips/pyvips/wiki/example-resize """ - out = pyvips.Image.thumbnail(filein, maxw, height=maxh) - out.write_to_file(fileout) - -def flip_image_horizontally(image_path): - """ - This function is created by a Large Language Model Gemini - Loads an image from the given path, flips it horizontally, and saves it back to the same file. - - Args: - image_path (str): The path to the image file. - """ - try: - # Load the image using OpenCV - img = cv2.imread(image_path) - - if img is None: - raise FileNotFoundError(f"Could not open or find the image at: {image_path}") - - # Flip the image horizontally (flipCode=1) - flipped_img = cv2.flip(img, 1) - - # Save the flipped image back to the same file, overwriting the original - cv2.imwrite(image_path, flipped_img) - - print(f"Image at {image_path} has been flipped horizontally and saved.") - - except FileNotFoundError as e: - print(f"Error: {e}") - except Exception as e: - print(f"An unexpected error occurred: {e}") - -def scan_petri_dish(label): - ''' - This part is for scanning a petri dish, placed on the left side of the scanner. - ''' - PETRI_DIR = os.path.join(DATA_DIR, "petridish") - - os.makedirs(os.path.join(PETRI_DIR), exist_ok=True) - os.makedirs(os.path.join(PETRI_DIR, "images"), exist_ok=True) - - timestamp = dt.datetime.today().strftime("%H%M%S") - - file_path_tmp = os.path.join(PETRI_DIR, "images", f"{label}_{timestamp}.tmp") - file_path = os.path.join(PETRI_DIR, "images", f"{label}_{timestamp}.jpeg") - - scan_brightness = 1 - command = get_scan_command(PETRIDISH_SCAN_CONFIG, brightness = scan_brightness, out = file_path_tmp) - - # send scanning command to the scanner - process = subprocess.Popen(command, stderr=subprocess.PIPE) - log.info( M.inf_high_res_scan) - log.info( command ) - v, error = process.communicate() - - if error: - log.error(error.decode()) - return - - """ Rename from .tmp extension to .jpeg extension""" - try: - os.rename(file_path_tmp, file_path) - except Exception as e: - log.error(f"Error during renaming {file_path_tmp} to {file_path}: {e}") - return - - PETRIDISH_JSON_PATH = os.path.join(PETRI_DIR, "metadata.json") - - try: - with open(PETRIDISH_JSON_PATH, 'r') as f: - metadata = json.load(f) - except (FileNotFoundError, json.JSONDecodeError): - metadata = {"images": [], "scanconfigs": []} - - current_id = len(metadata["images"]) + 1 - - new_image_data = { - "id": current_id, - "image_filename": f"{label}_{timestamp}.jpeg", - "comments": "", - "sample_id": "", - "sample_description": "" - } - - new_scanconfig_data = { - 'image_id': current_id, - 'PLATE-ID': label, - 'TIMESTAMP': timestamp, - 'SCAN_DEVICE': PETRIDISH_SCAN_CONFIG.device_name, - 'SCAN_FORMAT': PETRIDISH_SCAN_CONFIG.format, - 'SCAN_RESOLUTION': PETRIDISH_SCAN_CONFIG.resolution, - 'SCAN_DEPTH': PETRIDISH_SCAN_CONFIG.depth, - 'SCAN_BRIGHTNESS': scan_brightness, - } - - metadata["images"].append(new_image_data) - metadata["scanconfigs"].append(new_scanconfig_data) - - with open(PETRIDISH_JSON_PATH, 'w') as f: - json.dump(metadata, f, indent=4) - - - -def scan_high_resolution_and_save(label): - """ - 1. If not already existent create an directory for the image - 2. Create the scan command for the high resolution scan - 3. If the scan was successfull: rescale the file and save an additional .thumbnail file - 4. slice the image and delete the original file - 5. Show the thumbnail image to the user - """ - - scan_time = dt.datetime.today().strftime("%H-%M-%S") - scan_date = dt.datetime.today().strftime("%Y-%m-%d") - - directory = os.path.join(DATA_DIR, label) - scan_repetition = 1 # Default to 1st repetition - - try: - os.makedirs(directory) - log.info(f"Directory '{directory}' created successfully.") - except FileExistsError: - existing_files = glob.glob(os.path.join(directory, f"{label}_*.jpeg")) - scan_repetition = len(existing_files) + 1 # Next repetition number - print(f"You are saving repetition {scan_repetition} for plate {label}" ) - except PermissionError: - log.error(f"Permission denied: Unable to create '{directory}'.") - return - except Exception as e: - log.error(f"An error occurred: {e}") - return - - # Create path variables using scan_time for uniqueness within the day - general_filename = os.path.join(directory, f"{label}_{scan_time}") - scan_brightness = 3 - - # Create paths of tmp file and high resolution - tmp_file_path = os.path.join(f"{general_filename}.tmp") - high_resolution_save_path = os.path.join(directory, f"{label}_{scan_time}.tiff") - thumbnail_file_path = f"{directory}/{label}_{scan_time}.thumbnail.jpeg" - - command = get_scan_command(HIGH_RESOLUTION_SCAN_CONFIG, brightness = scan_brightness, out = tmp_file_path) - scan_starttime = dt.datetime.now().timestamp() - - # send scanning command to the scanner - process = subprocess.Popen(command, stderr=subprocess.PIPE) - log.info( M.inf_high_res_scan) - log.info( command ) - v, error = process.communicate() - scan_endttime = dt.datetime.now().timestamp() - scan_duration = str(dt.timedelta(seconds=scan_endttime - scan_starttime)) - print(f"Execution time of scan: {scan_duration}") - - if error: - log.error(error.decode()) - return - - """ Rename from .tmp extension to .tiff extension""" - try: - os.rename(tmp_file_path, high_resolution_save_path) - except Exception as e: - log.error(f"Error during renaming {tmp_file_path} to {high_resolution_save_path}: {e}") - return - - - """ Create a thumbnail file with .thumbnail.jpg extension and flip it horizontally""" - try: - resize(high_resolution_save_path, thumbnail_file_path + "[Q=98]", maxw = 12000, maxh = 8000) - flip_image_horizontally(thumbnail_file_path) - except Exception as e: - log.error("Error during resizing" + e) - - - """Display thumbnail file to user """ - try: - Image.open(thumbnail_file_path).show() - except Exception as e: - """ Im not sure if this raise it right implemented here """ - log.warning(M.war_cannot_preview_image + e) - - """ Slice the image and delete the original file """ - slice_image(high_resolution_save_path, label, scan_time) - - try: - os.remove(high_resolution_save_path) - except Exception as e: - log.error("Error during deleting the original file" + e) - - # add the scanplate to database - dh.add_plate_to_database(label, - scan_time, - '24', - HIGH_RESOLUTION_SCAN_CONFIG.resolution, - HIGH_RESOLUTION_SCAN_CONFIG.depth, - scan_brightness, - scan_date, - scan_repetition, - ) - - dh.create_blank_drymass_csv(label) - -def run_scanner() -> None: - scan_starttime = dt.datetime.now().timestamp() - - if ARGS.test_run: - log.warning("This is a test run. It bypass some critical features such as the check of label vality and lower the scan resolution") - - if ARGS.bypassed_label: - label = ARGS.bypassed_label - log.warning(f"Testcase: The label scanning was bypassed with this label: {label}") - else: label = decode_data_matrix(scan_barcode()) - - scan_datamatrix_time = dt.datetime.now().timestamp() - print(f"Execution time of label scan: {str(dt.timedelta(seconds=scan_datamatrix_time - scan_starttime))}") - valid, petri = check_label_vality(label) - - if ARGS.bypass_label_validation: valid = True - if not valid: return - else: log.warning("Testcase: The label vality wasn't checked") - - if ARGS.lower_scan_resolution: - HIGH_RESOLUTION_SCAN_CONFIG.resolution = ARGS.lower_scan_resolution - log.warning(f"Testcase: the scan resolution was set from {SCAN_RESOLUTION_PLATE} to {ARGS.lower_scan_resolution}") - - if ARGS.test_run: label = label+"_TEST" - if not valid: return - if petri or ARGS.petridish: scan_petri_dish(label) - else: scan_high_resolution_and_save(label) - - scan_total = dt.datetime.now().timestamp() - print(f"Execution of whole scan + slicing: {str(dt.timedelta(seconds=scan_total - scan_starttime))}") - - return - -def slice_image(input_path, label, timestamp) -> None: - - # Create directories - root_plate_path = dh.get_root_plate_path(label) - try: - os.makedirs(root_plate_path) - log.info(f"Directory '{root_plate_path}' created successfully.") - except PermissionError: - log.error(f"Permission denied: Unable to create '{root_plate_path}'.") - exit - except Exception as e: - log.error(f"An error occurred: {e}") - exit - - # Batch load the full image via pyvips - image = pyvips.Image.new_from_file(input_path) - - width = image.width - height = image.height - - # Hard coded values for the cell size of a 6x4 well plate - cell_width= width / 6 - cell_height= height / 4 - - print(image.width, image.height) - for x in range(0, 4, 1): - for y in range(0, 6, 1): - - WELLNAME = f"{DICT_PLATE_X_TO_LETTER[x]}{DICT_PLATE_Y_TO_FLIPPED_Y[y]}" - PATH_TIFF = dh.get_image_path(label, timestamp, WELLNAME, type = "tiff") - PATH_THUMBNAIL = dh.get_image_path(label, timestamp, WELLNAME, type = "thumbnail") - PATH_JPEG = dh.get_image_path(label, timestamp, WELLNAME, type = "jpeg") - - # Create TIFF and JPEG path if not already existend - os.makedirs(os.path.dirname(PATH_TIFF), exist_ok=True) - os.makedirs(os.path.dirname(PATH_JPEG), exist_ok=True) - - # Crop image using the following function: crop(left, top, width, height) - slice = image.crop(y*cell_width, x*cell_height, cell_width, cell_height) - slice.write_to_file(PATH_TIFF) - - # Create a THUMBNAIL file with .thumbnail.jpeg extension - """ Create a jpeg file in a flat directory with .jpeg extension""" - try: - resize(PATH_TIFF, PATH_JPEG+"[Q=100]", maxw = 2048, maxh = 2048) - except Exception as e: - log.error("Error during resizing" + e) - - try: - resize(PATH_JPEG, PATH_THUMBNAIL, maxw = 1024, maxh = 1024) - except Exception as e: - log.error("Error during resizing" + e) - -if __name__ == "__main__": - - logging.basicConfig(level=logging.INFO) - - run_scanner() - \ No newline at end of file diff --git a/img/doc_files.png b/img/doc_files.png new file mode 100644 index 0000000..d2ff027 Binary files /dev/null and b/img/doc_files.png differ diff --git a/img/doc_overview.png b/img/doc_overview.png new file mode 100644 index 0000000..720ec44 Binary files /dev/null and b/img/doc_overview.png differ diff --git a/img/doc_preview.png b/img/doc_preview.png new file mode 100644 index 0000000..e4d8fe5 Binary files /dev/null and b/img/doc_preview.png differ diff --git a/img/doc_scan.png b/img/doc_scan.png new file mode 100644 index 0000000..9eb51ef Binary files /dev/null and b/img/doc_scan.png differ diff --git a/img/icon-256.png b/img/icon-256.png new file mode 100644 index 0000000..b939ba8 Binary files /dev/null and b/img/icon-256.png differ diff --git a/img/icon-512.png b/img/icon-512.png new file mode 100644 index 0000000..62069e4 Binary files /dev/null and b/img/icon-512.png differ diff --git a/img/icon.png b/img/icon.png new file mode 100644 index 0000000..fea4ed6 Binary files /dev/null and b/img/icon.png differ diff --git a/img/icon.svg b/img/icon.svg new file mode 100644 index 0000000..ac38965 --- /dev/null +++ b/img/icon.svg @@ -0,0 +1,65 @@ + + + + diff --git a/printing_files/barcode_stamp.stl b/printing_files/barcode_stamp.stl deleted file mode 100644 index 074092c..0000000 Binary files a/printing_files/barcode_stamp.stl and /dev/null differ diff --git a/printing_files/custom_lid.stl b/printing_files/custom_lid.stl deleted file mode 100644 index bd88960..0000000 Binary files a/printing_files/custom_lid.stl and /dev/null differ diff --git a/printing_files/drying_rack.stl b/printing_files/drying_rack.stl deleted file mode 100644 index f3ec6cb..0000000 Binary files a/printing_files/drying_rack.stl and /dev/null differ diff --git a/printing_files/drying_rack_prusa_project.3mf b/printing_files/drying_rack_prusa_project.3mf deleted file mode 100644 index 84e319c..0000000 Binary files a/printing_files/drying_rack_prusa_project.3mf and /dev/null differ diff --git a/printing_files/well_plate_holder.stl b/printing_files/well_plate_holder.stl deleted file mode 100644 index d046108..0000000 Binary files a/printing_files/well_plate_holder.stl and /dev/null differ diff --git a/prototype/docker-compose.yaml b/prototype/docker-compose.yaml new file mode 100644 index 0000000..bc85b4b --- /dev/null +++ b/prototype/docker-compose.yaml @@ -0,0 +1,25 @@ +# ./docker-compose.yml +services: + scanner-lab: + build: + context: ./scanner-lab + dockerfile: Dockerfile + container_name: scanner-lab + # Pass the USB bus + udev info + devices: + - /dev/bus/usb:/dev/bus/usb + volumes: + - /run/udev:/run/udev:ro + # Make interactive shells nice + tty: true + stdin_open: true + # (Optional) run as root to sidestep permissions while prototyping + device_cgroup_rules: + - 'c 189:* rmw' # USB bus major; allow read/mmap/write for all minors + # optional: run as root while testing; later swap to non-root + proper groups + user: "0:0" + + security_opt: + - apparmor=unconfined # <-- helps on some setups + + restart: unless-stopped diff --git a/prototype/mock_image.pdf b/prototype/mock_image.pdf new file mode 100644 index 0000000..328b43d Binary files /dev/null and b/prototype/mock_image.pdf differ diff --git a/prototype/mock_image.svg b/prototype/mock_image.svg new file mode 100644 index 0000000..3fb6dd0 --- /dev/null +++ b/prototype/mock_image.svg @@ -0,0 +1,187 @@ + + + +mock_barcode_001 diff --git a/prototype/scanner-lab/Dockerfile b/prototype/scanner-lab/Dockerfile new file mode 100644 index 0000000..7047b5a --- /dev/null +++ b/prototype/scanner-lab/Dockerfile @@ -0,0 +1,24 @@ +# ./scanner-lab/Dockerfile +FROM python:3.11-slim + +# Tools + sane +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates wget tar usbutils udev \ + sane-utils libsane1 libsane-common libusb-1.0-0 \ + && rm -rf /var/lib/apt/lists/* + +# (Optional) Install Epson Scan 2 backend + non-free plugin +ARG EPSON_BUNDLE_URL="https://download-center.epson.com/f/module/e4b1664e-de6c-43d9-9045-b5c9e2f848fe/epsonscan2-bundle-6.7.80.0.x86_64.deb.tar.gz" +RUN set -eux; \ + mkdir -p /tmp/epson && cd /tmp/epson; \ + wget -O epsonscan2.tar.gz "$EPSON_BUNDLE_URL"; \ + tar -xzf epsonscan2.tar.gz; \ + cd epsonscan2-bundle-*.x86_64.deb; \ + yes | ./install.sh; \ + cd /; rm -rf /tmp/epson + +# Helpful: ensure backends listed once +RUN grep -q '^epsonscan2$' /etc/sane.d/dll.conf || echo epsonscan2 >> /etc/sane.d/dll.conf + +# Sit idle so you can exec in +CMD ["bash", "-lc", "echo 'Scanner lab ready'; tail -f /dev/null"] diff --git a/services/scanner-api/Dockerfile b/services/scanner-api/Dockerfile new file mode 100644 index 0000000..197e545 --- /dev/null +++ b/services/scanner-api/Dockerfile @@ -0,0 +1,77 @@ +FROM python:3.11-slim + +# --- system deps (add sane + common libs) --- +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates wget tar \ + sane-utils libsane1 libusb-1.0-0 libgphoto2-6 \ + libjpeg62-turbo zlib1g libtiff6 libopenjp2-7 libwebp7 \ + libqt5widgets5 \ + && rm -rf /var/lib/apt/lists/* +# Tools + sane +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates wget tar usbutils udev \ + sane-utils libsane1 libsane-common libusb-1.0-0 \ + procps psmisc libdmtx0t64\ + && rm -rf /var/lib/apt/lists/* + + +# (Optional) Install Epson Scan 2 backend + non-free plugin +ARG EPSON_BUNDLE_URL="https://download-center.epson.com/f/module/e4b1664e-de6c-43d9-9045-b5c9e2f848fe/epsonscan2-bundle-6.7.80.0.x86_64.deb.tar.gz" +RUN set -eux; \ + mkdir -p /tmp/epson && cd /tmp/epson; \ + wget -O epsonscan2.tar.gz "$EPSON_BUNDLE_URL"; \ + tar -xzf epsonscan2.tar.gz; \ + cd epsonscan2-bundle-*.x86_64.deb; \ + yes | ./install.sh; \ + cd /; rm -rf /tmp/epson + +# Helpful: ensure backends listed once +RUN grep -q '^epsonscan2$' /etc/sane.d/dll.conf || echo epsonscan2 >> /etc/sane.d/dll.conf + +# Sit idle so you can exec in +CMD ["bash", "-lc", "echo 'Scanner lab ready'; tail -f /dev/null"] + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PIP_NO_CACHE_DIR=1 \ + SCANNER_OUTPUT_DIR=/data \ + DEBIAN_FRONTEND=noninteractive \ + APT_LISTCHANGES_FRONTEND=none + +# --- install Epson Scan 2 bundle --- +# (Version pinned to the URL you provided; adjust as needed) +ARG EPSON_BUNDLE_URL="https://download-center.epson.com/f/module/e4b1664e-de6c-43d9-9045-b5c9e2f848fe/epsonscan2-bundle-6.7.80.0.x86_64.deb.tar.gz" + +RUN set -eux; \ + mkdir -p /tmp/epson && cd /tmp/epson; \ + wget -O epsonscan2.tar.gz "$EPSON_BUNDLE_URL"; \ + tar -xzf epsonscan2.tar.gz; \ + cd epsonscan2-bundle-*.x86_64.deb; \ + # The installer typically runs dpkg on included .deb files and resolves deps. + # Most variants are noninteractive already; piping 'yes' guards against prompts. + yes | ./install.sh; \ + cd /; rm -rf /tmp/epson + + + +# If needed, you can ensure entries exist: +# RUN grep -q '^epsonscan2' /etc/sane.d/dll.conf || echo 'epsonscan2' >> /etc/sane.d/dll.conf +# RUN grep -q '^epkowa' /etc/sane.d/dll.conf || echo 'epkowa' >> /etc/sane.d/dll.conf + +# --- create runtime user and give USB/scanner access --- +RUN useradd -m appuser \ + && groupadd -f scanner || true \ + && usermod -aG scanner,lp appuser + +WORKDIR /app + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY scanner.py main.py start.sh ./ +RUN chmod +x /app/start.sh + +VOLUME ["/data"] +EXPOSE 8000 + +ENTRYPOINT ["/app/start.sh"] diff --git a/services/scanner-api/main.py b/services/scanner-api/main.py new file mode 100644 index 0000000..fa696a9 --- /dev/null +++ b/services/scanner-api/main.py @@ -0,0 +1,392 @@ +from __future__ import annotations + +import copy +import logging + +import yaml +import os +import asyncio +from datetime import datetime +from pathlib import Path +from typing import Any, Dict, Optional +from unittest.mock import DEFAULT + +from fastapi import FastAPI, HTTPException, Body, UploadFile, File +from fastapi.middleware.cors import CORSMiddleware +from fastapi.staticfiles import StaticFiles +from fastapi.responses import PlainTextResponse + +from pydantic import BaseModel + +# ---- import your classes ---- +from scanner import BaseJsonGridScanner, MockJsonGridScanner, JsonGridScanner +from pydantic import BaseModel, Field +from typing import Any, Dict, List, Optional, Iterable +from pathlib import Path + + +SCHEMA_SQL = """ +CREATE TABLE IF NOT EXISTS images_json ( + id BIGSERIAL PRIMARY KEY, + inserted_at TIMESTAMPTZ NOT NULL DEFAULT now(), + data JSONB NOT NULL +); + +CREATE UNIQUE INDEX IF NOT EXISTS ux_images_json_dir_file + ON images_json ( (data->>'dirname'), (data->>'filename') ); + +CREATE INDEX IF NOT EXISTS idx_images_json_gin + ON images_json USING gin (data jsonb_path_ops); +""" + + +def _ensure_db_schema() -> None: + """ + Connect to Postgres and ensure the images_json table and indexes exist. + Safe to call multiple times thanks to IF NOT EXISTS. + """ + conn = _pg_connect_with_retries(PG_DSN) + try: + with conn.cursor() as cur: + cur.execute(SCHEMA_SQL) + finally: + conn.close() + +# ------------------------- +# App setup +# ------------------------- +app = FastAPI(title="Insect Scanner Device API", version="0.2.0") + +# CORS (adjust for your frontend) +app.add_middleware( + CORSMiddleware, + allow_origins=["*"], + allow_credentials=True, + allow_methods=["*"], + allow_headers=["*"], +) + +DEFAULT_CONFIG = """ +global: + enforce_barcode: false + +preview: + xyxy_mm: [0.0, 0.0, 210.0, 200.0] + file_format: "jpg" + dpi: 150 + +barcode: + xyxy_mm: [0.0, 0.0, 100.0, 100.0] + file_format: "jpg" + dpi: 300 + +rois: + - name: "overview" + xyxy_mm: [0.0, 0.0, 90.0, 90.0] + file_format: "jpg" + dpi: 300 + + - name: "whole" + xyxy_mm: [10.0, 20.0, 90.0, 80.0] + file_format: "tiff" + dpi: 600 +""" + +def _env_flag(name: str, default: bool = False) -> bool: + val = os.environ.get(name) + if val is None: + return default + return str(val).strip().lower() in {"1", "true", "yes", "on"} + + +CONFIG_PATH = Path(os.environ.get("SCANNER_CONFIG_PATH", "")) +USE_MOCK_SCANNER = _env_flag("USE_MOCK_SCANNER", default=False) + +# choose implementation +JsonGridScannerClass = MockJsonGridScanner if USE_MOCK_SCANNER else JsonGridScanner + +# ------------------------- +# Global state +# ------------------------- +TARGET_DIR = Path(os.environ.get("SCANNER_OUTPUT_DIR", "scanner_output")) +TARGET_DIR.mkdir(parents=True, exist_ok=True) + +# Persist here by default: /config.yaml +if not os.environ.get("SCANNER_CONFIG_PATH"): + CONFIG_PATH = TARGET_DIR / "config.yaml" + + +import psycopg +import json +import time + +PG_DSN = os.environ.get( + "PG_DSN", + "postgresql://postgres:postgres@db:5432/postgres?connect_timeout=5", +) +logging.info("PG_DSN host=%s db=%s user=%s", + os.environ.get("PGHOST", "db"), + os.environ.get("PGDATABASE", "postgres"), + os.environ.get("PGUSER", "postgres")) +def _pg_connect_with_retries(dsn: str, tries: int = 20, delay: float = 1.5): + last = None + for i in range(1, tries + 1): + try: + # autocommit=True so we can use simple INSERT/UPDATE without manual commit + return psycopg.connect(dsn, autocommit=True) + except psycopg.OperationalError as e: + last = e + time.sleep(delay) + raise RuntimeError(f"Could not connect to Postgres after {tries} tries: {last}") + + +def _db_upsert_many(items: Iterable[Dict[str, Any]]) -> None: + if not items: + return + # every item is a Python dict you want to store in JSONB column "data" + rows = [(json.dumps(item),) for item in items] # parameter is text -> jsonb in SQL + + + sql = """ + INSERT INTO images_json (data) + SELECT jsonb_strip_nulls(%s::jsonb) + ON CONFLICT ((data->>'dirname'), (data->>'filename')) + DO UPDATE SET + data = EXCLUDED.data, + inserted_at = NOW(); + """ + + conn = _pg_connect_with_retries(PG_DSN) + try: + with conn.cursor() as cur: + cur.executemany(sql, rows) + finally: + conn.close() +# +# UPSERT_SQL = """ +# INSERT INTO images_json (data) +# VALUES (%s::jsonb) +# ON CONFLICT ((data->>'dirname'), (data->>'filename')) +# DO UPDATE SET data = EXCLUDED.data; +# """ + +def raise_with_trace(exc: Exception, status_code: int = 500, msg: str = None): + import traceback + tb = traceback.format_exc() + + logging.error(tb) + text = f"{msg or type(exc).__name__}: {str(exc)}\n{tb}" + raise HTTPException(status_code=status_code, detail=text) + + +def _to_builtin(obj): + # make sure yaml dump won't choke on Paths or Pydantic models + from pydantic import BaseModel as _BM + if isinstance(obj, Path): + return str(obj) + if isinstance(obj, dict): + return {k: _to_builtin(v) for k, v in obj.items()} + if isinstance(obj, (list, tuple)): + return [_to_builtin(v) for v in obj] + if isinstance(obj, _BM): + return _to_builtin(obj.dict()) + return obj + +# Only expose user-facing keys +ALLOWED_PUBLIC_KEYS = {"preview", "barcode", "rois", "global"} +def _public_config(cfg: dict) -> dict: + return {k: cfg[k] for k in ALLOWED_PUBLIC_KEYS if k in cfg} + +def dump_compact_yaml(data: dict) -> str: + # Compact lists where possible (inline [a, b, c, d]) + return yaml.safe_dump(data, sort_keys=False, default_flow_style=None) + +def _load_initial_config() -> dict: + if CONFIG_PATH.exists(): + text = CONFIG_PATH.read_text(encoding="utf-8") + cfg = yaml.safe_load(text) + if not isinstance(cfg, dict): + raise ValueError("Persisted config is not a YAML mapping") + return cfg + # fallback to baked-in default + return yaml.safe_load(DEFAULT_CONFIG) +def _instantiate_scanner_from_config(cfg: Dict[str, Any]) -> BaseJsonGridScanner: + # ... your validation ... + # hand scanner a deep copy so it can’t mutate the caller’s dict + return JsonGridScannerClass(config=copy.deepcopy(cfg), target_base_dir=TARGET_DIR) +def _make_api_uri(path: Path) -> str: + """Return API-served URI (/files/...) from a filesystem path.""" + rel_path = path.relative_to(TARGET_DIR) + return f"/files/{rel_path.as_posix()}" + + +# Mount static file serving (so GUI can view files) +app.mount("/files", StaticFiles(directory=str(TARGET_DIR)), name="files") + +CONFIG_DATA = _load_initial_config() +SCANNER = _instantiate_scanner_from_config(CONFIG_DATA) + +BUSY_LOCK = asyncio.Lock() + +LAST_PREVIEW: Optional[Path] = None +LAST_SCAN: Optional[Dict[str, Any]] = None + + + +class BusyResponse(BaseModel): + busy: bool + + +class PreviewResponse(BaseModel): + uri: str + svg: str + ts: str + + +class ImageMetadataOut(BaseModel): + # Core fields we always return; extras are allowed + uri: str + thumbnail_uri: str + filename: str + dirname: str + md5: str + timestamp: Optional[str] = None + scan_metadata: Optional[Dict[str, Any]] = None + + class Config: + extra = "allow" # keep any extra metadata coming from your scanner + +class ScanResponse(BaseModel): + rois: List[ImageMetadataOut] + +# ------------------------- +# Helper +# ------------------------- +@app.get("/config", response_class=PlainTextResponse) +async def get_config_yaml() -> str: + # return sanitized, user-facing YAML + return dump_compact_yaml(_to_builtin(_public_config(CONFIG_DATA))) + +@app.post("/config", response_class=PlainTextResponse) +async def set_config_yaml(yaml_text: str = Body(..., media_type="text/plain")) -> str: + if BUSY_LOCK.locked(): + raise HTTPException(status_code=423, detail="Scanner is busy; try again later.") + try: + cfg = yaml.safe_load(yaml_text) + except yaml.YAMLError as e: + raise HTTPException(status_code=400, detail=f"Invalid YAML: {e}") + + # Validate by instantiating a scanner (you already deep-copy internally) + try: + new_scanner = _instantiate_scanner_from_config(cfg) + except HTTPException: + raise + except Exception as e: + raise HTTPException(status_code=400, detail=f"Config validation failed: {e}") + + # Swap globals + global CONFIG_DATA, SCANNER, LAST_PREVIEW, LAST_SCAN + CONFIG_DATA = cfg + SCANNER = new_scanner + LAST_PREVIEW = None + LAST_SCAN = None + + # Persist (atomic write) + to_write = dump_compact_yaml(_to_builtin(_public_config(CONFIG_DATA))) + tmp = CONFIG_PATH.with_suffix(".tmp") + tmp.write_text(to_write, encoding="utf-8") + os.replace(tmp, CONFIG_PATH) + + return to_write +# ------------------------- +# Endpoints +# ------------------------- +@app.get("/is_busy", response_model=BusyResponse) +async def is_busy() -> BusyResponse: + return BusyResponse(busy=BUSY_LOCK.locked()) + + +@app.post("/preview", response_model=PreviewResponse) +async def generate_preview() -> PreviewResponse: + if BUSY_LOCK.locked(): + raise HTTPException(status_code=423, detail="Scanner is busy, try again later.") + + async with BUSY_LOCK: + global LAST_PREVIEW + metadata = SCANNER.generate_preview() + path = Path(metadata["uri"]) + svg = Path(metadata["svg"]) + LAST_PREVIEW = path + return PreviewResponse(uri=_make_api_uri(path), svg=_make_api_uri(svg), ts=datetime.now().isoformat()) + + +# --- helper to rewrite FS paths -> API URIs --- + +def _meta_fs_to_api(meta: Dict[str, Any]) -> ImageMetadataOut: + """ + Takes one image-metadata dict from your scanner and converts: + - meta["uri"] (absolute/fs path) -> API URI (/files/...) + - meta["thumbnail"] (filename) -> thumbnail_uri (/files/...) + Leaves other keys intact. + """ + meta = dict(meta) # shallow copy so we don't mutate the original + # meta["uri"] is a filesystem path; turn it into an API path + img_path = Path(meta["uri"]) + meta["uri"] = _make_api_uri(img_path) + + # thumbnail is just a filename in the same directory + thumb_path = Path(meta["dirname"]) / meta["thumbnail"] + meta["thumbnail_uri"] = _make_api_uri(thumb_path) + + # Ensure required fields exist even if upstream changes + meta.setdefault("filename", img_path.name) + meta.setdefault("dirname", str(img_path.parent)) + meta.setdefault("md5", "") + + return ImageMetadataOut(**meta) + + +# --- updated /scan endpoint --- + +@app.post("/scan", response_model=ScanResponse) +async def scan_grid() -> ScanResponse: + if BUSY_LOCK.locked(): + raise HTTPException(status_code=423, detail="Scanner is busy, try again later.") + + async with BUSY_LOCK: + global LAST_SCAN + try: + result = SCANNER.scan_grid() + print (result) + _db_upsert_many(result) + + + # result: list of ROI metadata dicts + rois_out = [_meta_fs_to_api(m) for m in result] + + LAST_SCAN = { + "rois": [r.dict() for r in rois_out], + } + + return ScanResponse(rois=rois_out) + + except Exception as e: + raise_with_trace(e, 500, "Scan failed") + +@app.get("/healthz") +async def healthz() -> dict: + return {"ok": True, "time": datetime.now().isoformat()} + +@app.on_event("startup") +def _on_startup() -> None: + """ + Called by FastAPI when the app process starts. + Ensures DB schema is present before handling any requests. + """ + try: + _ensure_db_schema() + logging.info("Database schema ensured.") + except Exception as e: + logging.exception("Failed to ensure DB schema: %s", e) + # Optional: let the app fail hard if schema cannot be created + raise \ No newline at end of file diff --git a/services/scanner-api/prototype.py b/services/scanner-api/prototype.py new file mode 100644 index 0000000..8a6b5a0 --- /dev/null +++ b/services/scanner-api/prototype.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +""" +prototype_petri.py + +Prototype runner that: + 1) Instantiates the mock PetriDishScanner + 2) Generates a static preview image + 3) Scans barcode ROI to produce metadata + 4) Scans overview + the single 'whole' ROI + 5) Prints out where files were saved + +Assumes you have the classes available on the import path: + - BaseROIScanner, MockROIScanner, BaseGridScanner (with barcode support) + - PetriDishScanner (derived from BaseGridScanner) +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +# Import your implementations +# If these live in a local module like 'scanner.py', adjust the import accordingly: +from scanner import MockPetriDishScanner # type: ignore + + +def file_uri(p: Path) -> str: + return p.resolve().as_uri() + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description="Mock PetriDishScanner prototype") + parser.add_argument("--out", dest="out_dir", default="out", help="Target base directory") + args = parser.parse_args(argv) + + out_dir = Path(args.out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + + # Instantiate the mock Petri dish scanner + scanner = MockPetriDishScanner( + target_base_dir=out_dir, + + ) + + print("=== Generating preview (static path) ===") + preview_path = scanner.generate_preview() + print("Preview: ", preview_path, "->", file_uri(preview_path)) + + print("\n=== Scanning grid (barcode -> metadata, then overview + ROI(s)) ===") + result = scanner.scan_grid() + + print("Timestamp: ", result["timestamp"]) + print("Metadata: ", result["metadata"]) + print("Overview: ", result["overview_path"], "->", file_uri(result["overview_path"])) + for i, rp in enumerate(result["roi_paths"], start=1): + print(f"ROI[{i}]: {rp} -> {file_uri(rp)}") + + print("\nDone.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/services/scanner-api/requirements.txt b/services/scanner-api/requirements.txt new file mode 100644 index 0000000..979356a --- /dev/null +++ b/services/scanner-api/requirements.txt @@ -0,0 +1,6 @@ +fastapi +uvicorn +pillow +pyyaml +pylibdmtx +psycopg[binary] \ No newline at end of file diff --git a/services/scanner-api/scanner.py b/services/scanner-api/scanner.py new file mode 100644 index 0000000..b21bd5a --- /dev/null +++ b/services/scanner-api/scanner.py @@ -0,0 +1,909 @@ +from __future__ import annotations +import copy +import hashlib +import logging +import os +import uuid +import random +from abc import ABC, abstractmethod +from datetime import datetime +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple, Union +import base64 +from xml.etree.ElementTree import Element, SubElement, ElementTree +import shlex +import os +import shutil +import subprocess +import textwrap +import getpass +import json +from PIL import Image +from collections.abc import Sequence +import yaml # <-- needed for _load_config +from pylibdmtx.pylibdmtx import decode as dmtx_decode + +JsonLike = Union[str, Path, Dict[str, Any]] +def _read_host_id(path: str = "/etc/host_machine_id") -> Optional[str]: + try: + p = Path(path) + if p.exists(): + txt = p.read_text(encoding="utf-8").strip() + return txt or None + except Exception: + pass + return None +import re +import re + +_DEVLINE_RE = re.compile(r"^device\s+`([^']+)'\s+is a\s+(.+)$") + +def _parse_scanimage_device_line(line: str) -> Optional[dict]: + m = _DEVLINE_RE.match(line.strip()) + if not m: + return None + sane_device, desc = m.group(1), m.group(2).strip() + + # driver + if ":" in sane_device: + driver, rest = sane_device.split(":", 1) + else: + driver, rest = sane_device, "" + + device_name = None + usb_id = None + + # epsonscan2 style: ::: + if "::" in rest: + before, after = rest.split("::", 1) + if before: + device_name = before + if after: + usb_id = after + else: + # classic: : + usb_id = rest or None + + if not device_name and desc: + device_name = desc.split(" on ", 1)[0].strip() + + return { + "sane_device": sane_device, # <-- exact string to pass to scanimage + "driver": driver or None, + "device_name": device_name or None, + "usb_id": usb_id or None, + } + +def _debug_env(): + try: + who = subprocess.check_output(["bash","-lc","id && ls -l /dev/bus/usb/*/* 2>/dev/null || true"], text=True) + except Exception as e: + who = f"failed to gather id/usb listing: {e}" + return textwrap.dedent(f""" + ---- RUNTIME DEBUG ---- + EUID: {os.geteuid()} USER: {getpass.getuser()} + PATH: {os.environ.get('PATH')} + SANE_CONFIG_DIR: {os.environ.get('SANE_CONFIG_DIR')} + scanimage path: {shutil.which('scanimage')} + id/usb listing: + {who} + ----------------------- + """) + + +def _parse_xyxy_mm(xyxy: Sequence[float]) -> tuple[float, float, float, float]: + if (not isinstance(xyxy, Sequence)) or len(xyxy) != 4: + raise ValueError("xyxy_mm must be a 4-element sequence [x0, y0, x1, y1] in mm") + x0, y0, x1, y1 = map(float, xyxy) + if x1 <= x0 or y1 <= y0: + raise ValueError(f"Invalid ROI (x1<=x0 or y1<=y0): {xyxy}") + return x0, y0, x1, y1 + +def _load_config(config: JsonLike) -> Dict[str, Any]: + if isinstance(config, dict): + return config + if isinstance(config, Path): + text = config.read_text(encoding="utf-8") + elif isinstance(config, str): + p = Path(config) + if p.exists(): + text = p.read_text(encoding="utf-8") + else: + text = config + else: + raise ConfigError("config must be dict, str (yaml/json), or Path") + + # Try YAML (superset), then JSON + try: + cfg = yaml.safe_load(text) + if isinstance(cfg, dict): + return cfg + except Exception: + pass + try: + cfg = json.loads(text) + if isinstance(cfg, dict): + return cfg + except Exception: + pass + raise ConfigError("Could not parse config as YAML or JSON into a mapping") +class ConfigError(ValueError): + pass + +class ScannerNotFoundError(RuntimeError): + """Raised when the underlying `scanimage` command fails.""" + +class ScanImageError(RuntimeError): + """Raised when the underlying `scanimage` command fails.""" + + +def __init__( + self, + name: str, + xyxy_mm: Sequence[float], # <-- NEW: single quadruple + file_format: str = "jpg", + dpi: int = 300, + driver: str = "epson2", + usb_port: Optional[str] = None, + target_base_dir: Optional[Path | str] = None, + device_id = None, + host_id = None, + sane_device = None +) -> None: + # ROI attributes + self.name = name + self.xyxy_mm = _parse_xyxy_mm(xyxy_mm) + self.x0_mm, self.y0_mm, self.x1_mm, self.y1_mm = self.xyxy_mm # keep legacy attrs + self.file_format = file_format.lower() + self.dpi = dpi + + # Device / output + self.driver = driver + self.usb_port = usb_port + + self.device_id = device_id + self.host_id = host_id + self.sane_device = sane_device + + if target_base_dir is None: + self.target_base_dir = Path(Path.cwd()) / "scanner_output" + else: + self.target_base_dir = Path(target_base_dir) + self.target_base_dir.mkdir(parents=True, exist_ok=True) + +class BaseROIScanner(ABC): + def __init__( + self, + name: str, + xyxy_mm: Sequence[float], + file_format: str = "jpg", + dpi: int = 300, + driver: str = "epson2", + usb_port: Optional[str] = None, + target_base_dir: Optional[Path | str] = None, + device_id: Optional[str] = None, + host_id: Optional[str] = None, + sane_device: Optional[str] = None + ) -> None: + # ROI attributes + self.name = name + + self.xyxy_mm = _parse_xyxy_mm(xyxy_mm) + self.x0_mm, self.y0_mm, self.x1_mm, self.y1_mm = self.xyxy_mm + self.file_format = file_format.lower() + self.dpi = dpi + + # Device / output + self.driver = driver + self.usb_port = usb_port + self.device_id = device_id # (keep whatever you decide upstream) + self.host_id = host_id # (keep whatever you decide upstream) + self.sane_device = sane_device # (keep whatever you decide upstream) + + if target_base_dir is None: + self.target_base_dir = Path(Path.cwd()) / "scanner_output" + else: + self.target_base_dir = Path(target_base_dir) + self.target_base_dir.mkdir(parents=True, exist_ok=True) + + def _contruction_args(self) -> dict: + """Return all initialization parameters as a dictionary.""" + return { + "name": self.name, + "xyxy_mm": list(self.xyxy_mm), + "file_format": self.file_format, + "dpi": self.dpi, + "driver": self.driver, + "usb_port": self.usb_port, + "device_id": self.device_id, + "host_id": self.host_id, + "sane_device": self.sane_device, + "target_base_dir": str(self.target_base_dir), + } + # ---- public wrapper ---- + @abstractmethod + def _scan_roi(self, dest_path: Path) -> None: + """ + Concrete implementations should generate the ROI image + and write it directly to dest_path. + """ + raise NotImplementedError + + # ---- abstract inner implementation ---- + def scan_roi( + self, + timestamp: datetime, + final_filename: Optional[str] = None, + subdir: str = None, + extra_metadata: Dict = dict() + ) -> Dict[str, Any]: + """ + Wrapper that: + 1) decides the final output path (custom filename if provided) + 2) writes to a temporary file prefixed with '.tmp.' + 3) creates a 512px-wide JPEG thumbnail .thumb.jpg + 4) calculates md5 of the image BEFORE moving it into place + 5) atomically moves the tmp file to the final destination + 6) writes metadata JSON alongside the image + Returns: + dict metadata + """ + if not subdir: + date = timestamp.strftime("%Y-%m-%d") + datetime_ = timestamp.strftime("%Y-%m-%d_%H-%M-%S") + subdir = os.path.join(self.host_id, self.device_id,date,datetime_,self.name) + + out_dir = self.target_base_dir / subdir + out_dir.mkdir(parents=True, exist_ok=True) + + # Determine final filename + ts = timestamp + if final_filename: + filename = final_filename + else: + filename = timestamp.strftime("%Y-%m-%d_%H-%M-%S") + "." + self.name + "." + self.file_format + + final_path = out_dir / filename + tmp_path = final_path.with_name(f".tmp.{final_path.name}") + + # Clean leftover tmp + if tmp_path.exists(): + try: + tmp_path.unlink() + except Exception: + pass + + # 1) Let subclass write the image to tmp_path + self._scan_roi(dest_path=tmp_path) + + # 2) Compute md5 BEFORE moving + md5 = hashlib.md5() + with open(tmp_path, "rb") as fh: + for chunk in iter(lambda: fh.read(1 << 20), b""): + md5.update(chunk) + md5_hex = md5.hexdigest() + + # 3) Make a 512px-wide JPEG thumbnail from tmp file + # .thumb.jpg in the same directory + with Image.open(tmp_path) as img: + # keep aspect ratio; ensure RGB for JPEG + img = img.convert("RGB") + w, h = img.size + if w > 512: + new_h = int(round(h * (512.0 / float(w)))) + img = img.resize((512, max(1, new_h))) + thumb_name = f"{final_path.stem}.thumb.jpg" + thumb_path = out_dir / thumb_name + # write to a temp thumbnail then move, mirroring main image flow + tmp_thumb_path = thumb_path.with_name(f".tmp.{thumb_path.name}") + if tmp_thumb_path.exists(): + try: + tmp_thumb_path.unlink() + except Exception: + pass + img.save(tmp_thumb_path, format="JPEG", quality=85, optimize=True) + os.replace(tmp_thumb_path, thumb_path) + + # 4) Atomically move the main image into place + os.replace(tmp_path, final_path) + + # 5) Build metadata (basedir, filename, thumbnail, md5) + more_metadata: Dict[str, Any] = { + "timestamp": ts.isoformat(), + "dirname": str(out_dir), + "filename": final_path.name, + "uri": os.path.join(str(out_dir), final_path.name), + "thumbnail": thumb_path.name, + "md5": md5_hex, + } + metadata = more_metadata + metadata.update(self._contruction_args()) + metadata["scan_metadata"] = extra_metadata + # 6) Write metadata json alongside image: .json + metadata_json_path = out_dir / f"{final_path.stem}.json" + with open(metadata_json_path, "w", encoding="utf-8") as f: + json.dump(metadata, f, indent=2) + + # 7) Return only metadata + return metadata + + # ---- helpers ---- + @staticmethod + def _mm_to_pixels(width_mm: float, height_mm: float, dpi: int) -> Tuple[int, int]: + width_in = max(0.0, width_mm) / 25.4 + height_in = max(0.0, height_mm) / 25.4 + w_px = max(1, int(round(width_in * dpi))) + h_px = max(1, int(round(height_in * dpi))) + return w_px, h_px + + # def _filename(self, timestamp: datetime, scan_id: Optional[str]) -> str: + # sid = scan_id or uuid.uuid4().hex + # ts = timestamp.strftime("%Y-%m-%d_%H-%M-%S") + # return f"{sid}.{ts}.{self.name}.{self.file_format}" + +class ROIScanner(BaseROIScanner): + """ + Concrete ROI scanner that uses `scanimage` (SANE). + - Writes the raw image bytes directly to the provided tmp dest_path. + - Uses mm ROI from BaseROIScanner (x0,y0,x1,y1) and dpi to set the scan area. + - Raises well-typed exceptions on failure (missing binary, timeout, nonzero exit). + """ + + def __init__( + self, + *args, + mode: str = "Color", # "Color", "Gray", "Lineart" (depends on device) + timeout_sec: int = 240, # kill long-hanging scans + extra_scanimage_args: Optional[list[str]] = None, + **kwargs + ) -> None: + super().__init__(*args, **kwargs) + self.mode = mode + self.timeout_sec = timeout_sec + self.extra_scanimage_args = extra_scanimage_args or [] + + # Map Python-side file_format to `scanimage --format=...` + self._fmt_map = { + "jpg": "jpeg", + "jpeg": "jpeg", + "png": "png", + "tif": "tiff", + "tiff": "tiff", + "pnm": "pnm", + } + if self.file_format not in self._fmt_map: + raise ValueError( + f"file_format '{self.file_format}' not supported by scanimage. " + "Use one of: jpg, jpeg, png, tif, tiff, pnm" + ) + + + + def _scan_roi(self, dest_path: Path) -> None: + """ + Use `scanimage` to capture the ROI and write to dest_path. + Any error on stderr / non-zero exit → ScanImageError. + """ + + logging.info(f"scanning {self.name}") + print(f"scanning {self.name}") + # ---- Validate ROI & compute SANE area (in mm) ---- + x0, y0, x1, y1 = float(self.x0_mm), float(self.y0_mm), float(self.x1_mm), float(self.y1_mm) + logging.info([x0, y0, x1, y1] ) + if x1 <= x0 or y1 <= y0: + raise ValueError(f"Invalid ROI: ({x0},{y0})–({x1},{y1}) mm") + width_mm = x1 - x0 + height_mm = y1 - y0 + # SANE opts: -l (left), -t (top), -x (width), -y (height) — all in mm. + + # ---- Build command ---- + fmt = self._fmt_map[self.file_format] # jpeg/png/tiff/pnm + # Note: many devices expose "mode" and "resolution"; names can vary by backend. + # These flags are the generic ones used by scanimage. + + cmd = [ + "scanimage", + f"--mode={self.mode}", + f"--resolution={int(self.dpi)}", + f"-l", f"{x0:.2f}", + f"-t", f"{y0:.2f}", + f"-x", f"{width_mm - x0:.2f}", + f"-y", f"{height_mm - y0:.2f}", + f"--format={fmt}", + f"--out={dest_path}", + ] + + + cmd.append(f"--device-name={self.sane_device}") + + # Any additional backend-specific options (e.g., --brightness, --contrast, --source ADF) + cmd.extend(self.extra_scanimage_args) + + # ---- Execute, stream stdout → dest_path ---- + # We stream to file to avoid buffering huge images in RAM. + env = os.environ.copy() + env.setdefault("LC_ALL", "C") # English messages -> easier parsing/logs + + try: + logging.info(cmd) + print(cmd) + proc = subprocess.run( + cmd, + stderr=subprocess.PIPE, # capture for diagnostics + check=False, # we handle returncode ourselves + env=env, + timeout=self.timeout_sec, + ) + except FileNotFoundError as e: + # `scanimage` not found on PATH + raise ScanImageError( + "scanimage not found. Is SANE installed and on your PATH?" + ) from e + except subprocess.TimeoutExpired as e: + # Clean up partial file if we timed out + try: + if dest_path.exists(): + dest_path.unlink() + except Exception: + pass + raise ScanImageError( + f"scanimage timed out after {self.timeout_sec}s" + ) from e + except Exception as e: + # Any other OS-level issue + try: + if dest_path.exists(): + dest_path.unlink() + except Exception: + pass + raise ScanImageError(f"Unexpected error invoking scanimage: {e}") from e + + # ---- Check exit status and stderr ---- + stderr_text = proc.stderr.decode("utf-8", errors="replace").strip() + if proc.returncode != 0: + # Remove partial file on failure + try: + if dest_path.exists(): + dest_path.unlink() + except Exception: + pass + raise ScanImageError( + f"scanimage failed (exit {proc.returncode}).\n" + f"Command: {shlex.join(cmd)}\n" + f"Stderr:\n{stderr_text}" + ) + + # Some backends print warnings to stderr even with exit 0. + # You can choose to treat certain patterns as fatal; here we log or elevate if obviously bad. + if "error" in stderr_text.lower() or "failed" in stderr_text.lower(): + # Remove partial file when we detect an error-ish message + try: + if dest_path.exists(): + dest_path.unlink() + except Exception: + pass + raise ScanImageError( + f"scanimage reported an error despite exit 0.\n" + f"Command: {shlex.join(cmd)}\n" + f"Stderr:\n{stderr_text}" + ) + + # Final sanity check: destination file should be non-empty + try: + if dest_path.stat().st_size == 0: + raise ScanImageError( + "scanimage produced an empty file (0 bytes)." + ) + except FileNotFoundError: + raise ScanImageError( + "scanimage did not produce an output file." + ) + +class MockROIScanner(BaseROIScanner): + def __init__(self, *args, seed: Optional[int] = None, jpeg_quality: int = 90, **kwargs) -> None: + super().__init__(*args, **kwargs) + self._rng = random.Random(seed) + self.jpeg_quality = jpeg_quality + if Image is None: + raise RuntimeError("Pillow is required for MockROIScanner (pip install Pillow)") + + def _scan_roi(self, dest_path: Path) -> None: + """ + Generate a synthetic image for this ROI and write it to dest_path. + """ + # Geometry → pixels + width_mm = max(0.0, self.x1_mm - self.x0_mm) + height_mm = max(0.0, self.y1_mm - self.y0_mm) + w_px, h_px = self._mm_to_pixels(width_mm, height_mm, dpi=self.dpi) + w_px, h_px = max(32, w_px), max(32, h_px) + + # Synthetic image (random noise) + buf = bytearray(w_px * h_px * 3) + for i in range(0, len(buf), 3): + buf[i] = self._rng.randint(0, 255) + buf[i + 1] = self._rng.randint(0, 255) + buf[i + 2] = self._rng.randint(0, 255) + img = Image.frombytes("RGB", (w_px, h_px), bytes(buf)) + + # Save directly to the tmp path + if self.file_format in ("jpg", "jpeg"): + img.save(dest_path, format="JPEG", quality=self.jpeg_quality, optimize=True) + else: + img.save(dest_path) + +from abc import ABC, abstractmethod +from typing import Optional, Dict, Any, List + +class BaseGridScanner(ABC): + """ + Holds: + - a list of ROI scanners (grid cells) -> self.roi_cells + - a 'preview_roi' (BaseROIScanner) + - a 'barcode_roi' (BaseROIScanner) used by get_metadata() + """ + + def __init__( + self, + roi_cells: List[BaseROIScanner], + preview_roi: BaseROIScanner, + barcode_roi: BaseROIScanner, + driver: Optional[str] = "epson2", + + ) -> None: + self.roi_cells = list(roi_cells) + self.preview_roi = preview_roi + self.barcode_roi = barcode_roi + self.enforce_barcode = False + + # ---- Public operations ---- + + def get_barcode_metadata(self) -> Dict[str, Any]: + static_name = f"barcode.{self.barcode_roi.file_format}" + ts = datetime.now() + image_metadata = self.barcode_roi.scan_roi(final_filename=static_name, subdir="temporary", timestamp=ts) + image_path = Path(image_metadata["uri"]) + parsed = self._parse_barcode(image_path) + return parsed + + def _create_svg_preview(self, jpg_path: str | Path) -> Path: + """ + Build an SVG overlay: + - Embedded raster preview (JPEG) positioned by preview_roi mm box + - Barcode ROI in red (with black halo) + label + - Other ROIs in yellow (with black halo) + labels + """ + from xml.etree.ElementTree import Element, SubElement, ElementTree + import base64 + from pathlib import Path + + jpg_path = Path(jpg_path) + with open(jpg_path, "rb") as f: + img_b64 = base64.b64encode(f.read()).decode("utf-8") + + # ---- Colors & style knobs --------------------------------------------- + YELLOW = "#FFD400" + RED = "#FF2D2D" + BLACK = "#000000" + + ROI_STROKE_MM = 0.40 # visible colored stroke + ROI_HALO_EXTRA_MM = 0.80 # extra thickness for the black halo (behind) + TEXT_DEFAULT_MM = 3.5 + TEXT_HALO_MM = 0.8 + LABEL_PAD_MM = 1.5 + + # ---- Canvas in mm (A4) ------------------------------------------------ + svg = Element( + "svg", + { + "xmlns": "http://www.w3.org/2000/svg", + "xmlns:xlink": "http://www.w3.org/1999/xlink", + "width": "210mm", + "height": "297mm", + "viewBox": "0 0 210 297", + "version": "1.1", + }, + ) + + # ---- Place the preview raster ----------------------------------------- + px = float(self.preview_roi.x0_mm) + py = float(self.preview_roi.y0_mm) + pw = float(self.preview_roi.x1_mm - self.preview_roi.x0_mm) + ph = float(self.preview_roi.y1_mm - self.preview_roi.y0_mm) + + SubElement( + svg, "image", + { + "x": f"{px}", + "y": f"{py}", + "width": f"{pw}", + "height": f"{ph}", + "preserveAspectRatio": "none", + "href": f"data:image/jpeg;base64,{img_b64}", + "{http://www.w3.org/1999/xlink}href": f"data:image/jpeg;base64,{img_b64}", + } + ) + + def _rect_with_halo(parent, x, y, w, h, label=None, color=YELLOW): + """Draw a black halo rectangle behind a colored outline, then optional label.""" + # Halo (behind) + SubElement(parent, "rect", { + "x": f"{x}", "y": f"{y}", "width": f"{w}", "height": f"{h}", + "fill": "none", + "stroke": BLACK, + "stroke-width": f"{ROI_STROKE_MM + ROI_HALO_EXTRA_MM}", + "vector-effect": "non-scaling-stroke", + "stroke-linejoin": "round", + }) + # Main (front, colored) + SubElement(parent, "rect", { + "x": f"{x}", "y": f"{y}", "width": f"{w}", "height": f"{h}", + "fill": "none", + "stroke": color, + "stroke-width": f"{ROI_STROKE_MM}", + "vector-effect": "non-scaling-stroke", + "stroke-linejoin": "round", + }) + if label: + font_mm = max(2.0, min(TEXT_DEFAULT_MM, h * 0.25)) + text = SubElement(parent, "text", { + "x": f"{x + LABEL_PAD_MM}", + "y": f"{y + LABEL_PAD_MM}", + "font-size": f"{font_mm}", + "font-family": "DejaVu Sans, Arial, sans-serif", + "fill": color, # colored text + "stroke": BLACK, # black halo + "stroke-width": f"{TEXT_HALO_MM}", + "paint-order": "stroke fill", + "text-anchor": "start", + "dominant-baseline": "hanging", + }) + text.text = label + + # ---- Barcode ROI in RED ------------------------------------------------ + bx0, by0, bx1, by1 = map(float, self.barcode_roi.xyxy_mm) + _rect_with_halo( + svg, + x=bx0, y=by0, w=bx1 - bx0, h=by1 - by0, + label=getattr(self.barcode_roi, "name", "barcode"), + color=RED + ) + + # ---- Other ROIs in YELLOW --------------------------------------------- + for roi in self.roi_cells: + x0, y0, x1, y1 = map(float, roi.xyxy_mm) + _rect_with_halo( + svg, + x=x0, y=y0, w=x1 - x0, h=y1 - y0, + label=getattr(roi, "name", "ROI"), + color=YELLOW + ) + + out_svg = jpg_path.with_suffix(".svg") + ElementTree(svg).write(out_svg, encoding="utf-8", xml_declaration=True) + return out_svg + + def generate_preview( + self) -> Dict: + + ts = datetime.now() + static_name = f"preview.{self.preview_roi.file_format}" + image_metadata = self.preview_roi.scan_roi(final_filename=static_name, subdir="temporary", timestamp=ts) + svg = self._create_svg_preview(image_metadata["uri"]) + image_metadata["svg"] = svg + return image_metadata + + + def scan_grid(self) -> List[Dict[str, Any]]: # <-- fixed typing + """ + Returns: + { + "metadata": {...}, + "timestamp": , + "scan_dir": "///>", + "roi_paths": [Path, ...], + "metadata_path": Path + } + """ + # 1) metadata (scans barcode to temporary/) + barcode_metadata = self.get_barcode_metadata() + # scan_id = self._ensure_scan_id(metadata) + + # 2) timestamp + date dir + ts = datetime.now() + # 4) all ROIs (timestamped) — same directory + roi_metadata: List[Dict] = [] + for roi in self.roi_cells: + roi_metadata.append(roi.scan_roi(timestamp=ts, extra_metadata = barcode_metadata)) + + # 5) dump metadata as JSON file in the same directory + + return roi_metadata + + # ---- Abstracts & helpers ---- + + @abstractmethod + def _parse_barcode(self, barcode_path: Path) -> Dict[str, Any]: + raise NotImplementedError + + +class BaseJsonGridScanner(BaseGridScanner): + RoiScannerClass: BaseROIScanner = BaseROIScanner + def __init__( + self, + config: JsonLike, + target_base_dir: Optional[Path | str] = None, + driver: Optional[str] = "epson2", # <-- default driver at JSON-grid level + usb_port: Optional[str] = None, + ) -> None: + cfg = _load_config(copy.deepcopy(config)) + + info = self._probe_scanner_info() + driver: Optional[str] = info.get("driver") + usb_port: Optional[str] = info.get("usb_port") + # Prefer explicit device, then usb_port, then any ROI usb_port, finally driver + device_id: str = info.get("device_id") + sane_device: str = info.get("sane_device") + host_id: str = _read_host_id() + + # minimal presence checks + if "global" in cfg: + try: + self.enforce_barcode: bool = cfg["global"]["enforce_barcode"] + except KeyError as e: + logging.warning(cfg) + raise e + pass + + for key in ("preview", "barcode", "rois"): + if key not in cfg: + raise ConfigError(f"root: missing required key '{key}'") + if not isinstance(cfg["rois"], list) or not cfg["rois"]: + raise ConfigError("'rois' must be a non-empty list") + + for r in ["preview", "barcode"]: + cfg[r]["name"] = r + cfg[r]["driver"] = driver + cfg[r]["usb_port"] = usb_port + cfg[r]["target_base_dir"] = target_base_dir + cfg[r]["device_id"] = device_id + cfg[r]["host_id"] = host_id + cfg[r]["sane_device"] = sane_device + + preview_roi = self.RoiScannerClass(**cfg["preview"]) + barcode_roi = self.RoiScannerClass(**cfg["barcode"]) + + roi_cells = [] + for i, c in enumerate(cfg["rois"]): + if "name" not in c.keys(): + c["name"] = f"{i:02d}" + print(c) + c["driver"] = driver + c["usb_port"] = usb_port + c["target_base_dir"] = target_base_dir + c["device_id"] = device_id + c["host_id"] = host_id + c["sane_device"] = sane_device + roi_cells.append(self.RoiScannerClass(**c)) + + super().__init__( + roi_cells=roi_cells, + preview_roi=preview_roi, + barcode_roi=barcode_roi, + driver=driver, # <-- pass through to BaseGridScanner + ) + + self.config = cfg # optional: keep original config + @abstractmethod + def _probe_scanner_info(self) -> Dict[str, Optional[str]]: + """ + Best-effort probe of the attached scanner. + + Returns: + dict with keys: + - 'device_id': a stable identifier you want in paths/metadata + - 'driver': backend name (e.g. 'epson2', 'genesys') or None + - 'usb_port': full SANE device string (e.g. 'epson2:libusb:001:004') or None + + This method should not raise fatally; return None/empty/partial info if unknown. + """ + raise NotImplementedError + +class MockJsonGridScanner(BaseJsonGridScanner): + RoiScannerClass: BaseROIScanner = MockROIScanner + + def _probe_scanner_info(self) -> Dict[str, Optional[str]]: + # Pure mock + return {"device_id": "mock", "driver": None, "usb_port": None} + + def _parse_barcode(self, barcode_path: Path) -> Dict[str, Any]: + return {"scan_id": f"{random.randint(0, 9999):06d}"} + + +class JsonGridScanner(BaseJsonGridScanner): + RoiScannerClass: BaseROIScanner = ROIScanner + + def _probe_scanner_info(self) -> Dict[str, Optional[str]]: + + out = subprocess.check_output( + ["bash", "-lc", "scanimage -L"], + stderr=subprocess.STDOUT, text=True + ) + + for line in out.splitlines(): + parsed = _parse_scanimage_device_line(line) + if not parsed: + continue + sane_device = parsed["sane_device"] + driver = parsed["driver"] + device_name = parsed["device_name"] + usb_id = parsed["usb_id"] + return { + "sane_device": sane_device, + "device_id": device_name, + "driver": driver, + "usb_port": usb_id, + } + raise ScannerNotFoundError + + def _parse_barcode(self, barcode_path: Path) -> Dict[str, Any]: + """ + Read a Data Matrix from 'barcode_path' and return a dict payload. + Rules: + 1) Must decode exactly 1 symbol -> else raise ValueError + 2) Decoded payload must parse to a mapping (dictionary) -> else raise ValueError + """ + # 1) Load image + try: + try: + img = Image.open(barcode_path) + except Exception as e: + raise ValueError(f"Failed to open barcode image '{barcode_path}': {e}") + + # Optional: convert to grayscale to help decoder on noisy images + try: + img = img.convert("L") + except Exception: + # If convert fails for some reason, proceed with original + pass + + # 2) Decode Data Matrix + try: + results = dmtx_decode(img) + except Exception as e: + raise ValueError(f"pylibdmtx failed to decode '{barcode_path}': {e}") + + if not results or len(results) != 1: + found = 0 if not results else len(results) + raise ValueError(f"Expected exactly 1 Data Matrix, found {found} in '{barcode_path}'.") + + raw_bytes = results[0].data + try: + payload_text = raw_bytes.decode("utf-8") + except Exception as e: + raise ValueError(f"Decoded Data Matrix is not valid UTF-8: {e}") + + # 3) Parse into a dictionary (prefer JSON; fallback to YAML if you want) + parsed: Any = None + json_err = yaml_err = None + + try: + parsed = json.loads(payload_text) + except Exception as e: + json_err = e + if not isinstance(parsed, dict): + details = [] + if json_err: details.append(f"json: {json_err}") + if yaml_err: details.append(f"yaml: {yaml_err}") + detail_txt = "; ".join(details) if details else "payload is not a mapping" + parsed = {"sample_id": payload_text} + logging.warning(f"Decoded Data Matrix did not yield a dictionary. Returning instead {parsed}. {detail_txt}, {payload_text}") + + + # 4) Return the dictionary + return parsed + except Exception as e: + if self.enforce_barcode: + raise e + else: + logging.warning(str(e)) + return {} diff --git a/services/scanner-api/start.sh b/services/scanner-api/start.sh new file mode 100644 index 0000000..b3eda25 --- /dev/null +++ b/services/scanner-api/start.sh @@ -0,0 +1,15 @@ +#!/bin/sh +set -e + +# Ensure the mountpoint exists and is owned by appuser +mkdir -p /data + +# 1) relax perms on host-mounted USB nodes +if [ -d /dev/bus/usb ]; then + chmod -R a+rw /dev/bus/usb || true +fi + +chown -R appuser:appuser /data + +# Drop privileges to appuser to run the server +exec su -s /bin/sh -c "uvicorn main:app --host 0.0.0.0 --port 8000 --proxy-headers" appuser diff --git a/services/webapp/Dockerfile b/services/webapp/Dockerfile new file mode 100644 index 0000000..1112419 --- /dev/null +++ b/services/webapp/Dockerfile @@ -0,0 +1,47 @@ +# R 4.4.1 on Debian bookworm +FROM rocker/shiny:4.4.1 + +# --- System dependencies (minimal, no recommends) --- +RUN apt-get update && apt-get install -y --no-install-recommends \ + sudo \ + gdebi-core \ + # pandoc is already in rocker/shiny but harmless if reinstalled + pandoc pandoc-citeproc \ + libcurl4-openssl-dev \ + libssl-dev \ + libxml2-dev \ + libcairo2-dev \ + libxt-dev \ + libicu-dev \ + libsodium-dev \ + libmariadb-dev \ + xtail \ + wget \ + libpq5 libpq-dev \ + && rm -rf /var/lib/apt/lists/* + +# --- Use Posit Package Manager (stable, deterministic binaries) --- +ENV RSPM=https://packagemanager.posit.co/cran/__linux__/debian/bookworm/latest + +# Install base deps first (helps avoid cascading failures) +RUN install2.r --error \ + curl openssl R6 cli glue jsonlite + +# Install app packages (including httr2) +RUN install2.r --error \ + shinydashboard DT shinyjs sodium RMariaDB data.table httr2 lubridate \ + plotly ggplot2 htmlwidgets zoo fasttime gridExtra shinyBS xml2 \ + reactlog shinyFeedback shinyTime markdown + +RUN install2.r --error RPostgres DBI pool DT + + +# --- Shiny Server config and app --- +COPY shiny-server.conf /etc/shiny-server/shiny-server.conf +COPY shiny-server.sh /usr/bin/shiny-server.sh +RUN chmod 700 /usr/bin/shiny-server.sh + +COPY app /srv/shiny-server/app + +EXPOSE 8080 +CMD ["/usr/bin/shiny-server.sh"] diff --git a/services/webapp/app/app.R b/services/webapp/app/app.R new file mode 100644 index 0000000..adc96b2 --- /dev/null +++ b/services/webapp/app/app.R @@ -0,0 +1,557 @@ +# ========================= +# EntoScan Shiny Dashboard +# ========================= + +# ---- Packages ---- +rm(list = ls()) + +library(shiny) +library(shinydashboard) +library(shinyjs) +library(DT) +library(httr2) +library(magrittr) +library(jsonlite) +library(htmltools) +library(DBI) +library(RPostgres) +library(pool) + +# ---- Helpers & Globals ---- +`%||%` <- function(x, y) if (is.null(x) || (is.character(x) && length(x) == 0)) y else x + +# Backends / routing +api_base_internal <- Sys.getenv("API_BASE", "http://api:8000") # service-to-service +api_base_public <- Sys.getenv("PUBLIC_API_BASE", "http://localhost:8000") # browser-visible + +# External Label Maker UI +label_maker_url <- Sys.getenv( + "LABEL_MAKER_URL", + "https://darsa.info/EntoScan-labels/#fixme" # sensible default if env var not set +) + +# DB pool (close on session end) +db_pool <- pool::dbPool( + drv = RPostgres::Postgres(), + host = Sys.getenv("DB_HOST", "db"), + port = as.integer(Sys.getenv("DB_PORT", "5432")), + dbname = Sys.getenv("DB_NAME", "postgres"), + user = Sys.getenv("DB_USER", "postgres"), + password = Sys.getenv("DB_PASS", "postgres") +) +onStop(function() pool::poolClose(db_pool)) + +# Build absolute URLs for files served by /files (backend mounts /data → /files) +make_api_uri <- function(dirname, filename) { + rel <- sub("^/data/?", "", dirname %||% "") + paste0(api_base_public, "/files/", rel, "/", filename) +} + +# Busy modal helpers +show_busy_modal <- function(msg = "Working…", session = shiny::getDefaultReactiveDomain()) { + showModal( + modalDialog( + easyClose = FALSE, footer = NULL, + tags$div(class = "text-center", + tags$div(class = "spinner", icon("spinner", class = "fa-spin")), + tags$div(class = "msg", msg) + ), + size = "m" + ), session = session + ) + shinyjs::runjs("console.log('[UI] show_busy_modal');") + shinyjs::runjs("$('.modal').addClass('busy-modal');") +} +hide_busy_modal <- function(session = shiny::getDefaultReactiveDomain()) { + shinyjs::runjs("console.log('[UI] hide_busy_modal');") + removeModal(session = session) +} + +# /is_busy poller +fetch_busy <- function() { + req <- request(paste0(api_base_internal, "/is_busy")) |> req_method("GET") + res <- tryCatch(req_perform(req), error = function(e) e) + if (inherits(res, "error") || resp_status(res) >= 300) return(NA) + json <- resp_body_json(res) + isTRUE(json$busy) +} + +# Load config YAML from backend +load_config <- function(show_errors = TRUE, output = NULL) { + req <- request(paste0(api_base_internal, "/config")) |> req_method("GET") + res <- tryCatch(req_perform(req), error = function(e) e) + if (inherits(res, "error") || resp_status(res) >= 300) { + if (show_errors && !is.null(output)) { + msg <- if (inherits(res, "error")) res$message else tryCatch(resp_body_string(res), error = function(e) "") + output$error_msg <- renderUI(tags$div(style="color:#b30000;", paste("Load failed:", msg))) + } + return(NULL) + } + tryCatch(resp_body_string(res), error = function(e) "") +} + +# Query images_json and return a data.frame ready for DT +load_rois_df <- function(limit = 1000, offset = 0) { + sql <- " + SELECT + inserted_at, + (data->>'dirname') AS dirname, + (data->>'filename') AS filename, + (data->>'thumbnail') AS thumbnail, + (data->>'md5') AS md5, + (data->>'name') AS roi_name, + (data->>'timestamp') AS timestamp, + (data->>'device_id') AS device_id, + (data->'xyxy_mm')::text AS xyxy_mm_json, + (data->'scan_metadata')::text AS scan_metadata_json + FROM images_json + ORDER BY inserted_at DESC + LIMIT $1 OFFSET $2; + " + cat("[DB] Query images_json\n") + df <- DBI::dbGetQuery(db_pool, sql, params = list(limit, offset)) + + # Parse JSON columns and assemble display fields + parse_json_vec <- function(x, fallback) { + vapply(x, function(s) { + out <- tryCatch(jsonlite::fromJSON(if (is.na(s) || s == "") fallback else s), + error = function(e) fallback) + I(list(out)) + }, I(list(fallback))) + } + xy_list <- parse_json_vec(df$xyxy_mm_json, numeric()) + sm_list <- parse_json_vec(df$scan_metadata_json, list()) + + get_coord <- function(lst, i) if (length(lst) >= i) as.character(lst[[i]]) else "" + x1 <- vapply(xy_list, get_coord, character(1), i = 1) + y1 <- vapply(xy_list, get_coord, character(1), i = 2) + x2 <- vapply(xy_list, get_coord, character(1), i = 3) + y2 <- vapply(xy_list, get_coord, character(1), i = 4) + + sm_text <- vapply(sm_list, function(m) { + if (length(m) == 0) return("") + paste(paste0(names(m), "=", unlist(m, use.names = FALSE)), collapse = ", ") + }, character(1)) + + thumb_url <- make_api_uri(df$dirname, df$thumbnail) + file_url <- make_api_uri(df$dirname, df$filename) + + thumb_html <- sprintf( + "", + htmlEscape(file_url), htmlEscape(df$md5), + htmlEscape(thumb_url), htmlEscape(df$md5) + ) + + ts_num <- suppressWarnings(as.numeric(as.POSIXct(df$timestamp, tz = "UTC"))) + + df_out <- data.frame( + `Thumbnail` = thumb_html, + `ROI Name` = df$roi_name %||% "(unnamed)", + x1 = x1, y1 = y1, x2 = x2, y2 = y2, + `Timestamp` = df$timestamp %||% "", + `Device ID` = df$device_id %||% "", + `Scan metadata` = sm_text, + .ts_sort = ts_num, + check.names = FALSE, + stringsAsFactors = FALSE + ) + + cat(sprintf("[DB] Loaded %d rows\n", nrow(df_out))) + df_out +} + +# Render DataTable +render_rois_dt <- function(df) { + if (is.null(df) || !nrow(df)) { + return(DT::datatable( + data.frame(Message = "No ROI files found yet. Run a scan to populate the list."), + options = list(dom = 't'), rownames = FALSE + )) + } + DT::datatable( + df, + rownames = FALSE, + escape = FALSE, # allow thumbnail HTML + filter = "top", + options = list( + pageLength = 20, + lengthMenu = c(10, 20, 50, 100), + order = list(list(which(colnames(df) == ".ts_sort") - 1L, "desc")), + columnDefs = list( + list(targets = which(colnames(df) == ".ts_sort") - 1L, visible = FALSE, searchable = FALSE), + list(targets = 0, width = "110px") + ), + autoWidth = TRUE, + deferRender = TRUE + ), + class = "stripe hover compact" + ) +} + +# ---- UI ---- +ui <- dashboardPage(title = "EntoScan", + dashboardHeader( title = tags$div( + tags$img( + src = "img/icon.png", + height = "64px", + style = "margin-right:8px;vertical-align:middle;" + ), + "EntoScan" + ) +), + dashboardSidebar( + sidebarMenu(id = "tabs", + menuItem("Config/Preview", tabName = "config", icon = icon("cog")), + menuItem("Scan", tabName = "scan", icon = icon("camera")), + menuItem("Files", tabName = "files", icon = icon("folder-open")), + menuItem( + "Label Maker", + icon = icon("tags"), + href = label_maker_url, + newtab = TRUE # open in new browser tab + ), + menuItem("About", tabName = "about", icon = icon("info-circle")) + ) + ), + dashboardBody( + useShinyjs(), + tags$head( + tags$title("EntoScan"), + tags$link(rel = "icon", type = "image/png", href = "img/icon.png"), + tags$style(HTML(" + .roi-summary { margin-bottom: 10px; } + .roi-grid { display: flex; flex-wrap: wrap; gap: 12px; } + .roi-tile { width: 180px; border: 1px solid #ddd; border-radius: 8px; + padding: 8px; text-align: center; background: #fff; + box-shadow: 0 1px 2px rgba(0,0,0,0.04); } + .roi-tile img { width: 100%; height: auto; border-radius: 4px; } + .roi-tile .roi-name { margin-top: 6px; font-weight: 600; font-size: 14px; } + .busy-modal .modal-content { border-radius: 12px; } + .busy-modal .spinner { font-size: 48px; margin-bottom: 12px; } + .busy-modal .msg { font-size: 18px; } + #coord-tooltip { + position: absolute; background: rgba(0,0,0,0.75); color: #fff; + padding: 4px 8px; border-radius: 4px; font-size: 13px; + pointer-events: none; display: none; z-index: 10000; + } + ")), + # ========= JS: robust binding + debug logs ========= + tags$script(src = "tooltip.js") # <-- load external file + ), + + # 👇 Global tooltip lives ONCE (NOT inside preview output) + tags$div(id = "coord-tooltip"), + + tabItems( + # ---- Config ---- + tabItem(tabName = "config", + fluidRow( + box( + title = "Configuration", status = "primary", solidHeader = TRUE, width = 6, + actionButton("preview_btn", "New preview", icon = icon("camera")), + textAreaInput("yaml_editor", label = NULL, value = "", rows = 16, + placeholder = "Paste or edit scanner YAML here..."), + br(), br(), + textOutput("preview_ts"), + uiOutput("error_msg") + ), + box( + title = "Preview", status = "info", solidHeader = TRUE, width = 6, + uiOutput("preview_img") + ) + ) + ), + + # ---- Scan ---- + tabItem(tabName = "scan", + fluidRow( + column( + width = 12, + box(title = "Scan", status = "primary", solidHeader = TRUE, width = NULL, + actionButton("scan_btn", "Scan", icon = icon("play")), + br(), br(), uiOutput("error_msg_scan")), + box(title = "Results", status = "info", solidHeader = TRUE, width = NULL, + uiOutput("scan_summary"), uiOutput("scan_tiles")) + ) + ) + ), + + # ---- Files ---- + tabItem(tabName = "files", + fluidRow( + box( + title = "Files", status = "primary", solidHeader = TRUE, width = 12, + fluidRow( + column(6, actionButton("refresh_files", "Refresh list", icon = icon("rotate"))), + column(6, align = "right", + tags$small("Listing pulled directly from Postgres; images served from ", tags$code("/files"), ".")) + ), + br(), DT::DTOutput("files_table_dt"), br(), + HTML(paste0("

Open raw /files/ directory

")) + ) + ) + ), + + # ---- About ---- + tabItem(tabName = "about", + fluidRow( + box( + title = "About EntoScan", status = "primary", solidHeader = TRUE, width = 8, + uiOutput("about_content") + ) + ) + ) + ) + ) +) + +# ---- Server ---- +server <- function(input, output, session) { + # State + preview_url <- reactiveVal(NULL) + preview_ts <- reactiveVal("") + scanner_busy <- reactiveVal(TRUE) + prev_busy <- reactiveVal(NA) + modal_shown <- reactiveVal(FALSE) + + shinyjs::disable("preview_btn") + shinyjs::disable("scan_btn") + + # Poll busy & toggle UI + observe({ + invalidateLater(1000, session) + busy <- fetch_busy() + if (is.na(busy)) return() + + if (!identical(prev_busy(), busy)) { + cat(sprintf("[Poll] /is_busy = %s @ %s\n", busy, format(Sys.time(), "%H:%M:%S"))) + shinyjs::runjs(sprintf("console.log('[Poll] /is_busy = %s')", ifelse(busy, "true", "false"))) + } + + was_busy <- prev_busy() + prev_busy(busy) + scanner_busy(busy) + + shinyjs::toggleState("preview_btn", condition = !busy) + shinyjs::toggleState("scan_btn", condition = !busy) + + if (busy && !isTRUE(modal_shown())) { show_busy_modal("Scanning in progress…", session); modal_shown(TRUE) } + if (!busy && isTRUE(modal_shown())) { hide_busy_modal(session); modal_shown(FALSE) } + + # if device just became idle, refresh config silently + if (identical(was_busy, TRUE) && identical(busy, FALSE)) { + cfg <- load_config(show_errors = FALSE, output = output) + if (!is.null(cfg)) updateTextAreaInput(session, "yaml_editor", value = cfg %||% "") + } + }) + + # Initial config load once + observeEvent(TRUE, { + cat("[Init] Loading config...\n") + cfg <- load_config(show_errors = FALSE, output = output) + if (!is.null(cfg)) updateTextAreaInput(session, "yaml_editor", value = cfg %||% "") + }, once = TRUE) + + # Tab-enter hooks + observeEvent(input$tabs, { + cat("[UI] Switched tab:", input$tabs, "\n") + if (identical(input$tabs, "config")) { + cfg <- load_config(show_errors = FALSE, output = output) + if (!is.null(cfg)) updateTextAreaInput(session, "yaml_editor", value = cfg %||% "") + } else if (identical(input$tabs, "files")) { + df <- tryCatch(load_rois_df(limit = 1000), error = function(e) { cat("[DB] Error:", e$message, "\n"); NULL }) + output$files_table_dt <- DT::renderDT(render_rois_dt(df)) + } + }, ignoreInit = TRUE) + + # Files tab refresh + observeEvent(input$refresh_files, { + cat("[UI] Refresh files clicked\n") + df <- tryCatch(load_rois_df(limit = 1000), error = function(e) { cat("[DB] Error:", e$message, "\n"); NULL }) + output$files_table_dt <- DT::renderDT(render_rois_dt(df)) + }) +observeEvent(input$preview_btn, { + cat("[Preview] Starting preview...\n") + show_busy_modal("Applying config & generating preview…", session) + modal_shown(TRUE) + shinyjs::disable("preview_btn"); shinyjs::disable("scan_btn"); scanner_busy(TRUE) + + yaml_txt <- input$yaml_editor %||% "" + if (!nzchar(yaml_txt)) { + hide_busy_modal(session) + output$error_msg <- renderUI(tags$div(style="color:#b30000;", "YAML is empty.")) + return(invisible()) + } + + # Apply config + res_apply <- tryCatch( + request(paste0(api_base_internal, "/config")) |> + req_method("POST") |> + req_body_raw(charToRaw(yaml_txt)) |> + req_headers(`Content-Type` = "text/plain") |> + req_perform(), + error = function(e) e + ) + if (inherits(res_apply, "error") || resp_status(res_apply) >= 300) { + hide_busy_modal(session) + msg <- if (inherits(res_apply, "error")) res_apply$message else tryCatch(resp_body_string(res_apply), error = function(e) "") + output$error_msg <- renderUI(tags$div(style="color:#b30000;", paste("Apply failed:", msg))) + cat("[Preview] Apply failed:", msg, "\n") + return(invisible()) + } + normalized <- tryCatch(resp_body_string(res_apply), error = function(e) "") + updateTextAreaInput(session, "yaml_editor", value = normalized %||% yaml_txt) + output$error_msg <- renderUI(tags$div(style="color:#006400;", "Configuration applied.")) + cat("[Preview] Config applied\n") + + # Trigger preview + res_prev <- tryCatch( + request(paste0(api_base_internal, "/preview")) |> req_method("POST") |> req_perform(), + error = function(e) e + ) + if (inherits(res_prev, "error") || resp_status(res_prev) >= 300) { + hide_busy_modal(session) + msg <- if (inherits(res_prev, "error")) res_prev$message else tryCatch(resp_body_string(res_prev), error = function(e) "") + output$error_msg <- renderUI(tags$div(style="color:#b30000;", paste("Preview failed:", msg))) + cat("[Preview] Preview failed:", msg, "\n") + return(invisible()) + } + + payload <- resp_body_json(res_prev) + if (is.null(payload$svg)) { + hide_busy_modal(session) + output$error_msg <- renderUI(tags$div(style="color:#b30000;", "Preview missing SVG path.")) + return(invisible()) + } + + url <- paste0(api_base_public, payload$svg, "?t=", URLencode(payload$ts, reserved = TRUE)) + preview_url(url); preview_ts(payload$ts) + cat("[Preview] Preview URL:", url, "\n") + + output$preview_img <- renderUI({ + tags$img( + id = "preview-img", + src = preview_url(), + style = "max-width:100%;height:auto;border:1px solid #ccc;border-radius:6px;" + ) + }) + + # Defer the JS bind until after the DOM update is flushed + session$onFlushed(function() { + shinyjs::runjs("console.log('[Shiny] forcing tooltip bind'); if (window.__bindPreviewTooltip) window.__bindPreviewTooltip();") + }, once = FALSE) + + output$preview_ts <- renderText(paste("Last preview timestamp:", payload$ts)) + + # Force bind after the DOM updates + shinyjs::runjs(" + console.log('[Shiny] forcing tooltip bind (deferred)'); + setTimeout(function(){ + if (window.__bindPreviewTooltip) { window.__bindPreviewTooltip(); } + else { console.warn('[Shiny] __bindPreviewTooltip missing'); } + }, 0); + ") + + # If you prefer the custom-message approach, use this instead of runjs: + # session$sendCustomMessage('bind-preview-tooltip', list()) + # (and ensure tooltip.js registers a handler for 'bind-preview-tooltip') + + # modal closes via /is_busy poller +}) + + # Scan button: run scan and render tiles + output$scan_summary <- renderUI({ NULL }) + output$scan_tiles <- renderUI({ NULL }) + observeEvent(input$scan_btn, { + cat("[Scan] Starting scan...\n") + show_busy_modal("Full scan in progress…", session) + modal_shown(TRUE) + shinyjs::disable("preview_btn"); shinyjs::disable("scan_btn"); scanner_busy(TRUE) + + res <- tryCatch( + request(paste0(api_base_internal, "/scan")) |> req_method("POST") |> req_perform(), + error = function(e) e + ) + if (inherits(res, "error") || resp_status(res) >= 300) { + hide_busy_modal(session) + msg <- if (inherits(res, "error")) res$message else tryCatch(resp_body_string(res), error = function(e) "") + output$error_msg_scan <- renderUI(tags$div(style="color:#b30000;", paste("Scan failed:", msg))) + cat("[Scan] Scan failed:", msg, "\n") + return(invisible()) + } + + payload <- tryCatch(resp_body_json(res), error = function(e) NULL) + if (is.null(payload) || is.null(payload$rois) || length(payload$rois) == 0) { + output$error_msg_scan <- renderUI(tags$div(style="color:#b30000;", "No ROIs returned.")) + cat("[Scan] No ROIs returned\n") + return(invisible()) + } + + first <- payload$rois[[1]] + dev <- first$device_id %||% "(unknown)" + tstr <- first$timestamp %||% "" + meta <- first$scan_metadata + meta_html <- if (!is.null(meta) && length(meta)) { + tags$ul(lapply(names(meta), function(k) tags$li(tags$code(k), ": ", as.character(meta[[k]])))) + } else "(none)" + + output$scan_summary <- renderUI({ + tags$div(class = "roi-summary", + tags$p(tags$b("Device:"), dev, " | ", tags$b("Timestamp:"), tstr), + tags$div(tags$b("Scan metadata: "), meta_html), + tags$hr() + ) + }) + + tiles <- lapply(payload$rois, function(r) { + thumb_abs <- paste0(api_base_public, r$thumbnail_uri, "?t=", URLencode(r$md5 %||% "", reserved = TRUE)) + full_abs <- paste0(api_base_public, r$uri, "?t=", URLencode(r$md5 %||% "", reserved = TRUE)) + tags$div(class = "roi-tile", + tags$a(href = full_abs, target = "_blank", + tags$img(src = thumb_abs, alt = r$name %||% "ROI") + ), + tags$div(class = "roi-name", r$name %||% "(unnamed)") + ) + }) + output$scan_tiles <- renderUI(tags$div(class = "roi-grid", tiles)) + output$error_msg_scan <- renderUI(tags$div(style="color:#006400;", "Scan completed. See tiles above.")) + # modal closes via /is_busy poller + }) + + # Label Maker + output$label_preview <- renderUI({ NULL }) + observeEvent(input$gen_label, { + txt <- input$label_text %||% "" + size <- as.integer(input$label_size %||% 256) + hint <- input$label_hint %||% "" + if (!nzchar(txt)) { + output$label_preview <- renderUI(tags$div(style = "color:#b30000;", "Please enter label content.")) + return(invisible()) + } + img_url <- paste0(api_base_public, "/label/datamatrix?text=", URLencode(txt, reserved = TRUE), "&size=", size) + output$label_preview <- renderUI({ + tags$div( + tags$img(src = img_url, style = "max-width:100%;height:auto;border:1px solid #ddd;padding:8px;border-radius:8px;"), + if (nzchar(hint)) tags$div(style="margin-top:8px;font-weight:600;", hint) + ) + }) + }) + + # Fallbacks + output$preview_img <- renderUI({ tags$div("No preview yet. Click 'New preview' to generate one.") }) + output$preview_ts <- renderText({ "" }) + + about_reader <- reactiveFileReader( + intervalMillis = 1000, session = session, + filePath = "www/about.md", + readFunc = readLines + ) + + output$about_content <- renderUI({ + md <- paste(about_reader(), collapse = "\n") + html <- markdown::markdownToHTML(text = md, fragment.only = TRUE) + HTML(html) + }) +} + +shinyApp(ui, server) diff --git a/services/webapp/app/www/about.md b/services/webapp/app/www/about.md new file mode 100644 index 0000000..a597d82 --- /dev/null +++ b/services/webapp/app/www/about.md @@ -0,0 +1,232 @@ +![https://github.com/darsa-group/EntoScan/](https://img.shields.io/badge/github-%23121011.svg?style=for-the-badge&logo=github&logoColor=white) +EntoScan logo + + + +**Table of Contents** + +- [Flatbed Scanners For Entomology Uses and Limitations](#flatbed-scanners-for-entomology-uses-and-limitations) +- [What EntoScan Is](#what-entoscan-is) +- [Hardware Setup and Focus Considerations](#hardware-setup-and-focus-considerations) + - [Modify the Epson Perfection V850 Pro](#modify-the-epson-perfection-v850-pro) + - [Operational tips](#operational-tips) +- [Installing and Running with Docker](#installing-and-running-with-docker) +- [Interface](#interface) + - [Configuration and preview](#configuration-and-preview) + - [Using the preview](#using-the-preview) +- [Running Scans](#running-scans) +- [Result Files and Database](#result-files-and-database) + - [Metadata content](#metadata-content) +- [Printing Labels](#printing-labels) + + + +***EntoScan** is a free and open-source scientific tool developed in the [DARSA group](https://darsa.info/), at Aarhus University. +Please cite our publication if you use **EntoScan**.* + +## Flatbed Scanners For Entomology Uses and Limitations +Digital imaging is increasingly used in entomology to study insects, but capturing large numbers of specimens at scale remains challenging. +Traditional imaging approaches rely on specialised and expensive camera systems—often using motorised stages to tile images — which makes it difficult to standardise lighting, scale, and image quality across experiments. + +For small specimens and other biomedical applications, flatbed scanners have proven to be a powerful and affordable alternative. +However, a gap remains between proof-of-principle demonstrations and widespread adoption. +Scanners suffer from a shallow depth of field, which can limit their effectiveness, but this issue can be mitigated through simple hardware modifications. + +Another limitation lies in the proprietary software provided by scanner manufacturers, which is poorly suited for high-throughput biomedical imaging. +It is difficult to associate scanned images with external metadata, and users often need to devise their own visual labelling systems, such as two-dimensional barcodes. Moreover, images are not stored in a structured format that facilitates downstream analysis, and biomedical imaging frequently requires custom configurations—for instance, capturing multiple regions of interest within the same sample. + +## What EntoScan Is +EntoScan addresses some of the above limitations with a hardware and software stack for high-throughput imaging of individual or bulk of insects. +A modified flatbed scanner captures each specimen in a configurable layout. +The custom control software runs as a portable webapp. +2D barcodes are used to label scans and link images to relevant metadata. +The platform stores full-resolution images, thumbnails, and metadata in a structured database, +which ensures consistent naming and metadata handling, and can easily be queried to filter and display relevant images. + +This repository is organised around three services: +- `services/scanner-api`: a FastAPI service that talks to the scanner, keeps the configuration, generates previews, performs barcode-aware scans, and writes structured metadata. +- `services/webapp`: a Shiny dashboard that surfaces recent scans, previews, and metadata stored in the database. +- `services/db`: a PostgreSQL instance seeded with the schema EntoScan uses to persist scan records. + +## Hardware Setup and Focus Considerations +At the moment EntoScan is tested on **Epson Perfection V850 Pro**. +For optimal images, it is preferable to place insects as close as possible to the focal plane of the scanner: the top of the glass bed. +There are several ways to do that: +1. Place samples directly on top of the bed (optionally seal the edge of the bed with grout) +2. Replace the thick glass with an aluminium plate with a cut-out "frame" that can hold a container (e.g., petri dish or well plate) and offset it so that the floor of the container, where the specimens are, is on the focal plane. + +In some cases (e.g., sticky cards), placing the insects on the glass directly will be efficient and reliable. +However, specimens kept in liquid it is much more advisable to use containers such as petri dishes or well plates and move containers rather than individual insects, +Since this approach is less trivial, we describe how to modify the scanner: + +### Modify the Epson Perfection V850 Pro +- Power off the scanner, unplug USB, and work in a dust-free space. +- Remove the four screws that hold the factory glass platen, lift the glass with a thin spatula, and clean residual adhesive. +- Replace the glass with a 250 mm × 378 mm × 3.3 mm aluminium plate that has a 135 mm × 95 mm cut-out matching the well-plate footprint. Secure it with thin double-sided tape. +- Preserve the calibration reference: tape a white strip to the top edge of the aluminium so the scanner retains its exposure baseline. +- The cut-out lowers the specimen plane so insects rest closer to the optics' focal point; keep the plate flat to avoid defocus and artefacts. + +[//]: # () +[//]: # (### Fixtures and accessories) + +[//]: # (- `printing_files/well_plate_holder.stl`: PLA frame sized for 145 mm × 105 mm well plates; epoxy a 134 mm × 94 mm × 1 mm glass sheet to the underside so specimens sit flush and flat.) + +[//]: # (- `printing_files/barcode_stamp.stl`: round PLA insert that carries a removable label with the plate barcode; it occupies the bottom-right well (C6) for orientation.) + +[//]: # (- `printing_files/custom_lid.stl`: white PLA lid that suppresses ambient light; print two or three copies to rotate during the workflow.) + +[//]: # (- `printing_files/drying_rack.stl`: optional ASA part that keeps plates flat while drying insect specimens.) + +[//]: # (- Consumables: 24-well plates, removable Ø10 mm labels (Avery L6019REV-25), tweezers, ethanol for specimen storage, and Type 2D Data Matrix barcodes encoded with plate metadata.) + +### Operational tips +- Always power down before mechanical work, keep the glass surfaces spotless, and avoid bending the aluminium insert to maintain focus. +- Ensure the well-plate holder, barcode stamp, and lid are clean and dry; dust quickly degrades scan quality. +- If you replace the scanner model, measure the new platen carefully and replicate the cut-out dimensions so the ROI coordinates stay meaningful. + + +## Installing and Running with Docker +1. [Install Docker and the Compose](https://docs.docker.com/compose/install/) plugin, and make sure the host user can access the scanner's USB bus (the compose file shares `/dev/bus/usb` with the container). +2. On your machine, create a directory for EntoScan. +3. In this directory download `docker-compose.yaml` (from [our repository](https://github.com/darsa-group/EntoScan/)). +3. Create file named `.env` with these two variables: + ``` + # A to a writable directory on the host machine (your computer). + # This is where all the data and the database will live. + SCANNER_HOST_OUTPUT_DIR=/ABC/DEF + + # Whether you want to use a mock scanner + # Usefull if you want to test or develop the API without a physical device + USE_MOCK_SCANNER=1 + ``` +4. From your EntoScan directory, execute: `docker compose pull` +5. **Plug the scanner in your computer and turn it on** +6. Run `docker compose up` to launch the services. Check the output (it should show the services starting in order, with the web app last) +7. The web interface should now be available at on `http://localhost:8080`. +8. Use `docker compose down` when you need to stop the services. If you move the data directory, update `SCANNER_HOST_OUTPUT_DIR` so the container continues to write and read historical scans. + + +## Interface +Once the Docker services have started, the interface is served at http://localhost:8080. +On your browser, it should look like this: + +Interface overview + +### Configuration and preview + +The first tab in the dashboard is for configuration and preview. +The left hand side is the content of the current configuration file that describes how each scan is handled. +This is the most important aspect of the interface. + +The default template (YAML file) looks like: +```yaml +global: + enforce_barcode: false + +preview: + xyxy_mm: [0.0, 0.0, 210.0, 200.0] + file_format: "jpg" + dpi: 150 + +barcode: + xyxy_mm: [0.0, 0.0, 100.0, 100.0] + file_format: "jpg" + dpi: 300 + +rois: + - name: "ROI-1" + xyxy_mm: [0.0, 0.0, 90.0, 90.0] + file_format: "jpg" + dpi: 600 + - name: "ROI-2" + xyxy_mm: [100.0, 100.0, 190.0, 190.0] + file_format: "jpg" + dpi: 600 +``` +Key fields: +- `global`: Global variables defining the overall behaviour + - `enforce_barcode`: if `true`, the scanner fails if no barcode is detected +- `preview`: Special Region Of Interest (ROI) used to generate a preview; `xyxy_mm` is `[x0, y0, x1, y1]` in millimetres measured from the scanner origin (top-left). Choose a low DPI for fast feedback. +- `barcode`: Special ROI where a 2d barecode ([data matrix](https://en.wikipedia.org/wiki/Data_Matrix)) is expected; The decoded content (wither a just a string of characters or a JSON dictionary) is attached to every image under `scan_metadata`. +- `rois`: list (one or more) of regions to acquire subimages on. Each ROI must have: + - `name`: a unique and arbitrary name + - `xyxy_mm`: the coordinates of the ROI + - `file_format`: `"jpg"` or `"tiff"` + - `dpi`: the resolution in dot per inch + +[//]: # ( - and optional keys consumed by the ROI scanner such as `mode` (Color, Gray) or `extra_scanimage_args` for backend-specific overrides.) + +### Using the preview +After setting the configuration, you can press the preview button. The scaner then uploads the config and starts scanning. +When done, the interface displays, on the right hand side, the preview image and where regions described above would be. + + +Interface Preview + +Note that **you can display the coordinates in mm by hovering your mouse** on the image, which is useful to write or correct the `xyxy_mm` position fields. +In addition, that the current configuration is saved. You may want to copy it to your own file in case you or another user wants a new one. +## Running Scans +After satisfied with the configuration for your project, you most likely want to start scanning. +In the scanning page just press "scan", according to the number of ROIs and the resolution, it might take a moment. +When done, all the resulting images are displayed on this page. In addition, if a barcode was found and used: + + +Interface Scan + +the embedded metadata is also displayed, so you chan check it is correct. If so, you can continue scanning. +Note that, each image is named with a timestamp, and the ROI name. +In addition, each image is associated with a metadata file (JSON) that contains additional information such as the computer and scanner used, the driver, a checksum for the image, etc + +## Result Files and Database +In the "Files" tab, you can display the resulting images and their metadata in a filterable table: + + +Interface Files + +All of these images are stored in the location defined in `SCANNER_OUTPUT_DIR` (in your `.env` file, see Installation). +The file structure is as follow: +``` + +├── acabb787f1a24180912d8483cacbfdc9 +│   └── Epson Perfection V850 +│   └── 2025-11-06 +│   └── 2025-11-06_08-39-36 +│   ├── ROI-1 +│   │   ├── 2025-11-06_08-39-36.ROI-1.jpg +│   │   ├── 2025-11-06_08-39-36.ROI-1.json +│   │   └── 2025-11-06_08-39-36.ROI-1.thumb.jpg +│   └── ROI-2 +│   ├── 2025-11-06_08-39-36.ROI-2.json +│   ├── 2025-11-06_08-39-36.ROI-2.thumb.jpg +│   └── 2025-11-06_08-39-36.ROI-2.tiff +├── config.yaml +└── temporary + ├── barcode.jpg + └── preview.svg +``` +So, each image is stored in a directory with the following hierarchy: + +``` +///// +``` + +This structure is designed so that if users merge data from multiple scanners/machines in the same directory/file structure, +collisions (no duplicated path/filenames) should not happen. + +In each image directory is names after the ROI and contains: +* the image +* a thumbnail of the image for fast display (`.thum.jpg`) +* a JSON file with metadata (`.json`) + +### Metadata content +Each ROI emits a JSON companion with: +- `timestamp`, `dirname`, `filename`, `uri`, and `thumbnail`. +- Optical settings: `name`, `xyxy_mm`, `dpi`, `file_format`, plus the probed `driver`, `device_id`, `sane_device`, and `host_id`. +- Integrity: the image MD5 checksum. +- `scan_metadata`: the decoded Data Matrix payload (typically the plate label and replication identifiers). + +## Printing Labels + +We provide a [standalone webapp](https://darsa.info/EntoScan-labels/) to seamlessly generate labels that can be printed on standard paper or label paper. + + diff --git a/services/webapp/app/www/img/doc_files.png b/services/webapp/app/www/img/doc_files.png new file mode 100644 index 0000000..d2ff027 Binary files /dev/null and b/services/webapp/app/www/img/doc_files.png differ diff --git a/services/webapp/app/www/img/doc_overview.png b/services/webapp/app/www/img/doc_overview.png new file mode 100644 index 0000000..720ec44 Binary files /dev/null and b/services/webapp/app/www/img/doc_overview.png differ diff --git a/services/webapp/app/www/img/doc_preview.png b/services/webapp/app/www/img/doc_preview.png new file mode 100644 index 0000000..e4d8fe5 Binary files /dev/null and b/services/webapp/app/www/img/doc_preview.png differ diff --git a/services/webapp/app/www/img/doc_scan.png b/services/webapp/app/www/img/doc_scan.png new file mode 100644 index 0000000..9eb51ef Binary files /dev/null and b/services/webapp/app/www/img/doc_scan.png differ diff --git a/services/webapp/app/www/img/icon-256.png b/services/webapp/app/www/img/icon-256.png new file mode 100644 index 0000000..b939ba8 Binary files /dev/null and b/services/webapp/app/www/img/icon-256.png differ diff --git a/services/webapp/app/www/img/icon-512.png b/services/webapp/app/www/img/icon-512.png new file mode 100644 index 0000000..62069e4 Binary files /dev/null and b/services/webapp/app/www/img/icon-512.png differ diff --git a/services/webapp/app/www/img/icon.png b/services/webapp/app/www/img/icon.png new file mode 100644 index 0000000..fea4ed6 Binary files /dev/null and b/services/webapp/app/www/img/icon.png differ diff --git a/services/webapp/app/www/img/icon.svg b/services/webapp/app/www/img/icon.svg new file mode 100644 index 0000000..ac38965 --- /dev/null +++ b/services/webapp/app/www/img/icon.svg @@ -0,0 +1,65 @@ + + + + diff --git a/services/webapp/app/www/tooltip.js b/services/webapp/app/www/tooltip.js new file mode 100644 index 0000000..28aa463 --- /dev/null +++ b/services/webapp/app/www/tooltip.js @@ -0,0 +1,160 @@ +// www/tooltip.js +(function () { + // ---------- Config ---------- + const DEBUG = false; // set true only if you need diagnostics + const IDLE_DELAY_MS = 120; // tooltip updates after pointer stops this long + + // ---------- Tiny logger ---------- + const log = (...a) => { if (DEBUG) console.log('[Tooltip]', ...a); }; + const warn = (...a) => { if (DEBUG) console.warn('[Tooltip]', ...a); }; + + // ---------- Shared paper dimensions (in "SVG units"—mm in your previews) ---------- + let mmWidth = 210.0; + let mmHeight = 200.0; + + async function updateDimsFromSVG(imgEl) { + try { + if (!imgEl || !imgEl.src) return; + const url = imgEl.src; + const path = new URL(url, window.location.href).pathname; + if (!/\.svg$/i.test(path)) { log('Not an SVG; keep defaults'); return; } + + const resp = await fetch(url, { credentials: 'omit' }); + if (!resp.ok) return warn('SVG fetch failed:', resp.status); + + const text = await resp.text(); + const doc = new DOMParser().parseFromString(text, 'image/svg+xml'); + const svg = doc.querySelector('svg'); + if (!svg) return warn('No root'); + + // Prefer viewBox + const vbAttr = (svg.getAttribute('viewBox') || '').trim(); + if (vbAttr) { + const vb = vbAttr.split(/\s+/).map(parseFloat); + if (vb.length === 4 && vb.every(Number.isFinite)) { + mmWidth = vb[2]; mmHeight = vb[3]; + log('Dims from viewBox', { mmWidth, mmHeight }); + return; + } + } + // Fallback: width/height in mm + const wAttr = svg.getAttribute('width') || ''; + const hAttr = svg.getAttribute('height') || ''; + const mmRe = /^\s*([0-9]*\.?[0-9]+)\s*mm\s*$/i; + const wMatch = wAttr.match(mmRe), hMatch = hAttr.match(mmRe); + if (wMatch && hMatch) { + mmWidth = parseFloat(wMatch[1]); mmHeight = parseFloat(hMatch[1]); + log('Dims from width/height', { mmWidth, mmHeight }); + } else { + warn('No usable viewBox or mm size; using defaults'); + } + } catch (_) { + warn('Failed to derive mm from SVG'); + } + } + + // ---------- Utilities ---------- + function waitForElements(selectors, cb) { + const find = () => selectors.map(s => document.querySelector(s)); + const ready = (els) => els.every(Boolean); + + let els = find(); + if (ready(els)) return cb(...els); + + const mo = new MutationObserver(() => { + els = find(); + if (ready(els)) { mo.disconnect(); cb(...els); } + }); + mo.observe(document.documentElement, { childList: true, subtree: true }); + + // Soft timeout note (no console spam) + setTimeout(() => { if (!ready(find())) warn('Still waiting:', selectors); }, 1200); + } + + // ---------- Debounced tooltip binding (pointer events) ---------- + function bindDirect(img, tooltip) { + log('Bind direct', { src: img.src }); + + let idleTimer = null; + let lastEvt = null; + + const hide = () => { tooltip.style.display = 'none'; }; + + function computeAndShow(e) { + const rect = img.getBoundingClientRect(); + const inside = e.clientX >= rect.left && e.clientX <= rect.right && + e.clientY >= rect.top && e.clientY <= rect.bottom; + if (!inside) return hide(); + + const nx = (e.clientX - rect.left) / rect.width; + const ny = (e.clientY - rect.top) / rect.height; + const nxc = Math.max(0, Math.min(1, nx)); + const nyc = Math.max(0, Math.min(1, ny)); + const xmm = nxc * mmWidth; + const ymm = nyc * mmHeight; + + tooltip.textContent = `${xmm.toFixed(1)} mm, ${ymm.toFixed(1)} mm`; + tooltip.style.display = 'block'; + tooltip.style.left = (e.pageX + 12) + 'px'; + tooltip.style.top = (e.pageY + 12) + 'px'; + } + + function debouncedUpdate(e) { + lastEvt = e; + if (idleTimer) clearTimeout(idleTimer); + idleTimer = setTimeout(() => { if (lastEvt) computeAndShow(lastEvt); }, IDLE_DELAY_MS); + } + + const onEnter = e => debouncedUpdate(e); + const onLeave = () => { if (idleTimer) clearTimeout(idleTimer); hide(); lastEvt = null; }; + + img.addEventListener('pointerenter', onEnter, { passive: true }); + img.addEventListener('pointerleave', onLeave, { passive: true }); + img.addEventListener('pointermove', debouncedUpdate, { passive: true }); + + // Touch cleanup (hide when gesture ends) + img.addEventListener('pointerup', onLeave, { passive: true }); + img.addEventListener('pointercancel', onLeave, { passive: true }); + + return () => { + img.removeEventListener('pointerenter', onEnter, { passive: true }); + img.removeEventListener('pointerleave', onLeave, { passive: true }); + img.removeEventListener('pointermove', debouncedUpdate, { passive: true }); + img.removeEventListener('pointerup', onLeave, { passive: true }); + img.removeEventListener('pointercancel', onLeave, { passive: true }); + }; + } + + // ---------- One-shot binder that survives image swaps ---------- + function bindPreviewTooltip() { + waitForElements(['#preview-img', '#coord-tooltip'], (img, tooltip) => { + updateDimsFromSVG(img); + const unbindDirect = bindDirect(img, tooltip); + + // Rebind if #preview-img node is replaced by Shiny + const mo = new MutationObserver(() => { + const current = document.getElementById('preview-img'); + if (current && current !== img) { + unbindDirect && unbindDirect(); + mo.disconnect(); + // re-run after DOM settles + setTimeout(bindPreviewTooltip, 0); + } + }); + mo.observe(document.body, { childList: true, subtree: true }); + }); + } + + // ---------- Expose hook & auto-run ---------- + window.__bindPreviewTooltip = bindPreviewTooltip; + + document.addEventListener('DOMContentLoaded', () => { + bindPreviewTooltip(); + }); + + if (window.Shiny && Shiny.addCustomMessageHandler) { + Shiny.addCustomMessageHandler('bind-preview-tooltip', function () { + bindPreviewTooltip(); + }); + } +})(); diff --git a/services/webapp/shiny-server.conf b/services/webapp/shiny-server.conf new file mode 100644 index 0000000..5686bee --- /dev/null +++ b/services/webapp/shiny-server.conf @@ -0,0 +1,23 @@ +# Define the user we should use when spawning R Shiny processes +run_as shiny; + +# Define a top-level server which will listen on a port +server { + # Instruct this server to listen on port 80. The app at dokku-alt need expose PORT 80, or 500 e etc. See the docs + listen 8080; + + # Define the location available at the base URL + location / { + + # Run this location in 'site_dir' mode, which hosts the entire directory + # tree at '/srv/shiny-server' + app_dir /srv/shiny-server/app; + + # Define where we should put the log files for this location + log_dir /var/log/shiny-server; + + # Should we list the contents of a (non-Shiny-App) directory when the user + # visits the corresponding URL? + directory_index on; + } +} \ No newline at end of file diff --git a/services/webapp/shiny-server.sh b/services/webapp/shiny-server.sh new file mode 100644 index 0000000..2753cb7 --- /dev/null +++ b/services/webapp/shiny-server.sh @@ -0,0 +1,9 @@ +#!/bin/sh + +# Make sure the directory for individual app logs exists +env | grep -v 'IGNORE' | grep -v -P '^_' | perl -pe 's/^([^=]+)=(.+)$/$1='"'"'$2'"'"'/' >> /home/shiny/env.R +mkdir -p /var/log/shiny-server +chown shiny.shiny /var/log/shiny-server +export APPLICATION_LOGS_TO_STDOUT=true +export SHINY_LOG_STDERR=1 +exec shiny-server #2>&1 #>> /var/log/shiny-server.log 2>&1 \ No newline at end of file diff --git a/tools/README.md b/tools/README.md deleted file mode 100644 index a5ece45..0000000 --- a/tools/README.md +++ /dev/null @@ -1,61 +0,0 @@ -# Files folder - - -## Plate table -Path: ```tables/plates.csv``` -This table contains the following columns: -- `plate_id`: Id of the plate e.g. **2025-03-08_0001** -- `wellplate_type`: type of the well plate e.g. **24** -- `number_of_duplicates`: number of duplicates, usually as timestamps -- `thumbnail_extension`: extension of the thumbnail file. -- `duplicate_1`: timestamp in format HH-SS-MM e.g. **13-12-11** - - -## Insect table -Path: ```tables/insects.csv``` -This table contains the following columns: -- ```insect_id```: plate_id + well e.g. **2025-03-08_0001_*A1*** -- ```plate_id```: Id of the plate e.g. **2025-03-08_0001** -- ```well_name```: e.g. **A1** -- ```base_path```: directory where the images are stored (relative to the files directory) e.g. **data/** -- ```weight```: measured weight [mg] -- ```number_of_duplicates```: number of duplicates, usually as timestamps - - -**inherented content** -- The path of the images can be compiled with following formular: - ```base_path``` / ```plate_id``` / ```well_name``` / -- The file names within this directory are: - ```plate_id```\_(```DUPLICATE_N```)\_```well_name``` where *DUPLICATE\_N* you can insert all *duplicate\_n* cells where something is written. - - -# data-connector.py -following functions exists: - -## get_plate_csv_path() -> str: - return the path of the plate csv - -## get_samples_csv_path() -> str: - return the path of the insects csv - -## get_plate_thumbnail_path(plate_id: str) -> str: - -- ```plate_label```: label of the plate as a str value - -## is_plate_in_csv(plate_label: str) -> bool: - return if the plate already exists in the plate csv file - -## add_weight(plate_id) -> None: - return - - -## add_plate_to_csv(plate_label: str, timestamp: str) -> None: - # check if there is already something - return - -## get_plate_save_path(plate_label: str) -> str: - return "" - -## get_plate_slices_save_path(plate_label: str) -> str: - return "" - \ No newline at end of file diff --git a/tools/datasheets/drying_oven.ods b/tools/datasheets/drying_oven.ods deleted file mode 100644 index 2eec635..0000000 Binary files a/tools/datasheets/drying_oven.ods and /dev/null differ diff --git a/tools/datasheets/drying_oven.pdf b/tools/datasheets/drying_oven.pdf deleted file mode 100644 index 2d835a9..0000000 Binary files a/tools/datasheets/drying_oven.pdf and /dev/null differ diff --git a/tools/datasheets/weights.ods b/tools/datasheets/weights.ods deleted file mode 100644 index 3471489..0000000 Binary files a/tools/datasheets/weights.ods and /dev/null differ diff --git a/tools/datasheets/weights.pdf b/tools/datasheets/weights.pdf deleted file mode 100644 index b910a0c..0000000 Binary files a/tools/datasheets/weights.pdf and /dev/null differ diff --git a/tools/label.typ b/tools/label.typ deleted file mode 100644 index 01a3f98..0000000 --- a/tools/label.typ +++ /dev/null @@ -1,124 +0,0 @@ -/* TODO - -*/ - -#import "@preview/tiaoma:0.2.1": data-matrix - -// Calculating dimenstions -#let top-bottom-margin = 16.19mm -#let top = top-bottom-margin - 1mm -#let bot = top-bottom-margin + 1mm -#let left-right-margin = 11.25mm -#let column-gutter = (210mm - 150mm - (left-right-margin*2)) / 14 -#let row-gutter = (297mm - 210mm - (top + bot)) / 20 - -#set page(paper: "a4", margin: ( - top: top, - bottom: bot, - rest: left-right-margin ), - header: none) - -#let pad_zeros(n, width) = { - let s = str(n) - if s.len() < width { - return "0" * (width - s.len()) + s - } - return s -} - -#let barcode_label(entry, num) ={ - - circle(radius: 5mm, stroke: .5pt + black, - [ // regex which apply to match a year between 2000 and 2100, with or without the "petri_" prefix - #show regex("^(petri_)?(20[0-9]{2})"): it => [#text(size: 1.5pt, it)] - #set text(size: 3pt, font: "Ubuntu Mono", weight: "bold") - - #text( - baseline: 4pt, - [#entry]) - #v(1pt) - - #data-matrix( - entry + "_" + num, - height: 6mm, - width: 6mm, - options:(option-3: "square"),) - - #text( - [\_#num], - baseline: -4pt, - weight: "bold", - size: 4pt) - ]) - -} -// DATE, NUMBER_OF_PLATES, REPLICATIONS, FORCE_PRINT (true) -#let labels = ( - ("2025-02-01", 1, 1), - ("2025-02-12", 2, 1), - ("2025-02-17", 1, 1), - ("2025-03-07", 1, 1), - ("2025-04-01", 4, 1), - ("2025-04-10", 4, 1), - ("2025-04-11", 5, 3), - ("2025-04-29", 4, 2), - ("2025-04-30", 4, 2), - ("NA", 6, 1), - ("2025-05-07", 7, 2), - ("2025-05-08", 7, 2), - ("2025-05-09", 10, 2), - ("2025-05-14", 10, 2), - ("NA", 2, 1), - ("2025-05-28", 8, 2), - ("2025-06-03", 7, 2), - ("NA", 1, 1), - ("2025-06-12", 8, 2), - ("2025-06-13", 8, 2), - ("2025-06-18", 2, 2), - -) - -#let num_labels = labels.len() -1 -#let counter_printed_labels = 0 -#let row_content = () - -#for (i, entry) in labels.enumerate() { - let print-label = false - let skipping = entry.at(1) * entry.at(2) - if entry.len() == 4 { - if entry.at(3) == true{ print-label = true } - } - if i == num_labels {print-label = true} - if print-label{ - for j in range(entry.at(1)){ - let num = pad_zeros(str(j+1), 3) - for i in range(entry.at(2)){ - row_content.push(barcode_label(entry.at(0), num)) - } - } - - } - else{ - for i in range(skipping){ - row_content.push([]) - } - } - -} - - -#set rect(width: 100%, height: 100%, inset: 0pt, stroke: none) - - #grid( - columns: 15 * (10mm,), - rows: 21 * (10mm,), - inset: 0pt, - align: center + horizon, - column-gutter: column-gutter, - - row-gutter: row-gutter, - ..row_content.map(grid.cell), - - -) -