Skip to content

feat: add text_extraction activity type #125

Description

@siculo

Summary

Add text_extraction as a new activity type. Where tag_extraction
generates semantic tags describing the content of a photo,
text_extraction captures literal textual content visible in the
image — signage, billboards, menus, labels, street signs, documents,
text on buildings.

Motivation

Vision models already in use (qwen2-vl and similar) are capable of
reading text from images without additional providers or specialized
OCR models. The use case emerged from observing that a photo of a
billboard produces semantic tags ("billboard", "advertising") but
the actual text on the sign is never captured.

Text extraction adds a qualitatively different metadata layer that
complements tag extraction and can be run independently or in
sequence on the same project.

Scope

API

  • Add text_extraction as a valid value for the activity_type
    field in POST /api/projects/{id}/activities
  • Define the prompt used for text extraction (instruct the model to
    return only textual content found in the image, not a description)
  • Define the output format for extracted text (list of strings,
    or single block, to be decided)
  • Return 400 with unsupported_activity_type for unknown types
    (already specified in issue feat: update API paths and request shapes to new terminology, update plugin accordingly #92 — no new error code needed)

Plugin

  • Support text_extraction in the activity creation modal:

Documentation

  • Update API reference with the new activity type and output format
  • Update evolution document (already done in docs/)

Considerations

  • Activity context (issue feat: add context field to activity creation #122) is particularly useful here: the
    user can guide the model ("look for billboard text",
    "extract menu items") to improve extraction quality
  • Mixed-language text is handled naturally by multilingual models
  • Output quality depends on image resolution and font legibility

Acceptance criteria

  • POST /api/projects/{id}/activities with
    activity_type: "text_extraction" creates and queues
    an activity successfully
  • Extracted text is returned in the activity results
  • Plugin applies extracted text to the appropriate Lightroom
    metadata field
  • Existing tag_extraction activities are unaffected

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions