offpost/todo at main · HNygard/offpost · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
TODO for Cline.


Goal: make production usable


- [ ] Add history table including who did what
    - [ ] Special users for "system". Must point to some usage. Then handled in display.
    - [x] History table for emails
        Example actions
        - [x] Email received
        - [x] Email classified by user (include status type, status text, answer)
        - [x] Email classified by prompt (auto classification)
        - [x] Email sent
        - [X] Email ignored
- [ ] Old threads
    - [x] Remove extra labels
    - [ ] Open some labels to public
    - [ ] Clean up old threads by archiving them
- [ ] Update-imap - remake
    - [x] Direct access should list folders
    - [x] Read per folder
    - [x] Database table with status for imap folders
        - Fields
            - Folder name
            - Thread id
            - Last checked successfully for emails
        - [x] Create rows in update-imap.php
        - [x] Update last checked when successfully reading emails (process-folder in update-imap.php)
    - [x] Make sure update-imap logic works
    - [x] Create folder status in production
    - [x] Run a couple of folders and check that it works
    - [x] Schedule reading of folders
        - [x] Processing of folder in update-imap - Logging of outputs in own table
            - Table name: imap_folder_log
            - Fields
                - Folder name
                - Status
                - Message
                - Created at
        - [x] Run on schedule, last attempted first
    - [ ] Scheduled run of new folders
        - [ ] Move update-imap code to service class
        - [ ] Test service class
        - [ ] Add to cron job
    - [ ] Scheduled run of inbox
        - [ ] Move update-imap code to service class
        - [ ] Test service class
        - [ ] Add to cron job
    - [ ] Scheduled run of sent
        - [ ] Move update-imap code to service class
        - [ ] Test service class
        - [ ] Add to cron job
- [ ] Extract data from emails
    - [x] Extract plain text body from EML
        - [x] Make a scheduled task for extraction of information
        - [x] Make framework for what extractions to do - SQL based
        - [x] Extraction - use the code from file.php
        - [x] Test new task in production
        - [x] Run scheduled task
    - [ ] Extract HTML body from EML
    - [x] Extract PDF attachments as text
        - [x] Make code
        - [x] Test new task in production
        - [x] Run scheduled task
    - [ ] Extract image attachements as text (OpenAI or something)
        - [ ] Make code
        - [ ] Test new task in production
        - [ ] Run scheduled task
    - [ ] Extract other attachments as text (OpenAI or something)
- [ ] Automatic handling
    - Must have up-to-date ImapFolderStatus (DB table) to be allowed to run
    - Purring - Next email expected date (based on plan or that we have received an expected date from public entity)

- [ ] Views available for public
    - [ ] Public threads visible to non authenticated users
    - [ ] Front page with information

- [ ] Basic AI service integrations
    - [x] OpenAiIntegration class
        - Input: Model, prompt
    - [ ] Testing rig
        - [ ] Run for many models, collect results
        - [ ] Run for many prompts, collect results
        - [ ] Make some test data

- [ ] Email extractor
    - To be used as foundation for automatic classification and follow up
    - [x] New datatype ThreadEmailExtraction
        - extraction_id
        - email_id
        - attachment_id (nullable)
        - prompt_id
        - prompt_text
        - prompt_service
        - extracted_text
        - error_message
        - created_at
        - updated_at
    - [x] Service class for creating and updating ThreadEmailExtraction
    - [x] Overview page for admin to see all unclassified emails and attachments
    - [ ] Manually trigger extration prompts from overview page
    - [x] Integration with OpenAI for extraction using prompts
        - One prompt can depend on another.
        - Some prompts depend on the type of request (e.g. offentleglova, or custom election stuff)
        - [x] Map OpenAI response to text/JSON for saving as extraction
        - [x] Log usage of OpenAI. Token in/out, prompt, response
        - [x] Show case number in thread view

- [ ] Update-imap - folder creation should be done in class so that logic is the same in tests and update-imap.php
- [ ] GUI for replying to threads
    - [ ] Reply form with rich text editor
    - [ ] Template system for common responses
    - [ ] Attachment upload functionality
- [ ] New thread concepts
    - Reply to thread
    - Enable notiication for replies
    - Automatic follow up

- [ ] Database - run backup on schedule

TODO - medium term:
- [ ] Automatic follow up
    - [ ] Scheduled email sending system
    - [ ] Response tracking
    - [ ] Automated reminder system
    - [ ] Configurable follow-up rules