Repository navigation
Import from Tolgee through its API, background tasks with progress - #2
Merged
Merged
Conversation
- Background job queue in SQLite with two workers. Each batch saves progress and its resume point in the same transaction; jobs left running at shutdown resume on the next start, failed or cancelled ones can be retried. - Tolgee import: project API key or personal token, languages, namespaces, descriptions, tags, translated and reviewed texts, 500 keys per page with cursor paging and retries. Verified against a real Tolgee server. - Machine translation of whole languages now runs as a job. - Tasks page, progress strip on the project page, progress in the batch dialog. - Exports stream per language (ZIP and CSV written to the response), so large projects stay well under the container memory limit. - Load test: 160,000 keys import in about 8 s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Import big Tolgee projects (the target is ~160k keys in 5 languages) straight through Tolgee's API, as a background task with progress.
What changes
GET /api/jobs,GET /api/jobs/{id},POST /api/jobs/{id}/cancel,POST /api/jobs/{id}/retry.POST /api/tolgee/inspectandPOST /api/tolgee/import. Works with a project API key (tgpak_) or a personal token (tgpat_) plus project id; the app URL can be pasted as is. Brings languages, namespaces, descriptions, tags, translated and reviewed texts; 500 keys per page with cursor paging and retries. Into a new project or an existing one (overwrite optional). The key is kept only inside the encrypted job parameters.jobstable); verified upgrading the existing dev database.Verified
tolgee/tolgee:latestin Docker, seeded through its own API): 3,000 keys, 5 languages, namespaces, tags, a description and 50 reviewed texts all came across. This caught one real difference, now handled: with a cursor, Tolgee'stotalElementscounts only the keys left.LOWLEX_LOAD=160000 go test ./internal/server -run Load): 160,000 keys in ~8 s, key list under 20 ms during the import, nested-JSON and CSV exports of the full project in under 1 s with ~50 MB heap.