Translate content before ZIMing it

[benoit74]

There are situations where it would make sense to translate content before putting it inside a ZIM.

https://www.reddit.com/r/Kiwix/comments/1p6geda/translating_a_website_beforewhile_zim/

[ItzCobaltboy]

Hello! I am new around here but I would like to contribute to this!

Can you assign me this and guide me a little?

[kelson42]

@ItzCobaltboy What do you propose to do to implement this issue?

[ItzCobaltboy]

@kelson42 below is my proposed solution

Please lemme know if you have any questions or comments

---

In Short

Enable page translation to english with CLI argument --translate using Google Translate API, by reading/modifying WARC file just before conversion to zim file. Aiming to be non-intrusive to current pipeline.

---

Proposed Pipeline

Browsertrix crawl
        ↓
Original WARC
        ↓
[ optional translation step ]   // new addition
        ↓
Translated WARC
        ↓
warc2zim
        ↓
ZIM output

---

Plan

1. Add a CLI argument '--translate'.
   • When unset (default), Zimit behaves exactly as it does.
   • Otherwise an intermediate translation step is applied before calling warc2zim.
2. Before calling warc2zim.
   • Open the WARC file using warcio.
   • Parse only HTML content.
   • Translate relevant content (body/para/button text) using Google Translate API.
   • Rebuild WARC file safely with other metadata (HTTP headers, WARC metadata, etc) intact.
3. Return translated WARC file for warc2zim.

---

Scope of Translation

• We will only modify HTML Response bodies
• Do not touch
  • CSS
  • Scripts (JS)
  • Images
  • Non-HTML Content
• What gets translated: Visible text elements inside html
  • paragraphs.
  • labels
  • links
  • buttons
  • headings
  • etc

---

Files to be changed

1. zimit.py
   • Adding CLI flags and handling translation call
2. translator.py , new module that deals with complete translation pipeline of zimit.
   • Parsing HTML from WARC.
   • Calling Google Translate API with optimizations to reduce number of tokens.
   • Rebuilding and returning new WARC flle with metadata intact and text translated

---

Note: User will have to give their own API key for Google Translate Service, set as an ENV_VAR during starting of docker container preferably

[benoit74]

@ItzCobaltboy I kinda like the idea

Some more questions:

• do you think you can "easily" build a small quick and dirty PoC ZIM or is this too much? (no need for the code, no need to do that on a big website, just the resulting ZIM is fine)
• how big do you estimate the effort would be to implement this whole feature? I feel like this could be more a GSoC project than a "night-and-week-end" contribution, kinda worried about edge cases and subtle details we might have to handle, but I'm ready you show me I'm wrong. And we can definitely start with a v1 which works only on "simplest" cases.
• is there any kind of throttling imposed by Google Translate API?
• why Google Translate API (vs any other service, offer is quite broad nowadays)?
• how do you expect to identify visible text element inside HTML? I feel like this could be the major hurdle, but I'm not that experienced in that area ; I don't know how google translate achieve to do that on translate.google.com

[ItzCobaltboy]

@benoit74 Follow up (Sorry for the wait)

• Yess, I have looked into zimit as well as the python-libzim repos and I think onboarding myself to ZIM should be easy.
• Certainly this complete pipeline will take time to implement, I am not sure but I think it can be done in one month or something as per my availability.
• Google Translate itself doesn't have a throttling however one's billing amount may be one. Also Latency and timeout will need to be handled.
• I suggested Google Translate on top of my head, we can certainly switch to any other service/translation engine, preferably opensource, such as LibreTranslate, this can be run locally in Docker as well, which will make the runtime stack clean.
• Right now, usually most* of the important website text is wrapped in few tags, such as p, button, and others, we can parse it. and then we can translate specific string contents while not breaking the HTML structure. The approach needs refinement and may need changes but this is my idea of getting started
• I looked up into GSoC and I'm an eligible student who's looking to take the part! If comfortable, can I get your Discord or any other chat platform? I may be too early but if this issue is fit for GSoC scaled project, I'd love to explore the path while still building the feature.

[benoit74]

Nice, feel free to join our Slack : https://join.slack.com/t/kiwixoffline/shared_invite/zt-c5fgl342-VDYtm7fwY6w0BAeBqYxIGg

[ItzCobaltboy]

Can I have this issue assigned to me? I will start working on it and keep you posted!

[ItzCobaltboy]

• I suggested Google Translate on top of my head, we can certainly switch to any other service/translation engine, preferably opensource, such as LibreTranslate, this can be run locally in Docker as well, which will make the runtime stack clean.
• Right now, usually most* of the important website text is wrapped in few tags, such as p, button, and others, we can parse it. and then we can translate specific string contents while not breaking the HTML structure. The approach needs refinement and may need changes but this is my idea of getting started

We can also use the argos-translate library to run offline translation, thus creating a standalone setup, with safe HTML scraping using beautifulsoup, how does that sound?

[ItzCobaltboy]

@rgaudin any updates?

[benoit74]

I like the libretranslate / argos-translate approach. From what I undersrtand, libretranslate is indeed more a UI/API on-top-of argos-translate, so argos-translate should be fine.

I strongly encourage you to develop a quick-and-(probably)-dirty version locally to check if stuff fits well together and provide us with a first demo ZIM quickly. And once we are sure the approach works you can dive into details about how to code this correctly, which CLI inputs we want, handle probable edge-cases, etc ...