Feature/add google cloud translate by tanhaow · Pull Request #31 · Princeton-CDH/muse

tanhaow · 2026-02-17T19:11:58Z

Associated Issue(s): resolves #3 and #4 (The GCP API's performance is stable and doesn't have many errors. I think it's better to integrate GCP translation into the unified translate script that we already have.)

Changes in this PR

Added google_cloud_translate() function and registered google/translation-llm in the unified translate() interface to support translate text using Google Cloud's Translation LLM (TLLM) model
Added google-cloud-translate dependency to pyproject.toml
Added test script test_scripts/test_google_cloud_translate.py for API infrastructure testing
Added Google Cloud setup documentation to README.md

Notes

Authentication requires service account credentials via GOOGLE_APPLICATION_CREDENTIALS environment variable. Follow the instructions in README and configure it before running.

Reviewer Checklist

Follow the instructions in README to download Google Cloud credentials
Set up Google Cloud credentials: export GOOGLE_APPLICATION_CREDENTIALS="cdh-muse-6950d66acf83.json"
Run API infrastructure tests: python test_scripts/test_google_cloud_translate.py
Generate translation results with TLLM using the test script (test_scripts/test_translate.py) and the test dataset (test_cases.jsonl; dowloaded from Google Drive): python test_scripts/test_translate.py google/translation-llm test_cases.jsonl

Implements translation method using Google Cloud's Translation LLM (TLLM) model, providing state-of-the-art translation quality for Chinese, Japanese, Spanish, and English. The implementation is language-agnostic and uses ISO 639-1 codes. Changes: - Add google_cloud_translate() function in src/muse/translation/translate.py - Register google/translation-llm in unified translate() interface - Add google-cloud-translate>=3.15.0 dependency to pyproject.toml - Create test_google_cloud_translate.py for API infrastructure testing - Add Google Cloud setup documentation to README.md - Generate google_translate_results.jsonl with test translations Authentication uses service account credentials via GOOGLE_APPLICATION_CREDENTIALS environment variable. All tests passing with response times of 0.3-0.6 seconds.

Update script name and documentation to better reflect its purpose as an API connectivity test rather than a general Google Cloud Translate test.

Consolidate all test sections into a single streamlined output showing: - Credentials validation - Client initialization - API call success with timing - Model information - Source text and translation result

laurejt

Thanks for looking into the robustness of the google cloud translation API, but be sure to document this in the issue. Remember also that this issue should include a cost estimate for the Notion parallel sentence corpus.

Note: This PR will not capture #4. Additional updates will be needed to the translate_corpus.py script to fail early if errors are encountered with Google Translate. Do not begin working on #4 until this PR and #24 are merged/closed.

Requested Changes:

Move the google cloud translation code to a separate file and do not integrate it into the HuggingFace translation script.
Update the code so that credentials are assumed. There should be no information about the google cloud project hard-coded in the file. Instead use something like this line from the google cloud translate documentation:

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")

Update the documentation to describe the expected credentials set up / assumptions.

laurejt · 2026-02-18T18:36:09Z

src/muse/translation/translate.py

 # Supported models for the unified translate() function
 SUPPORTED_MODELS = {
    "tencent/HY-MT1.5-7B": "hymt",
    "facebook/nllb-200-3.3B": "nllb",
    "google/madlad400-7b-mt": "madlad",
+    "google/translation-llm": "google_cloud",
 }


Get rid of this global index. The underlying models default to the desired version. Instead the short names (i.e., the values of SUPPORTED_MODELS) should be the expected forms for specifying the model in the unified translate method.

However, this change should occur in a different PR (#24)

laurejt · 2026-02-18T18:43:10Z

src/muse/translation/translate.py

+    project_id: str = "cdh-muse",
+    region: str = "us-central1",
+    credentials_path: str | None = None,


Remove these inputs, these should be inferred from the active gcloud config

laurejt · 2026-02-18T18:44:48Z

src/muse/translation/translate.py

+        1. Service account JSON file (via credentials_path parameter)
+        2. GOOGLE_APPLICATION_CREDENTIALS environment variable
+        3. Application Default Credentials (gcloud auth application-default login)
+    """


Rewrite to describe assumed gcloud usage

laurejt · 2026-02-18T18:57:09Z

README.md

Move the setup notes to DEVELOPERNOTES.md

Update instructions to use gcloud: https://docs.cloud.google.com/translate/docs/authentication#client-libs

laurejt · 2026-02-18T19:08:51Z

pyproject.toml

  "numpy",
  "ipython", # Required by transformers for trainer functionality
  "transformers[torch, sentencepiece, tiktoken]",
+  "google-cloud-translate>=3.15.0",


Why this specific version? Only specify specific versions when it is absolutely necessary.

laurejt · 2026-02-18T19:09:58Z

google_translate_results.jsonl

Delete this. As we've discussed previously, no data files should be checked into the repo.

laurejt · 2026-02-18T19:12:09Z

test_scripts/test_google_api_connectivity.py

Remove this test script since it is not testing any code from the muse package.

laurejt · 2026-02-18T19:15:00Z

src/muse/translation/translate.py

+    try:
+        response = client.translate_text(
+            contents=[text],
+            target_language_code=tgt_lang,
+            source_language_code=src_lang,
+            parent=parent,
+            model=model_path,
+            mime_type="text/plain",
+        )
+    except Exception as e:
+        raise Exception(f"Google Cloud Translate API call failed: {e}") from e


This looks like an unnecessary try/except, remove it.

laurejt · 2026-02-18T20:53:23Z

@tanhaow As we discussed in the white boarding session, the google cloud translate code can also be folded into the general translate function. This commend supersedes my earlier comments.

…-cloud-translate

tanhaow · 2026-02-19T17:22:59Z

@laurejt Thanks for your review! It's ready for your second review. Changes:

Uses environment variables to config GOOGLE_CLOUD_PROJECT (cdh-muse) and GOOGLE_CLOUD_REGION (us-1)
Removed unnecessary try/except wrapper around API call
Removed version constraint from google-cloud-translate dependency
Deleted API connectivity test script
Moved setup documentation to docs/DEVELOPERNOTES.md with gcloud CLI authentication instructions
Updated all "HuggingFace model identifier" references in translate_corpus.py to "Model identifier (HuggingFace or Google Cloud model)"

laurejt

🚀

tanhaow added 7 commits February 17, 2026 13:37

Delete google_translate_results.jsonl

4f3f188

Rename test script to test_google_api_connectivity.py

10cf85b

Update script name and documentation to better reflect its purpose as an API connectivity test rather than a general Google Cloud Translate test.

Simplify connectivity test to single unified output

67fe2b2

Consolidate all test sections into a single streamlined output showing: - Credentials validation - Client initialization - API call success with timing - Model information - Source text and translation result

Update test_google_api_connectivity.py

ece44b2

Update comments

25c47f6

tanhaow self-assigned this Feb 17, 2026

tanhaow requested a review from laurejt February 17, 2026 19:12

Update test_google_api_connectivity.py

3738da8

tanhaow force-pushed the feature/add-google-cloud-translate branch from a306709 to 3738da8 Compare February 17, 2026 20:07

laurejt requested changes Feb 18, 2026

View reviewed changes

tanhaow added 2 commits February 19, 2026 11:57

Revise per @laurejt 's review

5adcb95

Merge remote-tracking branch 'origin/develop' into feature/add-google…

d4d5fcc

…-cloud-translate

tanhaow requested a review from laurejt February 19, 2026 17:18

laurejt approved these changes Feb 19, 2026

View reviewed changes

laurejt merged commit 4013379 into develop Feb 19, 2026
1 check passed

laurejt deleted the feature/add-google-cloud-translate branch February 19, 2026 19:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/add google cloud translate#31

Feature/add google cloud translate#31
laurejt merged 10 commits intodevelopfrom
feature/add-google-cloud-translate

tanhaow commented Feb 17, 2026 •

edited

Loading

Uh oh!

laurejt left a comment

Uh oh!

laurejt Feb 18, 2026

Uh oh!

laurejt Feb 18, 2026

Uh oh!

laurejt Feb 18, 2026

Uh oh!

laurejt Feb 18, 2026

Uh oh!

laurejt Feb 18, 2026

Uh oh!

laurejt Feb 18, 2026

Uh oh!

laurejt Feb 18, 2026

Uh oh!

laurejt Feb 18, 2026

Uh oh!

laurejt commented Feb 18, 2026

Uh oh!

tanhaow commented Feb 19, 2026

Uh oh!

laurejt left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tanhaow commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes in this PR

Notes

Reviewer Checklist

Uh oh!

laurejt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laurejt commented Feb 18, 2026

Uh oh!

tanhaow commented Feb 19, 2026

Uh oh!

laurejt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tanhaow commented Feb 17, 2026 •

edited

Loading