add LLM guidelines

[jpedataeditor]

LLM API Reproducibility Documentation

Here are links to technical documentation from major commercial LLM providers about reproducibility:

OpenAI

Official Documentation:

• OpenAI Cookbook: Reproducible Outputs with the Seed Parameter
• Azure/Microsoft Documentation: Reproducible Output with Azure OpenAI

Key Quote from OpenAI:

"Determinism is not guaranteed, and you should refer to the system_fingerprint response parameter to monitor changes in the backend."

Even with seed + temperature=0:

"There is a small chance that responses differ even when request parameters and system_fingerprint match, due to the inherent non-determinism of our models."

Google (Gemini/Vertex AI)

Official Documentation:

• Vertex AI: Experiment with Parameter Values
• Vertex AI: Model Reference (Inference Parameters)

Key Quote from Google:

"When seed is fixed to a specific value, the model makes a best effort to provide the same response for repeated requests. Deterministic output isn't guaranteed."

Anthropic (Claude)

Current Status:
Claude exposes temperature/top-p(/top-k) but no official seed parameter in the public API today. The recommendation is to minimize variance by setting temperature=0 and top_p=1, but Anthropic's documentation acknowledges that:

"Even with temperature 0.0, the results will not be fully deterministic."

General Technical Explanation

For a comprehensive overview of why determinism is difficult across providers:

• Keywords AI: How to get consistent and reproducible LLM outputs in 2025
• Unstract: Why is deterministic output from LLMs nearly impossible?

Bottom Line

All providers indicate that seed parameters and temperature=0 improve consistency but don't guarantee identical outputs due to:

• Floating-point arithmetic variations
• GPU non-deterministic operations
• Infrastructure changes over time
• Model updates and versioning