Production monitoring and post deployment tests

[zavelevsky]

Motivation:

detect broken live APIs as soon as possible.

First step:

Every time we deploy, run the following automated process:

1. Just before the actual deploy - take data snapshot from prod (basically call all APIs with different params and store the responses)
2. deploy
3. Take snapshots again
4. Compare (comparison can be strict or customized to tolerate small diffs where relevant)
5. Report any diffs - and manually inspect them

note: Maybe this be fast enough and timed to fit in our update window - minimizing the number of diffs