From 73c4a29f30269437e2347c70529c8b83e2fb9b97 Mon Sep 17 00:00:00 2001 From: Grey Moore Date: Thu, 29 Jan 2026 16:46:31 -0500 Subject: [PATCH 1/6] add a big warning at the top, a small warning later on, and tweak wording to still make sense --- source/includes/_bulk_data.md.erb | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/source/includes/_bulk_data.md.erb b/source/includes/_bulk_data.md.erb index 36521493e4a..b182145619f 100644 --- a/source/includes/_bulk_data.md.erb +++ b/source/includes/_bulk_data.md.erb @@ -5,6 +5,12 @@ This can be a helpful tool for answering high-level questions about member engag Once your data is in an external data warehouse replica analysts can use SQL to answer questions about activity or join it with data from other sources. + + We provide a set of automated bulk exports and webhooks, along with examples (linked below) on how to use them. It's possible to consume the Bulk Data API in its underlying format as CSV files in an S3 bucket or as a higher level @@ -20,8 +26,10 @@ your technical architecture. Since we expose the file events as standard HTTPS w ## Export schedule and webhooks Every night, we'll export the most up-to-date version of all of your data into a set of CSV files, one for each internal ControlShift table. The [data.full_table_exported](#webhook-endpoints-data-full_table_exported) indicates such an export. These full CSV files should _replace_ the existing data in your mirror database. +Refreshing your mirror database with the nightly full export is essential to ensuring an accurate copy of the data. -Additionally, once a minute, we'll produce CSV files with any new rows that have been _added_ to ControlShift's internal tables. The [data.incremental_table_exported](#webhook-endpoints-data-incremental_table_exported) webhooks indicates a set of these added-rows exports. Note that the incremental exports do _not_ include any updates or deletions of existing rows; you'll have to wait for the nightly export to receive fresh data with updates and deletions included. +Additionally, once a minute, we'll produce CSV files with any new rows that have been _added_ to ControlShift's internal tables. The [data.incremental_table_exported](#webhook-endpoints-data-incremental_table_exported) webhooks indicates a set of these added-rows exports. +Remember, the incremental exports do _not_ include any updates or deletions of existing rows; you'll have to wait for the nightly export to receive fresh data with updates and deletions included.