Consider moving from Google Sheets to CSV

[waldoj]

Google Sheets has no concept of pull requests, so we can't let people propose changes. While it's a convenient back-end, it's not very open. Also, it's slow. Without a loading indicator, people just assume that pages are blank and give up.

On the other hand, moving off of Tabletop.js is going to be a non-trivial change, and doing so may be counter to the spirit of the original project.

[ondrae]

For my next use of the census, I want to crowdsource a bunch of reviews during a national day of action. We'll be asking people to score the quality of the websites used to apply for a handful of important social services.

I was considering using Google Forms as the input.

Pull requests make sense if there is only going to be occasional outside updates.

[waldoj]

Hmm, good point—Google Forms certainly does lower the bar for submissions.

OTOH, it decouples the submission process from the maintenance process, which is to say there is an uncrossable chasm between "person who suggests new datasets" and "person who wants to help to maintain the data."

Now, I'm just thinking out loud here, but given that the crowdsourced data goes into a hopper for review anyway (IIRC, you created a worksheet just for submissions), I'm not sure that these two approaches are mutually exclusive. That is, submissions need to be reviewed and moved over to the worksheet that powers the site anyway. Perhaps, instead of moving them between Google Sheets worksheets, site operators would be moving them from a Google Sheets worksheet to an Excel worksheet (or whatever they're using to manage their Git-versioned CSV file)?

It seems like both of these approaches—letting people enter things into Google Forms and versioning data in Git—are both about the same thing: lowering the bar to contributing to a census. But I think they're about lowering the bar for very different types of users (non-devs vs. devs, editors vs. submitters), whose needs are perhaps incompatible?

[ondrae]

I'm happy to use either version. For the next census, I want to use whatever this becomes.

To continue the discussion though, I still think Google Docs works. We don't need a chasm, the spreadsheet containing the submissions can be the same sheet powering the webpage. Two changes required though. The first is we'd need to abstract out the questions. The second is that if there are more than one submission for a place and dataset, we use the most recent one.

You are right that there are different audiences, with either lots of input or limited. For yours, you have already done the work and there will be limited updates. For ours, the work will be done by lots and lots of volunteers submitting lots of updates.

We could make it overly complicated and have a flag that points to the datasource, either csv or spreadsheet. Actually Google Sheets lets you just add csv on to the end of the url and access it that way ...

[waldoj]

I still think Google Docs works. We don't need a chasm, the spreadsheet containing the submissions can be the same sheet powering the webpage.

I think the chasm is between submitting information and updating information. I only had maybe a dozen datasets inventoried and submitted by people, but I had many dozen corrected by people. That is, I would inventory a state, conclude that a given dataset didn't exist, and then somebody would want to update that record to say "yes, that does exist, and here's where it is." I was surprised to find that happening, but on reflection, I think it's because people prefer to fix something that is broken than create something new. Anyway, the point is that Google Forms is great for submitting new things, but not great for correcting existing things.

We could make it overly complicated and have a flag that points to the datasource, either csv or spreadsheet. Actually Google Sheets lets you just add csv on to the end of the url and access it that way ...

Huh. This is...pretty clever, actually! It abstracts away some of these problems. Folks can host their CSV right on the site, or they can host it on Sheets. The more I think about it, the more I like it.

For the record, I should list a few of the other reasons why I favor of leaving Sheets (or, as you suggest, abstracting it away):

• The Sheets API is not reliable. It has gone down before for many hours at a time, they could change it at any time (breaking any site using it), and Google rate-limits it, which would break a site just as it's getting a lot of attention.
• It's slow. It just takes a long, long time—on the order of multiple seconds—for data to load, mostly because the Sheets API's response is so bloated.
• Versioning of data is important. When adding or editing records in Sheets, I'm acutely aware that everything that I'm doing is changing the site in real-time, when what I'd really like is atomic updates (e.g., when I finish working on an inventoried datapoint, then I push out the changes).

I'm going to see if I can gin up some other folks who might be interested in using this package who have opinions about this. It'd be nice to get some other folks to weigh in here. This is an important architectural decision, and I have no particular reason to think that my position is a good one. :)

[creade]

I think we chose/stuck with google sheets not as much for the public/private interface but because not everyone on the team knew the requisite amount of git and github to update a static file. However it looks like github has rolled out some nice in-browser csv editing tools that might have broader appeal.