Show feature count while map loads

[matthew-white]

Right now if the submission map takes a while to load, the user only sees the message "Loading Submissions..." while the map loads. We don't show a count in the message, because while Frontend knows the number of submissions, it doesn't know the number of submissions with geo data. This issue is to improve the loading message so that it shows:

1. The number of features, i.e., the number of submissions with geo data
2. The number of features that have not yet been processed (the number whose extraction from the submission XML is not cached). If there are many unprocessed features, the map will take longer to load.

Frontend will request this information from Backend at the same time that it requests GeoJSON.

The goal of this issue is to give the user more information than they have now so that they can better anticipate how long it'll take the map to load.

[matthew-white]

@nicoleorlowski, what do you think the text of the loading message should say? How about something like "Processing X new Submissions and loading Y total Submissions" ?

[brontolosone]

Yes, I think we are aligned. A UI complication is that while we know how many features we have cached (in the DB), and how many fields we still need to process looking for features, we do not know the outcome of the processing yet. "Null" features are not served out — the source field of a submission may be empty, or contain invalid data.

So the preliminary counts we can give would be:

1. count of features already processed
2. count of fields still to process

The two counts may only incidentally add up to "total number of features eventually available on the map".
Thus, something like "loading 200 features" (by adding counts 1 and 2) would be misleading (or, dare I say, wrong), as the user may eventually only be given some fraction of that. Being shown 3 points on a map while the loading message implied "I'm going to give you 200 things" would not be great.

[matthew-white]

A UI complication is that while we know how many features we have cached (in the DB), and how many fields we still need to process looking for features, we do not know the outcome of the processing yet.

Ah thank you, I was wondering whether these counts would require Backend to do any XML processing. I was hoping not / that the response could be quite fast.

count of fields still to process

When you talk about the count of fields, you're talking about a count that increases with the number of submissions, right? Not just the number of fields in the form, I think? I'm thinking that this count is something like [number of submissions whose latest def is not in the geo extraction cache] x [number of geo fields in the associated form def] — is that right? For the loading counts shown in the UI, I think we'd only want to count a single submission once — not once for every geo field in the associated form def.

What do you think about a message like "Loading X existing Submissions with map data and scanning Y new Submissions for data" ? Is a message like that possible?

Being shown 3 points on a map while the loading message implied "I'm going to give you 200 things" would not be great.

Agreed. 👍 The loading message should leave the typical user feeling like they have a better understanding of the data situation and the expected load time, not feeling more confused about seemingly inconsistent counts.

I also wanted to say that I really feel like showing these counts is less important than investigating ways to return the GeoJSON more quickly. I know you're working on that already (e.g., with getodk/central-backend#1657), and I wouldn't want to distract from that work. If there are complications with these counts (e.g., with distinguishing between submissions vs. features, or with handling submissions edits), then I'm not sure how much time we should invest in this issue.

[brontolosone]

A UI complication is that while we know how many features we have cached (in the DB), and how many fields we still need to process looking for features, we do not know the outcome of the processing yet.

Ah thank you, I was wondering whether these counts would require Backend to do any XML processing. I was hoping not / that the response could be quite fast.

No XML processing required, it's basically just an early return after the already-existing processing planning stage.
It's not blazing fast, but fast enough — I'm averaging 15ms for the whole request/response locally in dev, for a collection of 10.000 submissions.

count of fields still to process

When you talk about the count of fields, you're talking about a count that increases with the number of submissions, right? Not just the number of fields in the form, I think?

Indeed. It's a count that increases with the number of submissions. The number of fields in the form would be not so useful for this use case 😆
In other conversations we've plotted for the demise of "get data from multiple different form fields in one request", so currently in the default case (thus unless you specify a bunch of field paths) it's one-field-per-submission, and in the future it'll always be. So with that change on the horizon (and since the map does not request multiple fields) you can proactively substitute "submission" for "field".

I'm thinking that this count is something like [number of submissions whose latest def is not in the geo extraction cache] x [number of geo fields in the associated form def] — is that right?

Not quite. It's the number of fields (err... "abstract fields", not "concrete fields" — a repeatgroup field will count as one field, but of course there can be many instances of that abstract field (say /a/b/c) in a submission (say /a/b[1]/c, /a/b[2]/c)) still to process, for that particular query. Change the query filters and this count may change along.

If you have valid geodata in the submissions (for specified filters), then (total-number-of-fields-already-processed + total-number-of-fields-yet-to-process) will equal exactly the number of geometry objects you're going to get, and in particular, if only one field is specified (or if no field is specified, thus in the default case) it will equal exactly the number of Features you're going to get.

If you don't have valid geodata in some of the submissions (for specified filters), then total-number-of-fields-already-processed will be the lower bound on the number of geometries you're going to get, and (otal-number-of-fields-already-processed + total-number-of-fields-yet-to-process) will be the upper bound. That's because we don't serve out empty fields. In theory we could still make a Feature in these cases, but with a zero-length geometries array (if that's geojson-legal?) — but how would you represent that on a map anyway.

For the loading counts shown in the UI, I think we'd only want to count a single submission once — not once for every geo field in the associated form def.

We're not counting submissions, but abstract fields. Yet, in the default case, and in the way the map uses the API, and in the future always+everywhere as soon you'll only be able to specify one path per request, these are indistinguishable, so in the UI you can talk about "submissions" instead of "fields" even though that'd technically be incorrect. ;-)

What do you think about a message like "Loading X existing Submissions with map data and scanning Y new Submissions for data" ? Is a message like that possible?

Ship it!

I also wanted to say that I really feel like showing these counts is less important than investigating ways to return the GeoJSON more quickly. I know you're working on that already (e.g., with getodk/central-backend#1657), and I wouldn't want to distract from that work. If there are complications with these counts (e.g., with distinguishing between submissions vs. features, or with handling submissions edits), then I'm not sure how much time we should invest in this issue.

No complications. It's done. Simple, fast, not a lot of code. It's just a matter of review and merging PRs in the right order. @alxndrsn is on it.