CQL parsing and query generation for jena-lucene

[recalcitrantsupplant]

The jena-lucene integration here will be updated to support a subset of CQL JSON.

The plan is then for prez to offer a mode, enabled by feature flag, to push the parts of CQL which can be passed through to Lucene, through to lucene, and continue to do CQL -> SPARQL translation on the others.

The properties that are lucene filterable (fields in lucene terminology) will be specified in the jena dataset config.ttl file. Prez should read/cache these on startup, and then use them when determining whether a filter can be pushed down. The property names would currently need to be referenced by iri token as a blank node is used for the property shape in the jena config See aiworkerjohns/jena#34.

(As a separate small PR we should extend Fuseki to support an endpoint which returns the RDF config of a dataset, such that prez can parse it)

There are three main changes required:

• update the CQL parser in prez to do the pushdown, at a high level, this will be modifying prez to not parse parts of the CQL when it is recognised as being a lucene filterable property.
• update the prez query generator to consume the JSON string in luc:query/luc:facet property functions, and modify the triples template if required to match the current facet/count triples that are generated. NB: the facet/query when used together should use a UNION to prevent getting a cartesian product of results e.g. https://github.com/aiworkerjohns/jena/blob/caae8d8249027500ca49ebbef5fe8755644c8870/demo/queries/05-combined.rq
• when the feature flag is enabled, do not generate a faceting query or count query in prez

NB the jena-lucene repo currently has a JSON microformat for filters/facets that is NOT CQL like, but the plan is to change it: aiworkerjohns/jena#22

The gswa branch is no longer in a state where it can be merged to main.
We will need two PRs, one to gswa branch and a second to main.

[edmondchuc]

Clarification from David: this piece of work is not to convert to the Lucene query syntax but the "non-query string" way where it's a structured filter in JSON. This depends on the issue aiworkerjohns/jena#22.

[recalcitrantsupplant]

I've updated the above - see strike through comment.

Some MVP simplifications that could be made:

• A mode where prez just passes a CQL filter through in its entirety and does not parse and determine which parts are for translation to SPARQL/which to push down. In the happy path case, all of GSWA's CQL filters would be passed down in their entirety - i.e. all filters available in the UI would be mapped to some lucene indexed field.
• With the change proposed in use IRIs for field names aiworkerjohns/jena#34 , the focus node iri, match (object), and once implemented, the field iri would be returned. These can form the sorts of abbreviated triples that GSWA wants, where they flatten sequence paths into single triples with "made up" properties. This would eliminate any parts of profiles that are currently used to expand the focus nodes back to the search terms they matched on.

[edmondchuc]

@recalcitrantsupplant besides updating the facet fields from string to IRI, does the rest of the examples look good to you?

[edmondchuc]

I'll use the Jena assembler to drive an endpoint that retrieves the list of available facet fields and add a new request query parameter to allow clients to specify exactly what facets to select, by IRI.

[recalcitrantsupplant]

I think so.

Not sure we need another endpoint for the list of available facet fields, could we not add it to the root endpoint?

[edmondchuc]

I think another endpoint is needed for clients to discover what facet fields are available. I'm not sure what is the root endpoint or how that would work. I was thinking clients would query /cql/facets and return a list of facet field IRIs and their labels.

[recalcitrantsupplant]

this should be the existing queryable endpoints