New expanded geospatial block by lubitchv · Pull Request #11507 · IQSS/dataverse

lubitchv · 2025-05-21T19:55:01Z

What this PR does / why we need it: Expanded geospatial block is proposed

Which issue(s) this PR closes:

Closes Feature Request/Idea: ISO 19115 versus Dataverse Geospatial Metadata Block (Proposed Changes) #10398

Special notes for your reviewer:
It is a new experimental geospatial block that extends the old one.

ofahimIQSS · 2025-05-27T15:14:01Z

@lubitchv Could you also add the new tsv file and the .properties file.
Does it also require an SQL migration script? Thanks!

lubitchv · 2025-05-27T20:41:54Z

@ofahimIQSS The new geospatial block is an extended version of an old geospatial.tsv. I can upload the new block as geospatial.tsv which will replace the old one or I can add it as a geospatial_new.tsv. In this case it may fail some tests since it is the extension of an old one and hence may conflict with it. What is better way to do? I think the end goal is update the existing block but I am not sure at this stage what should be done.

The new block does not need SQL migration script, since it is extension of old. I wrote an instruction in docs (in this PR) how to update existing installation with this extension of geospatial block.

ofahimIQSS · 2025-05-27T20:54:25Z

Thanks, @lubitchv ! We can go ahead and use geospatial.tsv as-is—no need to rename it. We'll test it on our end and roll it out accordingly.

lubitchv · 2025-05-28T22:01:25Z

I updated geospatial.tsv to a new expanded one. I also updated geospatial property file with new fields and also solr schema.xml I see that integration test is failing. Can I have a name of the test that fails?

pdurbin

Some feedback based on the latest commits. This PR is looking good!

pdurbin · 2025-06-02T14:34:49Z

+.. code-block:: javascript
+
+  curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file geospatial_new.tsv
+  curl "http://localhost:8080/api/admin/index/solr/schema" > new.xml
+  ./dataverse/conf/solr/update-fields.sh /usr/local/solr/solr-9.8.0/server/solr/collection1/conf/schema.xml new.xml
+  curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"


@lubitchv these upgrade instructions are great, thanks. Let's move them to a release note snippet.

I created the release snippet for upgrade.

pdurbin · 2025-06-02T14:37:06Z

+controlledvocabulary.spatialResolutionType.distance=distance
+controlledvocabulary.spatialResolutionType.vertical=vertical
+controlledvocabulary.spatialResolutionType.angulardistance=angularDistance
+controlledvocabulary.spatialResolutionType.levelofdetail=levelOfDetail


Here at the bottom let's track failing tests. This is what I'm seeing right now:

edu.harvard.iq.dataverse.api.DatasetFieldsIT.testListAllFacetableDatasetFields
edu.harvard.iq.dataverse.api.DatasetTypesIT.testUpdateDatasetTypeLinksWithMetadataBlocks
edu.harvard.iq.dataverse.api.DataversesIT.testListMetadataBlocks
edu.harvard.iq.dataverse.api.MetadataBlocksIT.testListMetadataBlocks

From a quick look they all have to do with the number of fields expected, like this:

JSON path data.size() doesn't match. Expected: <64> Actual: <71>

I fixed the integration tests.

pdurbin · 2025-06-02T14:38:57Z

 - Computational Workflow Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/computational_workflow.tsv>`__): adapted from `Bioschemas Computational Workflow Profile, version 1.0 <https://bioschemas.org/profiles/ComputationalWorkflow/1.0-RELEASE>`__ and `Codemeta <https://codemeta.github.io/terms/>`__.
 - Archival Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/archival.tsv>`__): Enables repositories to register metadata relating to the potential archiving of the dataset at a depositor archive, whether that be your own institutional archive or an external archive, i.e. a historical archive.
-
+- `New updated Geospatial Metadata block <https://docs.google.com/spreadsheets/d/1GMVIaQB6EqauNDqrPq09BBHd_HzUSD57oliTAoh3Ikw>`__: adapted for ISO 19115-3 format. It substitutes and expands existing geospatial.tsv metadata block. To use it, replace existing geospatial.tsv with the new one. To upload the block to the existing installation:


Let's check with @jggautier and others but I believe we we should overwrite "Geospatial Metadata" above rather than having a separate entry. In a release note snippet we can explain that the geospatial block has been updated.

On a related note, rather than "New expanded geospatial block" should the title of this PR be "Updated geospatial block" or "updated" or "improved" or "extended" or whatever. It's not really new, is it? It's bigger and better. 😄

pdurbin · 2025-06-04T21:07:07Z

-	eastLongitude	Easternmost (Right) Longitude	Easternmost coordinate delimiting the geographic extent of the Dataset. A valid range of values,  expressed in decimal degrees, is -180.0 <= East Bounding Longitude Value <= 180.0.		text	8		FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	geographicBoundingBox	geospatial
-	northLatitude	Northernmost (Top) Latitude	Northernmost coordinate delimiting the geographic extent of the Dataset. A valid range of values,  expressed in decimal degrees, is -90.0 <= North Bounding Latitude Value <= 90.0.		text	9		FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	geographicBoundingBox	geospatial
-	southLatitude	Southernmost (Bottom) Latitude	Southernmost coordinate delimiting the geographic extent of the Dataset. A valid range of values,  expressed in decimal degrees, is -90.0 <= South Bounding Latitude Value <= 90.0.		text	10		FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	geographicBoundingBox	geospatial
+	geographicCoverage	Geographic Coverage	Information on the geographic coverage of the data. Includes the total geographic scope of the data.		none	0		FALSE	FALSE	TRUE	FALSE	TRUE	FALSE		geospatial


Someone at standup noticed that you've changed displayoncreate for a lot of field from false to true:

pdurbin@beamish dataverse % git checkout develop Switched to branch 'develop' Your branch is up to date with 'origin/develop'. pdurbin@beamish dataverse % cat scripts/api/data/metadatablocks/geospatial.tsv | head | cut -f2,13 name geospatial name displayoncreate geographicCoverage FALSE country FALSE state FALSE city FALSE otherGeographicCoverage FALSE geographicUnit FALSE geographicBoundingBox FALSE pdurbin@beamish dataverse % git checkout 10398-geospatial-block Switched to branch '10398-geospatial-block' Your branch is up to date with 'lubitchv/10398-geospatial-block'. pdurbin@beamish dataverse % cat scripts/api/data/metadatablocks/geospatial.tsv | head | cut -f2,13 name geospatial name displayoncreate geographicCoverage TRUE country TRUE state TRUE city TRUE otherGeographicCoverage TRUE geographicUnit TRUE geographicBoundingBox TRUE

Can you please switch them back to false?

On a related note, there's a new API added in #11001 for 6.6, that allows you to change displayoncreate: https://guides.dataverse.org/en/6.6/api/native-api.html#set-displayoncreate-for-a-dataset-field . So maybe you can use that to set fields to true for your installation.

I replaced back displayoncreate to all the fields to FALSE

jggautier · 2025-06-09T12:29:30Z

Hi all! Just got back from vacation and noticed I was asked if this metadata block should be described in the "Experimental Metadata" section or if its description should replace what's in the "Supported Metadata" section about geospatial metadata.

@lubitchv could you write why you added a description in the "Experimental Metadata" section? From the discussion in this PR so far I couldn't get a sense of why.

And where ever the metadata block is described on this Appendix page, should it include information about how to upload the metadata block? I ask because this would be the first time we include that sort of information here, when I think it's usually in the release notes and in the Metadata Customization section of the Admin Guide.

jggautier · 2025-06-10T17:45:05Z

Just an update from my comments and questions yesterday after I got to chat briefly with @lubitchv this morning at the Dataverse Community Meeting at UNC. We agreed to try to find time to discuss more with more folks, including @amberleahey.

I should also say that I just noticed that "It is a new experimental geospatial block that extends the old one" is what's in the "Special notes for your reviewer" section of this PR's first comment, so it sort of makes sense to me why you'd want to describe it in the Appendix's "Experimental Metadata" section. And I think it would help to discuss with @pdurbin why he's encouraging you to instead replace what's in the Appendix's "Supported Metadata" section.

Hopefully we'll find time to chat this week during the conference.

coveralls · 2025-09-04T16:01:32Z

coverage: 24.33% (+0.2%) from 24.168%
when pulling a0eb3d5 on lubitchv:10398-geospatial-block
into d2b6a46 on IQSS:develop.

lubitchv · 2025-10-24T15:24:35Z

Can you let me know what integration test is failing?

qqmyers · 2025-10-24T15:45:21Z

It's a build issue - I merged develop into your branch to see if that helps.

amberleahey · 2025-12-12T17:52:39Z

Made a comment on #1592
DV-ISO Metadata Crosswalks.xlsx
ForReview_DataverseNewGeoMetadataBlock_12-2025.xlsx
Stay tuned...

qqmyers · 2026-02-03T15:28:28Z

-	country	Zimbabwe		247											
-	country	Åland Islands		248											
+#metadataBlock	name	dataverseAlias	displayName												
+	geospatial		Geospatial Metadata												


Have you considered a blockUri and/or termUri entries to make the association with 19115 clearer? As an XML standard they may not have a good mapping to URIs/RDF, but it might still make sense to add URI mapping even if they are of the form https://dataverse.org/vocab/iso19115/* (especially if not all fields in the block map to 19115)

Yes, our librarian Alicia @aliciuki is looking into it.

lubitchv · 2026-03-23T16:25:18Z

I updated the branch from the develop and I have continuous integration failed. Can you tell me which tests have failed?

pdurbin · 2026-03-23T18:39:07Z

I updated the branch from the develop and I have continuous integration failed. Can you tell me which tests have failed?

Hi @lubitchv, the failure is unrelated to this PR. I looked at https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/job/PR-11507/27/consoleFull and it shows this:

TASK [dataverse : Wait until localstack status is ready] ***********************
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (18 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (17 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (16 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (15 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (14 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (13 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (12 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (11 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (10 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (9 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (8 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (7 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (6 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (5 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (4 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (3 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (2 retries left).
FAILED - RETRYING: [localhost]: Wait until localstack status is ready (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 18, "changed": false, "content": "", "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://localhost:4566/"}

pdurbin · 2026-04-27T18:17:06Z

+	eastLongitude	Easternmost (Right) Longitude	Easternmost coordinate delimiting the geographic extent of the Dataset. A valid range of values,  expressed in decimal degrees, is -180.0 <= East Bounding Longitude Value <= 180.0.		text	8		FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	geographicBoundingBox	geospatial
+	northLatitude	Northernmost (Top) Latitude	Northernmost coordinate delimiting the geographic extent of the Dataset. A valid range of values,  expressed in decimal degrees, is -90.0 <= North Bounding Latitude Value <= 90.0.		text	9		FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	geographicBoundingBox	geospatial
+	southLatitude	Southernmost (Bottom) Latitude	Southernmost coordinate delimiting the geographic extent of the Dataset. A valid range of values,  expressed in decimal degrees, is -90.0 <= South Bounding Latitude Value <= 90.0.		text	10		FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	geographicBoundingBox	geospatial
+	expandedGeospatialMetadata	Geospatial Metadata - Expanded	Expanded metadata for geospatial datasets (ISO 19115-3)		none	11		FALSE	FALSE	FALSE	FALSE	FALSE	FALSE		geospatial


I'm confused by what I see in the UI for "Geospatial Metadata - Expanded":

It seems like I can't enter any information into it. Am I doing something wrong?

""Geospatial Metadata - Expanded" supposed to be like a title, to separate the blocks inside the block. So there is no data to enter.

Actually "Raster Spatial Representation" doesn't work for me either:

There's no way to enter info. 🤔

@lubitchv oh, interesting. So they're headings. @jggautier have you ever seen this?

Hey @pdurbin. No I haven't seen these headings.

Is this a case of the name of the parent field "Raster Spatial Representation" appearing above its child fields when the screen size is narrow enough?

I removed these these header fields "Geospatial Metadata - Expanded" and "Raster Spatial Representation" and Vector. We also improved the labels as well. So it is ready for review.

lubitchv · 2026-05-13T18:38:23Z

I have continuous integration test failing. Can anyone tell me what tests are failing?

new geospatial block

7e9ef14

pdurbin added this to IQSS Dataverse Project May 21, 2025

pdurbin moved this to Ready for Triage in IQSS Dataverse Project May 21, 2025

remove block

e17a01b

ofahimIQSS assigned lubitchv May 27, 2025

lubitchv added 7 commits May 28, 2025 10:54

expanded geospatial block

f85e079

geospatial properties

81e7459

geospatial properties fix

9b343cd

geospatial properties fix

e724514

geospatial.tsv

ab6d9a1

updated schema

93f5551

fix geospatial.tsv

96be722

pdurbin reviewed Jun 2, 2025

View reviewed changes

ofahimIQSS added the Size: 10 A percentage of a sprint. 7 hours. label Jun 3, 2025

ofahimIQSS assigned pdurbin Jun 3, 2025

ofahimIQSS moved this from Ready for Triage to Ready for Review ⏩ in IQSS Dataverse Project Jun 3, 2025

ofahimIQSS moved this from Ready for Review ⏩ to In Review 🔎 in IQSS Dataverse Project Jun 3, 2025

cmbz added FY25 Sprint 24 FY25 Sprint 24 (2025-05-21 - 2025-06-04) FY25 Sprint 25 FY25 Sprint 25 (2025-06-04 - 2025-06-18) labels Jun 4, 2025

pdurbin reviewed Jun 4, 2025

View reviewed changes

lubitchv added 4 commits June 5, 2025 16:35

geo false on creation

fd7072c

geo fix

9640193

int test naumber of fields

ea4327f

upgrade snippet

3d4c125

lubitchv and others added 5 commits September 4, 2025 17:22

Updated geoblock

11fb9f6

integration tests

d8920e1

Merge branch 'IQSS:develop' into 10398-geospatial-block

f8d0614

change to reolution

2302973

update geo block

92af478

Merge branch 'develop' into 10398-geospatial-block

0f34ee0

DS-INRAE mentioned this pull request Oct 30, 2025

Integrate Geographic Unit Field into Geographic Coverage Metadata Block #10502

Open

Merge branch 'IQSS:develop' into 10398-geospatial-block

1643e76

amberleahey mentioned this pull request Dec 12, 2025

Support for Importing and Exporting DCAT Metadata #1592

Open

amberleahey mentioned this pull request Dec 15, 2025

Feature Request/Idea: ISO 19115 versus Dataverse Geospatial Metadata Block (Proposed Changes) #10398

Open

lubitchv and others added 3 commits January 22, 2026 11:49

geospatial.tsv

a0eb3d5

Merge branch 'IQSS:develop' into 10398-geospatial-block

b79d38b

removed quotes

daae81a

qqmyers reviewed Feb 3, 2026

View reviewed changes

Merge branch 'develop' into 10398-geospatial-block

66ea07f

pdurbin reviewed Apr 27, 2026

View reviewed changes

Remove two fake fields

16a7096

pdurbin moved this from On Hold ⌛ to Ready for Review ⏩ in IQSS Dataverse Project May 26, 2026

lubitchv and others added 3 commits May 26, 2026 14:58

Merge branch 'IQSS:develop' into 10398-geospatial-block

63e520b

Integration testing

47cbce0

solr remove header fields

a1de833

pdurbin unassigned amberleahey and lubitchv May 27, 2026

Conversation

lubitchv commented May 21, 2025

Uh oh!

ofahimIQSS commented May 27, 2025

Uh oh!

lubitchv commented May 27, 2025

Uh oh!

ofahimIQSS commented May 27, 2025

Uh oh!

lubitchv commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jggautier commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jggautier commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lubitchv commented Oct 24, 2025

Uh oh!

qqmyers commented Oct 24, 2025

Uh oh!

amberleahey commented Dec 12, 2025

Uh oh!

qqmyers Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lubitchv commented Mar 23, 2026

Uh oh!

pdurbin commented Mar 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jggautier Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lubitchv commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

lubitchv commented May 28, 2025 •

edited

Loading

jggautier commented Jun 9, 2025 •

edited

Loading

jggautier commented Jun 10, 2025 •

edited

Loading

coveralls commented Sep 4, 2025 •

edited

Loading

qqmyers Feb 3, 2026 •

edited

Loading

jggautier Apr 27, 2026 •

edited

Loading