Skip to content
This repository was archived by the owner on Jul 25, 2026. It is now read-only.
This repository was archived by the owner on Jul 25, 2026. It is now read-only.

Benchmark containing undefined datasets cannot open #553

Description

@qjiang002

Issue from PR #540

Load this benchmark: https://dev.explainaboard.inspiredco.ai/benchmark?id=globalbench_ner

I got this error when loading this benchmark:

[0]   File "/Users/jiangqi/Desktop/Capstone/explainaboard_web/backend/src/gen/explainaboard_web/impl/db_utils/benchmark_db_utils.py", line 187, in <setcomp>
[0]     (x.dataset.dataset_name, x.dataset.sub_dataset, x.dataset.split)
[0] AttributeError: 'NoneType' object has no attribute 'dataset_name'

This is because this benchmark try to find all ner systems with 'system_query': {'task_name': 'named-entity-recognition'}, but there are systems with undefined/custom dataset, so their dataset is None. One way to deal with undefined datasets is to ignore undefined datasets in benchmark.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions