346 summarizer

[marcverhagen]

Added and adapted the summarizer from https://github.com/clamsproject/mmif-summarizer/. Also added some notes on how to add a CLI script.

[marcverhagen]

The checks keep failing. I think I got all type complaints from the tests fixed and removed unused code that was using non-existing methods. But I am unclear on how to deal with remaining issues. Many of them do seem like errors I have seen in the past where some attributes are not properly generated, but maybe they are all due to some type checking. For example, the first failure reported is

/home/runner/work/mmif-python/mmif-python/mmif/utils/summarizer/graph.py:43:20: error: in _init_nodes: No attribute 'documents' on str [attribute-error]
  In Union[Any, str]

        for doc in self.mmif.documents:

This refers to the following piece of code

    def _init_nodes(self):
        for doc in self.mmif.documents:
            self.add_node(None, doc)
            ...

Somehow the checker got the idea that self.mmif is a string.

I need to understand the check better before I can fix this.

[marcverhagen]

All remaining warnings are from pytype and I do not know how to deal with some of them (see for example the one listed above).

Do we have a policy on merging while some chacks fail?

[keighrim]

They are not warnings, but errors. I don't think we have a written policy on failing tests, but failing tests will make CI/CD pipelines fail and block any releases. Hence we can't merge a failing branch.

[keighrim]

I guess this should have a type hinting, so that pytype doesn't make the minimalist assumption (it being a str)

[marcverhagen]

Yeah, I figured I would have to do some more extended type hinting for those cases.

[codecov]

Codecov Report

❌ Patch coverage is 24.73461% with 709 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.56%. Comparing base (8e6426d) to head (51983c5).
⚠️ Report is 8 commits behind head on develop.

Files with missing lines	Patch %	Lines
mmif/utils/summarizer/summary.py	23.75%	292 Missing ⚠️
mmif/utils/summarizer/nodes.py	18.31%	165 Missing ⚠️
mmif/utils/summarizer/utils.py	19.88%	137 Missing ⚠️
mmif/utils/summarizer/graph.py	24.21%	97 Missing ⚠️
mmif/utils/summarizer/__init__.py	37.50%	10 Missing ⚠️
mmif/utils/cli/summarize.py	42.85%	8 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           develop     #350       +/-   ##
============================================
- Coverage    85.81%   65.56%   -20.25%     
============================================
  Files           15       22        +7     
  Lines         1903     2843      +940     
============================================
+ Hits          1633     1864      +231     
- Misses         270      979      +709     

Flag	Coverage Δ
unittests	65.56% <24.73%> (-20.25%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.