Add PngSuite tests to prevent pngcheck regressions

[ronaldtse]

This is a follow-up to #52. (please merge #52 before this, or we could rebase this after the merge of #52)

This PR adds the Bats (Bash Automated Testing System) framework and accompanying tests.

NOTE: A detailed description of the test framework and architecture is available in
the TESTING.md file, and here I'm pasting some relevant content from there.

This PR adds a hybrid CMake/Ceedling testing framework for the pngcheck to
ensure robust PNG validation across all platforms.

There are 2 categories of tests added:

• CLI basic tests (version, help, options, etc)
• Behavioral tests on executing pngcheck on images from Willem van Schaik's PngSuite of 176 images.

The full test suite is run across all test platforms supported in #52 on GitHub Actions.

The test fixtures of PngSuite are pulled in dynamically in order to prevent
licensing issues for this repository.

Testing pngcheck

Introduction

This document describes the comprehensive testing framework for the pngcheck
utility, which uses a hybrid CMake/Ceedling/Unity approach to ensure robust PNG
validation across all platforms.

Architecture

The test/ directory contains the test suite for the pngcheck utility, which
is used to guard against regressions on PNG validation.

The test files are sourced from the PngSuite
collection, which provides a comprehensive set of PNG test files designed to
test the robustness and compliance of PNG processing tools.

The test framework uses Unity C
testing framework with Ceedling
build system to run comprehensive tests against the pngcheck executable.

The CMake build infrastructure seamlessly integrates with Ceedling, allowing you
to run tests directly from CMake without needing to manually invoke Ceedling
commands.

Note: The CMake/Ceedling integration architecture follows best practices
adopted from the structure of the
apm32-ceedling-example
repository.

┌────────────────────────────┬──────────────────────────────┬──────────────────────────────┐
│  CMake (build)             │  CMake Targets (bridge)      │  Ceedling (test)             │
├────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│ ┌────────────────────────┐ │ ┌──────────────────────────┐ │ ┌──────────────────────────┐ │
│ │  Cross-platform        │ │ │  Test targets            │ │ │  Unity framework         │ │
│ │  compilation           │ │ │                          │ │ │  (C testing library)     │ │
│ │                        │ │ │ • test-all               │ │ │                          │ │
│ │ • linux (gcc)          │ │ │ • test-verbose           │ │ │ • assertions             │ │
│ │ • macos (clang)        │ │ │ • test-coverage          │ │ │ • test runners           │ │
│ │ • windows (msvc/msys2) │ │ │ • test-clean             │ │ │ • mocks (unused)         │ │
│ └────────────────────────┘ │ └──────────────────────────┘ │ └──────────────────────────┘ │
│ ┌────────────────────────┐ │ ┌──────────────────────────┐ │ ┌──────────────────────────┐ │
│ │  Build dependencies    │ │ │  Integration             │ │ │  Test management         │ │
│ │                        │ │ │                          │ │ │                          │ │
│ │ • system zlib          │ │ │ • auto executable path   │ │ │ • fixtures               │ │
│ │ • automatic fallback   │ │ │ • dependency management  │ │ │ • expectation generation │ │
│ │ • vcpkg (windows)      │ │ │ • environment setup      │ │ │ • report generation      │ │
│ └────────────────────────┘ │ └──────────────────────────┘ │ └──────────────────────────┘ │
└────────────────────────────┴──────────────────────────────┴──────────────────────────────┘

Overview

• Test files: C test files for CLI and PNG suite tests
• Fixtures: PNG files from the PngSuite collection
• Expectations: Expected outputs for each test case
• CLI tests: Static tests for command-line interface functionality
• PNG suite tests: Auto-generated tests using the PngSuite image test suite
• Helper functions: Reusable test utilities and assertions
• Configuration: Ceedling project configuration file (project.yml)
• Management tool: Ruby CLI to manage test files and expectations

As the Ceedling framework is written in Ruby, the test suite uses a Ruby
command-line tool (test/bin/pngcheck-test) to manage the test files,
expectations, and to run the tests.

Prerequisites

Overview

• CMake: Required for building the project and running tests
• Ruby: Required for running the Ceedling test framework
• Ceedling: Ruby-based testing framework (installed via Bundler)
• Unity: C testing framework (included with Ceedling)
• Python: Required for coverage analysis (optional, for coverage targets)
• gcovr: Python package for coverage analysis (optional, for coverage targets)

Installation

• CMake: Install from your package manager or download from
  cmake.org

• Ruby: Install from your package manager or download from
  ruby-lang.org

• Bundler: Install via RubyGems:

  gem install bundler

• Ceedling: Install via Bundler:

  bundle install

• Python: Install from your package manager or download from
  python.org

• gcovr: Install via pip:

  pip install gcovr

Quick start

Using CMake (Recommended)

# Configure pngcheck
cmake --preset Debug

# Build pngcheck
cmake --build build --target pngcheck --preset Debug

# Run all tests (PNGCHECK_EXECUTABLE is set automatically)
cmake --build build --preset Debug --target test-all

# Run tests with verbose output
cmake --build build --preset Debug --target test-verbose

# Run specific test suites
cmake --build build --preset Debug --target test-pngcheck-cli    # CLI tests
cmake --build build --preset Debug --target test-pngcheck-suite  # PNG suite tests

# Run tests with coverage analysis and generate HTML report (requires gcovr)
cmake --build build --preset Debug --target test-coverage

# Clean test artifacts
cmake --build build --preset Debug --target test-clean

Using Ceedling directly

# Install dependencies
bundle install

# Run all tests (requires pngcheck in PATH or PNGCHECK_EXECUTABLE set)
bundle exec ceedling test:all

# Run specific test suites
bundle exec ceedling test:test_pngcheck_cli   # CLI tests
bundle exec ceedling test:test_pngcheck_suite # PNG suite tests

Running tests in CMake

The project integrates CMake with Ceedling tests such that all test targets are
available via CMake. This allows you to run tests without needing to interact
with Ceedling.

Available CMake targets

Test targets

test-all

Run all Ceedling tests (alias: test).

The JUnit XML test report gets generated at
build/artifacts/test/junit_tests_report.xml.

If Ceedling is not installed, the target provides helpful instructions
on how to install it.

Ceedling is a Ruby-based testing framework, so you need to have Ruby and
Bundler installed. You can install Ceedling using Bundler:

# Ensure you have Ruby installed
ruby --version

# Install Bundler if not already installed
gem install bundler

# Install Ceedling through Bundler
# At the root of the project, run:
bundle install

test-verbose

Run all tests with verbose output.

test-clean

Clean Ceedling test artifacts (alias: clean-test).

test-coverage

Run tests with coverage analysis and generate HTML report (requires gcovr).

When gcovr is installed, coverage targets generate both console output and
HTML reports in build/artifacts/gcov/. The HTML report is located at
build/artifacts/gcov/index.html.

Note: Prerequisites need to be satisfied to use this tool (see
prerequisites).

test-pngcheck-cli

Run CLI-specific tests only.

test-pngcheck-suite

Run PNG suite tests only.

Targets for test case generation

pngsuite-status

Check the status of PNG suite test files. Shows which files are present and missing.

pngsuite-download

Download PNG test files from the PngSuite collection. This target automatically
downloads the required PNG files to test/fixtures/pngsuite/ if they are
missing.

Note: This target is automatically run as a dependency of test-all to ensure
PNG files are available before running tests.

generate-pngsuite-expectations

Generate expected output files for PNG suite tests. This target runs pngcheck
against all PNG files in the suite and captures the expected outputs to
test/expectations/pngsuite/.

Use this target when pngcheck behavior changes and you need to update the test
expectations.

generate-pngsuite-tests

Generate the test/test/test_pngcheck_suite.c file based on the PNG files and
expectations. This target creates the actual C test functions that will be
executed by the test framework.

CMake presets

The project includes the following CMake presets:

• Debug - Debug build with symbols
• Release - Optimized release build

They can be used to configure and build the project easily:

# Configure with presets
cmake --preset Debug          # Debug build with symbols
cmake --preset Release        # Optimized release build

# Build with presets
cmake --build --preset Debug
cmake --build --preset Release

Integration with Ceedling

A two-file approach is used to bridge the CMake build system with Ceedling's
test framework:

• CMakeLists.txt - Main CMake configuration file that includes the test subdirectory
• test/CMakeLists.txt - CMake configuration for the test suite

This integration provides the following features:

• Instead of needing to manually set the PNGCHECK_EXECUTABLE environment
  variable, the CMake integration automatically locates the built pngcheck
  executable and sets the environment variable for you.

• CMake automatically manages dependencies, ensuring that the pngcheck
  executable is built before running tests.

• The CMake targets provide a consistent interface for running tests across
  different platforms, without needing to manually invoke Ceedling commands.

• The CMake integration provides clear error messages if dependencies are missing,
  such as ceedling or gcovr, guiding users to install them.

Test structure

Test files

• test/support/test_helpers.c - Common test functions
• test/support/test_helpers.h - Header for common test functions
• test/test/test_pngcheck_cli.c - CLI functionality tests (manual)
• test/test/test_pngcheck_suite.c - PNG validation tests (auto-generated)

Directory layout

pngcheck/
├── CMakeLists.txt                  # Production build configuration
├── project.yml                     # Ceedling test configuration
├── TESTING.md                      # This documentation
└── test/
    ├── CMakeLists.txt              # CMake test configuration
    ├── bin/                        # Ruby CLI tool for managing tests
    │   └── pngcheck-test           # Ruby script to manage tests and expectations
    ├── lib/                        # Ruby libraries for `pngcheck-test`
    ├── test/                       # Test files
    │   ├── test_pngcheck_cli.c     # CLI tests (manual)
    │   ├── test_pngcheck_suite.c   # PNG suite tests (generated by test/bin/pngcheck-test)
    │   └── support/                # Test helpers
    ├── fixtures/pngsuite/          # PNG test files (176 files)
    └── expectations/pngsuite/      # Expected outputs

Test approach

Tests validate pngcheck behavior using simple prefix matching and exit codes:

• Exit code: 0 for valid PNGs, non-zero for errors
• Output prefix: "OK:" for valid files, "ERROR:" for invalid files

This approach focuses on core functionality rather than exact output matching.

CLI tests

Static tests for command-line options:

• Help option (-h)
• File handling (valid, invalid, nonexistent)
• Output options (-v, -q, -t, -p, -c, -7)

PNG suite tests

Auto-generated tests using the PngSuite collection:

• Valid PNGs: Various formats, bit depths, interlacing
• Invalid PNGs: Corrupted headers, bad checksums, truncated files

Note: The PNG suite tests are not committed to the repository due to
licensing considerations. Instead, the test management tool
(test/bin/pngcheck-test) downloads the PNG files from the PngSuite website
when needed. This ensures that the tests remain up-to-date with the latest PNG
suite files.

Test management

The test suite includes a Ruby-based management tool.

Note: Prerequisites need to be satisfied to use this tool (see
prerequisites).

To manage tests, run the Ruby CLI tool:

# Check status
bundle exec test/bin/pngcheck-test status

~~# Download PNG files (if missing) (add --force to redownload)~~
~~bundle exec test/bin/pngcheck-test download~~

# Regenerate expectations (when pngcheck behavior changes) (add --force to overwrite)
bundle exec test/bin/pngcheck-test expectations

# Generate test/test/test_pngcheck_suite.c
bundle exec test/bin/pngcheck-test generate

Adding tests

CLI tests

Edit test/test_pngcheck_cli.c and use helper functions:

void test_my_feature(void) {
    test_pngcheck_with_options("-option", "file.png", 0, "expected_content");
}

PNG tests

1. Add PNG files to fixtures/pngsuite/
2. Run bundle exec test/bin/pngcheck-test expectations
3. Run bundle exec test/bin/pngcheck-test generate

Troubleshooting

pngcheck not found: Build with make or cmake
Missing PNG files: Run bundle exec test/bin/pngcheck-test download
Ruby issues: Run bundle install

Credits

• This is a contribution from the Metanorma project (@metanorma) at Ribose (@riboseinc)

FYI @jbowler @ctruta @svgeesus

[jbowler]

I'm reviewing bats as a black box, but based on the "expectations" files this won't work; pngcheck is outputting a compression ratio at the end of the "OK" lines and that is almost certain to change.

The "x" files in PNGSuite also create an issue because the text in pngcheck output for errors is likely to change; maintainers do that.

I think this could be fixed by just checking for "OK" or "ERROR" at the start of the line. The exact output may be wrong but the point of pngcheck is to do a pass/fail and if @svgeesus feels like changing the somewhat abrupt "ERROR" then the CI tests can be changed too :-)

That said pngcheck returns an error code and that is the platinum in the sea of gold; automated testing relies on pngcheck to reliably identify possibly non-conformant (in the PNG sense) images. I guess what I'm saying is that it doesn't matter what it says, what matters is that it fails.

I suggest not checking output, which depends on many things, language for one, but just the return code for each of the PNGSuite images.

[jbowler]

Just to be clear: I'm intensely critical and over the years I've learned to do this only to people I respect.

[ronaldtse]

@jbowler point taken and I agree with the sentiments expressed.

Just to be clear: I'm intensely critical and over the years I've learned to do this only to people I respect.

Very honored for this... 😜 Just doing what we can to help out the community!

After some thought (and some time...) instead of BATS I'll rework this PR to use Ceedling which is apparently the leading C testing framework. Its component called "Unity" is a lightweight test library that is quite popular.

With Ceedling it would be possible in the future if pngcheck is to be refactored into smaller files that can be unit tested then the framework will allow testing those.

So please don't merge this yet, I'll re-push once done.

[jbowler]

So please don't merge this yet, I'll re-push once done.

Mark the PR as "draft" - see the top right of this page.

[ronaldtse]

Thanks! I wanted to do so earlier but couldn't find the button...

[jbowler]

Are your builds all using CMakeLists.txt or are they using the makefiles in some cases?

@ctruta, @svgeesus - CMakeLists.txt was only added 6 days ago but it seems to me that the makefiles should all be removed and only CMakeLists.txt used. It's confusing to have two ways of doing the same thing and I didn't see any documentation of what to use.

[ronaldtse]

@jbowler This PR aims to test both CMake and traditional Makefile workflows because this repo was documented to support both approaches.

[ronaldtse]

This is done and all builds/tests now pass across all supported platforms and the two allowed build platforms (CMake and traditional Makefile), including:

• Ubuntu 22/24 (amd64 and arm64)
• macOS 13 (Intel), 14 and 15 (arm64)
• Windows 2022 (Intel), Windows 11 (arm64)
• Windows MinGW (mingw32 and mingw64)

There are a few caveats that I couldn't solve:

• MinGW platforms do not currently run tests. For whatever reason, installation of Ceedling will hang the GitHub Action runners indefinitely. Hence I've commented out the test routine for mingw32 and mingw64. The build routines still run, of course.

• pngcheck warns when the zlib version of system is different from the one it is built with.

  zlib warning:  different version (expected xxx, using yyy)
  
  OK: ...
  

  This happens only on the macos-13 platform, and I haven't quite figured out how to resolve it. As a result, the output matching test code now skips all lines that contains "warning" and of course empty lines. The CLI tests now only match on the (1) error code and the (2) status string ("OK" or "ERROR").

• The current workflow for releasing code release.yml should be updated to use CMake to build release builds. I have not updated that workflow to use CMake releases. The latest build.yml which uses CMake can be the reference for that to happen.

[jbowler]

The local build picks up the installed pngcheck not the one that is built. I guess that doesn't matter in a Docker but it does for people developing the tests and it's confusing in any case.

[jbowler]

So far as the github actions are concerned by very basic test of pushing the changes to my own github repo are producing the results I expect; I deliberately introduced errors to make sure I get failures. The results are on the "Actions" tab here:

https://github.com/jbowler/pngcheck/actions

@svgeesus - I suggest you merge this now. It has adequate documentation for maintenance, although a little familiarity with rubygems helps (I now have sufficient after 30 minutes). The issue with picking up an installed pngcheck doesn't affect the github actions behavior and it's trivial to disable the actions ("workflows") - e.g. I just disabled one of them, "build-makefile".

[ronaldtse]

The local build picks up the installed pngcheck not the one that is built.

Yes, that is an issue that requires foreknowledge, it probably should be documented somewhere clear.

The environment variable PNGCHECK_EXECUTABLE is used to customize the path to the executable if the command is not the default PATH one.

[jbowler]

The environment variable PNGCHECK_EXECUTABLE is used to customize the path to the executable if the command is not the default PATH one.

So if I type "cmake", then type "make", then type "make test" I expect PNGCHUNK_EXECUTABLE to end up set appropriately.

The use of existing PATH is certainly a bug in Linux, et al. LDD_LIBRARY_PATH has the same problem. I don't know any way of getting round that bug. Life sucks.

From a cmake environment (I'd like to see the makefiles die without trace @svgeesus) I don't see why it can't be done; @ctruta (resident cmake expert!)

[ronaldtse]

if the direction is to get rid of Makefiles, we can create a target in CMake to run tests, perhaps linking with CTest. That would certainly make life easier if there’s only one build system.

[ronaldtse]

This PR is finally ready to go!

All tests in our fork are passing across all platforms:
https://github.com/metanorma/pngcheck/actions/runs/16271828679/job/45941408623

There are some caveats I wasn't able to solve in Windows CI, and they are now described in TESTING.md. Reproduced below for convenience.

CI troubles

Windows CI (windows-2022, windows-11-arm)

For some reason, the CMake custom target (in test/CMakeList.txt) fails to find
the test/bin/pngcheck-test script on Windows. Yet it works on Ubuntu and macOS.

In addition, Windows builds the pngcheck executable under the
build/{preset-name}/{preset-name}/ directory, while on Linux and macOS it is
under build/{preset-name}/.

As a result, the build.yml workflow does not use CMake to run tests:

• directly runs the test/bin/pngcheck-test script and uses the
  PNGCHECK_EXECUTABLE environment variable to point to the built pngcheck
  executable, instead of cmake --build build --preset Debug --target pngsuite-download.

• directly uses the ceedling command to run tests instead of the CMake targets:
  cmake --build build --preset Debug --target test-all.

[ctruta]

This is very useful, but let us sort out PR #52 first, and then we'll move over to this one.

[svgeesus]

Also, once this lands, I intend to add additional images to test out Third Edition chunks (and even Fourth Edition updates)

[ctruta]

Just like PR #52, this one, too, is a very extensive and thorough change, but I would like to make it simpler. I don't mind the history of three commits in #52 (and I also wouldn't mind if you squash those into a single commit, either), but these ones absolutely need to be squashed.

Let us integrate #52 first, and then, you can rebase these ones over that, and then we'll proceed with the review.

[svgeesus]

Let us integrate #52 first, and then, you can rebase these ones over that, and then we'll proceed with the review.

Thanks @ctruta for the review on #52, I just merged it. Now of course we have conflicts on this one :)

[svgeesus]

The test fixtures of PngSuite are pulled in dynamically in order to prevent
licensing issues for this repository.

That seems over cautious. The pngsuite license is very clear:

PngSuite

Permission to use, copy, modify and distribute these images for any
purpose and without fee is hereby granted.

(c) Willem van Schaik, 1996, 2011

Also, copying these in means that others can be added (I have several to add).

[ronaldtse]

@ctruta @svgeesus this is done!

Tests are failing now due to the inability to upload test results. This is because my account (PRs run according to the user's account) has no write access to the pngcheck repository. The identical PR on our fork, where I do have access, is passing:

• Add tests claricle/pngcheck#2

Here are the additional changes beyond the original PR post:

• PngSuite is now a permanent fixture. Removed dynamic download feature, associated targets and updated workflows.
• Added acknowledgment message to Willem van Schaik in TESTING.md

[svgeesus]

@ctruta could you review?

I want to merge this before adding Third Edition sample files as well.

[svgeesus]

I assume it would be easy in future to add additional directories under fixtures? I am thinking of the APNG testsuite, and third edition test files, for example.

[ctruta]

Following up, later-is-better-than-never.

@ronaldtse I recall you and I discussing about this being too big of a test matrix. So we landed #52 but now this PR clearly needs an update because many tests are failing. The tests are out of date because they check both the standard (CMake) builds and the legacy (Makefile) builds, and we agreed that we won't be re-enabling the Makefile builds.

This PR needed an update back then, and it still needs it now. Are you able to help us out with getting this integrated?

[ctruta]

I admit to knowing very little about Unity development, and even less so about Ceedling.
@ronaldtse, can you please clarify the necessity to have this as a test dependency, when (in my opinion) simply enumerating the various platforms and compilers that we want to test with CMake should in fact be sufficient?

[ctruta]

Thanks again for your effort, @ronaldtse. I agree the PngSuite fixtures are genuinely useful. However, after researching and taking into consideration the maintenance costs involving the Ceedling library, I've decided to implement the test infrastructure using CTest to keep dependencies within the standard C development toolchain.

I'm closing this PR, although I will credit your contribution when I add the test fixtures.