Phpunit PR coverage check

https://github.com/orbeji/phpunit-pr-coverage-check

This is a project I created to quickly check if a Pull Request has created all the necessary tests.

Until now, we have been using https://github.com/richardregeer/phpunit-coverage-check, which tells you the total percentage of code coverage and fails if it doesn’t meet the indicated percentage.

The problem we encountered with this is that often a small percentage remained uncovered, and suddenly, when you created a new PR, it would fail because there had been an accumulated loss of coverage percentage. So, even if your PR had the correct tests, this check would still fail.

In projects where we have full coverage, this is not an issue. However, in legacy projects where the percentage of code covered by tests is not as high as we’d like, this happens recurrently.

Therefore, I decided to create this tool, which focuses only on the code modified in the PR to check if they have test.

vendor/bin/pr-coverage-check check clover.xml 100 --diff=diff.txt --report=ansi

Just like with Richard Regeer’s tool, you can also specify the required coverage percentage. I honestly believe that the only appropriate value is 100%, and that’s how I have it configured in my environment. If someone submits a PR with untested code, it should not be accepted. If the untested code is due to being untestable for whatever reason, then a @coverageIgnore is more than justified. Nonetheless, I’ve left the option to lower the percentage in case someone doesn’t agree with this approach.

Ho does it work?

The tool essentially has two inputs: the coverage file and the diff with the changes from the PR. We get the coverage file by running the tests and saving the report, for example:

vendor/bin/phpunit --configuration phpunit.xml.dist --testsuite pipelines --coverage-clover 'clover.xml'

And to obtain the diff file:

git diff origin/main...origin/feature/lamevapr > diff.txt

To obtain the diff, the Git command may vary slightly depending on your workflow.

When generating the diff between two branches, it can be done in two ways:

Diff with 3 points

In this case, the changes are compared to the point where the branch was created. If your workflow involves merging the main branch at the most recent point, then running the tool in this way would interpret external changes as your own. This option is not appropriate in such cases.

Diff with 2 points

In this case, the changes in our branch are compared with the most recent changes in the destination branch. If we’ve merged develop into our branch, then the comparison will yield the correct changes.

Alternative way of working

Instead of providing the diff file, it can also be obtained directly through the GitHub or Bitbucket APIs.

In this case, the command would look something like this:

vendor/bin/pr-coverage-check check clover.xml 100 --pullrequest-id=1 --provider=Github --workspace=orbeji --repository=test --api_token=API_TOKEN

The reason for this alternative workflow is simply that I wanted to experiment with the GitHub and Bitbucket APIs. Both APIs allow you to obtain the diff of the specified Pull Request, so instead of calculating the diff beforehand with a Git command, it can be directly retrieved from the API.

Visualization of the results

When the tool finds that the specified coverage level is not being met, it can display which files and modified lines are not covered.

With the --report option, you can specify the type of report you want. The information in the report is always the same; only the format changes. The report information includes: file path and uncovered lines.

There are 3 different report formats:

ansi
comment
report

The first option, ansi, is the only one available if we run the command without Git configuration. This report shows a table in the terminal with the modified files and the corresponding lines that are not covered by tests.

root@bbfb2d246e64:/app# bin/pr-coverage-check check tests/clover.xml 90 --diff=tests/diff.txt --report=ansi
Coverage: 40%
--------------- ------------ 
File            Uncovered Lines
--------------- ------------ 
src/Dummy.php   19, 20, 26
--------------- ------------

The second option, comment, adds a comment to the Pull Request with a table showing the file and the lines that are not covered. This option is only available if you use the Git service configuration.

The last type of report is only available if the Git configuration is for Bitbucket. This type of report creates a report associated with the PR that includes information on the coverage percentage and annotations on the lines where coverage is missing:

Conclusion

This tool has been useful for maintaining the coverage level and quickly seeing which files are not covered. Most of the time, the issues I encounter are that the PR creator has only covered the happy path and has missed all the error cases. This way, I don’t have to download all the code and check if the tests at least cover all the cases.

As I mentioned at the beginning, if you have a project with full coverage, simply seeing that the percentage does not drop is sufficient. However, in projects where this maximum level has not yet been reached, this tool helps us to at least maintain the coverage level.