CoverBot
Create a scheduled pipeline to backfill unit tests for recently modified files
Overview
Tusk CoverBot runs as a scheduled daily, weekly, or monthly cron job and generates unit tests for recently modified files with low line coverage.
Think of this as Dependabot but for increasing unit test coverage across your repo.
CoverBot is complementary to the Tusk CI workflow which is triggered per commit to a PR. This feature is best suited for engineering organizations that have set unit test coverage goals.
Requirements
You need the following to begin using CoverBot:
- Team Plan and up
- Enabled repo in Tusk
- Working Tusk test execution environment
If you are already using Tusk to generate unit tests via CI, you meet all the requirements.
Reach out to the Tusk team via Slack or at support@usetusk.ai with your CoverBot schedule in mind (e.g., every Monday at 9am Pacific Time) to enable the feature.
How CoverBot Works
CoverBot is an async workflow so it is fully hands-off for your engineering team.
- At your pre-determined schedule (daily/weekly/monthly at set time), CoverBot creates a branch off of your default branch
- CoverBot runs a coverage script and finds recently modified files that have low line coverage
- CoverBot generates unit tests for the files and iterates on them to make sure they are passing and fully executable
- CoverBot creates a PR/MR with the line/branch coverage gains and test case details in the PR/MR description
- If no tests are generated, CoverBot deletes the branch from Step 1
- CoverBot assigns the PR/MR to the engineer who is in charge of reviewing Tusk PRs
- Assigned engineer reviews the PR/MR and approves the tests if no changes required
- Assigned engineer merges the PR/MR
FAQs
Who should review CoverBot PRs?
Who should review CoverBot PRs?
Some teams may already have a dedicated engineer for reviewing PRs by Tusk. If not, we recommend that the engineer who is on rotational duty be the person reviewing the CoverBot PR. On request, we can add a mapping for Tusk to auto-assign the PR according to the code owner.
Is there a limit to the number of files that Tusk will generate tests for?
Is there a limit to the number of files that Tusk will generate tests for?
The limit on number of files tested in the PR is 25. Let your Tusk representative know if you would like to change this default.
Why hasn't CoverBot created a PR at my pre-determined schedule?
Why hasn't CoverBot created a PR at my pre-determined schedule?
This scenario most likely happened because you have not yet merged the most recent CoverBot PR that was raised. CoverBot will only raise a new PR if there are no other CoverBot PRs still open for the associated test execution environment. We do this to prevent duplicate tests across CoverBot PRs.
Can I specify directories that CoverBot should not be writing tests for?
Can I specify directories that CoverBot should not be writing tests for?
Yes! Reach out to your Tusk representative or support@usetusk.ai and we can set this up for you.
Does CoverBot guarantee 100% line coverage for the files that it is backfilling tests for?
Does CoverBot guarantee 100% line coverage for the files that it is backfilling tests for?
Not at the moment. We are, however, working on allowing you to specify a coverage goal that CoverBot will use as a feedback mechanism when it is generating unit tests for files.
Does Tusk add failing tests to the CoverBot PR?
Does Tusk add failing tests to the CoverBot PR?
No, it does not. CoverBot’s goal is to increase code coverage; we want to make it as easy as possible for your team to merge its PRs.
Will CoverBot uncover edge cases in the system under test (SUT)?
Will CoverBot uncover edge cases in the system under test (SUT)?
It is unlikely. We’ve optimized CoverBot for generating only passing tests for the SUT’s happy path. This is because CoverBot’s goal is to increase code coverage as opposed to finding bugs. You still enjoy the benefits of preventing future regressions by merging CoverBot’s tests.