Platform
CoverBot
Create a scheduled pipeline to backfill unit tests for recently modified files
CoverBot is in beta. Reach out to your Tusk rep or support@usetusk.ai to get access.
Overview
Tusk CoverBot runs as a scheduled daily or weekly cron job and generates unit tests for recently modified files with low line coverage. Think of this as Dependabot but for increasing unit test coverage across your repo. Go from 0 to 100% code coverage over time with minimal effort. CoverBot is complementary to the Tusk CI workflow which is triggered per commit to a PR. This feature is best suited for engineering organizations that have set unit test coverage goals.
Requirements
You need the following to begin using CoverBot:- Team Plan and up
- Enabled repo in Tusk
- Working Tusk test execution environment
How CoverBot Works
CoverBot is an async workflow so it is fully hands-off for your engineering team.1
Navigate to CoverBot page
Go to Repositories > Repo Settings > CoverBots to view and manage your existing CoverBots for the repo. If you haven’t yet received access to create your first CoverBot, reach out to Tusk support
2
Add new CoverBot
Click Add CoverBot. Fill out the following fields:
- Bot name
- Test execution environment
- Schedule for day(s) and time to run
- Max no. of symbols to generate tests for
- Default reviewers (optional)
- File patterns for files/directories to focus on (optional)
3
Creates branch
At your pre-determined schedule (daily or weekly at set time), CoverBot creates a branch off of your default branch.If you want to see CoverBot’s test generation output before your next scheduled run, click the ellipsis icon on your CoverBot card and click Run Now.
4
Analyzes code coverage
CoverBot runs a coverage script and finds recently modified files that have low line coverage.
5
Backfills unit tests
CoverBot generates unit tests for the files and iterates on them to make sure they are passing and fully executable.
6
Creates PR/MR
CoverBot creates a PR/MR with the line/branch coverage gains and test case details in the PR/MR description.
If no tests are generated, CoverBot deletes the branch from Step 1.
7
Assigns PR/MR reviewer
CoverBot assigns the PR/MR to the engineer or group who is in charge of reviewing Tusk PRs. You can modify the default reviewer in the CoverBot settings panel.
8
Review and merge PR/MR
Assigned engineer reviews the PR/MR, approves, and merges the tests if no changes required. Otherwise, they check out the branch and make modifications before merging.
CoverBot is triggered on a test execution environment level. For example, if you’ve enabled CoverBot for two separate environments (e.g., frontend unit tests, backend unit tests), the bot will raise 2 separate PRs.
FAQs
Who should review CoverBot PRs?
Who should review CoverBot PRs?
Some teams already have a dedicated engineer or group for reviewing PRs created by Tusk. If not, we recommend that the engineer who is the “repo owner” be the person reviewing the CoverBot PR.
Is there a limit to the number of files that Tusk will generate tests for?
Is there a limit to the number of files that Tusk will generate tests for?
The limit on number of files tested in the PR is 25. Let your Tusk representative know if you would like to change this default.
Why hasn't CoverBot created a PR at my pre-determined schedule?
Why hasn't CoverBot created a PR at my pre-determined schedule?
This scenario most likely happened because you have not yet merged the most recent CoverBot PR that was raised. CoverBot will only raise a new PR if there are no other CoverBot PRs still open for the associated test execution environment. We do this to prevent duplicate tests across CoverBot PRs.
Can I specify directories that CoverBot should not be writing tests for?
Can I specify directories that CoverBot should not be writing tests for?
Yes. You can do this in the CoverBot settings panel in your repo settings.
Does CoverBot guarantee 100% line coverage for the files that it is backfilling tests for?
Does CoverBot guarantee 100% line coverage for the files that it is backfilling tests for?
Not at the moment. We are releasing a feature very soon that will allow you to specify a project coverage goal. CoverBot will use this goal as a feedback mechanism when it is generating unit tests for files.
Does Tusk add failing tests to the CoverBot PR?
Does Tusk add failing tests to the CoverBot PR?
No, it does not. CoverBot’s goal is to increase code coverage; we want to make it as easy as possible for your team to merge its PRs.
Will CoverBot uncover edge cases in the system under test (SUT)?
Will CoverBot uncover edge cases in the system under test (SUT)?
Yes, CoverBot will surface potential bugs that it detected in its PR body for you to investigate. However, it will not provide the failing test in “Files changed.”We’ve optimized CoverBot for generating only passing tests for the SUT’s happy path. This is because CoverBot’s goal is to increase code coverage as opposed to finding bugs. You still enjoy the benefits of preventing future regressions by merging CoverBot’s tests.