My Rust maintenance tool could report a perfect score with two analyzers missing. Together, those checks accounted for 30% of the grade. The output didn’t say they hadn’t run.
The tool is cargo upkeep. It combines six metrics into a weighted score: dependency freshness, security advisories, unused dependencies, unsafe code, clippy lints, and the minimum supported Rust version. One letter grade makes the result easy to scan. But that only works if the grade tells you what was measured.
Missing checks received perfect scores
The unused-dependency check uses cargo-machete, and the unsafe-code check uses cargo-geiger. Each carries 15% of the score. Both require separate installs.
When those tools were unavailable, their metrics defaulted to 100. A check that never ran contributed exactly as much as one that ran and found no problems.
Dependency freshness had a related problem. Its calculation used the total dependency count rather than the count successfully checked. Failed registry lookups could leave it with no useful comparison data and still produce a perfect score at its full 20% weight.
Reporting why a check was unavailable needed work too. The missing-tool detector expected Cargo’s no such subcommand message, but Cargo returned no such command. That mismatch classified missing tools as generic failures. The fix recognizes the relevant error wording and checks that it refers to the expected tool, so an unrelated command failure doesn’t get misreported as a missing analyzer.
A grade needs coverage information
Unavailable metrics now contribute neither credit nor penalty. Their weights leave the calculation, and the score is recalculated over the metrics that produced results.
That alone isn’t enough. A partial result can still earn an A, so the report also needs to say it’s partial. Here’s an excerpt from a run on the repository:
{
"score": 100.0,
"grade": "A",
"complete": false,
"measured_weight": 0.45,
"unavailable": [
{
"name": "Security",
"weight": 0.25,
"reason": "failed"
},
{
"name": "Unused dependencies",
"weight": 0.15,
"reason": "not_installed"
},
{
"name": "Unsafe code",
"weight": 0.15,
"reason": "not_installed"
}
]
}The A is based on 45% of the configured scoring weight. That isn’t 45% of the code or 45% of the checks. The security check failed, and two optional analyzers weren’t installed. Those missing metrics account for the other 55%.
This also means two A grades aren’t necessarily comparable. One might include the security analysis while the other couldn’t run it. The coverage information belongs beside the grade, not buried in a warning somewhere else.
If nothing can be measured, score and grade are null. There’s no useful number to assign in that case.
CI needs the same distinction
Previously, cargo upkeep quality returned a successful exit code even when every analyzer failed. A CI job checking only that code would pass without any analysis behind it.
The command now distinguishes three outcomes:
- A complete analysis exits successfully.
- A partial analysis with a score exits successfully by default, or fails with
--require-complete. - An analysis with no score always fails.
The full report still prints before a failing exit, so the CI log retains the explanation.
--require-complete means every metric must be successfully measured. Installing the optional tools isn’t sufficient if an analyzer then fails or a required data source is unavailable. That’s a useful policy for a runner configured to perform the whole analysis, but it needs to be an explicit choice.
I still want the grade, but I want to know what it covers. An A based on part of the analysis needs to look different from an A based on all of it.
Sources
- f0b1a0a — exclude unmeasured metrics from the score
- 37b737c — add –require-complete and the exit-code contract
- cargo-machete — the unused-dependency analyzer behind the 15% weight
- cargo-geiger — the unsafe-code analyzer behind the other 15%
- RustSec Advisory Database — the source for the 25% security weight
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].