Understanding reliability score
Once you upload a package SBOM file in CycloneDX format, Concert uses the Open Source Security Foundation (OpenSSF) scorecard tool to assess the security posture of the open-source software packages in your applications and generate a package reliability score based on the criticality of a package-related issue.
About the OpenSSF scorecard
The OpenSSF scorecard is used to assess the security posture of open-source software projects. It provides a standardized, automated method for evaluating projects based on a set of security best practices. The scorecard covers various aspects such as the project's maintenance, dependency management, and code quality. It aims to help developers and maintainers identify potential security risks and improve the overall security of their projects.
Concert uses the scorecard to analyze various aspects of an open-source project by using a set of predefined metrics and reliability checks. For each reliability check performed, the scorecard returns a score from 0 to 10 where a score of 10 indicates that the project has fully met the criteria for that specific check, while a score of 0 indicates that the project has not met any of the criteria. The scorecard also returns an associated risk level (critical, high, medium, and low) for each check, which determines the weight that is applied to the check score when calculating the aggregate reliability score.
- If the check indicates a Critical risk level, the scorecard applies a relative weight of 10.
- If the check indicates a High risk level, the scorecard applies a relative weight of 7.5.
- If the check indicates a Medium risk level, the scorecard applies a relative weight of 5.
- If the check indicates a Low risk level, the scorecard applies a relative weight of 2.5.
Using the weighted, risk-based score for each check, the scorecard generates an aggregate score, indicating the criticality of security risks within your software packages. Concert refers to this as the Reliability score. The weighted score makes it easy to effectively prioritize high-risk issues to ensure that the most critical vulnerabilities are addressed first.
| Name | Category | Risk Level | What it does |
|---|---|---|---|
| Binary-Artifacts | Cybersecurity | High | Determines whether the package is has binary artifacts or not. The only possible scores for this check are 0 (indicating binaries are present in the project) and 10 (ideal score, indicating no binaries are present). Learn more. |
| Branch-Protection | Secure development | High | Assesses whether the branch is protected in some capacity with GitHub's branch protection or repository rules settings. This allows maintainers to define rules that enforce certain workflow for branches, such as requiring review or passing certain status checks before acceptance to the main branch. A low score indicates that the package is vulnerable to intentional malicious code injection. Learn more. |
| CII-Best-Practices | Cybersecurity | Low | Determines whether the project has earned an OpenSSF (formerly CII) Best Practices badge the passing level or higher. Learn more. |
| Code-Review | Secure development | High | Determines whether the project mandates code review before pull requests are merged. A low score indicates a high risk of unintentional vulnerabilities or possible injection of malicious code. Learn more. |
| Dangerous-Workflow | Cybersecurity | Critical | Determines whether the project's GitHub action workflows have dangerous code patterns, such as untrusted code checkouts, exposed secrets, or use of untrusted inputs in scripts. A low score indicates a critical risk of exploitable code. Learn more. |
| Fuzzing | Cybersecurity | Medium | Determines whether the project uses fuzzing, as in the practice of feeding unexpected or random data into a program to expose bugs. A low score indicates a medium risk of possible vulnerabilities in the code. Learn more. |
| License | Policy | Low | Detects the presence of a published license file and its dependencies. A low score indicates
the potential lack of a license file in the top-level directory as well as the associated
dependencies. Learn more. Draft comment: erin.pelkey@ibm.com Is this correct? It's unclear what else, other than the existence of a license file in the top-level directory, that this check assesses. |
| Maintained | Secure development | High | Assesses activity within a project to help ensure that it is actively maintained. A low score indicates an archived project or one that has not been updated recently, presenting a high risk of unpatched vulnerabilities. A higher score indicates that there is at least one commit per week in the past 90 days. This check only succeeds if the project is more than 90 days old. Learn more. |
| Packaging | Cybersecurity | Medium | Determines whether a GitHub project is published as a package. A low score indicates a risk of missed security updates. Learn more. |
| Pinned-Dependencies | Cybersecurity | Medium | Determines whether a GitHub repository "pins" dependencies that are used during the build and deploy process. A pinned dependency is a dependency that is explicitly set to a specific hash instead of allowing a mutable version or range of versions. Learn more. |
| SAST | Cybersecurity | Medium | Determine whether the project uses Static Application Security Testing (SAST), also known as static code analysis. A higher score indicates the possible presence of unknown bugs. Learn more. |
| Security-Policy | Cybersecurity | Medium | Determines whether a project has published a security policy (SECURITY.md) and assesses the quality of its contents. A low score indicates a risk of insecure reporting of vulnerabilities. Learn more. |
| Signed-Releases | Secure development | Medium | Determines whether the project is cryptographically signing release artifacts by looking for a digital signature for each release. A low score indicates a risk of installing malicious releases of a package. Learn more. |
| Token-Permissions | Secure development | High | Determines whether the project's automated workflow tokens follow the principle of least privilege. A low score indicates a higher vulnerability to malicious code injection. Learn more. |
| Vulnerabilities | Cybersecurity | High | Determines whether the project has open, unaddressed vulnerabilities in the codebase or dependencies that use the OSV (Open Source Vulnerabilities) service. A low score indicates a higher risk of known vulnerabilities. Learn more. |
Based on the current UI, it looks like we assign a score of 0 if binaries are present. Shouldn't the opposite be true?