Introduction

Helm, developed by Medcrypt, emerges as a robust and precise SBOM (Software Bill of Materials) and Vulnerability Management Tool tailored to identify potential security risks. By leveraging Helm, Product Security Managers and Developers can streamline their focus towards refining their products and services, rather than delving into the intricate details of the third-party software they depend on. In this case study, we delve into a comparative analysis between Helm and a comparable tool, examining their efficacy in detecting vulnerabilities and mitigating risks.

Summary

For this case study, we utilized an SBOM from a Linux-based medical device to evaluate the efficacy of Helm in vulnerability detection. We found that Helm performed admirably, identifying 55 valid CVEs that posed risks to the device, aligning closely with the results obtained by the other tool. However, Helm demonstrated a significantly lower rate of false positives compared to its counterpart.

In the comparison, we observed that the other tool returned an overwhelming number of false positives, with 74% of the 215 CVEs it identified being irrelevant to the SBOM. This discrepancy can be attributed to the loose matching of dependencies by the other tool, resulting in inaccuracies in CVE identification. Conversely, Helm’s precision in matching dependencies ensured that all CVEs identified were relevant to the SBOM, thereby mitigating the risk of false positives.

The findings underscore Helm’s superiority in accurately detecting vulnerabilities while minimizing false positives, thereby enabling product security teams to effectively prioritize and address genuine threats without being inundated by irrelevant alerts.

Methodology

In this comparative analysis between Helm and the alternative tool, we utilized the same SBOM (Software Bill of Materials) and subjected it to both tools to evaluate their outputs. Each CVE (Common Vulnerability and Exposure) identified by either tool underwent thorough validation to ensure its legitimacy. This validation process involved verifying that the CVE pertained to a dependency listed in the SBOM and that it affected the correct version of said dependency.

In addition to CVE validation, we also examined other potential issues, such as whether the vulnerabilities impacted dependencies such as whether or not the vulnerabilities impacted dependencies running on the correct platform (in this case Linux). Notably, the alternative tool exhibited errors such as misidentifying dependencies and reporting CVEs that didn’t impact the version of the dependency that was actually included in the SBOM, while Helm did not.

Comparison

Helm detected 55 CVEs, whereas another tool identified 215 CVEs/GHSAs. Among these, only 1 GHSA lacked a corresponding CVE. The other tool found 160 more CVEs/GHSAs than Helm, but 155 of them turned out to be false positives. The majority of these false positives were due to incorrectly matching 50 dependencies, resulting in every CVE returned for those dependencies being a false positive. For instance, the tool incorrectly assumed that the dependency “@types/connect” referred to Adobe Connect, leading to 31 false positives. Unlike this tool, Helm ignored such dependencies, producing no false positives and accurately matching all CVEs for the dependencies it identified.

Implications for Post-Market Vigilance

Key Insights

1. Quality Over Quantity: While there is value in attempting to match only part of a dependency, especially when provided with partial data in an SBOM, this case study illustrates the risks of being overly permissive. The fuzzy matching feature, as employed by the tool in question, complicated the identification of vulnerabilities for Product Security Managers and Developers. It inundated them with a plethora of false positives, necessitating hours of sorting. In contrast, Helm’s more stringent matching criteria ensured that every CVE returned was pertinent to the SBOM. Consequently, Product Security Managers and Developers could efficiently focus on verifying if their products and services were vulnerable, rather than spending time double-checking the accuracy of their tools.
2. Prioritizing Real Vulnerabilities: Helm’s accuracy enables MDMs to concentrate on addressing genuine vulnerabilities that impact their products/services. In an ideal scenario, security teams would have ample time and resources at their disposal. However, this is rarely the case in reality. Security teams often operate within constrained budgets and tight deadlines. Helm’s precision streamlines the process, enabling security professionals to allocate their limited resources towards addressing the most critical risks.
3. Accurate Matching with Multiple Data Points: The majority of false positives generated by the tool in question were easily discernible by comparing the supplier listed in the SBOM with the supplier associated with the CVE. Relying solely on the dependency name, as demonstrated in the “ @types/connect” example above, led to an overwhelming number of irrelevant CVEs being returned by the tool. Helm, on the other hand, employs multiple data points for matching CVEs to dependencies, resulting in fewer false positives.

Conclusion

In this comparison, Helm stands out as the preferred choice due to its superior accuracy, outperforming the other third party tool despite yielding an equal number of valid CVEs. The integration of third-party tools like Helm and another system is aimed at enhancing the workflow for Product Security Managers and Developers, enabling them to focus on their primary responsibilities. While the alternative system offers impressive features and a user-friendly interface, its tendency for loose matching, as evidenced in this analysis, burdens users with unnecessary manual verification tasks, detracting from core business priorities.

Helm consistently demonstrates higher accuracy compared to its counterparts, including the alternative system under review. When evaluating third-party tools, Product Security Managers must prioritize not only security implications but also time efficiency. In a fast-paced and fiercely competitive environment, the risk of wasting developers’ time on redundant verifications poses a significant concern. Helm’s precision not only delivers comprehensive risk insights but also respects users’ time, allowing them to concentrate on critical business aspects essential for success.

Enhancing Software Security with Helm:

Incorporating Helm into your development process is crucial for seamless security integration. Choosing the right Software Bill of Materials (SBOM) vulnerability management tool is fundamental for compliance, cybersecurity, and operational integrity in today’s software landscape, where reliance on open-source and third-party components is increasing.

Key Considerations:

• Scalability: Helm efficiently handles projects of varying sizes and complexities to adapt to evolving needs, managing dependencies across different software projects.
• Seamless Integrations: Helm seamlessly integrates with existing development, security, and operational frameworks, automatically generating and updating SBOMs throughout the development lifecycle to enhance vulnerability management.
• Automation: Automated SBOM vulnerability management tools reduce manual efforts, minimize errors, and ensure continuous compliance and security.
• User Experience: Helm offers a user-friendly interface and intuitive workflows, enabling stakeholders of all technical expertise to effectively generate, read, and utilize SBOMs.
• Focused Analysis: Prioritizing security alerts by differentiating between actively used and unused software components streamlines risk management processes, enabling security teams to address vulnerabilities impacting operational security efficiently.

Interested in learning more about how Medcrypt helps medical device manufacturers meet regulatory requirements? Contact us at info@medcrypt.com and visit us at medcrypt.com to discover our full suite of medical device cybersecurity products and services.

‍

‍