Case Study: Optimizing Vulnerability Management: Helm Outperforms Competitors

Introduction

Helm is a comprehensive Software Bill of Materials (SBOM) and Vulnerability Management Tool designed for Medical Device Manufacturer (MDM) product security experts tasked with ensuring the safety and security of products in the field. With the increasing reliance on third-party software and hardware, the complexity of integration has escalated, making effective vulnerability management more critical than ever. Helm provides a robust solution to identify and manage vulnerabilities, ensuring both developers and users are protected. This case study compares Helm’s capabilities with those of an alternative SBOM management and Vulnerability Analysis tool, demonstrating Helm’s effectiveness in a fast-paced development environment.

Summary

An in-depth analysis revealed that Helm outperformed its competitor in several key areas of vulnerability detection. Helm identified more Common Vulnerabilities and Exposures (CVEs), and matched more dependencies correctly than the alternative tool. Notably, Helm did not report any false positives, unlike the alternative tool, which had a false positive rate of 7%. Overall Helm provided superior results in terms of both quantity and accuracy of detected vulnerabilities.

Comparison

Helm identified a total of 117 CVEs, whereas the alternative tool identified 104 CVEs. The competitor tool also found 7 proprietary security advisories* which could not be directly compared to a specific CVEs and were therefore excluded. Helm identified 13 more CVEs than the alternative tool (6 if you count proprietary security advisories). However 8 of the competitors tool’s CVEs were false positives, bringing the total number of valid CVEs to 117 for Helm and 96 for the competitor’s tool, a difference of 21 valid CVEs.

A significant part of this discrepancy is due to Helm’s successful matching of the dependency OS, which had 24 CVEs that the competitors tool failed to identify. Excluding the OS (non Linux or Windows embedded OS), Helm identified 2 additional CVEs that the competitor’s tool missed: CVE-2017–5130 and CVE-2020–8432.

Conversely, the alternative tool identified 5 CVEs that Helm did not, but 4 of these were false positives unrelated to the relevant dependency (libxml2). The final CVE identified by the alternative tool, CVE-2015–1819, referred to libxml instead of libxml2. Therefore, the alternative tool did not find additional relevant CVEs that Helm missed.

*For the sake of anonymity, we will refer to the alternative tool’s branded vulnerability findings as “proprietary security advisory.”

Methodology

The comparison involved running both tools using the same SBOM and evaluating their outputs. Each returned CVE was verified for relevance to the SBOM, ensuring accurate reference to the correct dependency and version to avoid false positives. Differences between the outputs were analyzed to identify where each tool excelled and where discrepancies existed.

Implications for Post-Market Vigilance

Key Insights

1. Accuracy of Results: Given the large number of dependencies used in modern products there are bound to be a large number of vulnerabilities that Product Security Managers must look out for. Helm’s high accuracy in returning relevant CVEs significantly reduces the time Product Security Managers spend filtering out false positives, unlike the alternative tool.
2. Scope of Dependencies: Matching as many dependencies as possible is crucial for keeping Product Security Managers informed about potential vulnerabilities that their products are exposed to. Missing even a single dependency, as demonstrated in this case study, can result in overlooking significant vulnerabilities (more than 20 CVEs) impacting end users.

Conclusion

Helm provides Product Security Managers with a highly accurate tool that identifies a broad range of relevant vulnerabilities. In this study, Helm outperformed a leading competitor by:

• Identifying more CVEs (117 vs. 104)
• Maintaining a higher accuracy (0 false positives vs 8)

The pressures on Product Security Managers and developers to deliver products, software and updates to market quickly and securely are high. Helm effectively addresses these challenges, offering a powerful solution that evolves with the changing security landscape.

Interested in learning more about how Medcrypt helps medical device manufacturers meet regulatory requirements? Contact us at info@medcrypt.com and visit us at medcrypt.com to discover our full suite of medical device cybersecurity products and services.

‍

‍