Medical devices use more software code than ever before. However, while software provides medical devices with more functionality and flexibility, it also introduces additional complexity that increases the risk of failure. About 20 percent of medical device recalls today are caused by software defects, and that number is rising.
The Federal Drug Administration (FDA) oversees the quality of medical devices sold in the United States, and companies wishing to release medical devices must obtain FDA 510(k) clearance. While investigating post-market failures, the FDA is focusing more on prevention and recommends static code analysis as part of the approach.
The value of complex defect detection
Modern static code analysis tools use sophisticated techniques to analyze source code to detect potential software defects. The tool attempts to analyze all logical paths in the code, providing more paths and code coverage than traditional forms of testing. Static analysis tools do not require any test cases and can even operate on code snippets to find potential program crashes, buffer overflows, memory leaks, data corruption, etc. Static analysis typically runs quickly and can report a range of potential bugs in a relatively short period of time (see Figure 1).
Figure 1: Static analysis can uncover potential problems early in the software development life cycle.
Static analysis tools do produce some erroneous results, commonly referred to as false positives and false positives, for various reasons. A false positive occurs when a static analysis tool thinks there is an error and there is no error. A false positive is when an error should be reported but not reported.
Most modern static analysis tools have to make a delicate trade-off between finding as many good results as possible with an acceptable level of accuracy and acceptable runtime. In other words, a noisy tool that finds every problem in a large number of false positives may be of limited value, as is a highly accurate tool that finds only a small subset of problems (see Figure 2).
Figure 2: Static analysis tools will not find all errors (false positives) and will report some errors that are not real (false positives). It is good analysis algorithms and proper analysis tuning that minimize missed errors and false reports.
Modern static analysis tools have improved analysis techniques to produce useful results with sufficient accuracy. Most organizations recognize that static analysis tools, while imperfect, provide significant value in most software development processes.
Take advantage of static analysis tools
Modern static analysis tools are relatively new to most medical device manufacturers. For many organizations implementing static analysis in their processes for the first time, understanding best practices helps get the most out of the tool in the shortest amount of time and with the least amount of rework.
Static analysis tools provide common settings for all types of codebases, and while they can find good bugs right away, simply tweaking the tool for the code can greatly improve the results (see Figure 3). This helps find more relevant bugs and reduces searches through false positives, which waste time and cause developer fatigue.
Figure 3: Almost every static analysis deployment should start with a solid tuning project. Tweaks lead to more and better errors and fewer false positives.
Many static analysis tools have their own source code parsers that may not understand or have access to all the code. Configuring the system to analyze all code or tweaking the system to identify interfaces that are not accessible at the time of analysis—such as individually verified third-party libraries—ensures results are optimal and repeatable. Achieving 100% code coverage is important to plug vulnerabilities that could increase risk.
Tuning helps find real problems. For example, telling a static analysis tool how the memory allocation mechanism works, or when a program exits so that the tool doesn't continue to track problems down a particular path, can help find new problems and remove bad ones. This can be a tedious process that requires specific expertise, but pays off in the long run.
Tuning is usually an ongoing process and should be reviewed regularly to ensure that the configuration is used consistently and to keep up with changes to the code and environment. Without continuous tuning, developers may miss important bugs and teams will waste time checking for false positives.
Many static analysis tools come with hundreds of checks covering a range of issues from concurrency to safety to C and C++ pitfalls. Many are not necessarily applicable to a given application. For example, why are C++ specific checks turned on when analyzing C code? Determining the right set of checkers takes some trial and error and expertise to see what is the best deal. Some areas to consider are: which types of checkers cause real problems, which checkers are prone to noise, and which checkers can be configured to be useful. Once a good set is finalized, it is locked so that it records and runs consistently.
In a real-world example that illustrates the value of the checker, a client wants to ensure that their static analysis system is running consistently, and requires immediate alerts when there is a discrepancy in the system. The developer creates a test suite with test cases for each checker. Whenever they make changes to the system, they run the test suite to make sure each checker is indeed working as expected. If that fails, they know they have a configuration issue that needs to be resolved. If the tests pass, the developers put the results into their design history files to prove that the system works the way they documented. This test suite not only provides customers with accountability and assurance, but also reduces their maintenance and administration costs.
Once comprehensive coverage has been achieved, the system tuned, and the breadth of analysis defined, developers can begin to use static analysis more effectively. For medical devices, a typical goal is to check every problem reported. Each question can be categorized in a number of different ways:
A problem that must be solved. It will have an appropriate priority to describe its importance and how it must be addressed during software development.
Problems that are correctly flagged, but are less likely to manifest as real-world errors, usually because the tool makes incorrect environmental assumptions. These types of classifications signal potential adjustment opportunities.
Issues that are erroneously flagged as bugs, either as false positives or outright bugs in analytics tools. These problems also portend adjustment opportunities.
Each of these cases must be carefully scrutinized. In particular, false positives should be checked for correctness. Every issue requires free documentation and a strong data retention policy for full accountability. These triage defect reports may be revisited during an audit or in a retrospective if material errors are discovered later in the process. Organizations often go back to statically analyzing defects to see how major bugs work through the process. It could mark a broken process or an opportunity to tweak the analysis to find better bugs.
Static analysis is typically run in developer sandbox builds and/or via a central build (see Figure 4). At the very least, it makes sense to analyze and evaluate the results before publishing. However, software development organizations shouldn't wait until the last minute to address the large number of bugs that may exist, especially when those bugs could have been addressed earlier as part of the specification process. Otherwise, the team could miss deadlines and change the code at the worst possible time.
Figure 4: Static analysis can be deployed in many different ways depending on business needs, environment, and tools used.
Organizations often automate static analysis as part of nightly builds or continuous integration builds. In this way, results can be reviewed frequently and acted upon as they arise. Others perform the bug discovery process earlier by enabling developers to analyze the code they are working on in a sandboxed environment. Developers can get immediate feedback on the quality of their code changes, then fix and verify defects before check-in. The faster the cycle time, the cleaner the code in the codebase.
No matter where it is running, the technical environment needs to be consistent to ensure the same results. Central and developer builds need to be consistent. Minor changes to analysis settings can result in more results being reported, and organizations don't need the additional burden of reviewing more issues that may be mostly false positives. Creating a highly automated system for developers will help ensure consistency.
Many medical device companies check not only source code into their repositories, but also their actual environment. In this way, traceability is available. The static analysis executable and all related configuration, status and other related items should also be checked regularly to ensure consistency and accountability.
Deal with the backlog
Most organizations start using static analysis after developing a lot of code. Generally, the more code, the more bugs are reported. Therefore, when rolling out static analysis, management must pre-allocate time to deal with the initial backlog of bugs.
It's best to do static analysis early in the development cycle to minimize the backlog, and then create a process to handle the backlog separate from the daily influx of bugs due to daily code changes. Reviewing defects takes time and should be appropriately divided among developers, or outsourced to a separate team to pick out defects that require work.
All development teams differ in their technical skill levels and how everyone on the team defines quality. In training and mentoring sessions, the most common arguments are:
"Yes, it's definitely a bug, but the code has been working so we don't want to change it."
"We shouldn't have code like this in our products."
"This situation never happens in real life."
"If we port the product to another platform in the future, this will become a mistake."
"If you spend a few extra minutes on this, you'll see that it's clearly a mistake."
Static analysis will provide various types of errors, from critical issues that must be resolved to warnings. Some organizations want to be opportunistic and only change code for provable bugs. Others proactively clean up the code and improve quality, even "fixing" warnings. Teams should be consistent in how they handle static analysis results. Review results, training/mentoring, and frequent communication are the keys to success.
When used properly, static analysis has proven to be very effective in improving the software quality of safety-critical code. Although approval is not strictly required, the FDA acknowledges its effectiveness. With proper planning, expertise, and a realistic investment, static analysis should yield a substantial return on investment and help deliver safe code to the market.
Reviewing Editor: Guo Ting