Complementary Testing Approaches






Complementary Testing Approaches

While we'd like to believe that an effective testing program should "find all the bugs", this is an unrealistic goal. NASA devotes more of its engineering resources to testing (it is estimated they employ 20 testers for each developer) than any commercial entity could afford toand the code produced is still not free of defects. In complex programs, no amount of testing can find all coding errors.

The goal of testing is not so much to find errors as it is to increase confidence that the code works as expected. Since it is unrealistic to assume you can find all the bugs, the goal of a quality assurance (QA) plan should be to achieve the greatest possible confidence given the testing resources available. More things can go wrong in a concurrent program than in a sequential one, and therefore more testing is required to achieve the same level of confidence. So far we've focused primarily on techniques for constructing effective unit and performance tests. Testing is critically important for building confidence that concurrent classes behave correctly, but should be only one of the QA metholologies you employ.

Different QA methodologies are more effective at finding some types of defects and less effective at finding others. By employing complementary testing methodologies such as code review and static analysis, you can achieve greater confidence than you could with any single approach.

Code Review

As effective and important as unit and stress tests are for finding concurrency bugs, they are no substitute for rigorous code review by multiple people. (On the other hand, code review is no substitute for testing either.) You can and should design tests to maximize their chances of discovering safety errors, and you should run them frequently, but you should not neglect to have concurrent code reviewed carefully by someone besides its author. Even concurrency experts make mistakes; taking the time to have someone else review the code is almost always worthwhile. Expert concurrent programmers are better at finding subtle races than are most test programs. (Also, platform issues such as JVM implementation details or processor memory models can prevent bugs from showing up on particular hardware or software configurations.) Code review also has other benefits; not only can it find errors, but it often improves the quality of comments describing the implementation details, thus reducing future maintenence cost and risk.

Static Analysis Tools

As of this writing, static analysis tools are rapidly emerging as an effective complement to formal testing and code review. Static code analysis is the process of analyzing code without executing it, and code auditing tools can analyze classes to look for instances of common bug patterns. Static analysis tools such as the open-source FindBugs[9] contain bug-pattern detectors for many common coding errors, many of which can easily be missed by testing or code review.

[9] http://findbugs.sourceforge.net

Static analysis tools produce a list of warnings that must be examined by hand to determine whether they represent actual errors. Historically, tools like lint produced so many false warnings as to scare developers away, but tools like FindBugs have been tuned to produce many fewer false alarms. Static analysis tools are still somewhat primitive (especially in their integration with development tools and lifecycle), but they are already effective enough to be a valuable addition to the testing process.

As of this writing, FindBugs includes detectors for the following concurrencyrelated bug patterns, and more are being added all the time:

Inconsistent synchronization. Many objects follow the synchronization policy of guarding all variables with the object's intrinsic lock. If a field is accessed frequently but not always with the this lock held, this may indicate that the synchronization policy is not being consistently followed.

Analysis tools must guess at the synchronization policy because Java classes do not have formal concurrency specifications. In the future, if annotations such as @GuardedBy are standardized, auditing tools could interpret annotations rather than having to guess at the relationship between variables and locks, thus improving the quality of analysis.

Invoking THRead.run. THRead implements Runnable and therefore has a run method. However, it is almost always a mistake to call Thread.run directly; usually the programmer meant to call THRead.start.

Unreleased lock. Unlike intrinsic locks, explicit locks (see Chapter 13) are not automatically released when control exits the scope in which they were acquired. The standard idiom is to release the lock from a finally block; otherwise the lock can remain unreleased in the event of an Exception.

Empty synchronized block. While empty synchronized blocks do have semantics under the Java Memory Model, they are frequently used incorrectly, and there are usually better solutions to whatever problem the developer was trying to solve.

Double-checked locking. Double-checked locking is a broken idiom for reducing synchronization overhead in lazy initialization (see Section 16.2.4) that involves reading a shared mutable field without appropriate synchronization.

Starting a thread from a constructor. Starting a thread from a constructor introduces the risk of subclassing problems, and can allow the this reference to escape the constructor.

Notification errors. The notify and notifyAll methods indicate that an object's state may have changed in a way that would unblock threads that are waiting on the associated condition queue. These methods should be called only when the state associated with the condition queue has changed. A synchronized block that calls notify or notifyAll but does not modify any state is likely to be an error. (See Chapter 14.)

Condition wait errors. When waiting on a condition queue, Object.wait or Condition. await should be called in a loop, with the appropriate lock held, after testing some state predicate (see Chapter 14). Calling Object.wait or Condition.await without the lock held, not in a loop, or without testing some state predicate is almost certainly an error.

Misuse of Lock and Condition. Using a Lock as the lock argument for a synchronized block is likely to be a typo, as is calling Condition.wait instead of await (though the latter would likely be caught in testing, since it would throw an IllegalMonitorStateException the first time it was called).

Sleeping or waiting while holding a lock. Calling Thread.sleep with a lock held can prevent other threads from making progress for a long time and is therefore a potentially serious liveness hazard. Calling Object.wait or Condition.await with two locks held poses a similar hazard.

Spin loops. Code that does nothing but spin (busy wait) checking a field for an expected value can waste CPU time and, if the field is not volatile, is not guaranteed to terminate. Latches or condition waits are often a better technique when waiting for a state transition to occur.

Aspect-oriented Testing Techniques

As of this writing, aspect-oriented programming (AOP) techniques have only limited applicability to concurrency, because most popular AOP tools do not yet support pointcuts at synchronization points. However, AOP can be applied to assert invariants or some aspects of compliance with synchronization policies. For example, (Laddad, 2003) provides an example of using an aspect to wrap all calls to non-thread-safe Swing methods with the assertion that the call is occurring in the event thread. As it requires no code changes, this technique is easy to apply and can disclose subtle publication and thread-confinement errors.

Profilers and Monitoring Tools

Most commercial profiling tools have some support for threads. They vary in feature set and effectiveness, but can often provide insight into what your program is doing (although profiling tools are usually intrusive and can substantially affect program timing and behavior). Most offer a display showing a timeline for each thread with different colors for the various thread states (runnable, blocked waiting for a lock, blocked waiting for I/O, etc.). Such a display can show how effectively your program is utilizing the available CPU resources, and if it is doing badly, where to look for the cause. (Many profilers also claim features for identifying which locks are causing contention, but in practice these features are often a blunter instrument than is desired for analyzing a program's locking behavior.)

The built-in JMX agent also offers some limited features for monitoring thread behavior. The ThreadInfo class includes the thread's current state and, if the thread is blocked, the lock or condition queue on which it is blocked. If the "thread contention monitoring" feature is enabled (it is disabled by default because of its performance impact), ThreadInfo also includes the number of times that the thread has blocked waiting for a lock or notification, and the cumulative amount of time it has spent waiting.



 Python   SQL   Java   php   Perl 
 game development   web development   internet   *nix   graphics   hardware 
 telecommunications   C++ 
 Flash   Active Directory   Windows