Testing Application fail-over using Virtual Hardware

Application availability is one of the key features of many designs. e-commerce web applications are a particular example where the application should be available to customers as much as possible. A common model to achieve this is to have a number of servers available. Should one server fail the other servers pick up the request and deliver the result to the customers computer.

Although this design is simple in concept and with a lot modern server systems straightforward to implement testing the failure scenarios is often a complex and demanding task. Typically this sort of testing is done manually by disconnecting a server from the network or terminating processes on the server.

Virtual hardware can provide a cost effective alternative and provides capabilities that are not available when using physical servers.
Lets take an example

1. The customer clicks on the ‘Add to basket’ for a widget
2. The request is serviced by server A and returns a page representing the basket containing the selected widget.
3. The customer clicks the ‘Proceed to checkout’ button
4. The request is serviced by server A and returns a page requesting payment details for the basket containing the widget.

A fail-over scenario might look something like this

1. The customer clicks on the ‘Add to basket’ for a widget
2. The request is serviced by server A and returns a page representing the basket containing the selected widget.
3. The customer clicks the ‘Proceed to checkout’ button
4. Server A has a network fault so the request is serviced by server B and returns a page requesting payment details for the basket containing the widget.

In production this might be caused by a rodent chewing through a cable at at critical moment or some other hardware fault. But how do we test this easily.

Virtual machines typically use a virtual network connection routed through a physical connection or completely virtualized on a single host server. With VMWare Server and Microsoft Virtual server this network connection can be controlled by the host or by an application controlling the host. The last option is the most interesting because it allows the connection to be controlled programatically in the test code.

A test might then look like (pseudo code)

result = http.request(“www.test.domain.com/AddToBasket?product=widget”)
AssertWidgetInResult(result)
VirtualHost.ServerA.Network = disabled
result = http.request (“www.test.domain.com/Checkout?orderid=” + result.orderid)
AssertPaymentOptionsDisplayed(result)

The ability to automate these tests should not be underestimated. More complex scenarios can be developed and executed easily. The test code can be run frequently perhaps as part of a continuous integration system. Execution time for the tests is now much faster - no manual interventions. Removing the manual steps makes the tests repeatable and less prone to execution error.

Developers, Quality Analysts and Architects gain a much greater understanding of how resilient their application is to failure which leads to a greater confidence that the application will behave as expected in a production environment.