Pass or Fail

In a project, I was given a task to test a system's Application Programming Interface (API). The system had Java and COM APIs. There was also documentation for the API, with descriptions for all its constructors, methods and constants.

I had to develop test scripts, using Java (for Java API) and JavaScript, Visual Basic, VBScript and C# (for COM API). I also had to run the test scripts for every new versions of the APIs during the project.

Main effort was applied to testing the Java API: it was tested completelly. The COM API was tested only for some selected, main functions.

Below is description of the Java API testing.

The testing approach is the following: in a test script, create a test function for each API method. The test function calls a method from the API and passes into the method different valid or invalid input parameters. The result returned by the method is compared with an expected value. If the actual and expected values are equal, the test function outputs a Pass status. Otherwise - Fail, and a defect shall be raised for the method.

Also in a test function, there is a check for an exception that the called method could throw. I check that an expected exception is thrown under defined conditions. Also I check that the text of the exception corresponds to the text in the documentation.

There may be cases when the documentation itself is outdated for some methods. Such defects should also be raised.

The test scripts can be executed from the command line, via a batch file, containing calls to all the test scripts. The test results are logged in a text file. The format for such a log file can be as follows:

Summary
-date time
-total number of test functions
-number of passed test functions
-number of failed test functions

Details (for each test function)
- test script name
- test function name with different input parameters
- Pass or Fail status
- Notes, if the test function failed

A script should throw an error if test data restoration failed during a test.

E.g. if in a test, a script creates new records, and the same number of records should be deleted in the end of the test, ensure that in the script, there is a check of the records number remained after the removal.

If a transaction needs to be run in a load test, say, in 4 times less then other transactions in the same script, then you can do the following:

if (rnd_skip < 4 && (strcmp(lr_get_attrib_string("is_load_test"), "yes") == 0) )
{
    lr_output_message("your comments...");
    lr_exit(LR_EXIT_ITERATION_AND_CONTINUE, LR_PASS);
}

where is_load_test is the parameter you need to create in the Run Time Settings dialog box. For a load test, its value should be "yes", and for a benchmark test - "no".

1. It is important to track the changes of the functionality that is being performance tested.

When attending a meeting, have a list of use cases and transaction names. Read aloud all the steps of each use case. Ask if anything from this has changed or will change in a new build.

If some functionality has been changed, need to update the use cases in the test specification and re-record test scripts. Get an agreement from the project manager and/or test lead regarding the changes in the test scripts.

2. A periodical manual check of the existing functionality is also important. Run the application under testing and go through all the steps that are covered by the use cases/test scripts. Check that between actual functionality and the transactions recorded in a test script the following corresponds:

• the response times (compare actual response time with the recent benchmark test results)
• contents (e.g. sections on a web page)

3. The test scripts should be re-recorded if there were significant changes in functionality.

It is important to control the jobs (processes) running in a test environment because they affect the system under testing.

The testers and the whole team should know and have a document with the following information for each job:

• Job name
• Job purpose and impact on configuration, functionality and performance of the system
• Who is responsible for the job and who starts/stops it
• When the job should be running

Also the team should be informed on every start/stop of the jobs, at least those that affect the system mostly.

It can be difficult to get values for a correlation parameter in HP LoadRunner if the parameter is located in a dynamic frame so that the order of the frame on the page changes for different users.

That is you need to know which frame number to put in the command

web_reg_save_param("param_name", "LB=some_left_boundary", "RB=some_right_boundary", "ORD=All", "NotFound=Warning", "RelFrameID=1.2", LAST);

One way to resolve this would be to find out in advance the frame number and store it in a DAT file for each user correspondently.

It is important to check that an automated load test provides valid figures. For this, you can manually check the performance of the application in the browser (if this is a Web application). A tool like HTTPWatch can help and show a list of requests and their duration, and the duration of the whole transaction.

This approach can be helpful because sometimes the automated load test might show wrong transaction duration due to various reasons, like

• outdated test scripts
• wrong test scenario configuration (Run-Time Settings)
• application uses some technologies that are not supported by the test tool. E.g. AJAX is not supported by LoadRunner's Web (HTTP/HTML) protocol, and the AJAX requests will be not recorded in the test script, which will cause, if not a transaction failure at all, but very likely smaller value of the response time.

So, at least periodical manual check of the application's performance within a release wll be helpful and save you from problems, if suddenly it will occur that there is a discrepancy between the actual performance and the load test results.

When you created and added a new script into an existing test scenario, you first need to run this new script for several iterations and ensure that it does not fail.

Otherwise, if you run the test without this preliminary check, it might happen that you will see the failure only in the end of the test, which will mean the test is not valid and the time is wasted (1-2 hours for a benchmark test).

There should be a release window for the test environment. It means that new builds are deployed into the environment only when it is approved by the test team.

In a test environment chat channel, a tester chats that the release window is open. Usually it is good to do in the end of the working day, e.g. at 17:00.

This time should be agreed with the Release team, and they should know that at this time daily the window opens.

Sometimes, the time might be changed depending on a situation, e.g. an urgent hot-fix deployment.

Helpful to chat in a test environment channel about work being done in the environment. E.g. when

• an automated test started/completed.
• release window is open (so that the relesae team would start deployment of new builds in to the test environement)
• there is a problem in the environment and the qa-support team should help
• need to discuss testing related questions, e.g. configuration, performance, issues