By Bob Boydston – UCI OIT Central Services

Prologue

My career in QA came about very inauspiciously. I graduated from the University of Southern California with my Master’s and had become the data analyst for my department. Working for a professor I created statistical programs in SAS and later in dBASE III Plus. It was through dBASE that I began my QA career. It all started when I saw an ad in a BBS with the message “dBASE dBASE dBASE – tester needed”. I applied for the job and after passing an exam in dBASE I got the job as a QA Analyst. Having been new to QA I began reading everything I could get my hands on QA. My department even brought in an expert to train us in the ins and outs of QA.

One of the things I began to appreciate was the need for negative testing. More and more I focused on “breaking the system”…as it is even today. When people would ask what I do for a living I would say, “I break things!” 🙂

So what is negative testing? It’s actually a number of things. It starts very simply at the “field” level. But there is also another important area called “State Based Testing.” Field testing is what you would think it is. Negative testing at the field level incorporates exercising field types, minimums and maximums, drop downs and date pickers to name a very few. State Based testing focuses on scenarios where the resulting “state” ends up with something awry, where a component ends up missing or misaligned or corrupted.

Field Testing

Negative testing encompasses different techniques depending on the types of fields, buttons, drop downs, date pickers and tabs to name a few. For each of the following object types negative testing can be performed. Take into account the “type” of field. Negative testing for a numeric field will be different than for a character field or a Boolean field. At the application level, many of these fields have prescribed rules. For instance, a numeric field may have an upper and lower threshold. Character fields may have a limit in the number of characters that can be entered. Some fields will require that a value be entered. Drop down fields may be mouse driven or may allow a user to enter a character to navigate to the desired drop down choice. Boolean fields also may have differing rules. Some require a True or False entry. Others require a numeric value…like 1 for true and 0 for false. Date fields may be free form while others one needs to enter a prescribed number of numbers for each date sub field like 2 digits for month, 2 for day and four digits for year. Date pickers also go by differing methods.

Mouse navigation testing evaluates if all the entry points are reachable. Sometimes the rendering of a screen precludes successful navigation. Tab navigation has become a real hot topic in recent years as it relates to Accessibility. For those with impaired motor and/or visual skills depend on accurate navigation via the tabs on a screen.

Another area that in some ways precludes normal QA testing is that of value injections into fields. For security purposes we want to inject xpaths, sql queries and html (Hypertext Markup Language) values in a field to see it exposes underlying information about the application and it’s infrastructure.

State Based Testing

State based testing depends on a finite state machine. An application can have many interconnected finite states guided by rules that transition from one state to another. For instance, SIS focuses on the status (or state) of a student. A student starts out as a Freshman and transcends to a Sophomore all the way to a Senior. Rules are set into motion as the student passes classes enough to move to the next state. For state based testing, QA can focus on the rules for each of the states. Additionally, using a negative testing focus QA can put a student in a compromised state. In other words, in a state that the application/system does not recognize.

Application Focus

For purposes of controlling the number of students enrolling at UCI at any one time, UCI has created what are called “Enrollment Windows”. According to a student’s status (or state) they are allowed to enroll during a specified window of time. For instance, athletes are usually the first to enroll. Seniors are second as they may have a need in taking particular classes in order to graduate. From there Juniors, Sophomores and Freshmen fill out the rest of the enrollment windows (there are exceptions to the rule but are not needed in this example).

From a negative testing perspective, it would be interesting to see if we could put a student in a state where he/she was NOT in any enrollment window. In other words, in the case our new Student Information System, it does not recognize the student’s state. This would be very informative in making sure a student does not slip into this state.

Security Focus

From a security perspective, negative testing can also be situational. As an example, make the assumption that in some way a user has broken into the application (the situation). From there the question QA will have is, what prevents the user from getting to the “crown jewels”. Applications that do not assess the credentials of a user at every turn in the application should be looked into to find ways to prevent this from happening.

Other ways of compromising a system in by looking at “deep links.” A deep link is a place in the application past the first screen (but usually much farther down). To simulate this, move to a screen deep into the application (i.e., getting to a deep link), copy the URL and log out. To test the deep link first bring down the browser and clear out all your cookies. Now enter the browser and simply paste the copied URL and hit Enter. If you have access to that page without logging in, then the application does not have the necessary safeguards in place.

Performance Focus

Typical performance testing examines the state of an application at designated levels of users accessing the application. For instance, what is the average amount of time for 10 users logging into the system at the same time. What about 20 users, 40 users, etc. Each of these “instances” can be called a state. It’s the place where the performance of an application has degraded. The reasons for degradation usually are multifaceted. How much memory on the system? What is activity of server at these different states? What about the efficiency of the database? Is there enough disk space?

Once we have qualified for each of these questions we can determine when the application basically cannot handle the next rung of testing. Does it happen at 100 users? 1,000 users? 20,000 users? Where? Once found out this gives development the opportunity to make the corrective actions.

Conclusion

One of the great dynamics in software development is the iconoclastic relationship between a development team and a quality assurance team resulting in differing points of view. In a nutshell one can say, Dev wants to know what works, while QA wants to know what doesn’t. Negative testing is the archetype of quality assurance. Without it the dynamic between dev and QA is broken.