By Bob Boydston and Tom Bindewald – UCI OIT Central Services

Over the last few years OIT Software QA has been involved in quality assurance testing and conducting performance measurements for various applications. In that period of time we have noticed that the results for these applications vary greatly especially in terms of how the QA testing and performance results can help a development team in identifying issues with an application’s overall quality and performance. What we have noticed is that those that employ Webservices have the easiest time in getting the application tested and once tested dealing with application performance issues.

Why? From a quality perspective, Webservices allow for each “method” to be tested individually. Through its interface each field in the method constrains the type of data that can be entered. This allows a test team to virtually specify the total number of test cases that are necessary. This includes both positive and negative types of testing.

From a performance perspective, each of the methods can be tested separately as well. Because the testing can be conducted from the level of a “method” then the results of a “method” allow for the development team to hone in on underlying issues. In other words, once the results come in the development team can quickly determine where there may be issues and focus on those methods that look problematic.

Webservices have two opposing types of application programming interfaces (aka API). One is called SOAP (Simple Object Access Protocol) and the other REST (Representational State Transfer). We don’t want to get into the battles between those who prefer one over the other but want to explain how they are used in terms of testing. As explained before you can test them at the method level (one operation) or from a scenario level. In fact, from a QA perspective you would want to test them at both levels; first at the method level and then from the creation of scenarios. The scenarios many times will emulate how the application will be developed, so it is important to know which methods will be used in the application and in what sequence. Ultimately, this is how the entire application gets tested. One example here at UCI is the UCPath project that utilizes ESBAT webservices both QA and Performance testing.

The most popular API is Rest. One of the main reasons is because REST allows for many types of data types and data formats like plain text, HTML, XML and JSON. While SOAP only uses XML. However, SOAP is excellent from the security level via the WS-Security extension.

Webservices from the Quality Perspective

So what is the process that makes the use of webservices an approach for better quality and better performance? Maybe the best way to explain is to show this by creating a sample application that is very simple for demonstration purposes.

Let’s say you have a web application that has the following features: User, User Role or User Profile, an Organization and the capability to Login and Logout. For each of these features we would want to utilize the four basic functions for handling data such as create, read, update and delete (also known as “CRUD” for those familiar with databases). For “Users” we’d want to perform functions like create a user, read a user’s information, update a user (or change information on a user) and delete a user.

So from a Webservice perspective we could create four methods (as explained above) or functions in the following manner: createUser, readUser, updateUser and deleteUser. Each of these methods would contain fields where data can be entered or modified.

Let’s look at createUser. One could imagine the following fields would be available: First Name, Last Name, Street Address, City, State, ZipCode and US Citizen. The first five fields would be alphanumeric fields. The ZipCode would be a numeric field. And the US_Citizen field would be a Boolean field.

To test this one would create positive tests for each of the fields, in other words, enter data that is constrained but appropriate for the fields. For the negative tests we would enter data that is not appropriate for that each field type. For the alphanumeric fields we could enter a Boolean values. For the (Zipcode) field we would enter a letter and Boolean value. Lastly, for the Boolean field I would enter both numbers and letters. We would, then, do the same type of testing for all other methods. How is this testing different than for other types of applications? It’s different because we’re testing to the constraints governed by the method. This means that the methods contain a limited range of functionality. Therefore, from a positive and negative perspective, you have tested every possible combination.

There other benefits to Webservice testing in that a tester can create scenarios by combining methods into suites. In this case, you could login, create a user, give the user a role, assign a user to an organization and then log out.

One other advantage of this scenario is that you can create a matrix of dependent methods. For instance, you cannot create a user role until you have created a user.

Webservice Tools

Two tools come to mind in the testing of webservices; SoapUi and Postman. While SoapUi can accommodate both SOAP and REST, Postman is essentially a REST client. Both tools allow for testing to occur at the method level and the scenario level. Also, both allow for validations to occur (pass or fail). Both have a free (open source) products but also applications with more convenient interfaces but at a price.

Webservices from the Performance Perspective

Many of the advantages of Webservices for QA are the same when measuring an application’s performance. The main advantage of using Webservices when testing for its performance is that the testing can take place at the level of a single method. Some methods are based at the business logic layer while others directly hit the database. Over the years of performance testing we have discovered, by and large, that database activity dominates application performance.

For database performance one can test using the CRUD methodology as it’s basis but especially in terms of retrieving data (get function) and updating or adding data (post or put function). Think of it this way, if you knew that hundreds or even thousands of users were attempting to retrieve data, you would want to know where potential bottlenecks may be hiding. Through Webservices you have the opportunity to discover this because you are able to isolate your search to a single method.

There are two popular applications that take advantage of webservices performance testing; mainly jMeter and LoadUi. jMeter is the most inclusive of the performance tools. The features are truly extensive. It is compatible with the following features: HTTP, HTTPS, SOAP, REST, FTP, JDBC (database connectivity), LDAP, mail, SMTP, POP3, IMAP, shell scripts, TCP, Java, multithreading and BeanShell.

LoadUi is a companion of SoapUi. Because it’s based on SOAPUI’s Webservice capabilities it allows for testing at the method level and at the scenario level.

Conclusion

Webservices have proven to be a boon not only in terms of an application’s quality but also in terms of its performance. In fact, it has become a standard within the IP industry. If you are building your application on webservices it allows you to build quality into your product as you create it. In the end, it is a great foundation for both quality and performance and OIT QA can be your partner in the process.

By Bob Boydston and Tom Bindewald – UCI OIT Central Services

What is Performance Testing Anyway?

Most think that performance testing pertains to such activities as measuring transaction response times, extent or stress testing of an application; however, performance testing covers numerous facets of an application’s behavior. One has to look at the total picture of the utility of an application. The focus of performance testing is user satisfaction; that should be the goal of software development teams given this is what users notice. The salient feature of an Application is its productivity as determined by its performance characteristics that included the design of the User Interface (UI / GUI), the application layer (Java, C#, C++), software libraries ¹, and database queries.

Therefore, the design and construction of the application is essential to its overall performance, which is the determining factor in the selection of the variety of performance tests conducted by the Quality Assurance team in OIT Central Services. It is with this perspective that we start of a series of blogs outlining differing aspects of application performance.

Additionally, adherence to establish coding practices ² is an essential component of an application’s performance. Using well known coding design patterns ² as the foundation for an application will result in logical coherent, concise, robust, and performant source code that is maintainable and understandable. Following the standards outlined in earlier articles in this blog under the rubric “Foundations of Software Quality” is a good guide to writing application source code that meets quality coding standards.

The QA team currently has two source code analysis tools: PMD and Findbugs ³ that inspect an application’s source code for programming issues. The reports generated by PMD and Findbugs provide developers with information to address a variety programming problems ⁴, including performance issues, throughout the development process.

Looking at a Layered Architecture

Currently, performance testing is focused on web applications using testing tools that are capable of simulating numerous concurrent users making simultaneous HTTPS requests or by invoking a web-service API layer as defined by the application. Typically (and recommended), the web application is comprised of a layered architecture: Web-browser client, Application Server running the various application components, and a database server. A layered architecture enables each of the layers web-application component to be decoupled from one another. In this way, each layer is independent of the other with a well-defined interface between each web-application component. For the most part the architecture is comprised of the Presentation (or Application) layer, the Business layer and the Data layer. These can be identified as (1) a Web-browser client, (2) an Application Server and application software, and (3) a Database server and/or data store server.

To Couple or to Decouple?

Obviously, in a simple application where there is only one screen and very few users, decoupling the application may not make a lot of sense. For more complicated applications decoupling definitely makes sense. There are many reasons why but one of them is to allow the application to engage in a particular activity ONLY when it is appropriate. For instance, when data is needed by the application layer, it can access the business layer, then the data layer to get that information for a particular request. The application is configured to allow the presentation layer to guide requests. Having to engage in all these layers simultaneously (thus coupling) would present a large drain on performance.

From a performance perspective, traditionally, most of the testing happens at the Business and Data layers. But there are applications that have “heavy” clients meaning their application layer has two disadvantages in terms of performance. First, is the amount of time it takes for a page to be ready for the user, i.e., rendering. Rendering can be influenced by design decisions. For instance, how much “coupling between the architectural layers” goes on before a page is ready for the user? How much “decision making” occurs in the client? Can much of the decision making occur in the business layer? Are there errors in the javascripts? Is it possible to get into an endless loop in the javascripts or between javascripts calling each other?

The Business Layer

So what is the business layer? Some have called it the “Service” layer. It is the encapsulation of what makes a business. Wikipedia identifies it as “real-world business rules that determine how data can be created, displayed, stored, and changed.”

If an application is built on Web Services then each service method portrays some business objective. Using Web Services as the basis of performance testing then the effort is straightforward. Many times a service method is dependent on some other service method. For instance, if you have a service method that creates a user instance, you might have another method that creates its profile determining its permissions and/or roles. The business purpose here is assure permissions and roles are assigned effectively for each user.

Other ways of testing performance of the business layer in terms of a business interest without using Web Services would be the length of time it takes for transactions to be created and/or accessed. In these cases, the data layer is used BUT the business interest is what is being tested.

The Data Layer

Performance testing the data layer can go in many directions. For applications that have “reporting” queries then testing those queries can be done by addressing the query directly like in a JDBC call. Other applications that use javascripts that distinguish the “data layer” in terms of arrays and other assigned variables. Most of our performance testing at the Data Layer revolves around database queries.

However, a greater number of web applications use javascripts to process the business layer actions predicated on information addressed via data layer. In these cases, testing how fast a screen renders can help in making sure that is essential in ensuring an acceptable user experience.

The Next Two Articles

This is the first in a series of articles about the foundation of performance testing. The next article will concentrate on the Business Layer and the one after that will deal with the Data Layer.

References

¹ Software Libraries – Database frameworks, developer and third party packages, and O/S system services.

² Design and code examples (Java, C++ … ) for various types of applications and systems …

http://www.engr.sjsu.edu/fayad/current.courses/cmpe202-spring2016/docs/CmpE202-SE-Links-Part-One-Fall2013/03-DesignPatterns/Design%20patterns%20in%20Java.pdf

³ Findbugs: https://wiki.oit.uci.edu/display/adcom/FindBugs+Information and PMD …

https://pmd.github.io/ , https://pmd.github.io/pmd-6.1.0/

⁴ There exist an anbount types of programming problems that impact perforamance such as: Threading, searching, caching techniquies, database framework issuse …

https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

By Bob Boydston – UCI OIT Central Services

Prologue

My career in QA came about very inauspiciously. I graduated from the University of Southern California with my Master’s and had become the data analyst for my department. Working for a professor I created statistical programs in SAS and later in dBASE III Plus. It was through dBASE that I began my QA career. It all started when I saw an ad in a BBS with the message “dBASE dBASE dBASE – tester needed”. I applied for the job and after passing an exam in dBASE I got the job as a QA Analyst. Having been new to QA I began reading everything I could get my hands on QA. My department even brought in an expert to train us in the ins and outs of QA.

One of the things I began to appreciate was the need for negative testing. More and more I focused on “breaking the system”…as it is even today. When people would ask what I do for a living I would say, “I break things!” 🙂

So what is negative testing? It’s actually a number of things. It starts very simply at the “field” level. But there is also another important area called “State Based Testing.” Field testing is what you would think it is. Negative testing at the field level incorporates exercising field types, minimums and maximums, drop downs and date pickers to name a very few. State Based testing focuses on scenarios where the resulting “state” ends up with something awry, where a component ends up missing or misaligned or corrupted.

Field Testing

Negative testing encompasses different techniques depending on the types of fields, buttons, drop downs, date pickers and tabs to name a few. For each of the following object types negative testing can be performed. Take into account the “type” of field. Negative testing for a numeric field will be different than for a character field or a Boolean field. At the application level, many of these fields have prescribed rules. For instance, a numeric field may have an upper and lower threshold. Character fields may have a limit in the number of characters that can be entered. Some fields will require that a value be entered. Drop down fields may be mouse driven or may allow a user to enter a character to navigate to the desired drop down choice. Boolean fields also may have differing rules. Some require a True or False entry. Others require a numeric value…like 1 for true and 0 for false. Date fields may be free form while others one needs to enter a prescribed number of numbers for each date sub field like 2 digits for month, 2 for day and four digits for year. Date pickers also go by differing methods.

Mouse navigation testing evaluates if all the entry points are reachable. Sometimes the rendering of a screen precludes successful navigation. Tab navigation has become a real hot topic in recent years as it relates to Accessibility. For those with impaired motor and/or visual skills depend on accurate navigation via the tabs on a screen.

Another area that in some ways precludes normal QA testing is that of value injections into fields. For security purposes we want to inject xpaths, sql queries and html (Hypertext Markup Language) values in a field to see it exposes underlying information about the application and it’s infrastructure.

State Based Testing

State based testing depends on a finite state machine. An application can have many interconnected finite states guided by rules that transition from one state to another. For instance, SIS focuses on the status (or state) of a student. A student starts out as a Freshman and transcends to a Sophomore all the way to a Senior. Rules are set into motion as the student passes classes enough to move to the next state. For state based testing, QA can focus on the rules for each of the states. Additionally, using a negative testing focus QA can put a student in a compromised state. In other words, in a state that the application/system does not recognize.

Application Focus

For purposes of controlling the number of students enrolling at UCI at any one time, UCI has created what are called “Enrollment Windows”. According to a student’s status (or state) they are allowed to enroll during a specified window of time. For instance, athletes are usually the first to enroll. Seniors are second as they may have a need in taking particular classes in order to graduate. From there Juniors, Sophomores and Freshmen fill out the rest of the enrollment windows (there are exceptions to the rule but are not needed in this example).

From a negative testing perspective, it would be interesting to see if we could put a student in a state where he/she was NOT in any enrollment window. In other words, in the case our new Student Information System, it does not recognize the student’s state. This would be very informative in making sure a student does not slip into this state.

Security Focus

From a security perspective, negative testing can also be situational. As an example, make the assumption that in some way a user has broken into the application (the situation). From there the question QA will have is, what prevents the user from getting to the “crown jewels”. Applications that do not assess the credentials of a user at every turn in the application should be looked into to find ways to prevent this from happening.

Other ways of compromising a system in by looking at “deep links.” A deep link is a place in the application past the first screen (but usually much farther down). To simulate this, move to a screen deep into the application (i.e., getting to a deep link), copy the URL and log out. To test the deep link first bring down the browser and clear out all your cookies. Now enter the browser and simply paste the copied URL and hit Enter. If you have access to that page without logging in, then the application does not have the necessary safeguards in place.

Performance Focus

Typical performance testing examines the state of an application at designated levels of users accessing the application. For instance, what is the average amount of time for 10 users logging into the system at the same time. What about 20 users, 40 users, etc. Each of these “instances” can be called a state. It’s the place where the performance of an application has degraded. The reasons for degradation usually are multifaceted. How much memory on the system? What is activity of server at these different states? What about the efficiency of the database? Is there enough disk space?

Once we have qualified for each of these questions we can determine when the application basically cannot handle the next rung of testing. Does it happen at 100 users? 1,000 users? 20,000 users? Where? Once found out this gives development the opportunity to make the corrective actions.

Conclusion

One of the great dynamics in software development is the iconoclastic relationship between a development team and a quality assurance team resulting in differing points of view. In a nutshell one can say, Dev wants to know what works, while QA wants to know what doesn’t. Negative testing is the archetype of quality assurance. Without it the dynamic between dev and QA is broken.

By Bob Boydston – UCI OIT Central Services

Prologue

This article is the second in a series called the “Foundations of Quality” that focuses on the infrastructure of a software team that needs to be in place for it to produce a quality application or service. The first dealt with the Code Promotion model. This article discovers the benefits of code reviews.

Top software houses in the world routinely conduct code reviews but not in the traditional time consuming way. Instead, developers are, increasingly, doing their code reviews using collaborative software such as GitHub, Bitbucket, Gerrit, CodeFlow, Collaborator, ReviewBoard and Crucible, to name a few.

So if code reviews are universally being conducted what are the results? Do they really help? What good are code reviews? Some people swear by them; others think they are waste of time. To be fair, both sides are right. So how can that be? It’s because it matters on HOW you do it.

In the olden days, code reviews were time consuming and did not net a lot of bugs or refactoring. Usually you had at least 3 people; the reviewer, the reader and the recorder. Many times you had even more people involved. The reviewer was the one who would choose the code to be reviewed. The reader tended to be the writer of the code (in some software houses it was actually the reviewer that read the code) and the recorder was the one who wrote up the results of the review as defects or refactoring endeavors. These code reviews were verbal and took time to conduct and all the “actors” had to be there at the same time. In other words, all coding activity for those participating in the code reviews would come to halt simultaneously.

But did they really produce results? To tell you the truth, not many issues or defects would come out of it. One of the reasons why there were not a lot of defects or refactoring was because in this process you could not review a lot of code because it took so much time to go through it. So the exposure to the code was limited. Many times you were seeing the “happy” path stuff or the stuff that a coder wanted you to see. For these reasons, code reviews were not always conducted.

Now days we have collaborative software and methods which reach out to much more code and more people. Now code reviews occur at the convenience of the person doing the reviewing. All “recording” activities are done at the point of a segment of code and are written eliminating the reader and the recorder.

Code Review Using Tools

One of the most important advantages of conducting code reviews using collaborative software is that it allows team members to see what others are doing on their own time. Not only do developers understand how and where their code fits into other parts of the application being developed but it also spreads around the “expertise” of the code to the team members. This helps alleviate the famous “developer in a room” scenario where a developer becomes the only one who understands his code. Or said differently, it increases the “Bus Factor.” Now you have more than one person exposed to the code. Importantly, it allows the reviewer to choose where to look.

There are couple of ways to conduct a code review using tools. The first is to get past all the obvious issues like missing brackets or no break in a switch. In QA we use two products called FindBugs and PMD (for java) to accomplish this task. The second is to take advantage of tools that allow for collaborative reviewing.

There are many tools out there but there are two this author thinks are worth looking at; GitHub and Review Board. Both have comparable features such as collaborative viewing and commenting, advanced “diff” viewers, comments of “diffs” shown with the file content and multi-line commenting. Also, each has its hallmark features that put it apart as well.

Review Board

Here are features that are noteworthy:

• Images can be dragged as part of your comment
• Treats text files like language based files
• Provides Markdown support
• PDF document support via Power Pack add-on
• Diff viewers
  • Syntax is highlighted depending on your programming language
  • Handles changes to the indenting of code
  • Tracks code that has been moved
  • Reveals what classes or functions you are in
• On demand expanding of the diffs
• Commenting images or PDFs
• Tracks issues in code that need attention
• Organizes reviews and discussions
• Comments of diffs shown with the file content
• Provides for “pre-commit” reviews

GitHub

Here are features that are noteworthy:

• Proposed changes start with a Pull Request
• Coder can request reviews
• Diff viewer allows for keyboard shortcuts for reviewing
• JS library detection
• Can review code changes since the previous review as opposed to all the changes
• Comments on context
  • Allows for comment of threads.
  • Ability to bundle your comments into one review
  • Comments can be suggestions or end up as requirements
• Protect branches by merging only the highest quality code
• Make status checks a requirement for repositories
• Cards: a snapshot of what people do that can be dragged as a “card”
• Notes: included as part of a stand-up
• Track and assign tasks
• Add milestones to a pull request
• Assign a pull request to one or more individuals

Conclusion

So do code reviews get results? Yes they do IF you use a collaborative tool such as those reviewed in this post.  As was mentioned in an earlier post, conducting code reviews is one of the pillars of quality code. So let’s make code reviews a standard for quality!

Note: Many thanks to Dana Watanabe for his help in distinguishing for me the many features of the two collaborative code review tools presented in this blog.

By Bob Boydston – UCI OIT Central Services

Prologue

This article is the first in a series called the “Foundations of Quality” that focuses on the infrastructure of a software team that needs to be in place for it to produce a quality application or service. Just like a house, it’s all about the foundation. A software team needs to heed the call for a firm foundation. From a quality assurance perspective, we will explore over the next few weeks the need for a code promotion model, code reviews, and release management, to name a few.

At a bare minimum, the Code Promotion model pertains to three essential environments: a Development environment, an Integration (or QA) environment, and the Production environment. There can be more environments depending on the need for more intensive types of testing.

A Word About Extra QA Environments and Release Management

Extra QA Environments

There are cases where QA can have its own environment where it does the testing before moving the code to the integration environment. This is usually done for applications that interact with other applications or services. In this case, you want to make sure the product behaves, as designed, as a standalone first before promoting it to the integration environment. Also, sometimes it is nice to have another QA environment where the code is one build behind so that comparisons can be made between the current one and the earlier one. This is very useful when debugging a problem.

Release Management

Promotion from one environment to the next environment can be done by the Development team but it is recommended that all promotions be conducted by the OIT Release team. By involving the Release team it does a number of things. First, it requires the Development team to make sure that it meets the standards of code promotion advanced by the Release team. Second, it frees the Development team from having to do the promotion meaning they can concentrate on their code development instead of the code promotions. And three, it prepares the Release team in handling promotions for an application focusing on configurations, prerequisite data connectivity and necessary services. Another big benefit is that once the Release team has promoted an application to the next environment, they become experts for deploying that application in the future.

Development Environment

So let’s look at each environment and see what kind of requirements and attention they need. The first is the development environment. This area is actually the least rigorous in terms of the promotion of code. Highly dependent on source code management system (see article entitled Comparing Source Code Repository Services), this area constitutes a “proof of concept” so much of the code base may be “out-of-date” or not totally in sync with the repository. This is where the code needs to be downloaded from the repository and the unit tests are run, especially on a build basis. The code cannot be deployed to the next environment until all the unit tests pass.

It is recommended that the promotion to Integration be performed by the Release team.

Integration Environment

The QA or Integration environment is just one step before Production. Because of that the Integration environment needs the “look and feel” of Production. In this way, once promoted to Production, there will be no surprises as it pertains to configurations, prerequisite data connectivities and necessary services. Until a decision is made to promote the code to Production the Integration environment becomes the environment of record. This is where functional (manual) and automation testing is conducted. This is also where automation tests are created and validated.

Configuration and Operational Data

Both the configuration and operational data between the Production and Integration data need to the same or as close as can be. Obviously, sensitive data such as vendor tax numbers and social security numbers are the exception where they need to be removed or written over. The intent of making sure the two environments look alike as much as possible is to remove any surprises once the code has been promoted to Production. For configuration files they also need to be in the “same” locations between the Integration and Production environments. Also, any “variable names” in the configuration files need to be same and the data needs to be formatted the same. This is helpful in debugging issues, and also in assuring that the promotion from one environment to another proceeds without a multiplicity of problems.

Cycle between Development and Integration

If bugs/errors are found or new functionality is added, then the cycle between Development and Integration ensues. Once bugs/errors are corrected, this is where they will be retested. For new functionality new automation tests are created and tested. It’s also the first place where the underlying architecture can be explored. Does redirection work? If a node goes down does the application correctly move a user from that node to a corresponding one without the user noticing it? What about performance? Is there a threshold where performance degrades rapidly as more and more users are logged in?

Promotion to Production

In terms of promoting the code base to the Production environment there are a have a number of steps that need to be passed before it is actually recommended for promotion. Some of these are standard based and others pertain to the types of tests that need to pass before promotion.

Standards Based 

In Software QA a standard has been set for all teams that it works with. The first relates to source code analysis tools. For Java based applications the code is run through FindBugs and PMD. FindBugs does not actually handle analyze source code. Instead it analyses bytecode. For instance, it evaluates when subtypes fail, nulls are returned, impossible casts, and infinite loops to name a few.  PMD provides true source code analysis like non-extant null checks, missing break in a switch or omitting a brace. PMD also provides Cyclomatic complexity information.

The second relates to a tool that analyses HTML and XHTML documents for well-formed markup. W3C Markup Validation Service is a great tool that finds all kinds of easy to fix issues.

So as code is handed over to Software QA it is analysed using these tools and reports are given to the development team.

Testing

Before an application or service is promoted to Production, Software QA is enjoined to provide testing. This is a combination of automation and functional tests. The tests are created using requirements and/or suggestions from Stakeholders (much like the feature files in the Selenium/Cucumber framework). These tests need to pass for the build to be eligible for promotion to Production. The only exceptions are when issues are addressed and a solution is agreed upon by all parties.

Again, it is recommended that the promotion to Production be conducted by the Release team.

Production Environment

Once an application or service is launched into Production, Quality Assurance’s role changes. Production is not a testing environment per se. Nonetheless, there are a number of actions that can be taken to assure that the newly launched application or service is ready for use. From a Software QA perspective, a smoke test called a “Health Check” should be run to validate that the application is running as expected. Usually, this means that the smoke test is non-intrusive in that records are not added or changed.

From a Release management perspective the application or service needs do the following:

• Implement Production Control “standards” for maintainability that are agreed upon
• Create and run Health Checks on a predetermined basis
• Identify log files and the agreed upon error levels set
• Establish external monitoring application and services
• Agree on the reporting needs at each level
• Provide documented instructions for Production Control to support the applications

Conclusion

These seem like simple suggestions but in truth, they help lay the foundation for quality. Without them, quality can be elusive.

By Seth Roby – UC Recruit Team

Bob recently shared a change that his team has made away from Cucumber and toward Selenium-Only testing. Given that Bob’s group is a standalone QA department that works on lots of different projects, who comes in at a fairly late stage of the game, and who often works without access to the development team, his decisions make sense. Still, there’s a lot of teams out there that do their own QA, and I wanted to offer a look at how we do things.

We on the Recruit team have been Cucumber users for years, and still use it every day: our 980-and-counting scenarios run multiple times a day, and must pass before any code merges to our deploying branches. Nothing hits our QA servers if it hasn’t already been run through our “cukes”.

Here’s how we suggest you handle Cucumber to make it an invaluable part of your toolchain:

Make your developers write tests, preferably first

Your product will have fewer regressions and more coverage, and you’ll discover issues earlier in the cycle as you plumb the possibilities. If you share your scenarios with your business analysts they can vet your assumptions and avoid rework. Don’t accept new code that isn’t tested, and don’t deploy code that isn’t tested.

Run a headless web browser

We use PhantomJS instead of using Selenium. Selenium requires a UI session, which means you have to dedicate more hardware to get parallel testing. We commonly run our test suite in six parallel sessions on a single web server with no UI, just PhantomJS. You’re giving up the ability to test different browsers, but you’re gaining speed and consistency.

Keep your codebase DRY as you go

Be aggressive about maintenance of your test suite. Testing code is only ever run while you’re executing the tests, so it is vitally important that it is as simple as possible; you can’t depend on unit tests here. Whenever you can, push complexity into the codebase proper by finding a way to make the test and the application do the same thing. The Growing Object Oriented Software Guided By Tests book is a great example of how to do this in practice.

Acknowledge that your test suite will inevitably grow

Make smart decisions about which tests to keep and which can be pruned. If you’re testing the same UI component in multiple scenarios, consider removing one. Better yet, remove both and add a scenario that tests the component divorced of its dependencies on the specifics of one page, more like a unit test running in the browser.

Push toward an assume-nothing data story

Every test should jump into the a database with only the bare essentials in it, and should use its setup to create the other resources it needs. We keep our statuses in the database all the time, but if your scenario needs a recruitment or a user you need to add that in using cucumber steps. Those steps are shared by a slew of scenarios, so we know they work, and they’re built to be flexible, so they can help you get started. Our “Given” tests are allowed to jump straight to the database to set up state, using FactoryGirl to build complex trees of objects; your framework/language of choice likely has a similar library. It allows us to avoid fixture tests, where fixing the data to make one test work endangers other tests in the suite.

Focus your testing on the golden path

The biggest bang for your buck is the scenario where everything is done correctly and the outcome is a success. If that breaks the feature is broken. Have a high level “if any error happens” scenario to ensure that reporting and transactions are honored, but almost all error cases should be handled in unit tests, where the plethora of possibilites can be examined in more detail.

Use page objects

These are a great tool to DRY up your code. They allow you to centralize your understanding about individual parts of the application, but more importantly they allow you to extract code from steps and into sharable libraries. Recruit uses very few page objects because they didn’t exist when we started, but we have lots of “component objects” that allow tests to work with specific reusable bits of the application.

Embrace JavaScript testing

Tools like Jasmine or Mocha run a different, more focused test suite, and can be run as a scenario in your Cucumber suite. You can get finer grained tests run in a lot less time, and be sure you cover all the cases, without having to set up as much state and depend on so many specifics. As Recruit has taken on more front end complexity we’ve begun to feel this pain, and are currently spinning up a Jasmine test suite.

Conclusion

Now, let me be clear. Setting up a test suite is a pain, and enforcing its care and feeding is a resource drain. We feel that discomfort, and we pay that price. We still have tests that sometimes fail for no apparent reason (to paraphrase Einstein, “Do not worry about your difficulties in Cucumber. I can assure you mine are still greater”), but the benefits we accrue are worth it.

Our team moves very fast and is unafraid to make major changes, because we have a comprehensive suite of tests acting as guardrails. Cucumber gives us the peace of mind to meet our stakeholders’ needs on the schedule they need to be met.

Cucumber isn’t free. Like all things in this industry it’s a tradeoff. But it’s one we’re very happy we’ve made.

By Bob Boydston – UCI OIT Central Services Team

Recently we moved from being a Selenium/Cucumber house to a Selenium Only house…yep we booted out Cucumber.  The reasons were many but much of it came down to the amount of maintenance we were involved in.  Our Software QA team now supports eighteen applications with hundreds of test scenarios and thousands of test cases and the number is growing. At one point all of them were written in Java using the Selenium/Cucumber framework. Actually, when we started using the Selenium/Cucumber framework it looked like the best solution, at the time, and it really was the best solution given the time frames we had to work with and the payoff we could muster.  Nevertheless, as time progressed and the number of projects we were supporting, simply maintaining the myriad of code bases grew more and more cumbersome. This is when we began to rethink our strategy and find ways to make our test automation maintenance less of a burden.

We identified a number of problems.  The most important ones were: (1) we had a lot of repeated code. We agreed this didn’t help solve the problem; (2) over time the scenario numbers and sizes grew; (3) this led to maintenance problems.  As things would change in an application we would have to update the code base to reflect the change. Because we had so much repeatable code, we would have to make the same change in multiple scenarios; (4) because of the size of these scenarios they would naturally run longer.  This meant that the scenarios in some applications (for instance KFS) would take hours to run. This was a problem because it would mean, many times, that we would find out about a problem late in the day.

Because of these issues we began to look for solutions.  Below are some of our findings:

1. One code base – less maintenance
   1. Issue: Repeated code using the Cucumber method. Each scenario would repeat much of the same code based on the necessary framework files. This made it a REALLY bad maintenance headache.
   2. Solution: Using the Selenium only solution we have all the scenarios running using just one set of code. A beneficial trade-off is that now we have immensely less code. All this means less maintenance.  When a change needs to be made, we go to just one place instead of many.
2. Retains page objects
   1. Issue: Until about a year and a half ago we would repeat all our DOM locations in the scenarios. Page objects were a GREAT upgrade to our code base as it allowed us to have ONE repository for our locations of the DOM. Before, in terms of KFS, it meant we had 93 files where the locations were stored.  In contrast, with page objects we stored them all in one place.  So, in the olden days, if a change was made on a location (as it often happens) then we would repeat the maintenance nightmare just to keep up.
   2. Solution: With the Selenium only solution we continue with the concept but instead of each of the page objects being stored in a separate files, they are integrated from the beginning. An added benefit with the new implementation we don’t need to re-compile the code whenever a location change needs to be made.
3. All scenarios run using one browser session – easier to run and faster
   1. Issue: In the old setup our tests would initialize the browser, go through the scenario and then once done bring the browser down. This would repeat for each scenario.  So the result was we had a number of false negatives if the browser failed to open or if tests started to collide during a run.
   2. Solution: In the new setup, we do all our testing in the same browser session. In this way we not only run faster but also, we save on all the false failures when the browser does not initialize or when an old browser session does not stop. There is also a safety catch that will restart a browser properly in the case of failed tests so that they don’t become stuck in the middle of a run.
4. Data driven testing – much easier to implement tests for a new project
   1. Issue: In the old system each time we needed to create a new scenario we would have to create all the files that supported the Cucumber setup like feature files, runner test files, step files and action files (if needed). Again we were strapped with creating duplicate code for each new scenario.
   2. Solution: Because the Selenium Only solution is data driven, we simply just add the data for the next scenario. As we move from project to project we have the heart of the system ready to go. All we need to do is create the page objects and fill in the data. So it’s application independent. It treats all applications alike. And for the most part there is VERY little “coding” to do to go from one application to the next.
5. Easier monitoring of the test status itself
   1. Issue: In the old system when a failure (true or false) would occur we could only “post-actively” know about it. So any solutions to a false failure would mean having to run it over again either via the script itself or by hand. This led to even more maintenance.
   2. Solution: From the beginning, we are monitoring what is happening as it is happening. So if a test fails we immediately re-run it using the same data.
6. Good for already established applications
   1. Issue: For established applications that have no automation tests, usually test automation starts out already behind
   2. Solution: Focus is on “data” driven tests; the creation and execution of tests go faster
7. Reporting is more accurate, customizable and extensive
   1. Maintenance of reporting
      1. Issue: Many times a test fails before actually executing the steps of the test. This happens, many times, because the browser does not launch. In terms of reporting it shows up as a failure. Although it is amendable by simply re-running and then reconstructing the report, it is a problem if the report was sent out to stakeholders automatically. Then an explanation needs to be sent out. More maintenance.
      2. Solution: Because the Selenium Only solution has self-monitoring capabilities a test can be re-run resulting in more accurate reporting.
   2. Reporting allows for customizations
      1. Issue: Sometimes when generating a report it is important to provide for things like snapshots or special tables. Doing so usually means doing it by hand. Again, more maintenance.
      2. Solution: Our reporting functionality allows for adding these capabilities via plugins. As an example, in our case, we take a snapshot of a test page and then compare the snapshot to the baseline. If there are errors we surround the error with red lines on the snapshot.

So we have shown that the new Selenium Only solution has answered a lot of our questions regarding easier maintenance because of we attended to the following:  one code base, use of page objects, having all the tests run in one browser session, making our tests data driven, providing for self-monitoring, finding a solution for established applications and proving for a better reporting framework.

If you have any interest in this new technology please contact us.  We’d like to share and help in any way we can.

Lastly, we want to thank Kyle Bartz who ran with these ideas and made them happen. Also, we want to thank Hoi Chiu for working with Kyle in bringing about a robust framework.

By Bob Boydston – UCI OIT Central Services Team

Tools are very important to software quality. From a software quality engineer/analyst perspective having a source-code repository service is paramount in providing a place to track software changes and ultimately the promotion of the code base from one environment to another.

What is a source code repository service?  What is the best one out there?  When you think of a source code repository version control and release management first come to mind.  But, there are other aspects of source code repositories that differentiate them.  Some are command-line based others based on web pages and then others utilize both.  Presented here are a comparison of the most used here at UCI OIT; mainly Subversion (SVN), TortoiseSVN, Git and Git with GitHub.

Apache SVN and TortoiseSVN

SVN and TortoiseSVN are great products in terms of managing repository (repo) branches.  If you are in Windows functionality is accessible via the right-click context menu.  They also have a command-line interface which is necessary for Linus/Unix/Solaris environments.  For those who are working on a Mac OS X machine there is no TortoiseSVN product. Instead, the SCPlugin can be used to access a SVN repository.

There are three features that are lacking in SVN. First, is with SVN’s dependence on a “central repository.”  If you are working remotely you cannot connect to the repository.  Second, there is no collaborative capability. So, if you wanted to fork off another repository the process is more entailed than other products.  This is due, partly, because there is no real web-based interface.   And third, it does not offer a distributed version control and code management system.

Git and GitHub

When you install GitHub you have to install Git as well, which means that you will be using both Git and GitHub. Git is an open source version control system command line tool and is used by many in the Linux/Red Hat/Unix community, as well as Windows from the DOS prompt. GitHub sits on top of Git and provides a user interface to Git. However, GitHub takes the Git functionality one step further. The “Hub” part bespeaks of collaboration. Some call it “Social Coding.”  It allows others to access your project and vice versa.  Also, it allows “forking” where one project can spawn another. GitHub is cross-platform meaning it can run on Windows, Mac and Linux/Unix/Red Hat OS’s.

So why all the commotion about GitHub?  Because it is collaborative! GitHub allows others to see your code.  As one has said, “GitHub lowers the barriers to collaboration. It’s easy to offer suggested changes to others’ code…” This is because there is a web-based interface.  In many ways, you are allowing others to review your code.  Also it allows you to fork your repository from someone else’s.

Unlike SVN, Git is not dependent on a “central repository.”  Instead, a working repository is copied to your machine. So if you are working remotely, you are working off your local repository. It enables you to work anywhere, even on a plane or a remote island.  This is called a distributed version control and code management system.

Who wins?

Because of its distributed system and possibilities of collaboration the Git/GitHub solution makes version control and release management easier and, to be honest, better in terms of making sure that the promotion of code meets modern day standards.

So what do you say?

This is a blog so please make your contribution. This is how a community of practice can find answers.

To get an understanding of all the commands/glossary found in Git and GitHub click on the following link: https://help.github.com/articles/github-glossary/