Automated Testing meetup @ Bigcommerce (David Vydra, GE Software)

Last week, Bigcommerce had the opportunity to host the Automated Testing San Francisco meetup in our office. David Vydra, Manager @ GE Software, was the speaker and the meetup was organized by Leo Cheng. David's central theme for the talk was the role of automated testing and best practices to follow for delivering quality software.

David started off the presentation with this statement: "Functional tests are the New Compiler". At Bigcommerce, where we use an agile development model and perform frequent code releases, we can relate to this statement. To ensure the best quality of software delivery, we want to catch bugs early, preferably using automated tests and the test results should be available in the shortest time possible to make deployment decisions.

A good automated testing setup should:

• Produce reliable test results
• Execute fast
• Support continuous integration(CI)

Good quality software is a direct result of testing. Automated tests should be the gatekeepers for software quality. In software paradigm, testing is frequently called as "Quality Assurance".

Where does Quality Assurance fit in?

Everybody in the product ecosystem, including developers and end users, value quality. It is most valued by the end users. And this is why Quality should be an important aspect of your deliverables. So who owns quality? Quality assurance is collectively owned by the team delivering functionality.

Quality checks for software has come a long way in the recent years. We are long past the days of purely relying on manual testing. It is not scalable for agile development teams where the focus is on shipping small feature sets and shipping often. David referred to "The Pipeline" your code goes through and becomes ready for deployment. Today, we have various tools that can help perform quality checks during the development process:

• Compilation
• Static analysis
• Unit tests
• Integration tests
• Functional or End-to-end tests
• Performance & Security testing
• Exploratory testing

With the help of repeatable and incremental builds, what David refers to as "Hermetic Build", CI systems can run all quality checks, except the Exploratory testing. Exploratory testing involves the intellectual thought process of a human to find bugs. A good example would be usability bugs.

Flaky tests - The bane of automated testing

Flaky tests(or flickering tests or flappers) tests are those which fail intermittently. Flaky tests are the ultimate bane of automated tests. Flaky tests result is loss of productivity and cause real bugs to be passed off as false negatives.
David talks about the idea of building a "flickering detector" where test is included in the CI only after it passes a 100 times. Flaky tests are not always caused by test issues, they often tend to hide a real bug.

Highlights worth calling out:

• Design test framework to allow parallel and random order of execution
• Use Docker to put your database with test data instead of creating it in the test, to help speed up tests execution
• Use mock/simulators for slow/unreliable services
• Use Domain Specific Language to test, when possible
• Use feature flags to control visibility of unfinished features
• Slower tests might indicate performance issues
• Functional tests are slow, mitigate that by batching your changes into the CI

It was a pleasure to host David at Bigcommerce, we are very thankful to him. Personally, I had a lot to learn from the presentation. I have found helpful suggestion to some of the problems we have been trying to solve at Bigcommerce.

Here's a recording of the meetup: