Finding the bugs in a software earlier can save time and money for a company. If there is fault slip through, then identifying the root case and doing respective changes in development can more cause time and money. Every tester, team and company targets to release a bug free product to it’s stakeholders — though it is a big challenge and developing a test strategy depends on project’s test needs and most importantly company’s quality mindset. Various testing approaches are useful for different testing levels.

In this post, I would like to propose a new term “E.A.T” — combining Exploratory and Automated Testing approaches. Here is what I think about it;

(Term “system” in this post is used alternatively for function or feature as well)

Exploratory Testing (ET)

Exploratory Testing approach is used by both new and experienced testers to identify bugs and quality improvements for a software product. New testers, in an attempt to explore more, think differently and can identify scenarios where software behaviour is unexpected. On the other hand experienced testers understand system behaviour and develop test scenarios accordingly.

Exploratory approach makes a tester to be broad minded and thinking beyond test prescriptions with freedom

ET approach offers several benefits and has been found to identify bugs hard to find with traditional test case based approaches. ET adds human element which can find various types of bugs from feature to GUI area. Testers who trust and utilizes their own instincts, knowledge, skills and experience, they use approaches like Exploratory Testing (ET). I think exploration of system and trying to find different behaviours is really important and testers do it consciously or unconsciously in other testing approaches or techniques as well.

Here is what I think when to use ET/Man approach.

1. Human interaction: ET approach is effective when testing user scenarios and user acceptance tests. GUI or features tests which are difficult to automate because of  software complexity.
2. Customer thinking: ET/Man is more closer to customer thinking, good or bad customer/users. Learn about the product and provide feedback as per customer experience.
3. Troubleshooting: The test results from AT will update you about what has failed but with exploratory approach you dig through the events and gather information to conclude the findings and develop more test scenarios.

Automated Testing (AT)

Some company’s rely mostly on Automated Testing (AT) for quality stamp and feel relaxed when they see all the green tick marks. Definitely AT can do the job in less time but there is a big danger lurking behind those green ticks, if left undiscovered. As part my daily routine, I have also been running AT several times and out of curiosity, when I explored passed test cases execution logs, I found bugs both in the AT code and the software itself as well. Sometimes AT code was outdated and was subjected to revision or update. (Sometimes, running ATs became boring) But the question is, should tester be happy to see green tick or become alert and asks — Is there something we might have missed?

How many times you have raised eyebrows over a green ticked test case?

Here are my thoughts when to use AT/Machine approach.

1. Save time and gain Speed: A faster execution of test cases saves time and money, all those day-to-day tests that are repetitive and takes longer execution time, should be automated if possible (and adds value). Example: Software Installations, loading data and configurations, backup and restore, upgrades, process fail-overs (HA) etc…
2. Reduce human error: Test scenarios for software with greater complexity are good candidates for test automation. Running several commands, loading user configurations, defining parameters and values with parallel actions are out of human capacity. There is more risk of human error than adding value to testing. Examples: Telecom, finance, aerospace and medical software.
3. Unit, component and Regression tests: Quality can be improved right from beginning of development, so if you have unit, component or regression tests in place then it can add a big value toward your quality goal. Every time there is code update, all these tests can be executed quickly and provide an early feedback over product/code quality. This will be facilitating development teams to keep an eye on code and product quality as well.
4. Load, Performance, Security, KPI etc.. tests: Automating the load, performance, security and KPI tests will make testers life easier and completes the test execution in less time. These tests require multiple actions in parallel and machines can really help to simulate a production environment. Example: Running tests with background traffic (user and data) will be more suitable for AT/machines

Exploratory and Automated Testing (E.A.T)

Now using the power of two, considering the testing needs and scenarios, let’s combine the pros of both ET and AT.

I am happy that most of the time I was able to do so. Let’s combine both approaches strategically to gain maximum benefit. The need for human mind and touch, can not be ignored, even if we have the most sophisticated and fast testing framework. There is always a value that human can add when it comes to testing.

Test Automation can save time and gives quick feedback on software quality, but don’t forget to update it.

An important point here. Both ET/Man and ATs/Machine can/should complement each by a closed feedback loop. You can use any testing method, technique or approach but key is to explore, automate and keep this cycle alive. when automating a test case, think about it’s future usage and value, return on investment perspective. If the implementation cost is high and the value you will gain is less in long run, then you need to think again and come up with other strategy that meets your need.

But you can start your exploration and see what scenarios you want to automate. As a tester, an advocate of quality, when I explore and find something interesting, I will see if it can be automated so I gain benefit of saving time and focusing my energies at other areas. When running AT, with exploratory approach in mind I can change the data attributes and observe system behaviour and then think about other scenarios to find faults. In this way ET/Man becomes the brain and AT/machine becomes the helping hand with power/speed — and together they find bugs, which must be found before customer finds it. Testers will love it because it is fun and freedom with responsibility.

Explore to automate and use automation results to explore further. Harness man and machine combination.

Once I was testing the system for performance values. I executed the automated tests to see if I can launch 3 Virtual Machines (VMs) (with all OS and software bundles loaded) within 4-6 mins. It went smooth but when I launched 8-10 VMs at the same time, it took 15+ mins to launch all, it was beyond the accepted time value. The performance values were different at different time intervals as well. This was just an exploration, upon investigation, there were certain operating system and middle ware high availability configuration parameters to be adjusted as per environment, this particular scenario was not tested for complete system earlier by development teams. After tuning the configuration timer values, the launch time for n VMs came under expected time range.

Another example, the system is performing and behaving good till 10th repetitive test but as soon as you run it for 30+, you might observe unexpected behaviours as soon system comes under stress and requires more resource to function properly. At this stage you can discover new scenarios that might be seen in production environment.

KEY:

The key is to combine both/different testing approaches to gain maximum benefit. If tester/QA person don’t find the bug the customer will and it will be late and costly for the company.  Discover beyond Green ticks with exploratory mind.

Inspirations:

 – Agile Testing

 – Testing pyramid

Handshake Image source