Test Pyramid – Base for Test Strategy

In this blog, we will understand the importance of following the Test Pyramid in all phases and aspects of Testing.

The main objective of implementing Test Automation is to shorten the feedback cycle and teams just starts automating every possible test condition irrespective of where it should be covered like Unit or Acceptance. So in the end we have so  many test cases and execution time is in hours which does not really add value but instead becomes a bottleneck. So lets understand the benefits of implementing Test Pyramid so as to cover all test conditions yet managing to execute everything in desired time.

What is Test Pyramid:

pyramid

 

 Unit Tests:

  • Why: To make sure code is developed correctly
  • What : Test Automation starts at Unit level.
  • Who : Developers & Technical Architects
  • When: Ideally before code is written. ( Failing Test case first) but at-least As soon as new code is written
  • % of total count : 75-85% of the total number of test cases
  • Where: Developer’s machine as well as the CI environment.
  • Frequency: Ideally before every commit but atleast Executed multiple times a day
  • ROI:
    • Highest. Quick to run, easy to maintain and modify.
    • Bug found by unit test case does not have to come via bug life cycle and can be fixed locally by dev.
    • Importance: Unit Tests form the bed for Testing and Test cases for any application.
  • Guidelines
    • Write unit tests for each and every test case – for models, controllers and views
    • Typical speeds should be around 5 minutes, with a maximum of 10 minutes
    • Cover maximum scenarios using “fast” test methodologies/types. For instance, write JavaScript unit tests to test client side validations, and avoid Selenium/Capybara based acceptance tests to do so

Service/Integration Tests

  • Why: To ensure communication between components are working
  • What : next level up from Unit Tests.
  • Who : Developers/Technical Architects/Testers
  • When: As soon as new API is developed or new API is used
  • % of total count : 15-5% of the total number of test cases
  • Where: Developer’s machine as well as the CI environment.
  • Frequency: Executed multiple times a day
  • ROI:
    • High. Quick to run, easy to maintain and modify.
    • bug found in this level does not involve UI, so saves a lot of cost.
    • Importance: Integration test are executed before GUI test to make sure that if its even worth executing GUI Tests.
  • Guidelines
    • Typical speeds range from 5 – 15 minutes

BDD Acceptance Tests

  • Why: To cover interactions and contracts between the system and external systems
  • What : next level up from Service Tests.
  • Who : SDET / Testers
  • When: New feature is developed and ready to test
  • % of total count : 10-5% of the total number of test cases
  • Where: Test Environment and CI.
  • Frequency: Executed multiple times a day
  • ROI:
    • Moderate to low. Slow to run, Expensive to maintain.
    • bug found in this level involve fixing the problems at all levels.
    • Scope of Fix can be in any layer of the application and also in any module
  • Importance: Acceptance tests are required to automatically check if user expectation will be met or not.
  • Guidelines
    • Keep the numbers downask yourself, if I can write a unit test for this scenario, should I write an acceptance test?
    • They run between 10 – 30 minutes.

 

Guidelines for Execution time:

Execution time for unit, integration and BDD really should depend on application to application and also from Team to team. To help decide the timeframe, ask yourselves:

  1. How frequently does your team check-in the code (remote push)? In the case of 2-3 times a day, test speeds of 10 minutes is acceptable. However if check-ins happen 20 times a day, 10 minutes is not acceptable. Making the test suite run 1 minute faster can save 1000 minutes collectively across the team.
  2. Can you run tests in parallel? One of the easiest ways to make the test suite run faster is to run them in parallel.

Guidelines for Consistency

Tests should NOT be fragile. Test should not fail without reasons. And re-running tests should not pass. Watch more closely on following the kind of automated tests,

  • Time dependent tests, these are the tests that run successfully when written and fail randomly based on time of the day or start failing after some time. Here is some help on how to write such tests
  • Test depending on external systems or services. If any kind of your test depends on services and system which are out of control from test, should be mocked.

You need to be cautious for

The inverted test automation pyramid

  • All too often, test cases that cannot be covered by developers in unit tests are directly automated on the user interface level, resulting in big sets of UI level automated tests that take eons to execute and maintain.

pyramid_inverted

  • In extreme cases, the middle layer doesn’t even exist in the overall test automation approach.

Hour Glass

  • In extreme cases, the middle layer doesn’t exist but is very thin in the overall test automation approach.

test_hour_glass

A lot of money is wasted unnecessarily on development and maintenance of the automated test cases.

My advice

  • Get familiar with unit testing, its benefits and the way the developers in your project use it. Try to understand what they test and what coverage is achieved in unit testing.
  • For those tests that are not covered by unit tests, try to find out whether the application you are testing offers an API that you can use to base your automated tests on.

I am confident that after you understand the test pyramid and strat applying it to your project then testing will be not be a bottleneck and the cost associated with overall testing of the project will go down drastically.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s