09/16/2021

Proof of Concept in Automation Testing

SHARE:

  • Linkedin Logo
  • Twitter Logo
  • Facebook Logo
  • Mail Logo

A proof of concept in automation testing requires preparation and a correct implementation guided by the best practices and experiences coming from the field.

If implemented successfully, it can drive the transformation towards automation with excellent results.

Prueba de concepto testing en monitor

What is a Proof of Concept?

A Proof of Concept (POC) is a technical term used in various industries to prove that an idea works in real-world environments. A POC is crucial in the decision-making process because its results can identify potential issues before they happen.

For example, imagine that you are asking for investments for a potential business idea. For investors, the idea and your business explanation might not be enough to prove that it is a profitable business. It is here where a POC is necessary to showcase the action plan working on a smaller-scale environment.

In automated testing, a POC is generally carried out when you are trying to adopt automated tests using a new tool. As a first stage in this transformation towards automation, the POC makes it possible to evaluate alternatives and results in a project with a limited scope.

Interested in carrying out an Automated Testing Proof of Concept? We can help you out!

Why is a Proof of Concept necessary in Automation Testing?

A well-executed POC makes it possible to evaluate different hypotheses and to validate some situations in a controlled environment.

Some of the examples include:

  • To identify the best tool for the type of application under test and the project objectives.
  • To understand the difference between manual and automated tests with regards to the results and the quality of the tests.
  • To understand what can and what cannot be tested automatically.
  • To fully grasp how much can be saved by using automated tests when compared to manual tests.
  • To estimate the long-term return on investment (ROI) of using automated tests.
  • To identify hidden problems and their impact on the expected results.

Implementing an automation solution implies a considerable amount of time and resources. Before embarking on this process, carrying out a POC can help validate the plan using a smaller amount of time with far less effort.

How can a Proof of Concept be implemented in Automated Testing?

Steps Proof of Concept in Automation Testing

1. Define the goals of the POC

First of all, it is important to define the goals of the POC; that is to say, what you want to accomplish when the proof of concept has been completed.

For this purpose, it is recommended to establish goals that can be measured and to understand when they have been met and when they have not. It is a good practice to indicate the expected results because it provides much needed clarity.

It is also possible to include the hypotheses you want to validate with the POC in order to gather the information necessary in responding to them.

These are some examples of goals and hypotheses:

  • To define and implement an automation framework.
  • To evaluate the technical feasibility of automation tests using a specific technological stack and architecture.
  • To determine the time saved by executing automated tests instead of manual ones.
  • To evaluate the execution capacities of the tests in different browsers.
  • To identify whether a specific type of error can be identified with automated tests.

You need to be able to compare the results of a POC against the current status of the tests. Knowing the current baselines will help you with the comparison later on.

2. Determine the scope of the POC

It is not necessary to cover all test cases in a POC, you must choose the most representative and important ones. These can include some of the key functionalities or the ones most used by end users.

The list provided below includes some of the criteria which can help with this selection:

  • Project duration: some of the best candidates usually include long-term projects or products which will need to be maintained over time.
  • Duration of the regression tests: if a considerable amount of time needs to be dedicated to regression testing during each testing cycle, the automation of tests can help reduce these times and improve coverage.
  • System stability: it is necessary to find a balance in this regard. The system must be stable; it is not recommended to automate tests for the first released version, but it is also ill-advised to wait for 100% stability, since this usually does not happen in reality.
  • System complexity: the complexity of the system and how much it can be tested is an additional factor that must be taken into consideration. For more complex systems, the automation of tests can require a higher effort, so it is necessary to carry out a detailed analysis of whether better results will be achieved with automated or with manual tests.

3. Selecting the tools

As part of the POC, a decision will need to be made regarding the automation tools to be used. In some cases, there might only be one tool, but it is also possible to evaluate several tools and compare the results.

These are some of the key factors that make it possible to determine which the most appropriate tools are:

  • Skills of the team using the tools: given that automated tests require more technical knowledge than manual ones, there are a lot of automation tools that require for testers to have programming skills in order to use them. There are others which make it possible for testers to use them without needing to code, and others which have a combination of both.
  • Technical skills and compatibility with the technology stack: there are tools which support different platforms and technologies, but some are designed to be used with specific ones.
  • Key functionalities and integration: the specific requirements that the tool must have. For example, the ability to generate reports, continuous integration, and a collaboration environment, among others.
  • Maintenance and reutilization: the tool’s ability to reduce maintenance efforts and to reuse tests. In this regard, some of the current tools have more advanced techniques which recognize changes in the applications and automatically adjust scripts.

Licensing type and budget: there are different licensing types, even open-source and free tools. It is important to evaluate the type of support offered in each case, and how much customization and extension is included with a license.

4. Automating tests

The automation stage is where test scripts are implemented in accordance with the above definitions.

Here are some recommendations for this stage:

  • Try at least one test case that you know will fail. This will make it possible for you to see how automated tests behave when a test fails because of a defect in the software. A good tool should be able to identify the error.
  • Document any detected limitations: having a record of any problems or limitations you have detected is a good practice to have information for the evaluation stage.
  • Apply changes to the system: it is ideal to have the chance to implement a system change in order to evaluate the test maintenance capacity.

5. Evaluate and present results

The final stage of a POC consists of evaluating results and presenting them to the stakeholders.

Among the main aspects to be evaluated and presented, you will need to consider:

Fulfillment of the goals

The first and most important task is to compare the goals and hypotheses of your POC, offering unbiased information regarding the results obtained.

Evaluation regarding the quality of the tests

By comparing manual and automated tests, you will need to show that there is no degradation in the quality of the automated tests.

Indicate which areas you can and cannot automate

Provide clarity regarding the areas or functionalities that can be automated and those that cannot be automated, either due to system or tool limitations.

Compare the tools used

If more than one tool was selected, provide a comparison indicating the advantages and disadvantages of each tool in addition to the functionalities they offer.

Evaluate timeframes for future estimations

Breaking down the time required by each testing activity offers information that is very valuable when estimating larger-scale projects. Ideally, you should be able to calculate the automation timeframes for a simple, medium, or complex functionality

Provide information regarding the ROI

Taking into account the results obtained and the estimations that were done, it is possible to determine the return on investment (ROI). This can be expressed in the amount of testing cycles it takes for automated tests to be profitable or in the overall summary of a project.

POC Results

After carrying out a POC, there are three possible scenarios that will need to be considered:

  1. The goals are met. With this result, the conclusion is that automated tests are beneficial and can be accomplished with the selected tools.There is no reason why a test automation project with a larger scope should fail.
  2. The goals are partially met and/or there are some difficulties. This situation implies that you were not able to achieve the best results for various reasons. It is recommended to go over the goals and requirements for the project, because they might have been too ambitious for a proof of concept. If there are any considerable technical limitations, make those explicit to evaluate any possible alternatives.
  3. The goals were not met. This is the worst-case scenario. If it happens, it is better for it to happen sooner rather than later. We suggest canceling the POC and approaching it again when the scenario for automated testing is more favourable. 

For the first scenario, the POC indicated that it is viable to move forward with automated tests and this has been approved by the stakeholders.

The next step would then be to work on a larger-scale project. The characteristics of this test automation project would have been validated during the POC.

Conclusions

A proof of concept in automation testing is a way to start these types of tests on a smaller scale with the lowest possible level of risk.

A well-executed POC makes it possible to evaluate different hypotheses and to validate different situations in a controlled environment.

In order to carry out a POC, you will need to follow the process detailed above to guarantee its success. The main stages are: defining the objectives, establishing the scope, selecting the tools, automating the tests and evaluating the results.

The results of a POC allow for well-informed decisions to be made in order to continue along the path of safe automated testing.

Interested in carrying out an Automated Testing Proof of Concept? We can help you out!