Question

I am trying to understand the difference between automation frameworks and automation testing tools. According to Test Automation in wikipedia a tool is dependent on the environment whereas a framework provides the structure for running the tool.

However, we have White which interacts with windows applications, then Selenium which works on browsers for web based applications. We call both of these “frameworks” but they both need to be run using a testing framework like NUnit or JUnit. So I feel they should be called tools because of this.

How exactly do we differentiate between an automation testing tool and an automation framework?

Answer 1

I now have better working experience on this and feel I can answer this myself. Basically a framework:

• provides a list of common library of functions,
• generates logs,
• provides the use of config files to make typical setting changes,
• and separates user test data from the code so that we are not
  hardcoding the actual test code,
• serves as a layer between the actual testing code and the raw language code. This allows makes maintaining the actual testing code much easier and simpler so that the engineer can make changes/fixes much more efficiently,
• separates the raw code from the testing code so that if general changes occur in the product then only selected functions can be modified instead of having it affect all the tests.

Now, in my original query, White was indeed a framework.

1. The reason is that if we try to “run” White it won’t do anything by itself. It is just providing us some functions that we can call and get the job done.
2. I mentioned that

but they both need to be run using a testing framework

thus implying that White needs to use NUnit or it is useless. I was wrong. White need NOT use a testing framework necessarily. We can also write basic procedural code that makes use of only white and nothing else and still it could do some meaninful task.

The testing framework that we added (NUnit) is only used to accomplish another separate goal (to keep track of different parts of the code (called tests) and to generate a final report of which tests passed/failed.

3. I also mentioned that

but they both need to be run using a testing framework like NUnit or
JUnit. So I feel they should be called tools because of this.

Here the misconception is the idea that “if it’s using a framework then it must be a tool”. The correct idea is that a framework can allow us to do some task and it can still use another framework to allow us to do even more tasks. For example, I’m using White framework to automate the windows dialogs. Now I add Log4j framework to add functionality for my own logs. Next I add a unit testing framework so that we can run the code (which uses both the previous frameworks) in the form of tests and generate results. Next, I add PNUnit framework and can now run the tests in parallel. See the idea?

Now the tools – these are nothing but the utilities that we are using while using the actual code – the code that will make use of all the above frameworks to do some actual worthwhile task. It could be the IDE or the platform which allows executing the code.