Scientific method in testing
begins here
Aristotle is believed to think that women had less teeth than men. At the moment we can't say for sure if he could not count or women in Greece of his time experienced problems with nutrition.
A fact.
To tell the truth best ever definition of scientific method is already provided by wikipedia and according to it
-- method should be empirical and have measurable evidence [2]
-- method should be proving or disproving a theory/assumption [3]
-- approach and analysis should be as unbiased as possible [4]
-- knowledge should be documented and sharable [4]
-- results should be reproducible [4]
All these map on what we know about testing perfectly. In fact any good testing professional has been doing this for years but did not always realise that there was a solid scientific background all the time. Let's look at each of the above points closely.
Well, being empirical. Unless you are in SQA department your job 100% emprirical anyway.
Measurability. In this case measuring is not about centimeters, but should be used in a bit wider sense, say, you do not expect your test to return some ok result, you expect it to return either specific number of lines, or no error message, or what not. But you need to be specific.
Being an applied technique testing is quite remote from most of theories and deals with more low-level thing like requirements and assumptions (derived directly from said requirements). As such your test is supposed not just to do something abstract but to either prove or disprove an assumption. Suppose that under certain circumstances some button is expected to tunr blue. Assumptions are:
-- button must look blue given preconditions are met
-- button should not look non-blue
-- etc
It is possible to talk endlessly on human biases how testers being human suffer from them severely. Here are some of them
-- fear of looking incompetent (and not reporting a potential issue)
-- fear of aggressive reactions (and not reporting a potential issue)
-- being lazy to deal with consequences (and singing it off in hope that by the moment it fires you'll be far away)
-- etc
Sharing knowledge. Ahh. Knowledge means power and unless you are lucky enough to work for a company with right corporate culture you will have to beg for it until you build your own informational capital and be able to trade. There is a lot of literature on business and functional analysis with a log of explanations of why documenting is important so I'll do it briefly here:
-- if information is documented and shared you do not have to waste anybody's time and effort to get it
-- if information is documented and shared it is less likely to be lost if key persons leave
-- if information is documented and shared it is much easier to make sure your tests are well aligned with requirements
-- etc
Reproducible results. Tiny but crucial difference between a bug and a glitch. To prove there is a bug it is important to be able to show that a very specific cause leads to a very specific result. Being inaccurate in this usually leads to non-reproducibles and I-do-not-wanna-know-it-works-on-my-machine stuff. Golden rule: make sure to reproduce it twice at least before you claim it works ok or it is a bug. As I mentioned earlier there are both false positive and false negative pitfalls.
-------------
Footnotes:
[2] "To be termed scientific, a method of inquiry must be based on empirical and measurable evidence subject to specific principles of reasoning." Source: http://en.wikipedia.org/wiki/Scientific_method
[3] "The chief characteristic which distinguishes the scientific method from other methods of acquiring knowledge is that scientists seek to let reality speak for itself, supporting a theory when a theory's predictions are confirmed and challenging a theory when its predictions prove false." Source: http://en.wikipedia.org/wiki/Scientific_method
[4] "Scientific inquiry is generally intended to be as objective as possible in order to reduce biased interpretations of results. Another basic expectation is to document, archive and share all data and methodology so they are available for careful scrutiny by other scientists, giving them the opportunity to verify results by attempting to reproduce them." Source: http://en.wikipedia.org/wiki/Scientific_method
-------------
To be continued in part 3
begins here
Aristotle is believed to think that women had less teeth than men. At the moment we can't say for sure if he could not count or women in Greece of his time experienced problems with nutrition.
A fact.
To tell the truth best ever definition of scientific method is already provided by wikipedia and according to it
-- method should be empirical and have measurable evidence [2]
-- method should be proving or disproving a theory/assumption [3]
-- approach and analysis should be as unbiased as possible [4]
-- knowledge should be documented and sharable [4]
-- results should be reproducible [4]
All these map on what we know about testing perfectly. In fact any good testing professional has been doing this for years but did not always realise that there was a solid scientific background all the time. Let's look at each of the above points closely.
Well, being empirical. Unless you are in SQA department your job 100% emprirical anyway.
Measurability. In this case measuring is not about centimeters, but should be used in a bit wider sense, say, you do not expect your test to return some ok result, you expect it to return either specific number of lines, or no error message, or what not. But you need to be specific.
Being an applied technique testing is quite remote from most of theories and deals with more low-level thing like requirements and assumptions (derived directly from said requirements). As such your test is supposed not just to do something abstract but to either prove or disprove an assumption. Suppose that under certain circumstances some button is expected to tunr blue. Assumptions are:
-- button must look blue given preconditions are met
-- button should not look non-blue
-- etc
It is possible to talk endlessly on human biases how testers being human suffer from them severely. Here are some of them
-- fear of looking incompetent (and not reporting a potential issue)
-- fear of aggressive reactions (and not reporting a potential issue)
-- being lazy to deal with consequences (and singing it off in hope that by the moment it fires you'll be far away)
-- etc
Sharing knowledge. Ahh. Knowledge means power and unless you are lucky enough to work for a company with right corporate culture you will have to beg for it until you build your own informational capital and be able to trade. There is a lot of literature on business and functional analysis with a log of explanations of why documenting is important so I'll do it briefly here:
-- if information is documented and shared you do not have to waste anybody's time and effort to get it
-- if information is documented and shared it is less likely to be lost if key persons leave
-- if information is documented and shared it is much easier to make sure your tests are well aligned with requirements
-- etc
Reproducible results. Tiny but crucial difference between a bug and a glitch. To prove there is a bug it is important to be able to show that a very specific cause leads to a very specific result. Being inaccurate in this usually leads to non-reproducibles and I-do-not-wanna-know-it-works-on-my-machine stuff. Golden rule: make sure to reproduce it twice at least before you claim it works ok or it is a bug. As I mentioned earlier there are both false positive and false negative pitfalls.
-------------
Footnotes:
[2] "To be termed scientific, a method of inquiry must be based on empirical and measurable evidence subject to specific principles of reasoning." Source: http://en.wikipedia.org/wiki/Scientific_method
[3] "The chief characteristic which distinguishes the scientific method from other methods of acquiring knowledge is that scientists seek to let reality speak for itself, supporting a theory when a theory's predictions are confirmed and challenging a theory when its predictions prove false." Source: http://en.wikipedia.org/wiki/Scientific_method
[4] "Scientific inquiry is generally intended to be as objective as possible in order to reduce biased interpretations of results. Another basic expectation is to document, archive and share all data and methodology so they are available for careful scrutiny by other scientists, giving them the opportunity to verify results by attempting to reproduce them." Source: http://en.wikipedia.org/wiki/Scientific_method
-------------
To be continued in part 3
No comments:
Post a Comment