Multi-skilled testing is coming our way...

CAT is using the following scenes just to represent thoughts in different way.

Scene 1: IT guru talk
4 gurus were sitting on a table with beer can. They were debating about some article they read on future of IT industry.
CAT sitting on nearby table overheard few dialogues -
..."In few years time three type of roles will survive: 
DT - Developer with (must to have)good testing background.
ArcMan - An Architect with basic understanding of management principles of cost, time, quality.
FT - Techical expert & Domain expert together."
...."There are going to be few players with the basic components hosted in cloud. Major development is going to be around binding such components from different vendors together."
...."Dedicated multiple roles might turn out as an overhead".

Scene 2: Staffing team discussing with CxO
Staffing: :It is tough to get NFR testing specialist in our cost & available time
CxO: Why do we hunt people everytime? Why not many of existing professionals trained so far to fulfill these special testing needs?
Training: It is tough to get people for training & grooming - from their existing schedule.
CxO: Is it unfair to expect multiple skills from experienced testing professionals? Are they going to work with MS office as core skillset?
All testing professionals should know at least one skill apart from functional testing & automation. Both are must for anyone with more than 5 years in this industry.
Additionally they can choose from one of the following:
Database
Security
Performance & other falling in this bucket (Load, Stress, Availability, etc).
Accessibility
Cross-browser

Scene 3: Agile team discussion - probably a daily meeting
Few dialogues from team meetings -
..."In this sprint we need to check the single user response & 10 user response for 5 shortlisted transactions"
...."I have never done such thing before & I do not have enough scripting skill as well"
...."Performance expert will guide and mentor for sometime"
...."We will help you with basic understanding of codebase & build & initial set-up in this sprint"
...."your progress on performance testing will improve with product's performance :)"
..."be prepared as approx 6 sprints from now security is going to come"

And many more such scenes... let us eyes open & read the situation. It is evident that if we(testing professionals) are in IT industry there is no escape from technology.

One of the following should be equally good along with both manual & autmated functional testing -
1. Database knowledge with some development ability (at least 1)
2. Scripting language with ability to write unit test methods (at least 1)
3. One of the non functional testing skill-set
4. Open Source - Most important. It appears that this is surely going to be a preferred choice in future.
Paid tool is cost to projects when there are open source tools -
Open source does it with bit more pain - can not be the reason to pay for tool, unless that paid tool is the only possible "technical" solution.

It is tough ask - but then changing scenario demands it, why wait for the last minute? Let us take first step at least!

Context Specific Defect Prediction

Defect Prediction is required.

It is easy to implement defect prediction model within context of your project. Although we are testing professionals and not statisticians, some common sense can help us to build a 'just enough' defect prediction model for our project.

CAT thinks that the following parameters are key contributors to the defect prediction model:
1. Nature of requirements
2. Defect injection factor for project (which is linked with development team)
3. Defect detection factor for project (which is linked with us)
4. Size under consideration e.g. 1 use case, 1FP, 1 Test Story Point, etc.

(you might wonder what is test story point - CAT will share its thought in upcoming posts on this).

A simple formula will help you achieve the defect prediction. Follow these steps:
1. Measure defect density (defects / size) from the "applicable" historic data.
2. Build Three level scale based on your gut feel (such that sum of all coefficients is 1)
e.g. Nature of requirements
Simple - 0.2, Medium - 0.3,Complex - 0.5
3. Measure the size for the project under consideration.
This size and the one in defect prediction should match.
e.g. both places should talk about use case or both places should talk about test story point.
I can not predict defects with density in FP (defects per FP) & current size in use case - unless I know relationship in FP and use case "for my project". Equations from any other source will not work as is unless it is "calibrated" for my project.
4. use simple formula
Predicted Defects = defect density * size
5. add our key factors to it
Enhanced Predicted Defects = (defect density) * (size)*(defect injection factor)*(nature of requirements)*(defect detection factor)
6. In the first implementation consider all factors at "middle" level.
This brings question to all intelligent minds - how do I equate both sides of equation?
We introduce additional factor for this.
7. New equation looks like this
Enhanced Predicted Defects = (defect density) * (size)*(defect injection factor)*(nature of requirements)*(defect detection factor) *(correction factor)
The first version will have correction factor = the value which equates both sides of equation.
8. Thereon conduct variance analysis after each release / cycle and based on entire team's understanding change rating for injection, requirement & detection.
Revisit the correction factor to equate both sides.
Applying correction based on variance is the best possible way to mature any home-grown model.
9. In next release / cycle - injection & detection factor should not change unless there is huge change in team composition / technology than previous release.
We can change only requirement factor.
10. Repeat this in logical cycles - after roughly Four rounds, one should get a model in good shape.

It is hard to find equation that gives exact number of defects before we start testing - we do not have Harry Potter's magic wand. It is "Prediction" model not a "Defect Detection Spell"!

Paid solutions, complex equations or tools may provide better results - how much better is the key driver to identify ROI & thus should be the deciding factor for choosing these tools / reports / equations.

Common sense says that next step is to include Five level scale instead of three to reduce inaccuracy band further.
When we are in execution - defect density & size should be in context of the current cycle. Historical data is useful only in the beginning.

There are much advance ways of predicting defects using a home-grown model. This was just one of those which hopefully should ignite the defect prediction culture in testing world.

Testing in agile projects

This topic is so huge that there will be series of posts to cover some aspects comprehensively.

I will begin with testing as a task in agile projects.

Agile projects function such that the agile manifesto is followed. The manifesto indicates that there is value in process, contracts, planning, etc. however, it advocates people, team and core work (of building product) over these.
People sometime do take undue benefit of this and call process-less or chaotic way of working as "agile".
It is recommended that novice reads the myths along with the manifesto.
Few links to understand myths -
http://www.theappgap.com/exploring-ten-myths-about-agile-development.html
http://stackoverflow.com/questions/1871110/agile-myths-and-misconceptions

Naturally testing as a task - "should retain" its basics even while working on agile projects. Writing test cases, reporting defects, conducting RCA, Defect Prevention, metrics, etc have its value addition in agile projects. The difference is just enough & context specific application of these concepts. It is the 'team' that decides what is needed & not a uniform governing process document or a manager / testing head like authority.

One needs to understand about 2-3 flavors of agile. To begin with scrum, XP, agileRUP, would be good enough in my opinion. The purpose of understanding these is to get to know the 'culture & environment' when a particular approach is followed. The culture & environment plays key role to change the style of testing in agile projects.

In upcoming series I will cover more about -

• Test Story Point - how story point can be made a test story point.
• Useful web sites, books, etc on agile testing.
• Testing metrics in agile projects.
• Some open source tools that might be useful.

I will expand this as I move further...

To end this today, I will say that - testing spirit remains unhurt in agile; it is just expressed in different way.

Minimum Testing

Recently, I had  discussion on identifying the Minimum testing required for a particular product. The historical data suggested that the the development process is matured. This was one of the reason the key stakeholders believed that the 'exhaustive testing' was not required & 'just enough' or 'minimum testing' was required.
While performing minimum testing, it was expected to measure the testing effectiveness.

Certainly The first step is to validate the expectation & if really there is need to perform 'just enough' testing then the following approach may be considered.

In my opinion, the concept of testing quadrants is useful to derive the solution.

The quadrant not just arrange the cases based on the testing types but it also recommend how and when it should be executed.

Let us assume that Q1 cases are already satisfying the quality criteria. 

Naturally,  the desired focus is on Q2, Q3 & Q4 cases. Even in these focused quadrants, all do not need equal priority. The analysis of historical defects & pain areas would suggest which among these three need higher attention.  The testing activities for the quadrant with higher focus need to start early - during the requirement stage. (Here clearly the benefit would come from moving the testing upstream).

Q2 & Q3 both target functionality; however Q2 focuses on positive, happy path & end to end flows - especially the explicit requirements. Q3 focuses on exploratory, negative test cases - the implicit requirements. 

During functional testing team should work with the agreed priority between Q2 & Q3 – (e.g. if explicit functionality is properly covered in unit testing, then higher attention on Q3 is expected).

Measuring the Effectiveness:

The expectation of “just enough” testing is driving the above approach. So the underlying assumption is that no exhaustive testing is required because of overall satisfactory & quality output from upstream phases.

While building "just enough" testing approach, the trade-off between time & (coverage + documentation) is achieved. Test execution is thus dependent on 'managed exploratory testing' (which is identified based on the nature of defects observed during earlier phases & the current phase).

The conventional mechanism (of measuring defect mapped to test cases, defect leakage, defect rejection, etc) should not be treated as the only indication of testing effectiveness.
Along with these measures, the time & testing thoroughness planned for each module / functionality also play an important role. Dependency from earlier phase/s in terms of vagueness, defects detected & leaked is also going to drive effectiveness of the next phase.

So a mechanism would be required -
a. to map the conventional measures with time & testing coverage allocated to each quadrant
b. to link the measures (like defect leakage) from all quadrants

we always UNDERSTAND REQUIREMENTS

I strongly believe that a testing professional keeps understanding the requirements - right from the proposal stage till the last activity in their assignment. I will share my opinion on "on the field" techniques / methods for this - in a separate blog.

Here I am showcasing how a simple "list" can do a wonderful work of
1. understanding "learning pattern" and its link with domain & technology.
2. understand the impact on quality.
(Above should be done by people with "technical" or "core testing" inclination.)
3. measuring effectiveness of KT & its time cost impact (this should be for managers / aspiring managers).

The dedicated phase most probably revolves around understanding the document (any form of requirement document) sharing the vision, expectations, etc from the products (both documented and not documented).

Post this phase - every week list down the "new" requirements (everyone has to maintain it individually).

• New - could be one that is present (in direct or implicit form) but not understood
• New - also could be one that is not specified

This difference should be understood well by all.
In free time / on weekly basis / with suitable frequency consolidate this list to arrive at "team's view" on new requirements.

Expand this list to create a table  (note - this table is for those requirements where team agrees that the requirement is new / "discovery") -

• domain
• technology
• # cases added / deleted / affected by the "discovery"
• possible impact on design (won't work for functional testing team without any visibility to development) 
• possible impact on data model (won't work for functional testing team without any visibility to development) 
• possible impact on code (won't work for functional testing team without any visibility to development) 
• projected impact on NFRs
• time (roughly) spent by whole team to grasp it (do not add a tracker for this! already you and your team are in enough trouble)

Be open to accept all comments - as understanding a requirements is "highly" subjective. BAs / SMEs / Client might have conveyed this in different way, other team members (including design / development team) might have noticed it. Definitely there will be few, which all "agree" as valid new requirements.

Whenever / if CRs or other commercial mechanism is raised for managing these - your table would help.

Technically - this table would help you to understand a pattern - that team is likely to miss requirements of a particular nature. If all projects provide 1/2 such patterns -
1. your organization would get a wonderful trend for a KT effectiveness module / program.
2. further analysis on domain & technology would provide valuable inputs to BAs, design & development team.

In my opinion, the management purpose of # of requirements, cost, impact etc should be separated from the team - this needs to be maintained and tracked by managers themselves.

So team should not relate # of new requirements to measure KT effectiveness, rather team should derive new patterns, solutions, testing techniques suitable to these "discovery" requirements - its only then your core testing knowledge would go up.

Opening!

About Me:
I am not one of those who have chosen & achieved software testing as their as career - I landed here. So far, I do not have any intention of being "manager" / "account head" / "delivery head" and so on - who mange, deliver, enrich the respective area in software testing engagements.

I am proud of my career. By now, for sure(!) I have understood that it is a technically challenging field. This field is far beyond spreadsheet or document based test cases.

I am aiming to reach a stage where I start providing a "sensible" and a "practical" software testing solution based on all that I know about software testing.

Reaching this stage in my opinion is one of the toughest task in this field.

Why I want to write the blog:
I am still young in this field to suggest something to someone & there is nothing great in "About Me" section that should make me write a blog. Well - still ironically I am going to write on Software Testing in this blog.
I am completely aware that there are already superstars / Tendulkar's of this field:- James Bach, Lisa Crispin, Cem Kaner and many more. But, I am not blogging to fill the gap of superstars / to be a superstar.


I am just trying to post -
1. what I feel & what I think others might be feeling about this field.
2. share the technical problems that I have faced (barring and masking everything that I should).
3. solutions to these problems whichever I know & that I can provide on a blog.

I have not thought about anything else that can be added here :) - probably time would suggest.