This nice song by R. Kelly is one of my favourite song
I believe I can fly...I believe I can touch the sky.
I think about it every night and day..
spread my wings and fly away...
Listening songs is one of my hobby and good thing is that you can listen to the music while doing any other stuff. right? (Gyming...walking...reading... relaxing...you name it. BTW I'm listening Waka Waka (This time from Africa) from Shakira right now, while I write this article :) ). Ever since I heard this song, I had felt that we should believe in ourselves, our thoughts and follow our dreams. If you can believe in something, you can definitely get it. You just need to believe in you..
When I started my career, as many of software graduates does, thought programming is the only elite job in software industry, but when I got experience, got interactions with several customers, understood importance of quality and understood pros and cons of programming (and testing as well), I realized that I'm more inclined for a testing role and I enjoyed it more than anything else:
#1 The scope for testing is very dynamic, technology independent, platform independent and that means you can try much more
#2 If you still enjoy programming, you can do coding, create tools that may help you do better job in testing, work on automation and still enjoy testing.
#3 Testing is a very analytical job that I always liked this since my college times
#4 Testing requires you to plan and execute tests tactically (using methods and techniques) and requires quizzing yourself all the time that makes it challenging and fun at the same time
#5 I wanted to be associated with Quality somehow, as I got more interested in quality field, after working in various areas QC, SQA, SEPG groups, part of CMM assessment team, Six sigma etc and I found testing as most interesting among all of these
#6 Every day comes pretty interesting for me while designing, defining and doing testing on different platforms, technologies and projects and I enjoy doing this all day long, every day
I believed in me and I do not regret a bit of my decision of selecting Testing as my career. I would convey this message to anyone who is working as tester and want to consider testing as their future career prospects. Testing as a career has way to go in future, just believe in yourself and go ahead with it...
I strongly believe that we should identify our interest areas and do something around the same area. You can deliver your best if you enjoy it. If you can believe in yourself, you can definitely fly and yes, sky is the limit.
And like Kelly sings...Hey, cause I believe in me...
Tuesday, June 29, 2010
Thursday, June 24, 2010
Prevention is a Cure - A Quality Perspective
For any product (and project) based enterprise software solution organizations, a lot of time is spent on Quality Inspection (Quality Control e.g. testing) and Quality Correction (e.g. debugging and fixing issues). The cost of these two areas can be moderate to huge, depending upon the size, diversity and usage of software applications in the respective business industry.
If we need to reduce this cost of overall quality, one of the better measures is by reducing and minimizing the total cost incurred on Quality Inspection and Quality Correction. With greater focus on Quality Prevention we can reduce the cost incurred on the overall product development and maintenance lifecycle and can produce increased level of quality.
Lets understand this with an example given below:
Cost of overall Quality = Cost of Quality Prevention + (Cost of Quality Inspection + Cost of Quality Correction)
With every small unit cost increase in Prevention method, we can reduce the cost of Quality Inspection and Correction to significant levels. In other words we can say that Total Cost of Quality Prevention is inversely proportionate to Total Cost of Quality Inspection plus Total Cost of Quality Correction. In other words:
Cost of Quality Prevention α 1 / (Cost of Quality Inspection + Cost of Quality Correction)
Prevention helps in reducing the probability of finding defect in production and enables some of the following benefits for any organization and not limiting to:
#1 Cost of prevention is much lesser then Cost of Correction as the defect is fixed at the initial phase itself
#2 Defects are found and fixed in earlier phases of development cycles and enables high quality products
#3 Cost of rework is reduced to a greater extent as number of defects is reduced to significant levels
#4 Product companies can focus on implementing more new requirements and innovations in product rather than focusing energy on maintenance, hot fixes and rework. With good quality there are always positive responses from customers who are happy and can be delighted as a result in longer run
#5 Happy and delighted customers provides return business and also are good references in market place resulting in more business opportunities to an organization
Paradigm shift
In software development life cycle, if we put more focus, energy and time on Quality Prevention area then it will significantly reduce the Total Cost of Inspection and Correction. There are few measures that we can take to make this happen. (Example below)
Success Criteria
Though prevention is a good method to improve overall benefits to organization, there are important areas that should be in place to enable the overall success in a longer run:
#1 Commitment of Management – Management should be committed to this goal. Management team should also provide their time to review the progress and to manage any identified risk. Without commitment from management, it is not possible to improve quality through Prevention methods as it may require significant level of investments initially.
#2 Support of Management – Quality focus should be followed in a top to down approach and must be supported by management on regular basis. This support should be extended to the team so that they are motivated and get adequate resources and tools to perform their work.
#3 Trainings – Everyone in the team should be trained adequately on the required processes. As the change may require significant alteration, in the way people work. Training needs should be evaluated frequently to help team understand on usage of various processes, tools and techniques.
#4 Attitude and Discipline – Everyone in the team must understand and agree to the common goal and should be motivated to carry the same level of standards. After all it’s the attitude and discipline that makes any improvement process successful.
#5 Continuous improvements – Prevention is part of continuous improvement process. It should be frequently evaluated by the teams and any required change should be adopted. Though change is good to follow but one also needs to be careful and should not apply changes too frequently. This may divert and distract the focus of the teams. It’s all about striking the right balance.
#6 Feedback from Customers – Though it is good to learn from internal processes and procedures, it is equally important to collect frequent feedbacks from customer. Also analysis of all defects reported by customers can be helpful in providing a good direction on where we should focus for improvements. Most of the successful software product organizations enables customer to provide their feedbacks through various forums, regular interaction sessions, user groups meetings, seminar, webinar etc.
If we need to reduce this cost of overall quality, one of the better measures is by reducing and minimizing the total cost incurred on Quality Inspection and Quality Correction. With greater focus on Quality Prevention we can reduce the cost incurred on the overall product development and maintenance lifecycle and can produce increased level of quality.
Lets understand this with an example given below:
Cost of overall Quality = Cost of Quality Prevention + (Cost of Quality Inspection + Cost of Quality Correction)
With every small unit cost increase in Prevention method, we can reduce the cost of Quality Inspection and Correction to significant levels. In other words we can say that Total Cost of Quality Prevention is inversely proportionate to Total Cost of Quality Inspection plus Total Cost of Quality Correction. In other words:
Cost of Quality Prevention α 1 / (Cost of Quality Inspection + Cost of Quality Correction)
Prevention helps in reducing the probability of finding defect in production and enables some of the following benefits for any organization and not limiting to:
#1 Cost of prevention is much lesser then Cost of Correction as the defect is fixed at the initial phase itself
#2 Defects are found and fixed in earlier phases of development cycles and enables high quality products
#3 Cost of rework is reduced to a greater extent as number of defects is reduced to significant levels
#4 Product companies can focus on implementing more new requirements and innovations in product rather than focusing energy on maintenance, hot fixes and rework. With good quality there are always positive responses from customers who are happy and can be delighted as a result in longer run
#5 Happy and delighted customers provides return business and also are good references in market place resulting in more business opportunities to an organization
Paradigm shift
In software development life cycle, if we put more focus, energy and time on Quality Prevention area then it will significantly reduce the Total Cost of Inspection and Correction. There are few measures that we can take to make this happen. (Example below)
Investments (Preventions) | Benefits |
Requirements 1. Detailed Requirements with use cases and case studies 2. Better and focused reviews 3. Involvements of customers | 1. Lesser time in understanding the requirements 2. Lesser time in architecture and design |
Design 1. Detail level of architecture and design 2. Proof of Concepts (POC) 3. Multiple design considerations 4. Detail level of design reviews | 1. Lesser time to complete coding 2. Better reuse of code and integration3. More option to build better designs 4. Higher scalability of design |
Coding 1. Coding standards and guidelines 2. Reusable frameworks 3. Detailed Unit test strategy 4. Code Reviews | 1. Less time to reuse 2. Person independent and lesser dependencies 3. Less time to complete Unit testing 4. Less time to complete Integration testing |
Testing 1. Detailed level of test strategy, test scenario 2. Usage of testing techniques 3. Detailed test data preparation 4. Detailed coverage of Regression test | 1. Less time to complete Functional testing 2. Less time to complete System testing 3. Increased ability to find defects before customer release |
Test Automation 1. Automation of repeated tasks 2. Automation of high priority modules | 1. Less time to complete tests |
Analysis of Production Defects 1. Analysis of Key trends in defects 2. Analysis of gaps in regression scripts 3. Improvements in regression test 4. Improvement in other development processes | 1. Timely fixing of common issues 2. Lesser time to complete Regression testing 3. Reduction in number of defects in future product releases |
Though prevention is a good method to improve overall benefits to organization, there are important areas that should be in place to enable the overall success in a longer run:
#1 Commitment of Management – Management should be committed to this goal. Management team should also provide their time to review the progress and to manage any identified risk. Without commitment from management, it is not possible to improve quality through Prevention methods as it may require significant level of investments initially.
#2 Support of Management – Quality focus should be followed in a top to down approach and must be supported by management on regular basis. This support should be extended to the team so that they are motivated and get adequate resources and tools to perform their work.
#3 Trainings – Everyone in the team should be trained adequately on the required processes. As the change may require significant alteration, in the way people work. Training needs should be evaluated frequently to help team understand on usage of various processes, tools and techniques.
#4 Attitude and Discipline – Everyone in the team must understand and agree to the common goal and should be motivated to carry the same level of standards. After all it’s the attitude and discipline that makes any improvement process successful.
#5 Continuous improvements – Prevention is part of continuous improvement process. It should be frequently evaluated by the teams and any required change should be adopted. Though change is good to follow but one also needs to be careful and should not apply changes too frequently. This may divert and distract the focus of the teams. It’s all about striking the right balance.
#6 Feedback from Customers – Though it is good to learn from internal processes and procedures, it is equally important to collect frequent feedbacks from customer. Also analysis of all defects reported by customers can be helpful in providing a good direction on where we should focus for improvements. Most of the successful software product organizations enables customer to provide their feedbacks through various forums, regular interaction sessions, user groups meetings, seminar, webinar etc.
Tuesday, June 22, 2010
How much quality is good enough?
Imagine if somebody says 'The quality of a product is 99% defect free'. Seems like WOW!! isn't it?
Think again. In term of deviation this (99% accuracy) stands at around 4 Sigma which means around 6000 defects per million opportunities. Now consider this:
3 Sigma is 93.3% accuracy - 66807 defects per million opportunities
4 sigma is 99.38% accuracy - 6210 defects per million opportunities
5 sigma is 99.97% accuracy - 233 defects per million opportunities
6 sigma is 99.99966% accuracy - 3.4 defects per million opportunities
7 sigma is 99.9999981% accuracy - 0.019 defects per million opportunities
Now let's compare some data:
Amazingly, mumbai dabbawalas stands at a whopping 7+ sigma (only 1 defect in 6 Million opportunities). This is a really amazing, considering that most of the workers (5000 workers and 170 thousand lunch boxes a day) are uneducated and uses only a color code system to identify lunch boxes. Who says to be high quality processes it needs to be complex. Instead it just needed to be simple and effective!! Isn't it?
Currently we do not see many IT organizations implementing Six Sigma level processes and only very few organizations implementing this practice. In future we might see such level of quality implemented in IT projects but that may be some time, from now. Many organizations desist from this practice as short term investment might be high and returns are primarily long term only.
BTW, you can also get sigma values for some of the things around you e.g.: 'Number of defects per million line of code' or 'Number of defect you found per thousand test steps' and check the sigma value for the same...Have fun.
Think again. In term of deviation this (99% accuracy) stands at around 4 Sigma which means around 6000 defects per million opportunities. Now consider this:
3 Sigma is 93.3% accuracy - 66807 defects per million opportunities
4 sigma is 99.38% accuracy - 6210 defects per million opportunities
5 sigma is 99.97% accuracy - 233 defects per million opportunities
6 sigma is 99.99966% accuracy - 3.4 defects per million opportunities
7 sigma is 99.9999981% accuracy - 0.019 defects per million opportunities
Now let's compare some data:
- If we send 3 Million letters to the users that means
- With 4 sigma we will lose 3000 letters
- With 6 sigma we will lose only 1
- For every week of TV broadcasting per channel
- With 4 sigma 1.7 hours of dead air broadcast
- With 6 sigma 1.8 seconds of dead air broadcast
- Out of every 500,000 computer restarts
- With 4 sigma around 4000 system crashes
- With 6 sigma Less than 2 system crashes
- Total flight canceled in US every 3 weeks
- With 3 sigma 964 flights cancellations in a day
- With 4 sigma 30 flights cancellations
- With 6 sigma 1 flight cancellations
Amazingly, mumbai dabbawalas stands at a whopping 7+ sigma (only 1 defect in 6 Million opportunities). This is a really amazing, considering that most of the workers (5000 workers and 170 thousand lunch boxes a day) are uneducated and uses only a color code system to identify lunch boxes. Who says to be high quality processes it needs to be complex. Instead it just needed to be simple and effective!! Isn't it?
Currently we do not see many IT organizations implementing Six Sigma level processes and only very few organizations implementing this practice. In future we might see such level of quality implemented in IT projects but that may be some time, from now. Many organizations desist from this practice as short term investment might be high and returns are primarily long term only.
BTW, you can also get sigma values for some of the things around you e.g.: 'Number of defects per million line of code' or 'Number of defect you found per thousand test steps' and check the sigma value for the same...Have fun.
Friday, June 18, 2010
What are the effective (and efficient) ways to acquire vast knowledge in a particular domain?
One of my friend, Inder Pal, posted a very good question on how to acquire knowledge in a particular domain. This is very important aspect for a tester as vast knowledge on a particular domain gives a big edge to the tester. This will help you become Subject Matter Expert (SME) within your organization and increases your overall credibility.
Here are my thoughts on this:
1. Getting involved in Business process testing - Getting involved during business process test helps understand the way business is done and is a effective way to learn a particular domain
2. Attending workshops with the customers and consulting groups - Attending workshops with the customers help understand various key configurations required to run the business. Given an opportunity this will help any tester. But these opportunities are not available to everyone and those who gets this opportunity should use to the fullest.
3. Participation in Alpha, Beta and UAT with customers - What else can give you a better exposure than working and participating in the testing done by any customer? You should observe and learn the key tests customers are exercising. Also try if you can get these acceptance scripts that you can use to improve your test repository.
4. Understanding of regulatory guideline (if existing on a particular domain) - If there are any regulations application on your domain and product then it is a must for a tester. The severity of any bug from a regulatory perspective will be much higher and it can have big impact on your organization and your customer business. Due attention should be given to learn regulations and guidelines. Any product that have good compliances to the regulatory processes get higher credibility in the industry.
5. Learning by doing specific degree, course, training, seminar, certifications etc - Check if there is any degree, course, training or seminar that can help you understand the domain well. Also you can do certain certifications if applicable to your domain. At the same time you need to research on the credibility of particular course and it is not effective and recognized in the industry then it will cost you both money and time, so you need to be careful if you are planning to go for one.
6. Internal or external trainings - There are many time organizations provide training to the employees. You should lookout for any possible opportunities to attend the same. Also there are external trainings that you can attend, but as I told earlier, you should be careful and should always check the cost, benefits and ROI from such investments.
7. Going through the knowledge base (past projects artifacts) available in a organization - Almost all organizations have some sort of knowledge base available within organization and you should make good usage of available resources. E.g. Test plans, good test cases, learning from past are some of the assets that you may use.
8. Interacting with your friends, relatives, blogs and forums - If there are friends and relatives who are also working in similar domain then try to seek information that may be useful to you. You can also subscribe to various blogs and forums related to your domain as these are very effective ways to learn and get clarifications to your questions and queries.
9. Attending conferences, seminars, webinars related to the domains and listening to the thought leaders - Try to see if you can attend any related conferences, seminars, webinars as these are very good source of getting valuables inputs for you to learn related knowledge.
10. Understanding and learning new researches, trends, advancements and regulations in a particular domain - This is a very important area that you should continuous and consistently follow. Learn new trends and advancement in domain will help you earn credibility within your organization and industry.
Here are my thoughts on this:
1. Getting involved in Business process testing - Getting involved during business process test helps understand the way business is done and is a effective way to learn a particular domain
2. Attending workshops with the customers and consulting groups - Attending workshops with the customers help understand various key configurations required to run the business. Given an opportunity this will help any tester. But these opportunities are not available to everyone and those who gets this opportunity should use to the fullest.
3. Participation in Alpha, Beta and UAT with customers - What else can give you a better exposure than working and participating in the testing done by any customer? You should observe and learn the key tests customers are exercising. Also try if you can get these acceptance scripts that you can use to improve your test repository.
4. Understanding of regulatory guideline (if existing on a particular domain) - If there are any regulations application on your domain and product then it is a must for a tester. The severity of any bug from a regulatory perspective will be much higher and it can have big impact on your organization and your customer business. Due attention should be given to learn regulations and guidelines. Any product that have good compliances to the regulatory processes get higher credibility in the industry.
5. Learning by doing specific degree, course, training, seminar, certifications etc - Check if there is any degree, course, training or seminar that can help you understand the domain well. Also you can do certain certifications if applicable to your domain. At the same time you need to research on the credibility of particular course and it is not effective and recognized in the industry then it will cost you both money and time, so you need to be careful if you are planning to go for one.
6. Internal or external trainings - There are many time organizations provide training to the employees. You should lookout for any possible opportunities to attend the same. Also there are external trainings that you can attend, but as I told earlier, you should be careful and should always check the cost, benefits and ROI from such investments.
7. Going through the knowledge base (past projects artifacts) available in a organization - Almost all organizations have some sort of knowledge base available within organization and you should make good usage of available resources. E.g. Test plans, good test cases, learning from past are some of the assets that you may use.
8. Interacting with your friends, relatives, blogs and forums - If there are friends and relatives who are also working in similar domain then try to seek information that may be useful to you. You can also subscribe to various blogs and forums related to your domain as these are very effective ways to learn and get clarifications to your questions and queries.
9. Attending conferences, seminars, webinars related to the domains and listening to the thought leaders - Try to see if you can attend any related conferences, seminars, webinars as these are very good source of getting valuables inputs for you to learn related knowledge.
10. Understanding and learning new researches, trends, advancements and regulations in a particular domain - This is a very important area that you should continuous and consistently follow. Learn new trends and advancement in domain will help you earn credibility within your organization and industry.
Thursday, June 17, 2010
How to manage situations when you have less time for Testing.
More often you find yourself constrained with time during testing phase, specifically when you are nearing the project release dates. With less time, obviously you have less number of options available with you and less number of scenarios that you can test and exercise. You need to see how you want to utilize this limited time available to conduct effective testing that allow you to find more issues. Some of the things that may help you overcome these situations are:
- Time is Important - When you run into a challenge and you have very less time and lot many tests to perform, first thing that you need to do is calm down!! Believe me you don't want to lose precious time at this moment by taking stress. Hours and even minutes do matters in these situations. If you can keep you head cool then you can think more effectively and productively.
- Be Positive - There will be situations you will be under tremendous pressure along with your team and most likely you are prone to dried up with extended hours of working. I would suggest you keep up the patience and try to be positive at this moment. Your motivation will not only charge you up but at the same time will have positive impact on other team members.
- Team Work - Team work is like a phenomenon. You can't express it but only feel it being a part. At this moment try to see if you can help any of your team member and at the same time check if you can seek help from others. You need to be smart and sensitive at the same time and understand that you need not to put burdens on others as well.
- Think from different contexts- Consider application from different contexts and perspectives e.g. business usage, usability, level of analysis, different conditions and behavior. Try to do end to end tests and then try to do various combinations of test data to get more coverage to the functionalities.
- Perform exploratory testing (ET) - Instead of doing more of scripted testing, if feasible, conduct more of exploratory testing. While doing exploratory testing try to observe the behavior of other areas while testing a particular functionality. You will be able to covers multiple functionalities together and help you conduct testing in more efficient manner. Switching between various modes to be more assiduous and make you feel the work more interesting. Refer - Exploratory Testing Polarities
- Work in sessions - Try to break down available time into small sessions. Conduct testing in sessions and then collate the results. Once you assess the results identify things needs to be tested in the session. For each session you should start with an identified list of tasks, so that you can focus on them one by one.
- Conduct Impact assessment - Try to identify impacts based on recent changes done in code during the last build, you might need help from development team for this and need to collaborate. This will help you identify the changes and will help you focus on areas which are impacted and are most prone to error.
- Risk-Based testing - You should focus more on modules which are high risk areas, have more defects (past data) and are fragile rather than over-testing the stable ones. This will certainly help you uncover most of the important defects. Refer here
- Have fun - Even though it may sound incongruous here, but believe me when you are challenged and then delivers, those are the moments you remember for long time. These may become good memories for you, down the line. I do remember lot of instances from my projects in past where we were faced with various challenges but seldom remember instances where things were easy, though these are quite few in number :).
Tuesday, June 15, 2010
A Friend named BUG...
Why a BUG is like a Friend to you...
1. When you find a bug, it brings you smiles and cheers
2. If you find a High severity bug you are elated...just like you meet up a close friend (and you feel like shouting in joy and happiness)
3. When a bug is closed you feel bad just like a friend saying 'good bye'
4. You feel bad when somebody says it is 'Not a Bug' as if somebody hurting your friend.
5. Your credit worthiness is highly increased if you log large number of confirmed bugs, just like you have high number of genuine friends
6. Your bug might be shared with others (duplicate bug) like some of your friends
7. Depending on the situations priority of bugs might get change like your friends you spend time with
8. Sometimes old bugs can be equally serious like few old friends (you meet after a long time)
9. Invalid bugs brings pain to you just like a false friend
10. Sometimes you need to 'handle with care', some of the bugs with emotions, you should treat equally a UI bug as compared to a Security bug just like your few friends
11. Last but not the least... Even though a bug is closed, it helps you keep good memories and great learnings, like some of the friends that you might have lost the touch...
Treat all bugs just like your friends and treat them well :)
1. When you find a bug, it brings you smiles and cheers
2. If you find a High severity bug you are elated...just like you meet up a close friend (and you feel like shouting in joy and happiness)
3. When a bug is closed you feel bad just like a friend saying 'good bye'
4. You feel bad when somebody says it is 'Not a Bug' as if somebody hurting your friend.
5. Your credit worthiness is highly increased if you log large number of confirmed bugs, just like you have high number of genuine friends
6. Your bug might be shared with others (duplicate bug) like some of your friends
7. Depending on the situations priority of bugs might get change like your friends you spend time with
8. Sometimes old bugs can be equally serious like few old friends (you meet after a long time)
9. Invalid bugs brings pain to you just like a false friend
10. Sometimes you need to 'handle with care', some of the bugs with emotions, you should treat equally a UI bug as compared to a Security bug just like your few friends
11. Last but not the least... Even though a bug is closed, it helps you keep good memories and great learnings, like some of the friends that you might have lost the touch...
Treat all bugs just like your friends and treat them well :)
Sunday, June 13, 2010
How to identify and manage Risks related to Testing...
There are several challenges and risks related to testing which may occur during your project execution. You should be careful and need to assess any possible risk coming into the project and should devise a mitigation strategy as early as possible.
This will help you manage your risks, help smoothing execution of project and help you release the product with best quality possible. I'm highlighting some of the risks that you might face as per SDLC phase for easier understanding. you should also evaluate risks in your project on a periodic basis as suitable, based on timeline of your specific project.
I also suggest you to publish frequent status report to your management team. This will allow you a platform and opportunity to convey the status and raise any concern/risk/challenge that you might face during the project. Many people think that sending status report is a waste of time but believe me if you utilize this tool effectively it will help a lot for smooth execution of your project.
This will help you manage your risks, help smoothing execution of project and help you release the product with best quality possible. I'm highlighting some of the risks that you might face as per SDLC phase for easier understanding. you should also evaluate risks in your project on a periodic basis as suitable, based on timeline of your specific project.
I also suggest you to publish frequent status report to your management team. This will allow you a platform and opportunity to convey the status and raise any concern/risk/challenge that you might face during the project. Many people think that sending status report is a waste of time but believe me if you utilize this tool effectively it will help a lot for smooth execution of your project.
- Risks in Requirement phase - Requirement must be good in order for any project to be successful. Testers should review the requirements in detail from various perspective
- Details are not available in requirements - All requirements should have sufficient details for an engineer to study them and able to build the required application. If requirements are not detailed then there might be lot of to-and-fro to understand the requirement which may consume crucial time during project execution
- Requirements are ambiguous - Requirements should be precise and clear. Any 2 person reading the requirement should be able to understand similar meaning out of the specification. If requirements are ambiguous then these can be either built incorrectly or crucial issues might slip from the test cycles
- Requirements are not testable - Each and every requirement should be testable. E.g. If a requirement is mentioned that 'UI should be good', then we may not be able to test the requirement as there can be different meaning for parameter 'good'. Similarly if there is requirement which says 'User should be able to login quickly'. It may have different meaning to different people.
- Changing requirements - Some times you are trying to hit a moving target, with requirement specifications changing every now and then. You should evaluate the impact on your deliverables and in case you find it a challenge then you should raise this
- Non functional requirements are not mentioned - Ideally requirement specification should mention non-functional requirements e.g. Performance, Security etc. If these are not mentioned then it may become subjective whether a system is in usable condition or not.
- Change Request - There might be change request coming into your project for which you do not get additional time. Any potential change request should be managed and come through a CR process and all impacts should be assessed before implementation of CR.
- Risks in Planning - Most of the project fails due to the lack and improper planning, even if you are preparing the plan, you should at least provide your inputs to your leads/manager before a plan is approved. Anyway you are the ones who will be executing the project and should correct any discrepancy during planning phase itself.
- Detail plan not present - Before we start the project we should plan our project efficiently. That means we should have broken all tasks to a manageable level and allocated resource and time for the execution of task. If this is not done properly, we might be setting up for the problem later
- Inadequate Resources - Many a times we see either adequate resources are not assigned to the project or they are assigned very late in the project. If this is the case then we should raise a risk and possible impact due to the same.
- Inadequate Test environment - Test environment is crucial aspect and we are expected to certify each and every possible environment combination. In case you do not get adequate test environment then you should raise this risk and in case you are not able to certify any particular combination, you should mention the same in the Validation report that you may prepare at the end of the project. Using this you should share the gap in execution as per the plan and share the impact with the management team
- Inadequate Time - Most of the project teams face this problem across SDLC. First of all you should budget adequate time for each tasks using a standard estimation technique. You should break down each task to a granular level and should block adequate time. Also budget some time for new change requests coming you way at the later phase of the project, you can also refer total efforts on CR from some of your past projects as references. Also plan some time for casual/sick leave for your team and add some buffer in the project. You should provide adequate rationale while doing so and negotiate with your management
- Inadequate Tools - In case there are tools planned in the project and are not available during execution, you should raise this risk and identify any other potential substitutes.
- Movement of resources across test teams - You will observe that many time resources are being shuffled across project teams. If this happens in your project then you should try to create shadow resource in your domain who have certain knowledge on techniques, tools and domain knowledge. Also try to create knowledge base documents across team and inculcate culture of knowledge sharing between team.
- Risks in Design phase - This is a important phase of build any requirements. You should try to participate as much you can during the design phase and should review the design documents thoroughly.
- Testers are not involved in designed - Many dev managers believe that testers do not play any role in design phase and try to keep them away. If this is the case then you should raise your concern and should try to get involved during design phase and provide your reviews. This will help you a lot while designing and execution of tests as you will have more perspectives to testing
- Designs are not robust - While reviewing if you encounter that design are not comprehensive, detailed and robust you should raise you concern and should try to ensure that nay open gaps in design should be implemented ideally before coding or test execution. Also you should consider that designs are built with sustainability and scalability.
- Performance is not considered during design - If performance consideration are not implemented in design then you should raise this appropriately
- Risks in Development -
- Unit test not comprehensive - Unit tests are important areas which can help identification of defects during early phase and effort of fixing an issue in code is also very less. In case you find many issues during testing then you should review unit test to see if they are adequately covering all possible gaps. In case you see any major deviation then you should raise this as a risk and help in correct this for future projects
- Features are broken in consequent builds - In case you identify that features working in earlier builds are being broken in next build while fixing some other issues. You may also want to get the impact assessment for any fix in the release (ideally in issue base directly). This will help you identify any possible impact in the build and then aligning your regression testing accordingly.
- Required documents are not provided by Dev - In case you do not get adequate documentation as planned e.g. release notes, mapping documents etc then you should raise this immediately so that correction can be done
- Risks in Test Design - While preparing test scripts and test data you should assess any challenge and raise it. You should assess any further risks in addition to the planning risks listed above:
- Test strategy is not devised - You should spend some time in determining test strategy so that you can proceed with design and execution of test. In case you determine any challenge during devising test strategy then you should raise this immediately for taking any corrective action. This may be any sort of dependency, tools availability, pre-reqs, coverage etc
- All Test scenarios are not covered - In case you determine that the test scripts are not robust and do not cover all test requirements then you should raise the risk and negotiate for adequate time to correct them
- Adequate test data is not present -If you observe that adequate test data is not prepared then you should update the data pool in your scripts
- Reviewers are not available - Review is one of the most important aspect to evaluate the quality of test design
- Risks in Test Execution - Test execution is one of the most important phases of any project, the results from this phase determines the quality and enables decision for the management for go-nogo. As testing team are owner of this phase, they should raise any risk that may hamper test execution and delay the release.
- Pre-requisites are not available - There might be some pre-requisites before you can start system test cycle. This can be in a form of tool, utility, stub, mapping document, test scripts, test environment, third party components etc. If these are not available then escalate immediately till you get all pre-reqs.
- High number of test builds - Due to high number of build you might be require to spend lot of time in installing multiple builds and due to this you will not provide adequate coverage which might result in issues slipping through test phase. You should work with Dev team and identify a build plan with a designated date and in case the number is increasing then escalate the same to you superiors.
- Many show stoppers are there in the build - You should conduct a smoke test round before you accept the build. This activity should be planned in you project. In case you find many show stopper issues then you should reject the build instead of spending time on the same.
- Many key requirements are not present in the build - Many a times you will get the build on time but when you start the testing you will realize that there are key requirements missing from the build. If this happens you should raise this immediately. For longer run you should ask release team to provide some sort of release notes which will have all requirements and known issues published by dev team. This will help you evaluate if the build is ready for system test.
- Regression time is not sufficient - You may end up doing routine testing and realize that there is not sufficient time to conduct full regression testing. In that case you should plan Risk Based Regression and highlight the same to the management in your status report. If there are genuine reasons you should negotiate either to get more testers to help you in testing or move the release dates while you can complete the required regression.
Tuesday, June 8, 2010
How much details Test Cases should have...
This is a very crucial aspect for designing the test cases. Even though it vary from situation to situation, but I'm giving some of the guidelines that may help you decide the optimal level of test case you should keep:
1. Test cases should cover positive, negative, boundary conditions, If it is a new requirement then we should try to cover full functional testing and if it is a patch or hot fix then we should focus more on Impact Assessment to the existing functions.
2. Wherever possible database testing should be planned have SQL related tests
3. We should keep number of steps to minimal and provide most of the tests data driven
4. Keep test data combinations in tabular format and try to make it compressed from easier maintenance and consistency
5. Use test techniques like DOE and Orthogonal array (also pair testing) for the scenario where we have large combination and data parameters
6. Identify and keep screenshots for test execution for future references and audits
7. Try to keep test cases modular and keep cross reference between different scripts for easier maintenance
8. In case you have related specifications available then you should try to keep references instead of repeating the scenarios e.g. For UI testing you can keep Style guides, for configuration testing, you can keep Configuration Baseline Document (CBD), for report validation, you can keep Report Mapping document etc for consistency purpose
9. Have different sets (and levels) of test cases focusing on Functional testing (Validation of functional requirement), Platform testing (Validity of different environment combination), Installation Qualifications for Upgrades (Upgrade from older version to new version) as per applicability to your situations
10. For Regression testing you should try to focus more on positive and business scenario level details
11. Following criteria should also be considered while deciding the level of test cases needed:
a. Test case detail dependency on the kind of lifecycle you are following e.g. Iterative vs. V-Model vs. Agile
b. Details of test cases depending on critically and priority of modules
c. Type of work is being planned - If you have your internal experts working vs if the testing work is outsourced
d. Level of available requirements and design specs
e. Level of testing performed functional, security, automation, performance
1. Test cases should cover positive, negative, boundary conditions, If it is a new requirement then we should try to cover full functional testing and if it is a patch or hot fix then we should focus more on Impact Assessment to the existing functions.
2. Wherever possible database testing should be planned have SQL related tests
3. We should keep number of steps to minimal and provide most of the tests data driven
4. Keep test data combinations in tabular format and try to make it compressed from easier maintenance and consistency
5. Use test techniques like DOE and Orthogonal array (also pair testing) for the scenario where we have large combination and data parameters
6. Identify and keep screenshots for test execution for future references and audits
7. Try to keep test cases modular and keep cross reference between different scripts for easier maintenance
8. In case you have related specifications available then you should try to keep references instead of repeating the scenarios e.g. For UI testing you can keep Style guides, for configuration testing, you can keep Configuration Baseline Document (CBD), for report validation, you can keep Report Mapping document etc for consistency purpose
9. Have different sets (and levels) of test cases focusing on Functional testing (Validation of functional requirement), Platform testing (Validity of different environment combination), Installation Qualifications for Upgrades (Upgrade from older version to new version) as per applicability to your situations
10. For Regression testing you should try to focus more on positive and business scenario level details
11. Following criteria should also be considered while deciding the level of test cases needed:
a. Test case detail dependency on the kind of lifecycle you are following e.g. Iterative vs. V-Model vs. Agile
b. Details of test cases depending on critically and priority of modules
c. Type of work is being planned - If you have your internal experts working vs if the testing work is outsourced
d. Level of available requirements and design specs
e. Level of testing performed functional, security, automation, performance
How you can grow in your testing career...
It is imperative for everyone to grow in their career with learning and understanding new aspects and improving knowledge and technical skills. Just like anything else we need to plan and follow what we want to achieve. I'm here by providing my inputs on how you can grow in your career and at the same time enjoy what you do the best. (Yes, I mean software testing):
- Identify your interest areas - This is very important aspect to consider and you should do what you want to do and not just do something which is imposed on you. Identify your strengths and identify what are the areas that you likes working most. From a career point of you identify whether do you want to work as a Individual Contributor stream or you want to Managerial stream. Both have their own pros and cons and you are the best judge of what is most suitable to you. Similarly from technical side, you should think about what kind of testing you want to perform e.g. Functional, Penetration, Performance, Security, Automation etc. All of these are equally good and you can follow similar growth levels in any of these fields. Select a career choice and work towards building relevant skills.
- Identify skills that you want to improve - Identify what are the skills you want to develop. Make short term or long term goals and follow them. e.g. for a short term you may want to learn something related to your work, domain or technology that may help you do your job better on a project or an assignment. For long term you may want to learn new automation tool or new language say Perl. It is important that you focus on few things rather selecting too many things at a same time and then failing to focus on something. You can take help from your friends, team members, leads, or other resources like internet, books, training course etc. This will help you in learning these skills better and faster.
- Learn new Technology, Tools, Techniques and Approach - Spend some time i learning the technology in which the product is built upon. Even though you may not needed to knwo this directly but rather it will give you knwoledge of how application works and will help breaking the same in more effectively. Always be on lookout for learning new techniques and approaches. This will help you improvise when needed as you will have spectrum of choices to solve one problem and you will become more effective. Believe that 'things can be better if tried differently'. Similarly learn new tools that helps that may help you more productive, some of the tools that you may want to focus might be, Automation tools, Performance testing tools, Security testing tools, Test data generator tools etc. You can some some of the tools that best fit your need and learn them.
- Challenge yourself - No challenge is better and fruitful than challenging yourself. It is the most important and enjoying than anything else. Identify few areas and try to make yourself sweat with the challenges. Try to do things faster in lesser time, try to learn something in short duration, try to minimize test cases (still keeping the effectiveness) etc. Also try to spend some time for physical activities to keep yourself fit by involving your-self to some sort of activity jogging, gyming or playing some sports.
- Read books and Read blogs of industry thought leaders - Try to spend some time of the day (may be an hour atleast) reading blogs on the net. It will give you new perspective to various situations that other people faces and it's a very good mechanism to get different perspective. Also read books related to your interest areas. This will help you clear any doubts and help you learn things comprehensively and in details.
- Share your thoughts with other and learn - Share your ideas, thoughts, knowledge with others within your teams, blogs, forums, groups. Discuss your point of view with others and see how people think about it. This will help you get lot of insights to certain areas and thoughts and in the process you will find new, interesting and useful skills, techniques, approaches and methods.
Remember - Knowledge is what separates success with luck...
Sunday, June 6, 2010
How to conduct effective Root Cause Analysis...
Root Cause Analysis is a very effective tool to conduct Causal Analysis. This process helps understand the problems which are being report and take corrective and preventive actions so that similar problems are fixed permanently and do not repeat in future.
Following are the steps to conduct effective Root Cause Analysis. Given below are the process steps with few guidelines to help complete RCA in a quick time and with effective and productive manner. These guidelines can be extended as needed for any specific scenario. The steps for conducting RCA are following:
A. Modularize Issue - Identify and categorize the issue, so that it can be assigned to a individual or a team
B. Identify Root Cause of Issue - Identify what exactly caused the issue to occur
C. Identify Corrective Actions - Identify what we need to do to provide immediate fix
D. Identify Preventive Actions - Identify what we need to do to prevent similar problem from re-occurring
E. Review with the team and Action items tracking - Identify and track actions items for the team
Let's understand each of these steps in details on what activities we should do in each of these steps:
A. Modularize Issue –
#1. Identify Modules and Sub-modules in the product to categorize the product
#2. Identify Modules wise SME in project team
#3. Identify issue to respective Modules and Sub Modules
#4. Based on respective Modules, assign to the respective SME or team
B. Identify Root Cause
#1. Review issue details
a. In Bug database
b. Service request id from Consulting/Customer Support
c. Customer Issue details e.g. QC system of customer
d. Any relevant email and other available information
#2. Define the problem
a. Understand the problem that is being reported based on available information
b. Understand the area of problem and steps to reproduce the problem
c. Identify any dependencies that may help in understanding the problem
d. If there is any gap in understanding talk to respective support, consulting or Dev resources
#3. Reproduce issue in test environment
a. Identify a respective QC environment on the release issue found e.g. if issue is found on version 5.1 then check issue in version 5.1 test environment
b. Reproduce in QC test env
c. Try to reproduce issue with the exact steps provided in the issue
d. If issue do not get reproduce then seek more information rather than giving up!
i. Send email to respective consulting or CS representative
ii. Talk to respective Dev lead/Developer or peer testers
iii. Other relevant sources that may help…
e. Smart Thought (If QC environment is not available)
i. Check any possible env with Dev team
ii. Check any available env with CS or Consulting
iii. If above options not available then prepare quick env with bare minimum components
(Note: It is highly advisable to keep latest release up-and-running to expedite issue reproduction process)
#4. Analyze issue on following parameters
a. Behavior of Design
i. Is this a known issue
ii. Do we have workaround conveyed to customer in release notes
iii. Do we have design document covering this scenario
iv. Have we prepared Impact assessment document covering this scenario (for patches)
v. Problem in logic
vi. Problem in algorithm
vii. Problem in coverage
viii. Problem in test strategy
ix. Problem in performance
b. Behavior of Coding
i. Code segment not correct
ii. Code segment not as per requirement
iii. Code segment has logical error
iv. Unit test strategy not covers this scenario
v. Unit testing not conducted for this scenario
vi. Unit testing not challenging the business scenario
vii. Unit testing not have adequate test data and test coverage
viii. Code review not done properly
c. Behavior of Testing
i. Scenario not part of test strategy
ii. Scenario not part of test scenario
iii. Scenario not part of test steps
iv. Was testing done for this issue
v. Was coverage enough for the issue
vi. Was Adequate regression testing done for the issue
C. Identify Corrective Actions
#1. Is there any workaround available
#2. Can workaround be suggested to customer
#3. Can a small and quick fix be provided to customer
#4. Does this require quick suggestion on IG and IQ steps that may help prevent the issue in near future
#5. How many customers are affected by this issue
#6. What is the Risk Priority Number for this issue (Severity x Occurrence-ability x Detect-ability)
#7. What it will take to fix the problem in the code branch
#8. What it will take to fix the problem in Main line code
D. Identify Preventive Actions
#1. Requirements
i. Was requirement was clearly stated
ii. Was requirement reviewed completely by team
iii. Was requirement review budgeted for this area
iv. How we will ensure that this problem do not occur in future
#2. Planning
i. Was this task planned in test strategy
ii. Was this task planned in mpp
iii. Was estimation correct (Have we missed some scenarios)
iv. Was enough time provided for the task in mpp
v. Was the plan followed with team and status updated correctly
vi. How planning can be made better for similar tasks
#3. Design
i. How similar logic/parameter/algorithm will be designed correctly in future
ii. How similar logic/parameter/algorithm will be implemented fully in future
iii. How can we prepare better test strategy during design
iv. How will these gaps can be identified during reviews
v. How can we plan and conduct good design practices
#4. Construction
i. How and why this scenario will be implemented correctly while coding
ii. How will we follow coding guidelines and checklist
iii. How will we add scenario part of Unit test strategy
iv. How will we ensure unit testing strategy is covers similar scenario
v. How will we ensure scenario missed in Unit testing
vi. How we will ensure all boundary scenarios are covered in Unit testing
vii. How Code Review done properly for the code to highlight the issue beforehand
#5. Test Script
i. How will we identify similar scenario part of test strategy
ii. Was this scenario present in Test Script
1. If Yes
a. Was this step executed properly
b. Why execution of step did not found the issue
c. Identify any missing test data
d. Identify any missing negative boundary condition
2. If No
a. Identify why this step not present in script
b. Identify why this was not caught in script review
c. Add this missing scenario to the script
iii. How will we ensure similar scenarios are not missed in review
iv. Are we considering past issues and test assets while designing test scripts
v. Are we considering learning from past (CAPA sheet) in next projects
vi. Is this scenario covered in Regression scripts
vii. Are existing regression scripts can discover the issue as-is
viii. What changes are required in Regression script
#6. Test Execution
i. Why this scenario was not planned in mpp
ii. Why this scenario was not executed
iii. Why we do not have complete proofs and available in VSS
iv. Why we do not have execution logs in VSS
v. Do this scenario need to be covered in Functional test script for next project
vi. Do this scenario need to be covered in Smoke test script for next project
vii. Do this scenario need to be covered in Regression test script for next project
E. Review with the team and Action Items tracking
#1. Once RCA is completed it should be reviewed with the team
#2. Any additional points and information should be updated in RCA
#3. A concrete action item should be identifed and planned for tracking
#4. The action item should have an owner who is responsible for completing the action items
#5. Action items should be reviewed by team on frequent basis (at least once a week/fortnight)
#6. Once action item is completed it should be marked as closed with details of artifacts which are updated
Following are the steps to conduct effective Root Cause Analysis. Given below are the process steps with few guidelines to help complete RCA in a quick time and with effective and productive manner. These guidelines can be extended as needed for any specific scenario. The steps for conducting RCA are following:
A. Modularize Issue - Identify and categorize the issue, so that it can be assigned to a individual or a team
B. Identify Root Cause of Issue - Identify what exactly caused the issue to occur
C. Identify Corrective Actions - Identify what we need to do to provide immediate fix
D. Identify Preventive Actions - Identify what we need to do to prevent similar problem from re-occurring
E. Review with the team and Action items tracking - Identify and track actions items for the team
Let's understand each of these steps in details on what activities we should do in each of these steps:
A. Modularize Issue –
#1. Identify Modules and Sub-modules in the product to categorize the product
#2. Identify Modules wise SME in project team
#3. Identify issue to respective Modules and Sub Modules
#4. Based on respective Modules, assign to the respective SME or team
B. Identify Root Cause
#1. Review issue details
a. In Bug database
b. Service request id from Consulting/Customer Support
c. Customer Issue details e.g. QC system of customer
d. Any relevant email and other available information
#2. Define the problem
a. Understand the problem that is being reported based on available information
b. Understand the area of problem and steps to reproduce the problem
c. Identify any dependencies that may help in understanding the problem
d. If there is any gap in understanding talk to respective support, consulting or Dev resources
#3. Reproduce issue in test environment
a. Identify a respective QC environment on the release issue found e.g. if issue is found on version 5.1 then check issue in version 5.1 test environment
b. Reproduce in QC test env
c. Try to reproduce issue with the exact steps provided in the issue
d. If issue do not get reproduce then seek more information rather than giving up!
i. Send email to respective consulting or CS representative
ii. Talk to respective Dev lead/Developer or peer testers
iii. Other relevant sources that may help…
e. Smart Thought (If QC environment is not available)
i. Check any possible env with Dev team
ii. Check any available env with CS or Consulting
iii. If above options not available then prepare quick env with bare minimum components
(Note: It is highly advisable to keep latest release up-and-running to expedite issue reproduction process)
#4. Analyze issue on following parameters
a. Behavior of Design
i. Is this a known issue
ii. Do we have workaround conveyed to customer in release notes
iii. Do we have design document covering this scenario
iv. Have we prepared Impact assessment document covering this scenario (for patches)
v. Problem in logic
vi. Problem in algorithm
vii. Problem in coverage
viii. Problem in test strategy
ix. Problem in performance
b. Behavior of Coding
i. Code segment not correct
ii. Code segment not as per requirement
iii. Code segment has logical error
iv. Unit test strategy not covers this scenario
v. Unit testing not conducted for this scenario
vi. Unit testing not challenging the business scenario
vii. Unit testing not have adequate test data and test coverage
viii. Code review not done properly
c. Behavior of Testing
i. Scenario not part of test strategy
ii. Scenario not part of test scenario
iii. Scenario not part of test steps
iv. Was testing done for this issue
v. Was coverage enough for the issue
vi. Was Adequate regression testing done for the issue
C. Identify Corrective Actions
#1. Is there any workaround available
#2. Can workaround be suggested to customer
#3. Can a small and quick fix be provided to customer
#4. Does this require quick suggestion on IG and IQ steps that may help prevent the issue in near future
#5. How many customers are affected by this issue
#6. What is the Risk Priority Number for this issue (Severity x Occurrence-ability x Detect-ability)
#7. What it will take to fix the problem in the code branch
#8. What it will take to fix the problem in Main line code
D. Identify Preventive Actions
#1. Requirements
i. Was requirement was clearly stated
ii. Was requirement reviewed completely by team
iii. Was requirement review budgeted for this area
iv. How we will ensure that this problem do not occur in future
#2. Planning
i. Was this task planned in test strategy
ii. Was this task planned in mpp
iii. Was estimation correct (Have we missed some scenarios)
iv. Was enough time provided for the task in mpp
v. Was the plan followed with team and status updated correctly
vi. How planning can be made better for similar tasks
#3. Design
i. How similar logic/parameter/algorithm will be designed correctly in future
ii. How similar logic/parameter/algorithm will be implemented fully in future
iii. How can we prepare better test strategy during design
iv. How will these gaps can be identified during reviews
v. How can we plan and conduct good design practices
#4. Construction
i. How and why this scenario will be implemented correctly while coding
ii. How will we follow coding guidelines and checklist
iii. How will we add scenario part of Unit test strategy
iv. How will we ensure unit testing strategy is covers similar scenario
v. How will we ensure scenario missed in Unit testing
vi. How we will ensure all boundary scenarios are covered in Unit testing
vii. How Code Review done properly for the code to highlight the issue beforehand
#5. Test Script
i. How will we identify similar scenario part of test strategy
ii. Was this scenario present in Test Script
1. If Yes
a. Was this step executed properly
b. Why execution of step did not found the issue
c. Identify any missing test data
d. Identify any missing negative boundary condition
2. If No
a. Identify why this step not present in script
b. Identify why this was not caught in script review
c. Add this missing scenario to the script
iii. How will we ensure similar scenarios are not missed in review
iv. Are we considering past issues and test assets while designing test scripts
v. Are we considering learning from past (CAPA sheet) in next projects
vi. Is this scenario covered in Regression scripts
vii. Are existing regression scripts can discover the issue as-is
viii. What changes are required in Regression script
#6. Test Execution
i. Why this scenario was not planned in mpp
ii. Why this scenario was not executed
iii. Why we do not have complete proofs and available in VSS
iv. Why we do not have execution logs in VSS
v. Do this scenario need to be covered in Functional test script for next project
vi. Do this scenario need to be covered in Smoke test script for next project
vii. Do this scenario need to be covered in Regression test script for next project
E. Review with the team and Action Items tracking
#1. Once RCA is completed it should be reviewed with the team
#2. Any additional points and information should be updated in RCA
#3. A concrete action item should be identifed and planned for tracking
#4. The action item should have an owner who is responsible for completing the action items
#5. Action items should be reviewed by team on frequent basis (at least once a week/fortnight)
#6. Once action item is completed it should be marked as closed with details of artifacts which are updated
Thursday, June 3, 2010
Do Software companies deliberately ship products with bugs?
Almost all of the software companies ships their products with a known bug list. It is important to understand what forces these companies are forced to follow this practice on regular basis. Also most of the companies have their own defined criteria to determine which bugs they can defer and deliver with the product release. Some of the possible criteria can be:
1. Bugs which are low severity and low priority
2. Bugs that may not cause any business impact
3. Bugs which have some level of workaround possible
4. Bugs which are existing (since old release) in the product
5. Bugs which are not high Priority and requires high level of efforts to fix them
These are some of the reasons that may allow some of the bugs to be deferred for a release. For any organization it is very important to determine good enough testing. Testing is good enough (This is also referred as Test Stop Criteria) if:
1. Requirements are correct and each of them are covered with at least one test scenario
2. All required functional and non-functional test scenarios are identified and are executed
3. All logical paths and conditions are exercised
4. All failed requirements are revalidated with respective regression test
5. All Identified issues planned for fix are fixed and validated
It is important to think on how as a tester you can help, contribute and voice over these areas and help improve quality within your Team and line of business:
1. Determine appropriate test strategy that maximizes the probability of finding defect
2. Try to build a Bug Deferral Criteria with your LOB, BU or product suite teams that is agreed upon by all stakeholders
3. If a defect is logged then try your best that the defect is fixed. you may not be able to do alone yourself, so use mechanisms like status reports, escalation and proper communication involving your superiors
4. If a bug is marked as deferred analyze if the bug not severe and has possible workaround. If you suspect that this bug can raise concerns from customers or decreases competitiveness in the market then raise your voice and escalate immediately. I'm sure if the alarm is genuine then it will be heard out
5. Ensure that the possible workaround is published in appropriate customer facing documents e.g. release notes, user guides, online help etc
6. Identify the means on how you can minimize the number of backlog issues. You will certainly need support from your superior and management for this.
7. Become game changers and identify measures on how you can contribute your best.
1. Bugs which are low severity and low priority
2. Bugs that may not cause any business impact
3. Bugs which have some level of workaround possible
4. Bugs which are existing (since old release) in the product
5. Bugs which are not high Priority and requires high level of efforts to fix them
These are some of the reasons that may allow some of the bugs to be deferred for a release. For any organization it is very important to determine good enough testing. Testing is good enough (This is also referred as Test Stop Criteria) if:
1. Requirements are correct and each of them are covered with at least one test scenario
2. All required functional and non-functional test scenarios are identified and are executed
3. All logical paths and conditions are exercised
4. All failed requirements are revalidated with respective regression test
5. All Identified issues planned for fix are fixed and validated
It is important to think on how as a tester you can help, contribute and voice over these areas and help improve quality within your Team and line of business:
1. Determine appropriate test strategy that maximizes the probability of finding defect
2. Try to build a Bug Deferral Criteria with your LOB, BU or product suite teams that is agreed upon by all stakeholders
3. If a defect is logged then try your best that the defect is fixed. you may not be able to do alone yourself, so use mechanisms like status reports, escalation and proper communication involving your superiors
4. If a bug is marked as deferred analyze if the bug not severe and has possible workaround. If you suspect that this bug can raise concerns from customers or decreases competitiveness in the market then raise your voice and escalate immediately. I'm sure if the alarm is genuine then it will be heard out
5. Ensure that the possible workaround is published in appropriate customer facing documents e.g. release notes, user guides, online help etc
6. Identify the means on how you can minimize the number of backlog issues. You will certainly need support from your superior and management for this.
7. Become game changers and identify measures on how you can contribute your best.
Tuesday, June 1, 2010
How to reduce number of tests using test techniques
Many a times we are required to perform validation with a large number of test scenarios that will require significant time and effort to cover the complete coverage. Most of us face this challlenge on a day to day basis due to following reasons:
1. Large number of Permutation and Combination of tests
2. Lack of resource and time available for testing
3. History data is not available that will help identify prioritization of test
To tackle this problem we may use test techniques that may help in reducing the number of tests and at the same time provide high level of coverage. This will ensure that we are using our resources effectively and optimally. one of the techniques that was used extensively in manufacturing domain and now used in software testing is Design Of Experiment (DOE).
DOE allows statistical analysis by experimenting behavior of different combinations of data. When you have large number of combinations then you can design your test data with Orthogonal Array (All Pair) techniques to reduce your number of tests. In this case instead of testing all n * n-1 combinations we create all possible pairs of data and test them. Let's take an example:
Suppose you have following parameters to test:
O/S
Win 2008
Red Hat Linux
Mac
Browser
IE
Mozilla
Print size
A4
A5
In this case if you take all combinations then you will get a total of 3*2*2 = 12 combinations, with all pair technique you can get this validated in only 6 combinations:
As the number of test combinations increased you will see lot of difference between all nth combination test and all pair test.
1. Large number of Permutation and Combination of tests
2. Lack of resource and time available for testing
3. History data is not available that will help identify prioritization of test
To tackle this problem we may use test techniques that may help in reducing the number of tests and at the same time provide high level of coverage. This will ensure that we are using our resources effectively and optimally. one of the techniques that was used extensively in manufacturing domain and now used in software testing is Design Of Experiment (DOE).
DOE allows statistical analysis by experimenting behavior of different combinations of data. When you have large number of combinations then you can design your test data with Orthogonal Array (All Pair) techniques to reduce your number of tests. In this case instead of testing all n * n-1 combinations we create all possible pairs of data and test them. Let's take an example:
Suppose you have following parameters to test:
O/S
Win 2008
Red Hat Linux
Mac
Browser
IE
Mozilla
Print size
A4
A5
In this case if you take all combinations then you will get a total of 3*2*2 = 12 combinations, with all pair technique you can get this validated in only 6 combinations:
TEST CASES | |||
Case | O/S | Browser | Print |
1 | Win | IE | A4 |
2 | Win | Mozilla | A5 |
3 | Linux | IE | A5 |
4 | Linux | Mozilla | A4 |
5 | Mac | IE | A4 |
6 | Mac | Mozilla | A5 |
As the number of test combinations increased you will see lot of difference between all nth combination test and all pair test.
There are tools also available but I have found tool provided by James Bach site as very effective and easy to use. There is a document inside the zip file that illustrates on how to execute the tool.
I would be more than happy to answer any related question on this.
Subscribe to:
Posts (Atom)