LanaFly

Redefining why we run Test Passes and Regress Bugs

2009-09-14T22:39:00.000-07:00

Redefining why we run Test Passes and Regress Bugs

I feel like running a test pass and regressing bugs has sort of lost their real intended purpose or potential. It has become something for management, and less about making the product better.

1) Test Passes

Right now, people will mind numbingly run through test cases - following test scripts exactly. They don't question what they ran before, what they are currently running or what they are going to run... basically the tester is half asleep. (funny thing is they were probably half asleep when writing the test cases in the first place!!) :)

Here are ways to get the most out of running a test pass!

NEW GOAL - if the testers are to run a full test pass (does not matter if its area owner or a contractor) - make it a requirement that they must find X number of bugs per day when going through the test pass.

What will happen:

The test cases they are running, now become a baseline (guideline), the tester now must push the tests to a new level to find bugs.
Test Cases are a simple entry point/code paths to consider.
The tester will step back at different points and look at a group of test cases - holes will be found!
Running a test pass will take a bit longer, but if the tester found a problem area - let them flush out the issues then and there.
You will sleep better at night knowing your testers were testing vs verifying!
Testers will now challenge the test cases themselves.

2) Regressing Bugs

Regressing bugs - typically testers try to get through them as fast as possible. They want to get them out of the way.

NEW GOAL -for every 3 bugs the tester regresses they need to find 1 bug. (or some ratio).

What will happen:

Pushes the tester to not only try the specific bug that was fixed, but to branch out and fully exercises other code paths that could have been affected by the fix.
Fixing bugs became just a little less risky - since the tester is trying to Break vs Verify.
Regressing a bug is no longer something to get out of the way or 'off your plate'. Fixed bugs tells the tester 'look at me... new code that has never been tested' which typically translates to potential bug farms!
Testers hopefully will not wait to regress their bugs, but see it as an opportunity to find more bugs. This means faster turn around and quicker stabilization.

With the new goals above - I guess you can think of test passes and regressing bugs as a way to perform focused ad-hoc testing!! (wow... that's sooo deep!!)

Testers are bitching - 5 simple fixes for 5 common complaints

2009-09-13T22:31:00.000-07:00

Testers are bitching - 5 simple fixes for 5 common complaints

Here are a couple of solutions/processes that can be put in place to stop some common complaints you might hear from testing!

Common Complaints from Test:

1. Problem - Tester says: What the f*!k? When did this get checked in? Why didn't anyone tell me? I don't know what is getting checked in?

Solution: EVERY CHECK IN MUST HAVE A BUG ASSOCIATED WITH IT. Any organization that does not have this requirement, sucks. Yes it is a little more work... but ask yourselves this... is there a checkin that is safe enough not to be tested. If it is alot more work... then there are serious problems...

2. Problem - Tester says: When did this feature appear?

Solution: Stop feature creep. Commit to a feature set for the release. If a new feature needs to be added because of feedback from beta testing, then testers need to be in the loop from the beginning(Testing should provide workarounds/risks/test estimates). But if features are being checked in, and are not related to anything else that is being done... it should be not allowed in. Remember Testing is already overwhelmed trying to ship the current product, they don't have time to entertain a new feature and bring it to stability. There is always next release to add the new feature.

3. Problem - Tester says: I only have time for surface testing

Solution: QA estimates need to be part of the release plan. If you are doing agile or waterfall, you have to have QA estimates for all features. If testers are saying they do not have enough time, there could be several reasons, but one place to start is to see if they were involved in the release plan.... Did Testers sign off on the release plan dates? If not, then... no wonder your product slips frequently OR lots of bugs are found by customers!

4. Problem - Tester says: We have no voice

Solution: QA Feature Progress Reports/Sign Off Sheet. This is really bad to hear... it can mean many things a)means testers are not informed of decisions being made, they are out of the loop. b) Or the testers do not feel like their profession opinion matters, or acknowledged or have an avenue to bring up concerns. c) OR testing is not providing information that is measurable (example they only provide gut feelings)

BIGGEST VOICE for Testers are providing Numbers!
Testing should be providing more then just gut feelings. They should provide a list of features and what they think the status of the feature is. This includes, incoming weekly bug rate, weekly fixed bug rate, list of areas still need to be tested, number of bugs still left to regress, number of bugs regressed. How the current week relates to the previous weeks.

Once testers provide numbers, everyone listens. :)

5. Problem: Tester says: I feel like I am out of the loop

Solution: Assign Feature Ownership. Sometimes testers are not in the loop because people do not know who they should inform on the Test side. Testers must be assigned features which they will be responsible for the feature from the beginning to the end. If there is more then 1 tester, and you have not divided up the areas of ownership, then it is total chaos. Developers have no idea who to talk to when there are code changes. Testers are only testing on a per bug basis, they are not seeing the feature as a whole. Overall quality sucks. No one is responsible for shipping a crappy feature. A feature never gets the full attention it needs. Only person that is in the loop might be the qa manager instead of the tester actually testing.

There are soo many benefits by assigning Features that its a whole blog in itself.

When to shake things around

2009-09-13T22:15:00.000-07:00

A) General Rule - After every release - switch Area Ownership

No tester should have the same area two releases in a row. If you have short releases, then it should be no more then a year or two.

Conditions before swapping areas:

Make it clear to the testers, that before handing off their area, the feature must be stable and any deliverables must be completed, automation/test cases etc.

Why switch up ownership?

Tester start to get desensitized.
They start to make assumptions that certain areas work.
Enthusiasm decreases - willingness to keep testing drops significantly.
They will not touch the area again unless there was a code change - and testing the fix in an area that was considered stable is minimal at best. (One of the reasons fixing bugs later in the cycle is very high risk)
Ignorance is Bliss - when an area owner finds a bad bug late in the cycle, there is a chance they might not enter it, in an attempt not to look like they suck. This is totally possible and chances of this increases if testers are questioned publicly on why a bug was not found earlier

What you get?

Spreading knowledge. You don't want only one person to know everything about a feature.
Increase number of eyes on an area. The new area owner has a different perspective on the product. The way they will start testing the new feature in how it relates to the old features they owned. As a result a whole new set of integration tests will be tested... which leads to new bugs. (assuming you encourage ad-hoc testing)
Excitement. If you make it clear to the new area owners not to assume anything was tested, and they are supposed to break it. It will bring a bit of excitement back, especially when the product has started to stabilize.
Surfaces problem testers - this is an awesome way to detect if a tester sucks. If the new owner starts to find a crap load of issues... you just covered your ass. You NOW know the old area owner needs a backup tester... which raises a RED FLAG! (time to give the old area owner a highly visible area which gets alot of traction by other testers.)
Saving Face - ** if a tester knows in a couple of weeks they will be handing off the area, they are probably going to start testing just a little bit harder to find the issues before someone else does. All testers hate it when someone finds a bug in their area... and they will try just a little bit harder to look like they suck!
Old feature owners never really let go. They will still be checking up on the feature every once in a while.

B) Other testers are finding bugs in another persons area

You really need to analyze the bugs being entered:

Are they basic bugs?
Are devs pissed off that the bug was not found earlier?(need to talk with them)
How long has the bug been around for?
Was the bug a regression? If so, how long has the regression been around for?

If you are frequently seeing bugs entered in a persons area - then something needs to be done. See if the tester is overwhelmed? Do they have too much on their plate? The features on their plate, is there a common integration link between them? Or do they own areas where if they are testing Feature A they are totally neglecting Feature B and both Features A and B are high risk?

If a tester has two high risk features, but there is no common integration point between them, you should assign one of the high risk features to another tester that has the bandwidth to look at it. (All testers do not have a high risk area)

C) Miscellaneous Reasons

I am just outlining some general rules in A) and B). Of course there are lots of other reasons to switch areas.

What you should get out of reading this blog post:

What I wanted to convey is switching area ownership is a good thing... and PLEASE don't be afraid of it. Testers LOVE it! Any if they don't like the idea... its because they are shitting their pants :) (NOTE: there are bad ways of switching areas... but doing A) is always good. And doing B) is for the quality of the product.... its stuff in C) which can cause problems.)

The Sweet Spot

2009-02-04T21:06:00.000-08:00

Out of all the years testing... there were these moments where everything aligned perfectly and as a result, my features were implemented, tested and stabilized in record time. These moments I am calling 'the sweet spot'. (Note: made up the term "the sweet spot")

My definition of 'the sweet spot' is:

when you have the right developer with the right tester, and they are both working on the same feature at the same time.

Sounds simple... but in reality it's pretty rare.

People these days are all hyped up about scrum, like its some magical solution to shipping faster. Its so complex (so many rules), that people think it must be the answer... eventually once you follow the path, you will see the light!? Lots of wishful thinking... and its missing the crucial key of what ships high quality products... the the emphasis of 'the sweet spot'. (yahoo one more rule to add!!!:)

What happens when you are in "the sweet spot":

Developer checks in code. The Tester and Developer are now in the sweet spot. They are both working extremely close. They go through this intense cycle of fixing and entering bugs as fast as they can. Both are 100% working on the same feature.

Developer fixes all bugs found regardless of severity, no punting/postponing/no backlog. They do not work on anything but bugs in the same area that the tester is working in. Developer does not fix other bugs in other area.

Tester is focused on testing the same area the developer is working on. They do not postpone testing to do automation or tools or writing test cases etc. They start pounding on it right away. They test from unit to integration and all the different testing themes.

Managers - life is easy for the managers. They just need to sit back and watch. This is the most stress free time for a manager. Their one responsibility is to make sure the dev and test keep the sweet spot going by removing external factors that could risk a context switch.

If someone asked the developer or the tester what they are doing, while in the sweet spot, both should be able to answer for each other. They are so close, they can finish each others sentences.

The Sweet spot should continue until the dev and test consider it done. Where both of them think that it can't get any better. Bug rate has slowed down to just a trickle.

Spike in bug count.

Time span from bug being entered, to fixed, to verified is very short. (Fast turn around)

Results of "the sweet spot":

Any feature that goes through the sweet spot is no longer a risk. You can check it off. Sure there might be minor issues, integration with other features that were implemented later. But its no longer an unknown, 80% or 90% should be working.

You can move the dev and tester to a new feature set to work on.

Two brain better then one. If you put an average dev with an average tester, and they go through the sweet spot... you WILL get ABOVE average results. Speed and stability will be way higher.

How to help line up everything such that "the sweet spot" can happen:

Dev and Test should be physically close together. Ideal is that they can talk to each other with out having to get up from their seats. (not a must, but boy it makes a HUGE difference)

Pair up the Dev with a Tester. State that they are a team. All information(meetings,emails) about the feature they are working on, they both included- no filtering.

Tell the developer and tester what should happen when the code is checked in.

DO NOT time bound the sweet spot. Commit to finishing - while incoming rate is high let them stay in the sweet spot. Reason it's ok not to time bound is because 100% of the tester and developers attention is on the product, it won't drag on because there are no other distractions all external factors are filtered out. AND another reason not to time bound is because if you cut it short, we all know it comes back under a different title...'bake time','integration phase', 'stabilization phase', 'code complete phase'... its that span of a couple of months before ship... where testers are finishing their testing they should have finished earlier but could not because it was time bound.

For the visual people out there... i made a super diagram of how things should go in Paint!!... when things went well for me in the past, its because it has the exact same flow below!

Look at the Tester - Non-Obvious ways to find bugs

2009-01-25T21:42:00.000-08:00

As a tester, I was always trying to figure out what area i should hit next. Where are the holes, the problem areas, the places that have bugs. One of the most efficient way is to watch the testers, figure them out, understand how they test, what their background is, etc.

Here is a list of testers I have seen over the years that you should look for. They all have a high potential for shipping low quality products if no one is watching them.

Note: the labels i gave them are my own and just for fun.

1) The Complainers

Listen to the hall way conversations. What areas or developers are the testers complaining about? If testers are complaining about a specific dev or area. Any kind of complaining demonstrates a problem with the dev test relationship. Sometimes you won't hear verbal complaints, instead you will notice that you never see the developer and tester talk.

Another reason testers might be complaining is because they are overloaded. They can't get to everything at once and no one is stepping into help them out. This causes the tester to do only surface testing, and deep in depth testing is not done just because of time constraints.

Types Of Bugs:

If the problem is dev/test relationship then there is always huge holes in areas that these two own and they will probably be very very very basic bugs.

If the problem is the tester is overloaded. Work with the tester, and say any testing you do is unofficial and under the table - but ask them what areas they are worried about etc. Any type of testing is bonus in their eye and it will alleviate some of the stress.

Reduce Risk:

For dev/test relationship: From a management point of view, you should definitely separate these two and assign this area to a new tester.

For the overloaded: move areas around. Bring other testers in. NOTE: i have randomly heard 'mythical man month' as an excuse not to do anything... what a bull. Get the area owner what they need or give them the option of forming a SWAT team - where they drive and pick out individuals to help and test specific areas that need to be covered.

2)The Underachievers

Look for the testers, where skills/experience do not match what they are producing. There can be a thousand reasons for this... but in the end, results are usually the same. If you see a tester that is not producing at the level they are capable of... that means they have checked out. (Time to move in on their area. :) ) Motivation and interest is gone, and they are probably just doing the bare minimum to stay afloat. You will know if they checked out by a simple test... Enter bugs in their area, if they don't react, they've checked out. (same test you would do if you are trying to see if an animal is dead, poke it with a stick and see if it moves :) ).

Types of Bugs:

All over the place. You never know when they lost focus. Look for recently fixed bugs, since bug regression is one of the first places that they will skimp on.

Management:

Got to move these guys to something new. To a new team, new product, fresh start. You might want to hold them tight and close to you because at one point they might have been awesome. But for some reason, the passion is dead, and you will never see it again (for you specifically)... cut the cord and let/help them move on.

4)Repeat offenders

If a tester dropped the ball on a feature they shipped before, they are going to do it again. Since they did not get fired the first time... why try harder? You will always find bugs in their area. Why? Because they are not getting it and don't care enough to try to get it. Typically these are senior testers. Funny thing is with these guys, you would never know by meeting them that they suck, because they don't think they did anything wrong. Sometimes these guys are big talkers with low bug counts or they are the guys that are just floating by.

Type of Bugs:

Everything under the sun. I have found that just following the specification design document found tons of problems. Other times I have found problems where the feature integrated with another product. You can always ask this tester what they hate the most, and you will probably get your answer of where to start looking for holes.

5)New Hires
I love new hires... motivation is high, work ethics high, they are young and energetic... only problem is they are new. When you are testing a new hires area, its best to actually teach them different techniques and let them run with it. I feel that its very important to give new hires the right feature set - set them up to learn what it takes to be a top tester. Features you should give a new hire, 1) is where alot of other features overlap it and 2) give them a high bug count feature. Reason to give overlapping/high traffic features, means other testers will uncover bugs that the new hire missed (reduce risk). And the reason to give the new hire a feature that typically generates a high bug count (like UI) is... it is easier to go deeper into the product if they have a UI entry point, so they can go top-down (I rarely ever see people go from bottom up (database level to UI)).

Types of Bugs:

Leave their areas alone. Better to take the opportunity to mentor them, show them what they missed etc. Don't take away the opportunity for them to practise finding bugs in their area.

5)Hurts to Think

When creating a test plan, it requires alot of analysis and thinking about their features and how it will integrate with the rest of the product. This requires thinking. So look at the test plan (if they are required). Definition of 'test plan' - is a document outlining how the tester is going to approach testing the area. Typically the testers that put no effort or thought into the test plan... will do the same when testing their area. It's the first sign there is going to be trouble.

Types of Bugs:

The type of problems you will see out of their area, are major redesigns later in the cycle. Huge integration misses.

6) The Verifiers

Verifying sucks. Its the worst thing ever. All they tried was very basic unit tests, which the developer already did. Basically its open season. I bet any test case that you try that even remotely pushed the limits will break. Instead of testing their area, its best to convert them to becoming testers. They can be converted over, it just might be that no one has mentored them, or showed them how to be successful. (This is all assuming that they want to be in testing, and have motivation to become a great tester)!

Type of Bugs:

Only thing that was tested was basic unit tests. Since there is so much testing left its best to mentor these testers and teach them how to be super stars if the motivation is there. You will see by bug count if they truly want to learn. And if they are all talk and no action then its time to start testing their area.

7)The Push Overs

The push overs are the testers that can't stand up for a bug. They find a serious issue, but get over ruled by their Program Management or Developer. Instead of pushing for the bug, they let it go. All three are at fault in making the wrong decision about a bug, but sometimes the tester claims innocent because in their world their responsibility ends at 'logging the bug'.

Types of Bugs:
The types of bugs you will find that make it to the final product are usability issues. Usability is a common type of bug where testers opinion is void and null because of the common saying 'your a tester, you are not a real world customer'.

Management:

It is best to pair up a push over with a senior program manager or senior developer. It usually the younger pm's that make the bad calls or the young developer that does not want to do the work.

8) The Keeper/Old Timer

The keeper is someone that has owned an area for several releases. People always think, this is good because they are the area expert, but its not!!! Totally opposite, its the worst thing you can do and you are in for trouble. They have become desensitized, no longer question anything. They no longer go through an area with a fine tooth comb. They skim over things, they test for 'it generally works'. Its the same old. Passions starts to ween, attention/focus starts to wander to other things.

It is better for the team, if this person takes a new area on and someone else takes over their old area. Why? 1) The keeper will have a hard time letting go, which means you will still see bugs coming in by the old area owner even though he has a new area (so you get two sets of eyes) 2)Expertise is distributed to more then one person on the team 3) You have a fresh set of eye looking at the feature 4) The keeper is less likely to get bored and leave your team.

Types of Bugs:

Regressions will not be found. Basic scenarios could be broken since the tester assumes they are working.

Weekly Bug Quota - why it's soo good in soo many ways

2009-01-22T22:42:00.000-08:00

I swear the best way to ramp up anyone is to give them ONE goal... hit a weekly bug goal. The magic number I always liked was 10 bugs a week minimum. And the rules to this game is, always hit it no matter what.

Who should this be applied too?

New Hires for sure, it simplifies their life and teaches them to focus on what makes them valuable... finding bugs.
Anyone that has not been through a full release cycle from start to end.
Any testers that are labeled as verifiers
Any tester that does not consistently produce results.
Any tester goes off and does there own thing and is leaving the product hanging.
Any tester that thinks they are done.

Why is a weekly bug goal good to have?

Teaches the tester to never take their eye off the ball, or at least in our case the product. It forces them to practice finding bugs. The tester should never go longer then a couple of days without being in the product.
Everyone knows how the product is doing (including devs and management). Its funny how people always assume the product is better when there are no bugs... when in reality you don't have a clue if its because the product is stable or the tester is not testing. By forcing the quota, you don't have to worry so much about the tester not testing.
Later in the product cycle, this quota is super important because it forces the tester to do integration testing - (where 80% of all bugs are integration). Alot of testers think that if their area is stable it vacation time! Not true, its heavy integration time. NOTE: I have noticed that very few testers spread outside their area, the mandatory bug quota forces them to branch out.
If you have hired someone to go through a test plan and verify. You should still apply the bug quota. It will apply a little bit of pressure, and force them to question the test cases they are running - makes them have the 'break the product' mentality. You will get higher quality testing, since they will deviate outside the test plan if they see something interesting.

Excuses you will hear (when they don't make the quota)
You will hear alot of excuses... here are a couple that come to mind:

"My area is stable" - my answer would be - "test other peoples area"
"I don't have UI" - my answer would be - "test other areas"
"I was regressing bugs" - my answer would be - "while regressing you should have been trying to find bugs" (regressing bugs is the best way to find bugs... so anyone says this statement probably did a really crappy job regressing the bugs!)
"I was writing my tool" - my answer would be - "write your tool, but also find 10 bugs. tool means nothing if you don't find bugs"
"I am blocked" - my answer would be - "test something else"

Great Developers from a Testers Perspective

2008-11-14T23:40:00.000-08:00

There are some dev's i loved working with! Here are some reasons why...

Communication
Bad Communication->Most developers and testers only communicate through bugs. One reason is Testers are frequently afraid of going to their developer for questions. Testers are even told sometimes not to bother their developer since the dev is really busy. IF the tester is blocked, they should not let anything get in their way of doing their job. Testers time is just as valuable as the Developers. If a tester is told not to bother the dev, then the tester should schedule office hours with the developer where they come in with a list of questions they need answered. (sorry got off track... but i feel its important to say this because i hate it when testers act like the victim)

TEAM work
Back to qualities of a great developer...

Great developers frequently discuss issues/designs with their testers.
They kept them up to date about changes coming up.
They tell their tester, area's of concern, area's the tester should focus on.
They understand the more the tester understands the feature, the better they could test it.
The great developers always included the tester on emails and meetings that discussed their features.
Great Developers do NOT take bugs in there area personally
Great Developers know they need great testers to challenge them. They know you can't ship a product with out them.

Basically... the developer and the tester are 1 unit... 1 team. They should do everything together.

Why communication/team work sometimes fails...
Sometimes testers might not be included because they can be a distraction. They will start breaking the feature where the purpose was to design it. The tester needs to just be reminded of this fact... and everything will be ok. Don't just not invite the tester because they just break everyones idea... just tell them what you need from them before the meeting is started and it will go smoothly.

Here is a list of things I have seen great developers do:

writes simple well designed code - oh how nice it is to do white box testing on a feature that has a simple design!
rewrites crap! Code might not have started off as crap, but as the bugs were being checked in, the code became patchy hard to follow, alot of exceptions to the rules added all over the place. The great developers, will stop, and revisit the area and refactor/clean it up!
Comment their code
Design a flow chart of how it might go and let the tester loose on it.
THEY NEVER do hacks (or very rarely). They will write code to fix the problem the right way even if it takes them alot longer. They understand that putting hacky code in, still have to go through testing which will find alot of bugs around it, and eventually they will have to go back and do the right fix, which will again have to be tested... (so the dev that tried to save 1 day of work or 1 week of work, actually caused more work on themselves and all other people involved.
I am curious - what is the life span for a hack? how long on average does it actually stay in the code? What ever the answer - short or long half life... you can't win... it always comes back to bite you in the butt... in sooo many ways!
Almost forgot - making the feature testable!!!! YAHOO!! this can be done by providing hooks for automation, providing some extra UI. If a feature is a black box, put a light bulb in it... light it up! The area will get stabilized faster, and both you and the tester can move on to the next feature!!

Bug Bashes - how to make them less sucky

2008-11-14T00:18:00.000-08:00

Bug Bashes are when a bunch of people (cross disciplines) test the product. I have found that they can be useful if done correctly. (note: i personaly don't like them, but when they are done correctly, i hate them less:) )

1. Product needs to be stable.

There should have been a couple iterations between the dev and tester that own the area.

Unit testing should have been completed.

The tester that owns the area should have a goal that no one will find any bugs because they already found it. If your tester feels this way about the product, then schedule the bug bash.

What happens if you have a bug bash too soon. It is similar if you ship a product too soon... a lot of noise. Alot of people getting frustrated because they are hitting basic bugs, duplicate bugs in the database. The area owner dev and test waste a ton of time trying to repro and filter the bugs. Majority of the bugs are not repro or dup's. You are lucky to get one good bug if your product was not really ready for the bash... when 50% of the bug bashers hit 1 basic bug, 90% of the bugs entered are all around that bug... sort of like an open gate way to hell.

2. Guided Bug Bashes
If you really want to get something out of the bug bash, you need to provide guidence. Otherwise, 90% of the people are going to spend 80% of the time all going through the exact same path (which is probably covered by automation anyways). At the end of an unguided bug bash, you end up with nothing! You don't know where people were testing, you don't know what they tried, you don't know anything... basically you wasted peoples time and got no results (or superficial results that the product is good to go).

Details you should provide to the testers who are going to participate in a bug bash.
a) Provide a goal - set the tone - example... "let stop ship" - 'be malicious' - 'break the build'
b) Provide Areas of Risk -
provide details of why an area might be risky
provide how to get to the risky area
provide various entry points to your feature set
list past bugs in the area - *** super important
recent fixes
c) Provide a list of Integration Points - how the feature integrates with the rest of the product
d) Bug Template - provide a template for entering bugs. Along with a bug template, you should provide a definition list example Toolpane, ToolPart - so the testers can describe the bug with the correct definition.

3) Instead of Bug Bashes
I typically don't like bug bashes - when i am participating, half of the time it is installing and configuing, and the second half is me just figuring out the feature set. I feel very unproductive and felt like the 4 hour crunch was more just a surface ramp up on the feature set.

Instead of Bug bashes - i prefer SWAT teams - highly specialized, individually pick persons to do focused testing in specific areas. There is no reason why everyone has to bug bash at the same time. Before I start talking more about alternatives to bug bashes, I will write a future post...

4) Prizes
Yup prizes. You are taking people away from their work to help you do your work. Prizes are great motivators! It can be as simple as winning a huge plastic penguin! I think people love to win, they will do anything... cheat, lie. This is the kind of mentallity you want the testers to have when bashing on your feature set. They are less likely to verify, and more likely to try and break the product... push the limits.

Top Testers - are there common traits?

2008-10-01T21:00:00.000-07:00

I know there are common traits among top testers... because I can relate to them and I can pick them out really really fast. They stick out like a sore thumb among the rest of the testers. Funny thing is that it does not matter what type of product... top tester are always top testers.

Here is a list of non obvious traits. (I know I have used Non-Obvious alot so far in my blogs, but i feel like i am writing stuff that other people don't write... and i am not sure why that is?)

Highly Competitive
Top testers are extremely competitive. They want to be the best. They produce more by large margins compared to everyone else. There is no shade of grey. They don't just dominate the bug count, they dominate at everything. They write the most automation, they write the best test cases and the most complete test suites, they write the best most complete test plans (test design specifications). If average tester produces X, the top tester will produce 1000 times more and faster (with higher quality).

I have to admit, I was(and still am) like this. But the top testers I met over the years, are always trying to grow other testers to be mini-me's. They would love if everyone tried their best. Its alot more fun when you have closer competition. :) Plus they can feed off each other. Its amazing what happens when one top tester feels threatened by another top tester... both tester productivity goes through the roof!!! BUT its not talked about, and they totally work well together because there is a mutual respect and they both have the same goal of shipping a high quality product. BUT in secret deep down, they will work longer and harder, just to be one bug ahead of the other tester.

Extreme Ownership
Top testers treat their product as their baby. Notice I said 'product'. They take a sense of ownership of ensuring every thing is covered. Typically testers just test their area, and then they are done, they don't care if the rest of the product sucks... all they know is they are done. Top testers, branch out, they reach very deep and very wide. Its like they are creating vines that grow and branch out touching every part of the product. Eventually the top testers will know the state of every feature, they are the walking bug database. When they see a bug, they know without looking at the code, what is causing the bug.

Always Analyzing
Top testers are always analyzing the product... when i say always... i mean... when they go home their minds are still trying to break the product mentally. When they go to sleep they are still trying to break the product. By morning, they have a whole new set of idea's they want to try to break the product.

Breaking the Product
All top tester try to break the product all the time. It can look like it comes easy since they can find so many bugs quickly. The key is every test case they try, it is always to break the product. "what happens if I try this, in combination with this... can i break it?". Notice the "what happens if"... that means they are not sure about what the out come will be because they pushed something to the limits, or are combined features in a unique way.

I am trying to think what people that verify say... and I can't think of anything... you know why? because verifying takes no thought process... steps are already outlined for you, result is already defined, the tester just has to execute them. I think when people verify, its like they are watching a tv show but not really watching it... they are just staring at something. Can you tell i hate to verify?... i actually can't do it... it hurts too much... it takes the fun out of the game...

No Assumptions - question everything
This is a huge one. To become a top tester, never assume anything. Never assume the dev is right, never assume feature is implemented they way it was specified in a document, never assume the technology your feature is using works, never assume someone else will test it, never assume a fix by a senior dev is safe, never assume things work.
Since top testers question everything under the sun, they end up finding everything under the sun!

Tangents
I personally have problems staying on topic when talking... i am always taking tangents. But I am not talking about verbal skills here. Top testers while testing can easily take tangents, they might be testing one area, then notice something odd, and go down a completely different route. They don't test one area 100% and then go to another area. They are all over the place. If you look at their bugs... they will be all across the board. This typically happens once their specific feature set is stable... and then they get to go out and play with the rest of the product... adhoc/exploratory testing. They are very fluid, moving around from one area to another... they see highways between feature sets where typical testers only see walls.

I know there are more... but I can't think of them right now...

Who should be a testers best friend?

2008-09-29T00:18:00.002-07:00

Answer: The developer.

The person a tester should talk with the most on the entire team is their developer. They should be in constant communication. It does not matter if you are in agile or waterfall development.

New hires, that are new to testing, are so surprised when I tell this to them. They are often shy and afraid to talk with the developer (often intimidated). And the developer is so deep in writing their feature, they don't notice the awkwardness :)

I also, firmly believe the tester should sit near the developer. I have worked in places where there is a separate dev/testing hall (hallway of testers and a different hallway of developers) and I worked at a place where I got to sit beside the developer.

From my experience the amount of information I absorbed sitting near the developer was amazing. I could just listen, or provide real time feedback on discussion that took place, provided valuable feedback on bug fixes (verbally told the developer things they should watch out for), helped to be a sounding board on design decisions. And I know if i said just a little further, the interaction would have been limited to only when i initiated the conversation or when the dev just checked in, or when i found a bug.

For me, I know once I sit in my seat, i hate getting up... its like a context switch and its hard to reorient yourself when you come back to your desk and have to remember what you were doing... (especially if you are like me and have a thousand windows up) :)

How to be a Great Tester

2008-09-28T23:52:00.001-07:00

... coming soon

Non Obvious ways to find bugs - Look at the Bug List

2008-09-28T21:33:00.000-07:00

Here are someways to use the Bug List to help you find bugs and figure out the most unstable areas.

Its not just for tracking bugs. If you are monitoring the in-coming bugs, you can get alot more information about the feature and product state. Here are a couple of things I have learn to look for when monitoring the in-coming bugs.

FYI... all the top testers, always monitored the bug list - knew every bug entered and the cause of them :)

1) Old Bugs - look over old bugs.

Bug Count Per Feature - If you are inheriting an area, look for areas that do not have very many bugs. This could indicate that there was minimal testing done. No one ever writes a feature that is perfect... its not possible.

Bug Count vs Type of Feature - you need to look at how many bugs were entered in an area. Does the feature have UI or is it only api's? Does the feature have heavy integration with another product? Is the feature legacy code or brand new? The expected bug count should go up or down depending on how you answer the questions above.

Higher Bug Count
UI
Integration with 3rd party products
Customizable applications - UI and API level
New Features

Lower Bug Count
API (only because UI typically exercises this code, so it all depends to goes through the code path sooner)
Legacy Code
Code that has no integration.
Database level

2) Watch new bugs being entered - they will give you an idea of the types of bugs being found, and what to try in your area.

A)Investigate for repeats - If a dev made a mistake, see if you can apply it somewhere else in the product? Typically the same kind of mistake occurs in several places. These mistakes can be done by the same developer or across multiple developers. If its a very simple bug like a feature do not work with Unicode characters, then its probably a product wide bug, where the developers have not has sufficient training in this area. Make sure you see determine if its a product or isolated case.

B)Is it a very basic bug? - if the bug is very basic but the feature has been checked in for a long time... test it(its for sure a bug farm)!!! There are several things it could indicate when a basic bug is found late in the cycle -

Could simply mean the Tester is not testing their area
Area might not have an owner. I have found huge testing holes in the product because i looked into why a simple bug was found late in the cycle. Typically, holes occur when one feature is providing data to another feature, example web services. Who tests what? Typically both testers think the other is testing the feature, when neither are. Test Contracts can help, as long as everyone is using the same terminology.
Tester has become numb. Tester might have been aware of the issue, but no bug was ever logged. This happens very frequently, tester goes and talks with the dev or pm, and they punt on the issue saying... "well if you put in the bug we are just going to won't fix it". Unless you switch up the testers (get a new set of eye), these types of bugs make it to production... and cause alot of usability issues. So if new bugs are entered, and they are basic and they are entered by someone other then the area owner, I think its about time to switch up area ownership.
Tester might have known about the bug, but was afraid to put it in because it exposes the fact that they did not test their area. Again, changing area ownership is the best thing here... because the new tester will not be worried about covering their ass while testing. AND yes this totally happens in the real world.... because, there is no way to prove a tester did not do their job unless someone else finds the bug. And the poor performers are not going to risk entering a basic bug... its like shooting themselves in the foot - their job could be at risk - at least they will be interrogated by everyone on why it was not found earlier.

C) Is there a sudden spike in bug count? - and the spike does not correlate to feature check in! - I hate when this happens. Its cause by something external. Typically it happens a month or so just before review time. You need to take note of those testers that improve productivity around review time. I have noticed that simply testing in their area lights a fire under their ass! So a simple resolution is put bugs in their area ever once in a while just to keep them going.

D) Watch out for missing spikes in bug count - There should always be a huge spike in bug count right after a feature gets checked in. If there is not then something needs to be done right away. Possible reasons are:

Tester is swamped with other areas. This is really bad. Give the area to a different tester, the longer the time between feature check in and fixing bugs, the worse the dev's are at fixing them. When a feature is just implemented, its all in the dev's head... is all in RAM. To delay putting in bugs, means that the dev's moved on and swapped out the RAM. To swap back in when your tester might have enough time is just bad for the product.
Tester does not have a clue how to test the feature. This can be because they are new, do not know the technology, or not technical enough. This is where you want to pair the tester with someone Senior. They can help review the test plans or help create one, and provide idea's on how to break the new feature.

D) Random bug that does not make sense - Have you ever read a bug... and say 'what?'... 'how is that possible'... 'shit'... When i see these bugs, it usually means that i made an assumption that something was working or it worked in a particular way.

You need to revisit your feature set, and see what the impact it has on your testing if you find you made an assumption. KEY is that you have to go back - when you get this gut feeling! I usually feel like the floor was taken out from under me, they way i visualized the entire system was wrong. You need to go back... and enter bugs, even if you don't want to because everyone will know you missed something. Better you find the bug then some other tester and better you find the bug before the customer does!!!

E) Regressing bugs with NO new new bugs being entered - When regressing bugs you should try to find new bugs. You should try to find 3 bugs for every 1 bug you entered. I know its not realistic, but it keeps you mind switching to 'verify mode'. You should still keep the 'break it' mode while regressing bugs.

Testers commonly have bug regression nights, where they have to go through a crap load of bugs. You should review bugs that were closed by the tester, and pick out once that you know should have taken longer then 5 mins to regress, or ones that are high risk. Talk with the tester to see what they tried, or go around and play with the feature yourself.

I know there are more things to look for... but I can't think of them right now... :)

What is testing

2008-09-28T19:50:00.001-07:00

I want to define what I think testing is. I find it really important to drill this into new hires. Its the first thing I talk about with them. Note: this is my definition... (i don't know if other people think the same way... but i know it works, you will ship high quality products following it)

Testing:
Testing is all about finding bugs. Bugs are only found if you are trying to break the product. If at somepoint you start to verify the product, then you are not testing. So I guess "testing" == "breaking"... in my world. :)

Non Obvious ways to find bugs - Look at the developer

2008-09-28T19:46:00.000-07:00

There are alot of different ways to find bugs. Typically they are done by a theme like boundary testing, error testing, internationalization...etc. Then there are the non-obvious ways to find bugs.

1) Look at the developer - Understand your developers - Goal here is to find bugs, so I am listing traits to look out for. Note: i have made up these categories - so take them with a grain of salt :)

Just out of College - these developers make alot of mistakes but they learn very very fast. They typically don't test their code or they don't know what to test for. Since they are so new to writing code for production, basically any testing theme will find bugs.

Things to try:

Error/Boundary - New dev's typically just follow the spec (if there is one). If valid input is 1 - 10, then that is all they will code for. They typically never code for crazy inputs like maxing out the field or value. At the beginning when they see bugs on their plate with 'pushing the limits' type bugs they will often fight back and say 'no user will try this', but just fight back. These bugs have to be dealt with in the code of because they will cause problems later down the road. They can lead to crashes, security bugs etc.
Integration Test Cases - new developers are barely staying afloat, they are only familiar with the code they have just written. This is where your job as the tester comes in... its your job to teach the developer (indirectly through bugs) how the rest of the system works. It sounds funny to say teach, but the developers are sort of like horses with blinders on (can only see what is in front of them)... even if you tell them to watch out for something, they won't remember. That is where the bug queue will be their reminder and the teacher.
All testing themes - accessibility, internationalization, localization, security... etc. For a college hire, this is all new, and the tester has to be VERY VERY thorough in testing their area.

Dev/Test/Feature Match up:

I personally think it is good to pair up a new dev with a senior tester.
Or you can give the developer a very visible feature. Where the entry points to his feature gets hit by alot of testers.
Sometimes, the senior tester can overwhelm the new dev. So testers, be careful... you want to have a good working relationship.

The Cowboy's - these guys are the worst. They pump out code like no tomorrow - alot of code turn. They don't think or plan what they are doing. They break other peoples code alot, and they think they are the shit. What makes these dev's different then the principle/architect devs is that they never learn. Its like they keep going in circles. They can just as easily write code and then delete it and write it again, but with no improvement.

Things to Try:

Automation: You have to save yourself from always trying the very basic tests at every build. With this developer, you will have to repeat your testing thousands of times. You have to automate your areas right away. Next step is to make a requirement that the dev must run your automation before he checks in.
Regressions: Regression rate will be very very HIGH. These dev's never learn. When you get a build, you must have the mentality of - "for every 1 bug i regress i have to find 5". They will put their fingers into everything, so you have to also.
Source Code Changes - you need to view source code changes for these guys. This will help you focus your attention on areas that they probably broke.
Stay Close - tester has to stay very close to this developer. Try to get the developer to talk about how he is going to fix the bug. Provide areas that you think will break because of his implementation. This process will force the developer to slow down and start thinking before he writes the code.

Dev/Test/Feature Match up:

You have to match a senior tester with this type of developer. The senior tester will know how to push back on bugs and they will be more efficient in finding the problem areas right away.

Weak Dev's - these are the dev's that are just not getting it. Typically the other dev's will know who they are, and will assign area's that are not high risk, and isolated. But all features are risky.

Things to Try:

Security Bugs - even though the area might be isolated, and not a common case - the best security bugs can be found in these areas. Its almost like everyone does less... code reviews are not as tough, the tester paired up with the dev are not as tough on the feature. This is where you will probably find the best security bugs... just because the passion is not their... no one cares for it! And since no one cares for this area, the type of bugs that will guarantee a fix are the security type bugs! :)

Dev/Test/Feature Match up:

Since these dev's are sort of disengaged from the rest of the dev's and feeling low, its good to pair them up with a tester that is a new hire. New hires are very excited, and they will help the sinking dev up a level. A senior tester will just break the weak dev even more.

Short Attention Span - These dev's don't have ADD, but they do have problems sitting in the same area for a long time. These are the devs that get bored really fast. The like to write new code, but hate to spend time to fix the bugs with them. And when they do fix the bugs, they cause alot of regressions because they don't take the time to property check their fix before checking in.

Things to Try:

Depth Testing - you have to go really deep in your testing. Test all layers of their feature. Eventually you will want to look at the source. If the features integrate/use another product you have to understand what the other product can do... because your dev does not care. They are the reactive vs pro active dev.
Fit and finish bugs - Since the dev will want to write the skeleton and move on to something new, you will find alot of fit and finish bugs. Example, xml is not human readable, alert messages are crappy, dialog looks like shit.
Usability - the dev might have followed the spec, but that does not mean it's shippable. The Tester needs to look at usability/flow of the product. Does it make sense? Will end users understand how to use the feature?

Half Ass Dev - their bar is "it barely works". Everything they do is half ass. If you try a test, only the main scenario works and everything else is broken. Usually these devs have a chip on their shoulder or they are managers of other dev's. They either have been at the company for a long time, or they have had crappy testers in the past. They think they are soo smart... reason i say this is because they are willing to leave their code half done, because they will just pass off the rest of the work to their reports. They are in for the big bang (look at what i did, but with minimal effort) and make the direct reports to clean up their mess.

Things to Test For:

Bugs - just put in lots of bugs. You have to bring them down a notch. Show them how crappy the feature is. Your goal is to put them on the top of the list for most bugs in their area. And you have to do this soon after the feature was implemented/checked in. Otherwise they will blame other dev's.

Dev/Test/Feature Match Up

Senior tester for sure. Someone that will challenge them.

Reactive Dev - Area's that these dev's own take the longest to stabilize. Dev's will just sit around waiting for bugs in their area. They won't look at what they have done, to see how they could do things better. Its typical when a tester finds a bug, they will only fix that one bug. They won't bother to look at other places in the code that have the same problem, they wait until the tester finds them.

Note: I am not sure if the Reactive Dev is because of the dev or because of the features they have?

Things to try:

I don't know yet. Funny eh? This type of dev can be created because of the situation he is in.
Integration testing - if the developer is doing a feature that is touched by everyone - example upgrades/migration (and he is the dev that got stuck with it). That does not mean that you have to be the only tester testing it. You should try to have pushes across the team, and have each individual tester testing how their area integrates with the feature.

OK - that's it for now - happy bug finding :)

About Time

2008-09-28T19:39:00.000-07:00

I think it is about time to do a brain dump of things that i have learned over the past 8 years. :)

Note: I never read articles or books about testing, never attended a conference. So I have no idea what other people are saying about Testing. All my posts are conclusions I have formed from analyzing the world around me. Any techniques I describe, are techniques I have used personally and have worked for me. :)