Non Obvious ways to find bugs - Look at the Bug List

Here are someways to use the Bug List to help you find bugs and figure out the most unstable areas.



Its not just for tracking bugs. If you are monitoring the in-coming bugs, you can get alot more information about the feature and product state. Here are a couple of things I have learn to look for when monitoring the in-coming bugs.

FYI... all the top testers, always monitored the bug list - knew every bug entered and the cause of them :)

1) Old Bugs - look over old bugs.

Bug Count Per Feature - If you are inheriting an area, look for areas that do not have very many bugs. This could indicate that there was minimal testing done. No one ever writes a feature that is perfect... its not possible.

Bug Count vs Type of Feature - you need to look at how many bugs were entered in an area. Does the feature have UI or is it only api's? Does the feature have heavy integration with another product? Is the feature legacy code or brand new? The expected bug count should go up or down depending on how you answer the questions above.

Higher Bug Count
UI
Integration with 3rd party products
Customizable applications - UI and API level
New Features

Lower Bug Count
API (only because UI typically exercises this code, so it all depends to goes through the code path sooner)
Legacy Code
Code that has no integration.
Database level

2) Watch new bugs being entered - they will give you an idea of the types of bugs being found, and what to try in your area.

A)Investigate for repeats - If a dev made a mistake, see if you can apply it somewhere else in the product? Typically the same kind of mistake occurs in several places. These mistakes can be done by the same developer or across multiple developers. If its a very simple bug like a feature do not work with Unicode characters, then its probably a product wide bug, where the developers have not has sufficient training in this area. Make sure you see determine if its a product or isolated case.

B)Is it a very basic bug? - if the bug is very basic but the feature has been checked in for a long time... test it(its for sure a bug farm)!!! There are several things it could indicate when a basic bug is found late in the cycle -

1. Could simply mean the Tester is not testing their area
2. Area might not have an owner. I have found huge testing holes in the product because i looked into why a simple bug was found late in the cycle. Typically, holes occur when one feature is providing data to another feature, example web services. Who tests what? Typically both testers think the other is testing the feature, when neither are. Test Contracts can help, as long as everyone is using the same terminology.
3. Tester has become numb. Tester might have been aware of the issue, but no bug was ever logged. This happens very frequently, tester goes and talks with the dev or pm, and they punt on the issue saying... "well if you put in the bug we are just going to won't fix it". Unless you switch up the testers (get a new set of eye), these types of bugs make it to production... and cause alot of usability issues. So if new bugs are entered, and they are basic and they are entered by someone other then the area owner, I think its about time to switch up area ownership.
4. Tester might have known about the bug, but was afraid to put it in because it exposes the fact that they did not test their area. Again, changing area ownership is the best thing here... because the new tester will not be worried about covering their ass while testing. AND yes this totally happens in the real world.... because, there is no way to prove a tester did not do their job unless someone else finds the bug. And the poor performers are not going to risk entering a basic bug... its like shooting themselves in the foot - their job could be at risk - at least they will be interrogated by everyone on why it was not found earlier.

C) Is there a sudden spike in bug count? - and the spike does not correlate to feature check in! - I hate when this happens. Its cause by something external. Typically it happens a month or so just before review time. You need to take note of those testers that improve productivity around review time. I have noticed that simply testing in their area lights a fire under their ass! So a simple resolution is put bugs in their area ever once in a while just to keep them going.

D) Watch out for missing spikes in bug count - There should always be a huge spike in bug count right after a feature gets checked in. If there is not then something needs to be done right away. Possible reasons are:

• Tester is swamped with other areas. This is really bad. Give the area to a different tester, the longer the time between feature check in and fixing bugs, the worse the dev's are at fixing them. When a feature is just implemented, its all in the dev's head... is all in RAM. To delay putting in bugs, means that the dev's moved on and swapped out the RAM. To swap back in when your tester might have enough time is just bad for the product.
• Tester does not have a clue how to test the feature. This can be because they are new, do not know the technology, or not technical enough. This is where you want to pair the tester with someone Senior. They can help review the test plans or help create one, and provide idea's on how to break the new feature.

D) Random bug that does not make sense - Have you ever read a bug... and say 'what?'... 'how is that possible'... 'shit'... When i see these bugs, it usually means that i made an assumption that something was working or it worked in a particular way.

• You need to revisit your feature set, and see what the impact it has on your testing if you find you made an assumption. KEY is that you have to go back - when you get this gut feeling! I usually feel like the floor was taken out from under me, they way i visualized the entire system was wrong. You need to go back... and enter bugs, even if you don't want to because everyone will know you missed something. Better you find the bug then some other tester and better you find the bug before the customer does!!!

E) Regressing bugs with NO new new bugs being entered - When regressing bugs you should try to find new bugs. You should try to find 3 bugs for every 1 bug you entered. I know its not realistic, but it keeps you mind switching to 'verify mode'. You should still keep the 'break it' mode while regressing bugs.

• Testers commonly have bug regression nights, where they have to go through a crap load of bugs. You should review bugs that were closed by the tester, and pick out once that you know should have taken longer then 5 mins to regress, or ones that are high risk. Talk with the tester to see what they tried, or go around and play with the feature yourself.

I know there are more things to look for... but I can't think of them right now... :)

Non Obvious ways to find bugs - Look at the developer

There are alot of different ways to find bugs. Typically they are done by a theme like boundary testing, error testing, internationalization...etc. Then there are the non-obvious ways to find bugs.

1) Look at the developer - Understand your developers - Goal here is to find bugs, so I am listing traits to look out for. Note: i have made up these categories - so take them with a grain of salt :)

Just out of College - these developers make alot of mistakes but they learn very very fast. They typically don't test their code or they don't know what to test for. Since they are so new to writing code for production, basically any testing theme will find bugs.

Things to try:

• Error/Boundary - New dev's typically just follow the spec (if there is one). If valid input is 1 - 10, then that is all they will code for. They typically never code for crazy inputs like maxing out the field or value. At the beginning when they see bugs on their plate with 'pushing the limits' type bugs they will often fight back and say 'no user will try this', but just fight back. These bugs have to be dealt with in the code of because they will cause problems later down the road. They can lead to crashes, security bugs etc.
• Integration Test Cases - new developers are barely staying afloat, they are only familiar with the code they have just written. This is where your job as the tester comes in... its your job to teach the developer (indirectly through bugs) how the rest of the system works. It sounds funny to say teach, but the developers are sort of like horses with blinders on (can only see what is in front of them)... even if you tell them to watch out for something, they won't remember. That is where the bug queue will be their reminder and the teacher.
• All testing themes - accessibility, internationalization, localization, security... etc. For a college hire, this is all new, and the tester has to be VERY VERY thorough in testing their area.

Dev/Test/Feature Match up:

• I personally think it is good to pair up a new dev with a senior tester.
• Or you can give the developer a very visible feature. Where the entry points to his feature gets hit by alot of testers.
• Sometimes, the senior tester can overwhelm the new dev. So testers, be careful... you want to have a good working relationship.

The Cowboy's - these guys are the worst. They pump out code like no tomorrow - alot of code turn. They don't think or plan what they are doing. They break other peoples code alot, and they think they are the shit. What makes these dev's different then the principle/architect devs is that they never learn. Its like they keep going in circles. They can just as easily write code and then delete it and write it again, but with no improvement.

Things to Try:

• Automation: You have to save yourself from always trying the very basic tests at every build. With this developer, you will have to repeat your testing thousands of times. You have to automate your areas right away. Next step is to make a requirement that the dev must run your automation before he checks in.
• Regressions: Regression rate will be very very HIGH. These dev's never learn. When you get a build, you must have the mentality of - "for every 1 bug i regress i have to find 5". They will put their fingers into everything, so you have to also.
• Source Code Changes - you need to view source code changes for these guys. This will help you focus your attention on areas that they probably broke.
• Stay Close - tester has to stay very close to this developer. Try to get the developer to talk about how he is going to fix the bug. Provide areas that you think will break because of his implementation. This process will force the developer to slow down and start thinking before he writes the code.

Dev/Test/Feature Match up:

• You have to match a senior tester with this type of developer. The senior tester will know how to push back on bugs and they will be more efficient in finding the problem areas right away.

Weak Dev's - these are the dev's that are just not getting it. Typically the other dev's will know who they are, and will assign area's that are not high risk, and isolated. But all features are risky.

Things to Try:

• Security Bugs - even though the area might be isolated, and not a common case - the best security bugs can be found in these areas. Its almost like everyone does less... code reviews are not as tough, the tester paired up with the dev are not as tough on the feature. This is where you will probably find the best security bugs... just because the passion is not their... no one cares for it! And since no one cares for this area, the type of bugs that will guarantee a fix are the security type bugs! :)

Dev/Test/Feature Match up:

• Since these dev's are sort of disengaged from the rest of the dev's and feeling low, its good to pair them up with a tester that is a new hire. New hires are very excited, and they will help the sinking dev up a level. A senior tester will just break the weak dev even more.

Short Attention Span - These dev's don't have ADD, but they do have problems sitting in the same area for a long time. These are the devs that get bored really fast. The like to write new code, but hate to spend time to fix the bugs with them. And when they do fix the bugs, they cause alot of regressions because they don't take the time to property check their fix before checking in.

Things to Try:

• Depth Testing - you have to go really deep in your testing. Test all layers of their feature. Eventually you will want to look at the source. If the features integrate/use another product you have to understand what the other product can do... because your dev does not care. They are the reactive vs pro active dev.
• Fit and finish bugs - Since the dev will want to write the skeleton and move on to something new, you will find alot of fit and finish bugs. Example, xml is not human readable, alert messages are crappy, dialog looks like shit.
• Usability - the dev might have followed the spec, but that does not mean it's shippable. The Tester needs to look at usability/flow of the product. Does it make sense? Will end users understand how to use the feature?

Half Ass Dev - their bar is "it barely works". Everything they do is half ass. If you try a test, only the main scenario works and everything else is broken. Usually these devs have a chip on their shoulder or they are managers of other dev's. They either have been at the company for a long time, or they have had crappy testers in the past. They think they are soo smart... reason i say this is because they are willing to leave their code half done, because they will just pass off the rest of the work to their reports. They are in for the big bang (look at what i did, but with minimal effort) and make the direct reports to clean up their mess.

Things to Test For:

• Bugs - just put in lots of bugs. You have to bring them down a notch. Show them how crappy the feature is. Your goal is to put them on the top of the list for most bugs in their area. And you have to do this soon after the feature was implemented/checked in. Otherwise they will blame other dev's.

Dev/Test/Feature Match Up

• Senior tester for sure. Someone that will challenge them.

Reactive Dev - Area's that these dev's own take the longest to stabilize. Dev's will just sit around waiting for bugs in their area. They won't look at what they have done, to see how they could do things better. Its typical when a tester finds a bug, they will only fix that one bug. They won't bother to look at other places in the code that have the same problem, they wait until the tester finds them.

Note: I am not sure if the Reactive Dev is because of the dev or because of the features they have?

Things to try:

• I don't know yet. Funny eh? This type of dev can be created because of the situation he is in.
• Integration testing - if the developer is doing a feature that is touched by everyone - example upgrades/migration (and he is the dev that got stuck with it). That does not mean that you have to be the only tester testing it. You should try to have pushes across the team, and have each individual tester testing how their area integrates with the feature.

OK - that's it for now - happy bug finding :)