#21Days21Tips

On LinkedIn, The Test Chat Community recently ran a campaign called #21Days21Tips, asking software testers to share a tip a day. I hadn’t initially planned on joining in, but after seeing a few of the first day posts, I decided it would be good exercise for my brain since daily posting isn’t usually something I tend to do regularly. So, without further ado, here’s a compilation of my entries for this campaign.

Day 1: Learn to write for your audience(s)

Unlike programming, testing doesn’t have a default work product. The closest we get is bug reports, and even those aren’t universally used, or even desirable.

What IS common, is that whatever our work products are, they are usually some kind of written communication. So, it’s important to consider who we’re writing for:

-What is their level of knowledge about what you’re working on?
-What do they care about?
-Why do they need to hear from you?
-How can you be clear and concise, without leaving out information they may need?

Reading about writing, and practising it regularly, will help you get better. A couple of books I’d recommend looking at are:

-Weinberg on Writing: The Fieldstone Method, by Jerry Weinberg
-On Writing, by Stephen King

Day 2: Don’t just focus on tech stacks

I’m a generalist by nature, so new technologies are always going to catch my attention, and often at least a little interest. The thing is, technologies come and go. So do development processes, management systems, even testing methodologies.

By all means, learn the appropriate tech stacks to make your work as effective as possible. Also consider, there are many other aspects of software use that have very little to do with the stack, such as:
-User Experience
-Accessibility
-Fitness (Is this actually good for what it’s supposed to be?)
-Ethics (eg. Could/does this do harm to someone? Does it commit some kind of fraud?)

Learning outside of the technical world, and even the testing world, will help you become a stronger tester. You will have a more diverse perspective in your own head, and that’s rarely a bad thing.

Day 3: Don’t just confirm

Testing is about more than just confirming that some piece of functionality does “what it says on the box”. Certainly, that can be part of the testing effort. As a friend of mine once said, you want to make sure it does what it should do, and not do what it shouldn’t.

Now, what does “should” mean in your context? That’s something you need to figure out. Talk to everyone you can, and not just programmers or software managers. Talk to marketing and sales, talk to finance, talk to HR even! Anyone who might help you build a better model of the product is fair game.

Consider for example, an app developed by a car company, used to control the vehicle in certain ways, such as remote start, unlocking/locking doors, etc. Now, what it “should” do seems pretty straightforward. What about the things it shouldn’t do? Should it let you stop the car’s engine while driving? Should it let you open the tailgate of an SUV while the car is moving? If multiple users have the same app, should it let both of you send commands?

This is by no means a complete definition of testing, or even an approximation of one. This is just one small aspect to consider.

Day 4: ABTN – Always Be Taking Notes

Human memory is notoriously fallable. There’s a good reason it’s not usually considered good evidence in legal proceedings – it can be manipulated surprisingly easily, and even without outside influence, can be altered with the passage of time through other experiences.

Thus, it’s absolutely critical for testers to be taking notes as they work. Mind you, I don’t just mean during actual testing activity – I mean any time you’re doing something related to your work. Meetings, conversations, research, even daydreaming! All these can be incredibly useful sources of information.

How you take your notes is a matter of personal preferences. A lot of folks like digital tools like Evernote, Google Keep, etc. to track their notes. Personally, I like a paper notebook and a pen or pencil. I find the freedom of a blank (or lightly dotted) page much more useful for my way of thinking.

(shameless self promotion, I wrote a blog post about my preferences here: https://lnkd.in/dhDBX_Bm)

One other thing about notes – whatever your method or medium, be sure to keep track of where your notes are. It can be incredibly frustrating to know you have notes about something important, and can’t find them.

Day 5: Tools are good, minds are better

We all use tools to test. Arguably, even the computer you use to conduct the tests on software is a tool, albeit a necessary one. Automation is another tool, and like any other tool it can be useful when it is applied in a fit-for-purpose way. However, none of the tools out there are as important as your mind.

Consider a simple tool, outside of the context of software: a hammer. The hammer, sitting on a workbench or in a toolbox, is effectively useless without a human wielding it. Further, the effectiveness of the use of that tool is highly determined by the mind of the user: knowledge about how much force to apply to various situations (eg. driving nails into studs in home construction vs. fine furniture), where to hold the hammer to balance the power and control, etc. These are all things that depend on the mind – the brain, the nervous system, sensory systems, etc.

Now, consider the tasks that a hammer performs. The hammer might indeed be the best tool for the task at hand, but what if one isn’t available? Again, the mind comes into play. Thanks to our minds, we may be able to devise a way to handle the task without the necessary tool, or to decide to wait on the task until the tool is available.

Choosing the right tools for the occasion can be challenging at times, and it is easy to fall into a trap of thinking that the tool defines your work. You are not the tool, you are the mind using the tool.

Day 6: Be cautious of metrics

100% test coverage. 50 test cases. 18 bugs.

What do all these actually mean? In short, very little, especially without context.

When you see a metric, ask yourself:
-What is this trying to measure? Does the metric actually measure what it’s supposed to?
-Is what it’s measuring a relatively standardized unit of measurement?
-Is this a leading or lagging indicator?

Consider “100% test coverage”. 100% of what? Paths through the system? Functions? The tests we think covers the entire product (but realistically doesn’t – it’s impossible to test everything with limited time and resources)? It’s meaningless. Now, if you say we completed 100% of the tests we planned, that might be different. If the plan was agreed upon by the stakeholders, then they have some context as to what was done.

Likewise, with measuring the number of test cases or bugs, what is the size of either of these? Are all test cases and bugs of equal size, priority, and (in the case of bugs) severity? Of course not! Bug counts by severity MIGHT give you some indication on a current state, but even that is a lagging indicator if you’re interested in understanding the quality of the software.

As for test cases, some are going to be simple, and others are going to be complex. Should they all be lumped in together? If that’s so, what’s stopping your testers from making everything small, so it looks like they’ve completed a TON of testing, when in fact it’s been some pretty basic stuff and that’s about it?

As a side note, there’s a quote that crops up regularly: “You can’t manage what you can’t measure”, attributed to W. Edwards Deming. Thing is, that’s a misquote. The actual quote is “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” So perhaps, instead of burning a bunch of time worrying about measurements of things that can’t (or shouldn’t) be measured, find another way to get the information you need.

https://lnkd.in/daDBtc78

Day 7: Learn and Teach, Teach and Learn

Today’s tip applies to software testing, and can also apply to nearly any skill you have.

When you learn something, I think you have at least some ethical obligation to share what you know. If someone else has facilitate this learning, whether directly as a teacher, or indirectly as the producer of some kind of material, they have spent time and effort to help you get better at that skill. If you’ve learned something yourself, you should share what you’ve learned, because that’s how we all get better. Now, what that sharing looks like is going to be highly specific to your context – maybe it’s doing a talk, or making a YouTube video, or writing a blog post. There are lots of ways to share your knowledge, find one that works for you.

One of the upsides of teaching, is that you learn. On the face of it, this might seem counter-intuitive, since many educational systems assume that the instructor is a complete expert in the topic. The reality is, none of us are so expert at something that they cannot learn more. By teaching, in any format, we are forced to re-examine what we know, and how well we know it. As well, students are an excellent mirror to examine what we know, and how deeply we understand a topic.

Seek out opportunities to learn, including teaching, and it becomes a virtuous cycle.

Day 8: Change it up

Inattentional blindness can creep up on even the best tester, especially if the application you’re working on hasn’t changed much in a while. This is when it’s more likely that you’ll miss something that you likely would’ve discovered.

Here are a few different ways to combat this:
-If your organization has other applications to test, ask if you can take some time away from your current project to test another one
-Approach the application in a way you’ve never tried before; maybe no one has been worried about performance, but that doesn’t mean you can’t explore how the application performs under serious load
-Take a break! If you can manage it, take some time off work, and forget about what you’ve been working on
-Pair with a tester that is newer to your organization, and let them direct your work

Day 9: People first

You are a person, as am I, as is the programmer who wrote the code for the application you’re testing. In fact, everything we do in testing can be traced back to people, both for better and worse.

Understanding people is, I think, one of the most valuable skills a tester can cultivate. When you understand people:

-You are more likely to be kind when you find an issue, which helps to foster a positive relationship with the programmer, including them working with you to help make things better for your customers
-You are more likely to correctly anticipate possible issues, especially if you are aware of your own biases as well as others
-You are more likely to build a better mental model of what the people who might use your product care about, and focus your testing effort on those things

The late Gerry Weinberg said, “No matter how it looks at first, it’s always a people problem.” (The Secrets of Consulting). However, he also said “No matter how it looks, everyone is trying to be helpful.” (Quality Software Management: Volume 2, First-order Measurement) I think these two quotes can help remind us that while people may create the problems, people ALSO want to help.

Day 10: Know when (or when not to) stop, Part 1

Before I start, I’d like to say that I’ve been remiss in not acknowledging The Test Chat Community for kicking off #21days21tips. This has been an excellent learning opportunity, and I’m incredibly grateful for their efforts in building the testing community.

When will testing be done? I suspect that every tester has, at least once in their career, been asked this question. It’s a fair one – stakeholders are waiting on information to make a decision, and yours is important.

In a literal sense, testing is never done. We could test to infinity, should we so choose. With the restrictions of limited time and resources, we must choose when we will stop.

A common stopping point is some kind of deadline, often something at or near a release date. It’s also common to start with that date, and work back to try and figure out what can and should be tested in the allotted time.

There are also unanticipated stopping points. If you encounter a bug that significantly blocks your testing, or would make any further test results suspect, then it may be the right time to stop. Then again, maybe you can reasonably test around the issue. Making these kinds of decisions requires a lot of context about the project, so don’t be afraid to bring other people into the conversation.

Day 11: Know when (or when not to) stop, Part 2

You’ve just found a really cool bug, and you rush to see the project manager to get it assigned out to be fixed. What happens when they say “Nope!”

Bug Advocacy is something that testers should understand, even if they aren’t directly involved in the decision-making about what gets worked on. We must know when to push for something to be addressed, especially when we believe it to be a serious problem or something that we have a particular insight about.

However, we must also be careful not to advocate strongly for every single bug. People will start to ignore you; as the saying goes, if everything is top priority, then nothing is top priority. Further, you damage your hard-won credibility; why should someone take you seriously about something else testing related, when you’re constantly pushing for immediate fixes to minor issues.

When you’re about to go charging in, to push for a bug to be addressed, stop for a moment and consider the context of the project. What else are programmers working on? What are the deadlines? What is the severity of this bug?

Day 12: Understand priority and severity

How bad is a bug? Without context, that question is like asking “how long is a piece of string?” Not all bugs are equal in severity, nor is their priority always obvious. WE might think this a bug is a showstopper, imagining the possible negative scenarios and warning of those consequences, but that may or may not be the bigger picture.

Severity and priority are two different, though related, criteria when evaluating a bug. Severity is how bad the bug is. Severity can be further broken down to impact (how many users are affected), and likelihood (how likely is it that a user will encounter the issue). The matrix of impact and likelihood can be a useful heuristic to evaluate severity.

Priority is how important is it to fix it, as well as how timely that fix needs to be. One of my favourite heuristics for evaluating priority is the Eisenhower Matrix, which asks us to evaluate based on whether or not we consider the bug Important or Not Important, versus Urgent or Not Urgent. In my view, testers can express an opinion on priority, but should not be the arbiters of it, unless they will also be the ones fixing the bug.

Day 13: Treat all automation like code

Automation, regardless of whether it’s intended as a one-off simulation or repeated checks, is code. Even the “codeless” tools aren’t really codeless, they just abstract the writing of code from the user.

If we accept that automation is code, then it should be treated like any other programming project. There needs to be people invovled who understand programming, as well as people who understand testing. The focus should be to keep things simple whenever possible. Automation code should be stored in the same system as your product code, although it’s probably a good idea to keep it in it’s own repository.

Likewise, test your code. Make sure that the results you get accurately reflect what’s happening. To put it another way, don’t just say “The dashboard says it’s green, so we’re good.” Just because the automation may never be seen by a paying customer, doesn’t mean it’s not vital to make sure it’s a quality product.

Finally, remember that, just like in real life, the technical solution is not always the best one. Be sure you’re choosing the right way to spend your business’ time and money.

Day 14: Testing is a distinct discipline

Software testing is a distinct discipline, and practitioners of that discipline can play an important role in helping the product be the best it can be.

Programming primarily relies on a mindset of building, and that’s not a bad thing. In fact, I think it’s critical to being able to make decisions about what to work on, implementation, and trade-offs. There’s also a tremendous value in focusing on the positive – bringing what we believe we can make into reality.

Testing, on the other hand, relies on a mindset of questioning. We don’t seek to add anything to the product, although sometimes feature ideas are a byproduct of our work. Our goal is to understand what’s being (or been) created as best as possible, and communicate that understanding to anyone who needs to know.

No one, not even the programmer who wrote the code, can be completely familiar to the smallest detail with everything that their code interacts with, especially indirect contacts. Any of these contacts could potentially be involved in triggering a bug, and if the programmer spent all their time trying to deal with those cases, nothing substantial would get built.

Be aware of the natural tension between the two mindsets, and that sometimes one or the other will (and should) hold sway over a project.

Day 15: Requirements are fallable

Software requirements can be a useful guide to getting a sense of what stakeholders care about, which should guide what gets built as well as what should be focused on in testing.

The challenge with requirements, is that they’re imprecise. First, they’re using written language. Language, no matter which one is used, has a certain ambiguity. I suspect that most of us have experienced a situation in which we use a word that means one thing to us, and another thing to someone else. Written language lacks the tone of verbal communication, as well as the non-verbal cues that we aren’t always aware of.

Another lack of precision in requirements, is that they are always incomplete, and occasionally incorrect. The lack of completeness is primarily due to assumptions and tacit requirements. It would be difficult for anyone to document every single assumption they’re making, and even if they did, it’s unlikely that would be read closely by everyone. Tacit requirements face similar challenges around documentation. As for incorret requirements, whomever is requesting a particular feature may not have reflected deeply on what need they’re trying to address.

I don’t really have any tips on how to address this, other than to be wary of “testing to the requirements” without giving some additional thought as to what else might need to be looked at.

Day 16: Develop your interoception

Data is a good thing, and you should definitely try to gather the best quality data you can when you’re trying to make a decision. That being said, it’s not the only thing that should guide your testing. Feelings, and your senses, are at least as valid as data, and sometimes far more valuable.

You may have heard about “gut feelings”, or been told to “trust your instincts”. To me, these are referring to interoception, which is a sense that is often overlooked. One definition of interoception is “the sense of the internal state of the body. This can be both conscious and non-conscious.”

Interoception is part of the complex interplay between the body, brain, and nervous systems. If we learn to pay attention to the signals it provides, we can better identify when something isn’t right. Besides being a generally useful life skill, this can help with testing by letting us know that we should dig into a particular area, or that some behaviour of the software isn’t quite what it should be.

Mindfulness and breathing exercises can be useful in helping improve one’s interoception. Individual experiences may vary, so try different practices and find what works for you.

Day 17: Run shallow, run deep

Shallow testing, also known as touring, can be just as valuable as deeper testing.

Tours, especially when you aren’t familiar with the software, can be a great way to get a sense of the design sense (or lack thereof) , which can help inform the Consistency with Self heuristic. They can also alert you to some of the more obvious bugs, which tend to crop up in early versions, or when a new programmer has joined the project.

Deeper testing is also valuable. It gives you a chance to really understand the intricacies of a feature, what it interacts with, and where its boundaries are. This is where some of the nastier bugs can hide, and I think it behooves us to try our best to discover them.

Both of these approaches are useful, depending on the context of the project, and where you fit in that project. As well, if one of these tends to be your default, consider changing it up to help keep your skills sharp.

Day 18: Get out of your head

As I’ve said previously, software testing is a pretty cerebral job. You’re working with complex mental models, while trying to identify biases and risks, all in your head.

The problem with being so inward-focused, is that you can forget that our brains and bodies aren’t really separate; each requires the other working well for us to bring our best efforts to whatever we’re doing.

If you haven’t already, find some physical activity that helps you get out of your head. Yoga, running, weightlifting, and martial arts are just some examples.

FWIW, I like Aikido, because it presents an interesting riddle: in order to learn a technique, you must think and pay attention to what you’re doing; in order to perform the technique well, it must become nearly automatic.

Day 19: Keep costs in mind

Automated checks, when used appropriately, can be a useful tool as part of a tester’s overall strategy for testing a piece of software. Something that often seems to get missed, are the costs associated with implementing that automation.

First, there’s the cost of creating them. If you want it done in a timely fashion, it needs to be the primary task of one or more people to write those checks. Timeliness will also be impacted by their coding skill, as well as their understanding of the tools and/or frameworks being used.

A (usually) connected cost are the tools involved. Yes, Selenium and the like are FOSS (Free Open Source Software), but many tools are the opposite.

Another cost to consider is the maintenance. This includes updating the checks when something changes, periodically testing the checks to make sure they actually still work (ie. haven’t started sending you false positives), and reviewing the checks to make sure they still provide value.

As well, you may have to consider the costs of the testing environment. Mobile apps need to be tested in a mobile environment; there are some great platforms for this, so you don’t have to build your own device farm, but they cost money. If your development team is using a CI/CD pipeline, chances are good that builds are handled in the Cloud, and time for builds or checks in the Cloud aren’t usually free.

Do your homework before you jump into automated checks. You’ll be glad you did.

Day 20: Small things can make a big difference

One of my favourite bug stories is the time that the letter é in the username of an email address crashed an entire production database. When our DBA told me the cause, I was blown away, for two reasons:

1) Up until then, I didn’t know that characters outside of the usual English ones were even allowed in email addresses. Turns out, the RFC does allow for it in the username part of the address.

2) Something that small, an unexpected ASCII character, could cause so much havok.

Since that time, I’ve been far more careful about watching for small things, especially when it comes to inputs, but also when I’m told that something is a “small change” to the code.

Day 21: Relationships are at the heart of testing

You can have all the technical and testing know-how in the world, and without good relationships with the people in your project, your skills will be wasted.

Building positive working relationships with developers, management, and pretty much everyone involved in the project you’re testing can make the difference between success or failure.

Consider this scenario: you find a weird bug, and you can only reproduce it some of the time. You want to work with a developer to try and figure out what’s going on, so you can better understand why it hasn’t been reliably reproducible.

If you have a positive relationship with developers, they’re more likely to choose to help you. After all, you interrupting their work isn’t always going to be ideal for them. If you haven’t spent the time building those kinds of relationships, you’re a lot less likely to get the help you are looking for.