Setting up Testing at an Engineering Org

Setting up Testing at an Engineering Org

Engineering organisations, those that predominantly focus on building new tech, have their own challenges when trying to set up testing. They’re less established in terms of processes, with the engineering teams being much more in charge of what’s built and how it’s done. So how do we work in these organisations and set a strategy for testing?

I’ve worked as the first tester in a couple of these types of companies, setting up testing from scratch, and here are the tips that I’ve learned along the way.

Your scope is wider and deeper

The testing within an engineering organisation will likely need to span across all functionality and deep through the stack. Engineers will want information about where the bugs lie from a technical standpoint in order to easily diagnose them and fix them.

DIG INTO THE LOGS – you’ll have to be prepared and plan to chase data through services and systems. Ask the teams to show you where the service logs are and how to access and interpret them, this way you can start to diagnose the issues seen and support your team in speed to fix them.

PAIR WITH DEVELOPERS – have a plan to pair with developers when an issue is found, this will help you to see how issues are diagnosed so you can start to do that yourself. This will help you to push your testing deeper and find more technical issues to engage your team with (adding value).

BENCHMARKING – it’s likely that non-functional and some areas of quality just won’t have been thought about before, so you need a plan to benchmark these areas. Understand how to capture actual information about the quality of something (what tools and techniques will help you) and have a plan for how to communicate your findings.

Your stakeholders are engineers

More frequently than in an enterprise organisation, your stakeholders are all going to be engineers and developers. How we communicate to them will be important in building rapport and engagement as unless we “seem technical” we can lose credibility and support. Engineers can be very opinionated and in an engineering org will have more to say about how things are run and work.

UNDERSTAND UNDERSTANDING – have an early plan to assess the lay of the land, what testing have engineers seen before and did it work for them (this will help us see where to apply education). Start to see what problems engineers want solving and who’s likely to be an ally vs. who’ll be a detractor, initially we can just focus on supporting allies for quick wins.

USING THEIR LANGUAGE -we’ll win over engineers if we’re seen to be technical (or engaged in the engineering side of things). That means having a plan for being able to talk through the stack and about engineering topics; I do this by asking questions and making notes or by pairing with people. Try to attend architecture and engineering discussions to learn more about the system too. Working to engage with engineering terms also shows that you’re interested in their work and can be a trusted advisor that’s on the same side as them.

PRAGMATISM – one thing that always comes up is being told “your bar for quality is too high” which may come from people just never talking about quality before. We need a plan to communicate what good enough means, when to push for quality and when to back down. Talk to people to understand a view of good enough and also whether the code being written is planned to be refactored soon (if it is, we can probably live with lower quality for now). Communicate ideas about performance and security needing massive refactoring and rearchitecting if it’s not built in and confirm whether that’s okay. Oh and don’t try to aim for fixing every single bug, know when something just isn’t an issue and leave it be (or document it separately as design / tech debt).

TALK TO THEM – you’ll do better when you talk to engineers and engage with them, rather than when you throw documents at them. Talk to the team about strategy, or have a call about bugs rather than documenting and sending just a link. This allows you to build rapport, allows the engineers to have a say in whether something is important and also unpick the technical side of the issue.

Things move much more quickly

You’re much more likely to have proper agile delivery and incremental change in an engineering organisation, which means we need ways to make sure we give ourselves enough time for testing. It also means ways of working can change from sprint to sprint because people are empowered to make changes quickly.

TEST JUST ENOUGH – there won’t be enough time for exhaustive testing so you need a plan to work out what to test. Work out what good enough means and set the scope of your testing (and priority) to keep on top of what needs testing. Make sure to communicate that scope to the team so they can support and have an input on priority calls.

WORK WITH THE PIPELINE – moving fast usually means a pipeline of merges (continuous delivery & integration) so the plan and strategy needs to work with that. make sure automation is in the same repo as the developed code, is called by the pipeline and can be used for meaningful diagnosis of issues (with failing tests preventing the merge or deployment). Don’t recreate tests that developers are already writing, talk to the developers about what to add as unit tests (or even help them by setting a scope of what should be tested).

KEEP COMMUNICATING – things change a lot, so information might become outdated; that includes information about how we test or what quality looks like. This means we need to make sure we’re kept up to date by attending discussions or having a plan to communicate assumptions and get them challenged. For example, use the daily stand up to find out if anything has changed functionally or architecturally (as apposed to it just being an “I did things” update).

TEST EARLIER – prevention is better than cure; do risk analysis on things earlier through Triforce (3Amigos) or risk analysis of work to find and prevent issues happening. Find ways to test headlessly and have a plan for pulling and running your own environments from branches to allow you to get testing as early as possible.

Testing isn’t a given

You can’t assume that testing is happening, the right testing is happening or even that people want testing in an engineering organisation. Everyone will want to do the right thing and create good code, but they might not see the value in testing or testers; this could be because of bad previous experiences or a lack of education for what testing can do. If there’s been no testing, then people may not know to call on testing for supporting / making decisions on “is this good enough”.

YOU HAVE TO SELL TESTING – one thing we need to do early on is to have a plan for how we market testing to people which means selling the value, scope and types of testing. This is a lot more about the where, what and why as opposed to the how (which is what we usually focus on). We have to work to win people over and make sure that people can see the value in testing, so we have to be pragmatic and tailor the messaging to make it meaningful to people. Remember that you catch more flies with honey rather than vinegar, so win people over with positive attributes of how testing can help and don’t try to blame, make people feel bad or suggest people are wrong.

YOU HAVE TO SHOW VALUE – don’t get too hypothetical and just write documents or give talks 9those are great, but don’t only do them). Be prepared to roll up your sleeves and show people your testing and testing outcomes first hand by getting stuck in practically. It may be better to hold a loose strategy / approach in your head, get some actual testing done to see what’ll work and then document a strategy for people; then they’ll have seen things and can understand it more. Don’t be tempted to write a theoretical approach using a template, create something based on the risks and needs that the team actually has: “There is this risk – we will solve that using these things”.

PUSH YOURSELF INTO DISCUSSIONS – as mentioned above, in an organisation where people aren’t used to working with testers you’ll have to push to be included. That means having a plan to (and being prepared to) jump into meetings and being present and engaged. You’ll have to teach people that testers will join engineering calls, or decision making calls and have an opinion or ask questions; if you sit in them all quietly and don’t put forward opinions or seek information then you’ll be teaching people that testers don’t bring much to the table. To this end, have a plan for what conversations to target and how you can help / what you want to get out of them so that you can be seen as adding value.

Things change a lot

Refactoring, changing requirements, changing quality needs and greenfield development all means that your product and code base is likely to change a lot. This is at odds to established products at enterprise organisations, where change may not happen frequently at all. It’s important to have a strategy that works with the changing nature of things.

THE RIGHT APPROACH – scripted tests will be less useful when things change a lot because you’ll constantly be reworking things, instead we need approaches that are more flexible. Focus on exploration heavy testing that still documents things in a consistent way so that you can uncover information but also have the flexibility needed for your testing.

THE RIGHT AUTOMATION – changing code and features means that automation is less your friend than usual because tests will become obsolete. Have less tests on areas that are more static and create closer to the code tests (unit / API) for ease of maintainability. Frequently I find that the UI is the most likely to change, with a lot of redesigns and placeholder objects, so potentially steer away from a UI driven automation suite.

FAST REGRESSION – lots of code merges means lost of risk of things breaking, we need an approach for fast regression testing (potentially giving us confidence in a couple of hours). Where automation may not be feasible, you have to develop other methods for regression testing like using exploratory charters, reusing your API calls from existing tests and taking a risk based approach to cover what unit tests / automation doesn’t. I usually look at some core features and provide a general health check of the happy paths of these in slack / a wiki page to show what was regression tested and allow for decision making.

You’ll have to do glue work

Glue work is those thankless tasks that hold a team together. Usually invisible, we may not even realise they’re happening (or have to happen) in a team, meaning that they don’t always have a defined owner. In an organisation where testing and quality hasn’t been happening professionally before, we’ll need a plan to identify bits of glue work that need to happen and to pick them up.

ARE PEOPLE COMMUNICATING – engineers might not talk to others and start making assumptions, meaning the risk of doing things incorrectly and lowering quality. We need to have a plan to ensure product and engineering are on the same page and communicating at the right level, a plan to make sure engineers are talking to other engineers and other teams to be aligned and that the team is communicating and demoing to stakeholders. This also means spotting where other testers may or may not be communicating and supporting that or even checking in with colleagues to make sure they’re okay and being heard.

ARE THINGS BEING DOCUMENTED – decisions, strategies, risks, assumptions, ways of working… who’s making sure these are documented and we’re all adhering to them? Engineers frequently don’t write down processes or socialise them, leading to assumptions being made and quality being impacted. You’ll need a plan to ensuring that guidelines and documentation happens, working with others or getting it done yourself.

MEETING PEOPLE WHERE THEY ARE – at least initially, because we’re on a sales drive for testing we’re going to have to meet people at where they are rather than in the middle. We’ll have to make the compromises, work how others want, educate ourselves on what others are doing and be a coach and trainer. That means having resilience in the short term and planning for how to move to where you can meet in the middle (or have people move over to your side).

More Blogs

LinkedIn