Is your “testing in production” strategy failing?
“ I don’t always test my code, but when I do, I do it in Production”
I’ve heard this too many times. And every time, I hear clanging bells. No, not the kind of bells at churches or temples. More like the ones at the fire brigade.
Yes, we know that technology is evolving at lighting speed and the time to market keeps tightening its girdle. Stakeholders want teams to release robust and reliable deliveries as soon possible. For some teams this means testing their application directly in production.
For some, this approach may work. For others this approach fails miserably.
Leo Tolstoy’s book Anna Karenina begins:
“All happy families are alike; each unhappy family is unhappy in its own way.”
This concept maps as well to software deployments as it did to dysfunctional 19th-century Russian families. When a rollout goes as planned, it looks like every other rollout: no errors to remediate, no disrupted services to restore, just new bug fixes or features available for use. On the other hand, if a deployment goes poorly, any number of scenarios might unfold.
All though there are many schools of thought on testing in production, the fundamental is if you are to settle for living on the edge you’ve simply got to make provisions to ensure you do not fall off the cliff with no protective gear on.
Pre-release environments are too sanitised: pre-release environment that testing is conducted on is not close enough to the Production Environment. The difference in configuration, datasets and assets in the production environment can alter the experience your end user can receive. Therefore, the closer you can test to the environment your user will be in, the better understanding you will have of your product quality.
Rollback strategy: if there is a strategy that allowed the team to continuously deploy, monitor the changes in core metrics and roll back if there was a problem, then you could take the risk to find certain problems later in the process. Even if that meant post release.
Fix forward: as a last resort, you might need to forgo the rollback plans, and simply fix the issue in-place. The challenge with deciding to fix an issue in place is until you know how to fix the issue — until you can diagnose and remediate the issue, and then deploy a fix — your customers will continue to experience its effects.
Automation where necessary: teams should have a selected test cases to run in development environment(s) and focus more on automated and monitoring to run in production. Quality engineers will then have more time to test the important areas in the meantime and give the right type of feedback to the developers.
