Testing In Production

I owe these thoughts to a couple of lunches with Trrish. She and I have both worked with various computer systems at CU for many years. For those of you who don’t work with computer systems, here’s a word of explanation about the title.

The way systems work goes is that you have a number of different environments that you use for different purposes. Say you have a web site, for example. You’ll have one copy of that web site, well, out on the web, where people can see it and use it. In systems lingo, that copy is called “production.” But you might have another copy that only you can see, which you use to add new stuff to your site. You’d call that copy “development” — it’s the copy of the site that’s still being developed. Once a new piece is developed and finished, you’d roll that piece into production. If you’re a big organization, with a big web site, you probably want to test that stuff out before it goes into production. But you’d want to do it in an environment that isn’t changing all the time (unlike the development environment), so you’d create a “test” environment. You may want an environment that’s exactly like production that you can use as a staging area, to make quite sure something works before you move it to production, so you’d create a “pre-production” environment. And so on.

If you want to try something new, you do it in your development environment, test thoroughly in your test environment (iterating back to development to fix the bugs you find), stage it in your pre-production environment, and only then do you move it into production. One of the cardinal sins in systems is to do your testing in production, because if the change you’ve made breaks something, that mistake happens in public, affecting your actual customers. Sometimes, for some ridiculous dysfunctional reason, you may find yourself having to do it anyway, but it is a bad, bad idea.

How unfortunate, then, for a systems person to become a parent, and discover that parenting consists of nothing but testing in production! Where should I send my kid to school? Or should I homeschool instead? Which doctor’s advice should I listen to about my kid’s issues? They don’t all agree. What battles should I choose to take on, and which ones should I let slide? How strict should I be? When I have an agenda, should I just impose it as an authoritarian? Be transparent about it and try to ally with my kid to make it happen? Subtly sneak it through? Which circumstances should change my answers?

I don’t know! I’m testing in production! Worse yet, if my changes introduce bugs, it’s quite likely that I won’t see them until much later, when lots of damage has already been done. There’s no way to test anything beforehand to see if it causes harm, so you just have to go with your instinct and hope for the best. It’s no wonder parenthood feels so nervewracking some of the time — you’re responsible for (to you) the most important person in the world, so the stakes feel incredibly high. You have no choice but to make critical decisions — and doing nothing is also a decision. And if those decisions blow up in your face, they do so in production — no test scenarios, no rollbacks, no dry runs.

My dad is very fond of this Calvin & Hobbes strip:

I guess this is just my version of that.

As a kid gets older, this metaphor may start to break down. After all, when there’s a problem in production, it’s all-hands-on-deck until that problem gets fixed. We don’t expect production to fix itself. (Well, maybe if production is Watson…) Increasingly, though, as kids get older, we do expect them to fix some of their own problems. At some point, intervention is probably detrimental. At what point would that be? Which problems should kids have to fix themselves, and when? Damned if I know. See above.