I've been thinking a lot about testing methodology...
# thinking-together
t
I've been thinking a lot about testing methodology recently because I'm writing a piece on my blog about it, and I realized this community must have a ton of things say about it, especially theorizing the fundamentals. I care about this a lot because in my experience a lot if not most time is spent testing, and doing testing better is a serious productivity improvement. For example, when should tests be deterministic vs random? When should one use unit tests vs integration tests (I suspect it's much deeper than "lots of unit, few integration")
g
i’ve found that all kinds of tests are needed. What a type test does not do is just as important. Integration tests should not check do in depth correctness of the responses. Unit tests should be limited in scope. micro benchmarks for algorithms, macro benchmarks for cpu/memory tuning. squeeze testing for testing failure condition. sandbox testing with artificial traffic. statistical analysis of production traffic on canary deployments. there is no silver bullet.
👍 3
testing of the human element is good as well. run scenario of typical/expected failures and test the procedures for handling them.
w
Regression testing: if it broke once, it's liable to break again.
a
If you do random tests, for the love of Cthulhu keep the seeds.
☝️ 3
j
I often recommend Chapter 6 of The Practice of Programming (pdf here) for hard won wisdom on testing.
💡 1
k
@Jack Rusher That link was useful and timeless enough that I made some notes for myself. Thank you!
❤️ 2
Here are some opinions I wrote down yesterday, which happen to be disjoint: * Tests aren't always useful. There are legitimate reasons to avoid automated tests in some situations. * Testing first is a great way to get into the habit of writing tests. But the key is the tests, not when you write them. * In each test, take care to separate the thing being tested from setup and checks (I hadn't heard of AAA; thank you) * On conventional stacks, try to write code in a way that minimizes the need for testing I/O (my Mu is designed to eliminate this bullet) * On conventional stacks, test one piece at a time more than whole assemblages (also "called 10 unit tests for every integration test"; again obsolete on Mu: http://akkartik.name/post/tracing-tests) * Never DRY tests, only production code. With tests it's more useful to be able to see everything at a glance. If the test is too long, perhaps you need better interfaces in your production code. * Coverage is useful as a signal to check once in a while. It becomes less useful if you constantly look at it. If you happen to look and find that coverage is low, focus on doing the above bullets more diligently. 100% coverage is not desirable.
👏 1
t
What's Mu?
k
I'll DM you a link so it doesn't distract this thread.