I bet their unit tests are all green

I never trusted unit tests. Unit tests are made to test what you already write in code – you add an if because you mentally want to test something – one does not test everything because it’s next to impossible and absolutely impractical to do so.

Today I have two examples. The first is the Youtube Playlist – as some of you know, I made a list of 290 items with Romanian pop music. And I noticed something: after the 100th item, the list in the right kept jumping one more than it should. So much so that I even notice how they jump: the list in the right starts at position 99 + 2*(100 – index).

I’m not sure they fixed this bug, I don’t even care about it, but it’s funny because they also use the list in the right to display thumbnails for the next song (they don’t have a button for the previous song because their mom was mean and put them in the corner often, and who cares what the clients want?). And I was talking about this with a programmer friend, and we discussed how I’m pretty sure that all their unit tests will turn green.

A similar bug with Valve’s Steam – as a test I reinstalled a lot of games (you can now, by selecting multiple items in the games list, finally!). Their list management was a mess – and still is a mess, and you can prioritize certain downloads, but the list just doesn’t work as it should. The bugs are too many to mention, really, from bad rendering of items to bad data management, but suddenly, after going under 40 items, the list worked smoothly.

Again, I’m sure their unit tests work perfectly.

So what’s my problem with unit tests? Two problems, in my opinion: the first is that one writes unit tests that will never fail (because they mentally tested that when implementing things) or unit tests that will fail too fast, as soon as you try to refactor something. And don’t tell me that unit tests make refactoring easier, because they don’t. I am yet to see a project that runs better because of unit tests, and I’ve seen a lot of projects.

It’s the reason I hate today’s development techniques – they promise but don’t deliver. And things don’t really work as advertised. Engineering by buzzwords doesn’t deliver.

19 Replies to “I bet their unit tests are all green”

  1. Unfortunately, some developers are using similar arguments to avoid writing any kind of automated tests (whatever the name – integration, system tests etc).
    In my opinion, “unit” tests are useful when they tests invariants, real business rules, not implementation details – ex.: if I had to write a method that finds the optimum path in a graph, I might write several unit tests – they won’t prove that the algorithm is correct, just that it’s not broken in several common cases.

    As you said, unit test cam quickly become a burden when they test stuff like – “this controller method is calling that repository method when.. ” 🙂

  2. It’s the reason I hate today’s development techniques – they promise but don’t deliver.

    In the case of unit tests I think the problem is that people assume they promise something, without bothering to read what it actually says on the tin. Unit tests are nothing like formal proof of correctness, and I don’t think any lucid person ever said they were. That doesn’t make them useless.

  3. Automated tests are useful when they are enforcing business rules which otherwise, for a developer, are not clear from documentation (if any).

    I also saw developers basically testing if new keyword gives back a POCO object. Utterly useless.

  4. I hear very similar arguments quite a lot. I think you are missing the point of unit tests here. I think they are absolutely crucial for any professional software project. I just want to comment on the 2 major flaws in your logic.

    1) “tests that will never fail” – unit tests are needed to test that the coding I have written (or will write – TDD) is correct. You are, right the test are always write if you don’t make mistakes. With the same argument, you can state that “double entry bookkeeping” is a waste of time – in the end it is the doing the same twice.

    2) “fail as soon as you try to refactor something” – in that case your unit tests are simply written poorly. The main purpose of unit tests is to enable refactoring. If you are testing the public API of this unit, you should be able to refactor the whole implementation without breaking the unit tests.
    Refactoring is improving the internal structure without changing the behaviour. Without unit tests – how do you want to assure that the behaviour is not changed?

  5. Everyone who says that they can write unit tests better doesn’t really practice what they preach – I’ve seen the unit tests of TDD evangelists and they are pure crap. In theory yes, things are better with unit tests, in practice they are better off without unit tests.

    I’m not for „not testing” – tests are a good thing. But if you test that a certain piece of code does what it should, if an if ifs correctly and if a for loops correctly, that might be a waste of time, and it generally is as it will not allow you the freedom to change things significantly.

    Component testing is the way to go. Yes, automated tests. I believe in automation. But tests are not fixing bad programming, no matter what Uncle Bob thinks.

  6. Yup, that’s what I’m saying.

    Point is that if a huge business cannot handle a 100+ items list, how can they handle security? I bet their unit tests are all green there as well 🙂

  7. I am not saying that my unit tests are perfect, but they are worth the time. They help me during development and later safeguard the code again changes from others or myself in 6 months.

    Again, there are quite a few things I could mention, but I will restrict myself to 2 again

    1) I hope we agree on that code must be executed in order to be sure that is does what is expected. That is the more true for languages where you don’t have a compiler to find at least obvious mistakes. But if your application takes a lot of time to start, unit test can really be beneficial in keeping your roundtrip times low.

    2) It is much cheaper to test many different aspects of your unit with a unit test. I agree that testing all of them is not realistic. With a larger application, it is not feasible to do this with other tests. Even if possible, it is often too time consuming.

    In some environments, unit tests are crucial. In others they might be nice to have, but in any case I consider them to be best practice. Just like “double entry bookkeeping” for accountants and “hand washing” for meds. Both professions took quite a long time to accept those practices.

  8. We agree on both points. However, the UTs are not a guarantee for anything. That’s what I’m saying here – they unit tested, and I expect Google to be the kind that unit-tests a lot (because their developers have enough time, their firm doesn’t really create many new products).

    The point I’m trying to get across is that you create tests for things that you know they should work (let’s not discuss TDD, TDD is just brain-damage). There’s no „good way” to write your tests and make sure everything you write is bug free. Not even 100% code coverage can guarantee this. I’ve seen 100% code coverage with tons of bugs. And at the scale we build software today, with a gazillion frameworks under us, there are no guarantees.

    I’m absolutely for automated testing, I’m only laughing at the misguided people who think automated tests are enough.

  9. Property tests for the win. Golden tests instead of unit tests where there are no real properties to check.

  10. I have a different opinion on TDD, but I have my doubts we are talking about the same. Talking about TDD when you don’t see the benefit of good unit tests (yet) is kind of pointless anyway.

    I agree on the point that unit tests are not everything. Not even automated tests. Therefor the “test pyramid” is seen as best practice. It includes some manual tests on the very top. But it relies on a good foundation of unit tests.

    I just want to point out one more misconception I face often: “because their developers have enough time”. Unit tests are nothing optional that can be thought of on top if there is time. In my eyes, it should be part of proper work. I can think of no other profession where you would label the verifying of their work as optional if there is time. Mason, carpenter, electrician – you name it.

  11. Fairly big project, they were testing getters and setters just for code coverage sake. All tests green of course!

  12. I started to write unit tests because (surprise!) I wanted to test my code before integrating it into the product. This happened pretty late in my developer life (like… 4 years ago, and I’ve been in the business for ~14 years). After that I noticed that my style of writing code changed, because I wanted my code to be unit-testable. I write unit-tests because I can catch my bugs earlier.
    …but to make sure that my feature is reliable I always write integration tests.

    “It’s the reason I hate today’s development techniques” – unit testing is not a “today dev technique”, it’s surprisingly old. Also, methodologies don’t “promise” anything. They’re just process descriptions and it’s up to the developer (manager?) to follow them “to the letter” or not.

  13. The discussion is pretty old I guess and the debate is old enough, that no one actually knows what business value brings to the table and I am talking about unit-test and test coverage.
    Now if you are writing tests just for the sake of writing them and have them all green, than you should just drop it as it is a waste of time and you are far more happy to do other things than this. However if you are writing test, in a critical process of your app and try to narrow all the corner case in that particular part of code, than kudos to you, but still, some odd narrow case may happen and you still did not cover it. In that case your are happy because you can write a unit-test that imitates the bug and see what happens, also the modifications inside the code will make you fell a little bit more secure when you write all the unit test again.

    So to wrap it up, unit-test are having some utility, but not as a big one as everyone thinks, it is not a silver bullet and especially when you are working in a big app with a lot components that need to be in place just see if a small feature is valid or not. Regarding the code coverage, this metric is also a pretty faking one, in terms that you can write a single unit test that covers everything, but does not test a damn narrow case.

  14. Methodologies do promise a lot of stuff, all the time, because that’s why you use them. They don’t deliver, because there’s always „you didn’t do it right”. It’s always easy to tell people that they didn’t follow the methodology right.

    I also write unit tests because it helps me test my code faster than just running the application. But I don’t rely on unit tests exclusively, and they become legacy way too fast for my taste (I like to refactor things often). I tend to do more integration tests than unit tests, because units are quite easy to check, but even if each unit works perfectly the integration is most always problematic.

  15. Yup, that’s pretty much my point too – code coverage is not as important as people think it is. I’m going generally for integration testing, and development techniques that will allow me to avoid a series of bugs that are most often to occur in my type of application.

    The point with unit testing is that it misses the things that the developer missed while implementing as well. If you never care to check for null pointers you probably wont unit test with nulls either. UTs with 100% coverage work when the only branches happen inside your code. But some branches, some exceptions, don’t happen because you have a branch somewhere and a flow that you didn’t test, but because the data you input is not well suited for your flow. And refactoring becomes harder, despite what the big guys say, when you have a gazillion unit tests testing (probably) the things that would’ve worked anyway.

  16. Your rant makes you come across as a developer that has been told he needs to write unit tests, but can’t bring himself to do it, and now tries to find excuses for not doing it.

    “I never trusted unit tests. Unit tests are made to test what you already write in code – you add an if because you mentally want to test something”

    Unit tests are not only about testing what you already wrote in code. Most of their value comes from what they can offer to the next dev that sees your code – the senior dev that is called to refactor it, or add features on top of it, or fix your bug. And “next dev” is in most cases “you from the future“– you, returning to some code that you don’t even remember you wrote.

    And when you look at some code for the first time, trying to fix a bug in it, it’s nice to be able to open, on your second monitor, a set of unit tests that list everything the initial dev claims their code does. For example: “List should jump to the next item”. Because you can then:
    – add one more test (“List should still jump to the next item after the 100th item”)
    – see your new test fail
    – change one line in the code to fix the bug
    – see all tests – including your new one – pass
    – get the hell out of there 5 min later

    Try to see software applications as alive and evolving. Try to think “I’m doing this for future me” when writing your next unit test.

  17. Hi, thank you for the involuntary compliment you paid me, thinking I’m some young, inferior developer. Thank you for the advice. If you want to know where you’re wrong do let me know.

  18. The reality, at least mine, is that none of the test categories are a true answer to software quality. But altogether they can increase quality to a level you can be comfortable with. You start with static code analysis, perform code reviews, run unit tests, then component tests, functional tests, performance etc. Each step in a continuous cycle will bring you closer to what you define for yourself as a quality target.
    So yes, only unit tests are probably useless. It takes a lot of discipline and experience to write good code.

Comentariul tău