Our product is a distributed system. The modules I work on are fairly new, quite rigorous, well tested. They were developed with recent best practices in mind. Other modules can be considered as legacy software.
While I’m vigilant about everything that happens within modules I’m responsible for, I’m under constant pressure to work with bad data sent to me from the other modules. At heart, I’m a “Fail Fast” principle developer and as a result , when problems arise I usually am able to eliminate the possibility of error in my modules. It’s not so much about blame, just saving wasted effort in chasing bugs in the wrong places.
But the argument I keep coming up against is: “We can’t let this stuff fail in production, the customer expects this to work, why don’t you work around this problem”. And this would be an argument for robustness: be liberal in what you accept, conservative in what you send.
I should also note that these are mostly intermittent problems. We see them in integration tests but they are hard to reproduce. Timing and concurrency are involved.
I’m having a hard time balancing between the two principles. Part of it is my worry that if I start allowing and propagating exceptional data, I’m inviting trouble and I won’t have as much confidence in my system. But I can’t argue against keeping the system working even if other modules are sending me wrong data. The reason other modules aren’t getting fixed is that they are too complex and fragile, while mine still appear clear and safe. But if I don’t resist the pressure, my modules will slowly be saddled with the same problems I’ve been rejecting until now.
I should say that the system is not “crashing” in production, but my module may simply display an error to the operator and ask them to contact support. A crash would be a big problem, but if I’m reporting the error clearly, then isn’t this the right thing to do? I suspect that my peers just don’t want the customer to see any problems, period. But my module is rejecting data from other modules within our product, not customer input. So it seems to me that we are just not tackling problems.
So, do I need to be more pragmatic or hold my ground?
Thanks everyone. The case that prompted this question ended well, and partly thanks to insights I got from the answers above.
My initial reaction was to stick to fail fast, but I thought about this some more, and had reached the conclusion that one of the roles of my module is to provide a stabilizing anchor to the rest of the system. That does not necessarily mean accepting bad data, but surfacing problems, isolating them and handling them in a transparent manner until we find a solution.
I planned adding a new handler and code path for this case, which would properly execute as if it was a special use case that was previously undocumented.
We had a discussion where I reiterated the need to deal with the problem at the boundary, but was also willing to help. I outlined my plan to the other side, because I had a suspicion that my position was viewed as overly pedantic, and that the solution was perceived as me only having to turn off spurious validation of harmless data, even if it was incorrect. In reality though, the way I work is largely data driven, so I explained why it has to be correct and how behavior is driven by it and how in accommodating this data I will be implementing a special code path.
I think this gave weight to my position and it led to a more thorough discussion of the other side’s aversion to fixing the data. It turned out that it was more of a weariness of dealing with an error prone legacy system than an actual obstacle. There was a relatively simple solution, it was just scary to make a change, a mindset that’s fairly entrenched.
But having aired all challenges and possible solutions, we eventually agreed to fix the data, and so far it seems to have solved our problem. Our integration tests are now passing consistently, but we have also added logging and will continue to monitor it.
In summary, I think that for me, the synthesis of both principles is that fail fast is essential for surfacing problems. But once they do surface, robustness means providing a transparent path to continue operation in a way that does not compromise the system. I was able to offer that, and by doing so, won some goodwill from the other side and got the data fixed in the end.
Again, thanks to everyone that responded. I’m too new to rate comments, but I do appreciate all the perspectives presented.