This series of articles should give some ideas on how to make a good plan to handle errors in a radio or TV broadcasting environment. The articles should be read in the order they appear here, where this is the first article.
We broadcasters believe that we are so good at redundancy just because we have learnt the sentence ”there must be two of everything”. Piles of hardware have been sold and installed which just sits there, consuming power and doing nothing. Hardware ready to take over, in case of an error with some other hardware. This take over should preferably happen automatically without anyone noticing, and if some human interaction is required, then it should just be in the form of an “if it doesn’t work, press this” red button. This seems to be the idea of redundancy among many broadcasters. The result is that all this expensive extra hardware doesn’t help when it’s finally needed. In fact we could just as well turn most of it off. Then we must not forget that adding redundancy complicates things and sometimes also introduces some failures of its own.
Redundancy is fine, and for a radio or TV station where dead air can result in a loss of income or prestige, redundancy is absolutely necessary, but it has to be made right.
The basics is easy: Identify all devices that may fail and replace them with redundant devices that can detect that they don’t work and switch to a backup device. When you have got rid of all single point of failures like this, then you have a perfect redundant system. That doesn’t sound too hard, does it? But it will cost you a lot of money. At the same time, it is not possible to make a 100% redundant system – there will always be a least one singe point of failure left (and in reality a lot more) – this will be explained shortly. Apart from this, a lot of redundancy can make it very difficult to find an error, and too much redundancy can also introduce more error – sometimes more errors than you would have had if you had less redundancy. Perhaps some of the money saved because you didn’t buy all this redundancy can then be spent on qualified people who can do the right thing in case of a problem, instead of redundancy that has a much more limited set of possible actions to remedy a problem.
Many parameters go into the evaluation of how much redundancy you should have. What is the cost of down time compared to the cost of redundancy? How much down time can be accepted? Can some redundancy give other benefits, such as easier access to service it so that software upgrades don’t always have to happen at 3 at night? All this is worth considering, but will vary from place to place, so I will not go further into them here.
In the following articles I will discuss a number of issues that must be addressed in order to achieve redundancy that actually works. The cases here have been collected from my own experience or by looking at other industries. The broadcast industry has a lot to learn from other industries when it comes to redundancy and security. This is only natural – technical problems on a radio or TV station usually only gives some mess in the program schedule and perhaps some lost advertising income – but nobody dies or get injured, and usually an error on the radio or TV is forgotten very fast. Good for us. But it means that broadcasters are more undisciplined and broadcasting equipment is not tested thoroughly like equipment that would go into an airplane or a nuclear power plant.
