That's a good point. Guess I'm naturally biased with Matlock being a ref, but the quality is very much impacted by the pool you have to choose from. The sad reality is that most refs pack up long before they get to the FL, or even the NL.
These days, the abuse starts with aggressive parents on the touchline watching their children play. If you can be arsed with dealing with that, you can gradually make your way up the steps, and move on to bring abused by adults instead. So that could take the form of yelling, swearing, being shoved, being hit, having some fat bloke take his shirt off to hit you (why do people do that?), death threats, and being threatened in the car park after the game. Really, why would anyone bother to give up their evening or part of their weekend for £25 - £30 plus travel just to face that?
All of the above are regular occurrences, and unless you're exceptionally passionate about reffing or stupid, why on earth would anyone do it? That's the root of the problem.
Most leagues are desperately short of refs, and that's why. By the time you get to NL level most refs will have dropped out (including many really good ones) and there's a small number left, some of whom won't be of the standard we'd all want.
Also, afraid I don't buy the biased refs thing at our level, it's down to human error or incompetence.