True, but I don't see that as much of a problem as I think most regulars play and rate both bad and good maps and if a person of the kind you describe give a fair rating, others will most likely give the same or similar ratings. If no one else rates it, his rating will be the only one that counts and therefore have a 100% effect on the calculated rating.
A system like the one I have described wouldn't do wonders and some might say it's unfair, but I believe it would slightly increase the accuracy of map ratings and implementation is therefore worth it.