I also agree with this idea, though I have serious doubts that it would be implemented.
I would also say that, were it to be implemented, moderators that won't do their job correctly could be "impeached", by either the voters, the other mods, or the admins, so it's not some tyranny by the voters who can constantly vote mods that won't do anything into power (though, having been around here for a few years, I think most members are logical enough adults that they would pick good mods).