Your logic seems sound, but is flawed.
There weren't any left-handed daimyo (or any other rank).
The Japanese are a very organised and regimental race. Probably less so now than then. Left-handedness simply wasn't tollerated.
Swords were always worn at the left hip. This meant that those wearing swords knew how to pass each other by (right side top right side) without their tsuka clashing. If they did then that would be taken as an insult or a threat and a duel would have to ensue.
Throughout the Japanese Martial Arts today, you'll see that all bar one aspect is done ambidextrously (probably thanks to Professor Kano of Judo, Professor Funakoshi of Karate and O-Sensei of Aikido.) The one notable exception is sword work (and other weapons, such as, I think, naginata). All sword work is done right handed. Always.