So is it a 5mm gap... or a 1mm gap???
Either way... the bushing is supposed to be "clamped" by the bolt in the knuckle in a way that the bushing won't pivot in the knuckle.
When a washer is introduced on either side of the bushing, (trying to fill a gap)... that creates a separate potential pivot point... which could cause wear on the arm or washers, (or knuckle) ...eventually causing slop.
Too hard of a washer... and it will rub/wear the knuckle... or the aluminum on the sides of the bushing.
Too soft of a washer... and the washer will wear... and you end up with play/slop.