I do have a spring pulling the fork forward like drawn in on that picture.
Ok, I didn't know what you mean by "sw arm" at first.
My switch is like you think, it's just a little lever that gets depressed by the brake lever when it comes all the way up. It just stopping a tiny bit short and not depressing it all the way. I was thinking about just putting something on the lever (or switch as you suggested) so that it would get depressed sooner, but I wanted to make sure that brake returned all the way too.
That's why I figured a spring would help with that last inch it needs.
Plus the spring I have on the opposite side of the fork is pretty strong so I didn't think using a brake return in the other direction would do harm.