Hmm, primer as a final finish? I like the idea of the matte finish, but...most primers, while heavy in pigments and other 'fillers' are very low in the varnish/sealer department. It will not protect bare metal against the elements for long as it is porous. The main job of primer is to form a bond between bare metal and paint, it gives the paint something to grip. Any bare metal that you prime should be coated with a paint of some kind, even a cheap spraycan of clear will extend the life of your metalwork.
Pro70z28 is right, block sand for a smooth finish, its more work, but looks much better and is easier to control for featheredgeing and avoiding scratchs. Good luck