You need only remove the loose stuff. Then paint over whatever rust and metal is left behind.