Most of the new high solids/low VOC primers and paints lay on with quite a bit of orange peeling. You could probably thin a lot more than 10%, but clean air laws limit manufactureres to the amount of volatile hydrocarbons in the paint, that is why they list such low numbers. Before thinning more, though, I suggest you try a less volatile thinner such as MEK. MEK will give the primer a little more time to level out before it starts to set. Check with the manufacturer for compatibility.

For future reference, you can lay down any epoxy primer on fiberglass. After that, use whichever paint system you want. The epoxy primer will act as a barrier which prevents chemicals that may cause fiberglass to deteriorate from coming in contact with the fiberglass.