It sounds like you have a vacuum problem. Do you have all of the vacuum ports either being used or plugged. Also, assuming you have a '55 or '56 distributor the vacuum advance should be ported vacuum (above the throttle plates), not manifold. These distributors - pre '57 are only vacuum advance - not centrifugal, so it really makes no difference at idle (static only) as whether it's hooked up or not, but during operation when you do have some venturi vacuum action, that is when you need it hooked up. A 650 on a 312 is a bit big but I have run bigger on them and successfully - but that's ancient history now !!!