Welcome to CHR!

To follow up with Dave's suggestion - looking at the front of the carb, the vacuum port on the left is timed and vacuum comes on after idle. the port on the right is manifold vacuum, i.e., present at idle. I'd recommend the one on the right. Also - as simple as it may sound, perhaps there is a vacuum leak somewhere as it certainly sounds like a vacuum issue. Try a little squirt of card cleaner around the base of the carb and if there is any difference in the idle speed you have a leak. Also go along the intake manifold where it meets the heads. The manifold bolts are notorious for coming loose and can cause vacuum leaks.

Good luck and keep us informed of what you find.

Again, Welcome to CHR,
Glenn