Losing a bit of vacuum via an intake leak would affect your idle speed, wouldn't it? I wonder if you have an oil leak that is also sucking air?