That hole goes all the way to the inside of the cast area where the pump rod is. It's open and oil WILL squirt from it.
As you probably already know, it can make a big mess.
Like you said, it does it after you have dogged it a bit, as the pump rod rides on the cam it can pump the oil. This keeps the rod lubed in it's journal. If the bolt is removed the oil has to go someplace so it take the path of least resistance.
Get a new bolt about 1/2" long, add some sealing tape or paste and thread it in.