If you and your car can deal with the weather, go it will likely still be a good time. Especially if you like shopping the swap meet or vendors. If it raining Saturday night, they will likely cancel the track run. Cold weather doesn't damper the atmosphere but rain will.

If it's pouring with no sign of relief, then skip it. The infield has a habit of becoming a mud bog. Unless of course, your car is a mudder.