Sounds to me like your new battery ain't so new. You have a dud. If you bought it new, bring it back for warranttee, if you bought it used, go buy a new one. Your alternater is working because after it runs for a bit, the battery has enough charge to start it. However the battery runs down after a short period.

Other than that, you might have a short.