That main valve should shut it off tight. The only time we have had a loss of the gas is when we forget to turn off that main one. Since you are swapping out tanks each time it seems to rule out a valve problem, unless, unknown to you, they are simply refilling yours and giving it back to you. Those small tanks don't hold much, so you are probably just using more than you thought.

Don