Tech's right, manufacturing tolerances aren't exact enough to permit cam and lifter swaps like that. The lifter bores can be a bit out fore and aft, too, which causes misalignment of wear patterns.
You state that it had lifter noise, that'll speed up the wear, too. Just bite the bullet and tear it down to tiny pieces and start over. It'll need a thorough cleaning.