I wouldn't call Harry a non-believer. He and The Man Upstairs may not see eye-to-eye, but Harry's certainly aware of Him and His power. How could he not be, having fought alongside Michael Carpenter?
More accurately, Harry feels like he's committing a taboo by involving himself in the affairs of a faith which, by and large, would like to pretend he didn't exist, or do something to remedy his existence. He doesn't like the idea that he might be seen as rubbing God's nose in it. "Hey, here I am! A spellcaster, doing Your job for you!"