I was sort of thinking the same thing because I had read the book by Oliver Sacks (neurologist.) I forget which book of his it was that had the prosopoagnosia in it. I think it was "The Man Who Mistook His Wife For A Hat".
He had another section about patients who had an injury to the part of the brain where visual information is processed. That part was destroyed. So:
1) the people couldn't see anymore (the eyes worked fine, but the connection was broken);
2) the people lost all memory of seeing and all memories that were visual in nature
3) the patients were unable to actually conceive of the idea of seeing.
So when you said that the people in that universe all had a facial recognition deficiency, and they didn't realize it, I was thinking it was something like the above. They are incapable of even imagining that they have a facial recognition deficiency.
