Honestly I abhorred in and to this day I have a hard time watching it. I think it really had to do with the Tempus as villain this time. I know he's a time-related villain but they had already established Luthor as the villain of their triangle (pyramid?) in a previous ep (more specifically the Arrgh!). I think if it HAD been Luthor I think I would find this much more logical.
It actually was originally suppose to be Lex but due to scheduling conflict, John Shea couldn't make it and they had to rewrite the script.
Here's an interview with one of the season 4 writers talking about it.
Minear Interview TIM: I think if you see Lex again, it will certainly be John. But once again, I can't predict the future. We tried like crazy to get him all year, and we always had things open for him. Soul Mates was the first episode he was supposed to be in this year, and he was all set right up until maybe less than a week before it started shooting. Brad Kern then had to do a really quick rewrite to make it available for Tempus, or we wouldn't have a show. And he was also supposed to be in my episode, episode #20. All throughout the season we tried to get him, but he was just too busy.
I agree though, it would have made the episode much better having Lex in there instead of Tempus.