First of all, let me put your mind at ease. Having this thought does not make you a villain. It makes you a long-term partnered adult with a pulse. Most people don't wake up one morning and think, "Ah yes, swinging - this will clearly solve everything." Curiosity like this usually shows up more quietly. It's often less about wanting other people and more about wanting something back: playfulness, aliveness, the feeling that sex is still a place of discovery.
When I am with said people, we are both fully enjoying ourselves, but it is impossible for me to climax. At first, I thought maybe it was just nerves or my partner didn't have a technique that matched my desires (not a problem, that's where communication comes in!), but even after we part, neither my husband nor I can get me to climax.