AITA for accusing my fiancé of cheating?

I might be bat shit crazy here and if you feel I am, just tell me please. I feel like I'm losing my fucking mind. Back 6 months ago my fiancé convinced me to move back to his home town. I should have just stuck with my "no" but I gave in because I was due to have a baby in 2 months and moving back to his home town ensured that we were surrounded by family in the event of us needing help (including my family, as they were now 40 minutes closer to us). We moved away from this area 6 years ago because of the drama and the drinking habits (all of our friends did nothing but party 24/7 and to be blunt, both him and I were too weak to turn it down and we wanted to change our lifestyle).

Now, since moving back here he has gotten close to a guy he works with. We will call him Joe. This guy is 12 years older than us, has 6 sons, married, owns his home on the water, has a bunch of "toys" (dirtbikes, sleds, four-wheelers, boats, etc) and my fiancé is soaking it up. He now gets to do all this stuff that he never got to do before and has a buddy who constantly offers to do cool shit with him on his dollar and it's heavily impacted his morals (imo) because now all he thinks about is what cool shit he will get to do next. And make no mistakes, I AM invited, but it's not fun for me to sit there and watch them drink and play on all their toys.

But when I started declining invites, this guy's wife did too because she is now pregnant. The guys have, in the past 6 months, stayed out until sometimes 5am drinking and partying over to random peoples houses that this guy knows at least 5 times. Twice since I had our child. My fiancé is in full contact with me the entire time but that makes zero difference. When he is here he's a dynamite dad and partner. But if his buddy calls, he jumps and runs basically. Well... last week I had a girlfriend over and my fiancé picked her up for me (she was drinking). He had his buddy with him. Apparently this guy told my friend that he cheated on his wife a month ago and gave the excuse that he was drunk and he never would have done it sober. His wife does know, I guess. But the thing is, a month ago my fiancé was constantly with him. He was distant and acting super depressed all the time. And then for 2 weeks, he refused to go with this guy at all. So I asked my fiancé why this guy felt the need to tell my friend that to begin with and he started clamming up and giving me half assed answers and in the moment, I said "you were acting weird all last month and you were with him constantly, so you knew he was cheating on his wife because you were with him". He just went silent. So I said "did you cheat on me?" (His buddy is very open about getting his friends involved in 3-ways in the past). He kept saying no, he didn't and I just stopped asking questions because I was sick to my stomach at this point. But he's been acting weird ever since and whenever I asked him what was wrong, he would just say he was tired. But last night he finally said "I can't look at you the same after you accused me of cheating". Considering he is with this guy constantly, stays out for as long as he pleases and given the circumstances, I feel it was safe for me to ask questions. But either way, he thinks I'm wrong.