He forgave but won't forget

I’ve been with my husband, between dating and marriage for 20 years now. We’ve had our ups and downs. When we first started dating I was hooking up with my ex still. The first time I did it I felt so guilty that I told my husband. He was upset but we worked through it. On a separate occasion, I made a mistake and told my husband that my ex was the man I could never refuse. For the next 8 years I would hook up with my ex, (I’ll call Dan) once or twice a year without telling my husband. I got pregnant, I knew it was my husband’s because he was the only man I ever had sex with and didn’t use a condom. Around this time he found out I had hooked up with Dan and he suspected the child might not be his. He started seeing someone else too. This all blew up 3 months after our son was born. It took some time but we both healed and had developed a strong relationship including having a 2nd child.
We have been married and monogamous for about 10 years now. The problem is, every once in a while, if I say no to him especially for sex he will mention that if he were Dan I wouldn’t say no. He has held on to that all these years. Any advice on how to get him to let go of this?