Am I jumping to conclusions?

Yes, you are jumping to conclusions!! You’ve only gone out twice. You have admitted you like each other and agreed not date other people and focus on this. That’s a great start! Now don’t start freaking out just because he’s had to cancel a date. Do you really think he would offer to talk on the phone and watch your favorite show together if he were going out with another woman instead? There is all kind of stuff going around where I live. We’ve had to reschedule a dinner with friends over 3 times in the past month because either my kids or their kids or one of us was sick!! My advice is to give him the benefit of the doubt until he does something not to deserve it. You can’t drag your baggage from other guys flaking into this relationship! No man wants to deal with the sins of another man. 
Everyone is nervous to get emotionally invested and get hurt. But it’s way TOO soon after 2 dates to be texting him something like that. You are going to sabotage this if you don’t get a grip on your emotions. You should be sitting back and enjoying getting to know him. And in your mind you should know you were fine a few weeks ago before you met him and you will be fine if this doesn’t work out. You are putting too much pressure on him and yourself this early.

I honestly don’t understand why woman feel the need to share their feelings all the time? Its a woman’s job to watch, listen and observe someone long enough before they can know if they are the real deal or not. This is basic common human sense with anyone you don’t know well enough whether its someone you’re dating, a new neighbor, a new co-worker, etc. Treat him the same way you do with any stranger and allow it to unfold naturally.

All you should be doing at this point-in-time is taking ‘mental notes’ to see if he’s just not feeling well and steps right back in, or fades out. When you let go of the need to control people and allow their colors to show early on it makes dating less stressful. I know its hard when you like someone but this is when you need to be less anxious or worried and just let it end early instead of dragging it out if its not meant to be dragged out.