Why does he flirt but not make a move?

I have this guy friend- N. I’ve known N for a few years now, but only through his dad and brothers (all of whom I have worked with or currently work with). But our communications never went beyond seeing each other occassionally at the bar or in passing. About 6 months ago, N applied for a job opening where his brothers and I currently work and he was hired. Over the last few months we have become friends and gotten to know each other better.

About a month ago things changed drastically. N and I were hanging out at a friend’s house and we were left alone. He began talking to me and telling me about his life at home and his relationship with his brothers and his dad- and it’s not good. As he is talking to me and sharing with me, he’s crying and very upset. I do my best to comfort him and offer him support.

After that night, I’ve noticed my feelings for him growing more and more. He flirts with me like crazy at work- one of his brothers even made a comment to my best friend that it seemed like N has some interest in me- and always drops hints at going out and doing something outside of work. But when I say “Well if you want to go bowling/to the movies/to the bar just let me know. I’m down!” he doesn’t ever make a move.

I guess my big question is… why is he showing signs of interest, but backing off when I give him the option of asking me out? Should I just go for it and ask him myself? My only concern however is I don’t want to ruin our friendship or my friendship/working relationship with his brothers. I feel like he likes me, my best friend said she thinks he likes me, and one of his brothers even says he thinks N likes me. So why isn’t he making a move? I’ve sent very clear and obvious signals to him that I’m interested. I just haven’t outright said it due to fears of rejection/ruining our friendship.

How do I proceed with this?