Nah. 20 to 23 isn’t a big deal. When I was 18 I dated someone who was 20 and by the time we broke up he was 24 and I was 22. Those differences get even less significant the older that you get.
I would maybe run it by your friend before you ask out her brother, but other than that, good luck! Go forth and be brave!