Neighbor hit on me (not physically) made me have flashback of sexual abuse?

My neighbor tried to hit on me he is 54, married, his children are my age how do I tell him no today he was fixing our fidge and asked me for me and my mothers number, he called me an hour after fixing the fridge and said he wasnt living with his wife anymore and asking me how old I was when I said I was 27 he said wow I didnt know you were that old so when I thought this conversation was gonna get a bit out of neighbor mood I said to him Im gonna let my mom know you called and I got off the phone... you would say go tell his wife but my other married neighbor is always at her house when the neighbor who called me is gone.... I dont get down like that but I dont wanna rush off the phone without given him a concrete dont call me anymore so what should I say...I told my mom I dont want him to do any work for us anymore because him hitting on my took me back to older men hitting on me and a time that I was sexually abuse so it scares me... but she laughed and said he hits on every young girl around but I am not comfortable am I being too serious about this