You built walls around yourself after a relationship that broke you. He's not breaking those walls, he's walking straight through them.