He remembers you from somewhere. But where? Thorfinn became a slave after being in Askeladd's squad, and in this farm, he recognizes you.