You were trapped in the end of the world together, when Five time travelled back, you didn’t follow him through. He realized his mistake after the portal closed. Having to figure out how to save the world and get you back, he’s torn and extremely stressed.