Being a concubine of an emperor that favors you and doesnt treat the others harshly isn't as bad as it seems. Maybe rumors are really mostly considered lies for a reason.