Training items add EXP gained when training characters. Since they're also considered valuable, use them on your Rare characters up. You can use up to 5 training items at once and you can even select the same type. You can get more Training Items by stepping on Item Tiles in quest stages, as daily log-in rewards, or completing missions.