As long as the bearing is packed correctly there is no need to fill the empty area...