I am not sure since when (probably quite recently) it is possible to grant direct resource access to federation principals without using GCP Service Account as a middleman.

It would be great to support this use case here.

When I try to skip service account email annotation I'm getting this error

Which at the same time is wrong (should point to service account annotation, not expiration annotation).