Is Core Web Vitals just a ranking factor?

It will be a ranking factor in 2021. Obviously, it will already impact conversion as various researches are pointing out.

At the same time, pages that are passing the Core Web Vitals assessment will also see their chances increase of become part of the Google Top Stories. Having AMP pages isn't the only requirement anymore. Passing Core Web Vitals will do the job as well.

Chromium is always busy improving and catering to optimal user experience. They are experimenting with different types of fast page labelling. Passing the Core Web Vitals assessment might yield more benefits in the future as well. For example, reduced pay-per-click (PPC) costs.