The primary reason to virtualize is to maximize the “bang for your buck” on your hardware. Containers are great, but have their limits.
So long as you have a desire to learn it (and the budget), I say dive in with Proxmox and see how you can put that hardware to use. VMWare ESXi is more common in a business/enterprise setting, but costs money to for anything beyond the basic functionality after the evaluation period.
I second Paperless NGX. I have been using it for a few years, and it has been working great!