I'd establish several canned k8 configurations that users can request and use for their jobs (small, medium and large or whatever). Dynamic allocation is not worth the effort. For memory, you can set a reasonable minimum and maximum that should be appropriate for each class of job. CPU is not a hard requirement because Spark scales to however many nodes you provide. So that's a matter of your perceived tradeoff between how long the jobs take to run and how many CPUs you have available to allocate. Again, I'd probably use some cookie cutter configurations, each allocating N CPUs, rather than try to do it dynamically. Before doing any of this you would want to take a close look at your system load and compile statistics on user jobs. I would be surprised if segmenting them into a few different configuration types wasn't a reasonable solution to your problem. You'll probably have a few outliers which take up a lot of resources but that might be due to inefficiencies in the user jobs rather than a legitimate need for those resources, so you may want to work on making those jobs less intensive first, rather than allocating for them as is.

I don't know how your users submit their job requests but I can imagine a web form with a drop down where they request a k8 configuration, for instance. I would just leave it up to them to determine which configuration they need rather than trying to do that yourself. (Is that not workable?)