Telling Linux kernel *not* to use certain CPUs

I'm trying to run some benchmarks on a multicore machine and I'd like to tell the Linux kernel to simply avoid certain cores *unless* explicitly told to use them. The idea is that I could set aside a handful of cores (the machine has 6 physical cores) for benchmarking and use cpu mask to allow only benchmark processes onto the given cores. Is this feasible?