How can we create an isolated addreess space per process without highmem in 64bit?

my question may be silly. But even after some searching, I still didn't get why highmem is unnecessary in 64bit architectures. As far as I searched, in x64bit, there is no highmem. In other words, all the physical memory is directly mapped with virtual memory. But at the same time, all the processes will have their own address space starting from 0x00000 to sth for various reasons including security. If all the physical memory is already directly mapped with each piece of virtual memory, How can we create a separate and isolated address space for each process without changing mapping between physical memory and virtual memory?