Should I keep all Python libraries only in the virtual environment?

I am working on Debian Stable Linux (12.9) and it is otherwise working very well. I am also using Python3 for small data analysis tasks. I have been installing Python3 libraries in virtual environment using "python3 -m pip install pkg" command. However, on checking on Synaptic package manager, I found that Pandas and Numpy libraries/packages are installed from Debian repository and hence system-wide. Should I remove them from system and install them in virtual environment instead? I am worried that removing them from the system may break the system itself. Thanks for your insight.