Voice control: disable dictation mode

I'm using Voice Control on my laptop, and I'm facing an issue. Whenever I have the cursor in a writable area (e.g. Visual Studio Code, Skype or Slack chat prompts, etc.), the dictation mode starts writing random words. This happens when I'm listening to music or video tutorials, and it becomes frustrating to delete the unintended text. Is there any way to use only the command mode and avoid this problem? Thanks!