Grep only numbers, not the alphanumeric entries

I have a list of values like: 1 2 3 4 Ak123 Ak23 Ak147 1Apple 2Apricot 3Mango 4Orange I just want to execute a simple grep command to list me only the numbers. i.e. 1, 2 , 3 and 4. I tried this command - grep -Ein --color '^\s*[0-9]' test.txt but it returns the alphanumeric also. Instead I tried omitting the character by this command: grep -Ein --color '^\s*[0-9][^A-Z]' test.txt but it gives 0 results.