Can't hear people unless switched to speakerphone on Galaxy S4

I have Samsung Galaxy S4 and I cannot hear callers unless the speakerphone is on. The caller can hear me well but there is only a very faint which is inaudible sound back. I have gone to all the settings but it does not change anything. How can I fix this?