Nexus 4 phone speaker not functioning

The speaker phone of my new Nexus 4 is not working. I am able to hear the calls without the speaker phone, but on switching to speaker mode I hear voice from speaker for a fraction of second and then I hear no voice and it shifts back to speaker off mode even though it shows that the speaker is on. I have no problem otherwise in the hearing on speaker as I am able to listen to music etc. I tried reformatting my phone and it solved the problem temporarily, but the problem reoccurred within a few days. My phone is only a month old, and is fully updated to Android 4.2.2. Is there a permanent solution to this?