Linked Server Connection Issue to SQL Server AlwaysOn Listener

I have SQL Server 2008R2 Standard Edition. I am making a Linked Server connection to a Listener of an AlwaysOn Availability Group. However, when I run a query from SSMS I intermittently get this error: OLE DB provider "SQLNCLI10" for linked server "Listener,4600" returned message "Unable to complete login process due to delay in opening server connection". Msg 7303, Level 16, State 1, Line 7 Cannot initialize the data source object of OLE DB provider "SQLNCLI10" for linked server "Listener,4600". I thought it may be a driver issue but OLE DB looks to be installed. Any idea what I may be missing?