Activity Monitor Blocked by Firewall. Set Static Port?

We are seeing an issue with not being able to open Activity monitor for remote SQL servers. The Activity Monitor is unable to execute queries against server . Activity Monitor for this instance will be placed into a paused state. Use the context menu in the overview pane to resume the Activity Monitor. The RPC server is unavailable. (Exception from HRESULT: 0x800706BA) (mscorlib) It appears some firewall is blocking it, When we look at firewall logs (on the network), we are seeing blocks from 496xx. And its different port for each server. Is anyone aware if we can make this a static port?