I have been trying to place two virtual machine with heartbeat. osboxes and osboxes2. When I run nmap localhost this is the output: Nmap scan report for localhost (127.0.0.1) Host is up (0.00017s latency). Not shown: 991 closed ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 53/tcp open domain 80/tcp open http 443/tcp open https 631/tcp open ipp 3306/tcp open mysql 9050/tcp open tor-socks 10000/tcp open snet-sensor-mgmt Here is the tcpdump of such interface eth0: 18:38:22.267817 IP 192.168.141.135.48748 > 192.168.141.255.694: UDP, length 315 18:38:22.268640 IP 192.168.141.135.38234 > osboxes2.694: UDP, length 315 18:38:22.269421 IP osboxes2 > 192.168.141.135: ICMP osboxes2 udp port 694 unreachable, length 351 As you can see udp port 694 unreachable. When i try to ping osboxes from osboxes2: PING 192.168.141.137 (192.168.141.137) 56(84) bytes of data. 64 bytes from 192.168.141.137: icmp_seq=1 ttl=64 time=0.284 ms 64 bytes from 192.168.141.137: icmp_seq=2 ttl=64 time=0.291 ms 64 bytes from 192.168.141.137: icmp_seq=3 ttl=64 time=0.681 ms ha.cf file from osboxes: #Arquivo de log de debug: logfile /var/log/ha-log #Arquivo de log debugfile /var/log/ha-debug #Para onde vai os logs logfacility local0 #Frequencia em segundo de batimentos cardicados keepalive 2 #Tempo indica a morte do node deadtime 25 #Tempo que o heartbeat deve esperar por beats (nao o beat dos beatboxes) warntime 10 #Tempo maximo para declarar o outro servidor morto initdead 50 #Porto de sincronia udpport 694 #Endereco de broadcast da rede bcast eth0 #Nao entendi. Se for preciso vai la procurar depois. ucast eth0 192.168.141.137 #Determinar se o servidor volta para o master caso ele responda auto_failback on #Nome dos nodes do cluster node osboxes2 node osboxes haresourcers from osboxes: osboxes 192.168.141.135 apache It's worth noting that osboxes(principal node) is 192.168.141.135 and osboxes2 is 192.168.141.137. I guessing the problem resides in the port 694 not opened since it is unreachable.

I get this error from pacemaker after i change apache from http to https. now my ocf::heartbeat:apache resource is not find status page. I generate SSL certificate separately for 3 servers. Everything was working fine when running on http but as soon as I added the (self-signed) SSL certificate pacemaker Apache (ocf::heartbeat:apache): Stopped And error shows Failed Actions: * Apache_start_0 on server3 'unknown error' (1): call=315, status=complete, exitreason='Failed to access httpd status page.', last-rc-change='Mon Sep 21 16:22:37 2020', queued=0ms, exec=3456ms * Apache_start_0 on server1 'unknown error' (1): call=59, status=complete, exitreason='Failed to access httpd status page.', last-rc-change='Mon Sep 21 16:22:41 2020', queued=0ms, exec=3421ms * Apache_start_0 on server2 'unknown error' (1): call=197, status=complete, exitreason='Failed to access httpd status page.', last-rc-change='Mon Sep 21 16:22:33 2020', queued=0ms, exec=3451ms /etc/apache2/sites-available/000-default.conf ServerAdmin webmaster@localhost DocumentRoot /var/www/html Redirect "/" "https://10.226.***.***/ " SetHandler server-status ServerAdmin webmaster@localhost DocumentRoot /var/www/html Redirect "/" "https://10.226.179.205/ " Order deny,allow Deny from all Allow from 127.0.0.1 *pcs resource debug-monitor --full Apache* Operation monitor for Apache (ocf:heartbeat:apache) returned 1 > stderr: + echo > stderr: + printenv > stderr: + sort > stderr: + env= > stderr: AONIX_LM_DIR=/home/TeleUSE/etc > stderr: BXwidgets=/home/BXwidgets > stderr: HA_logfacility=none > stderr: HOME=/root > stderr: LC_ALL=C > stderr: LOGNAME=root > stderr: MAIL=/var/mail/root > stderr: OCF_EXIT_REASON_PREFIX=ocf-exit-reason: > stderr: OCF_RA_VERSION_MAJOR=1 > stderr: OCF_RA_VERSION_MINOR=0 > stderr: OCF_RESKEY_CRM_meta_class=ocf > stderr: OCF_RESKEY_CRM_meta_id=Apache > stderr: OCF_RESKEY_CRM_meta_migration_threshold=5 > stderr: OCF_RESKEY_CRM_meta_provider=heartbeat > stderr: OCF_RESKEY_CRM_meta_resource_stickiness=10 > stderr: OCF_RESKEY_CRM_meta_type=apache > stderr: OCF_RESKEY_configfile=/etc/apache2/apache2.conf > stderr: OCF_RESKEY_statusurl=http://localhost/server-status > stderr: OCF_RESOURCE_INSTANCE=Apache > stderr: OCF_RESOURCE_PROVIDER=heartbeat > stderr: OCF_RESOURCE_TYPE=apache > stderr: OCF_ROOT=/usr/lib/ocf > stderr: OCF_TRACE_RA=1 > stderr: PATH=/root/.rbenv/shims:/root/.rbenv/bin:/root/.rbenv/shims:/root/.rbenv/bin:/usr/local/bin:/home/TeleUSE/bin:/home/xrt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb > stderr: PCMK_logfacility=none > stderr: PCMK_service=crm_resource > stderr: PWD=/root > stderr: RBENV_SHELL=bash > stderr: SHELL=/bin/bash > stderr: SHLVL=1 > stderr: SSH_CLIENT=10.12.116.46 63097 22 > stderr: SSH_CONNECTION=10.12.116.46 63097 10.226.179.205 22 > stderr: SSH_TTY=/dev/pts/0 > stderr: TERM=xterm > stderr: TeleUSE=/home/TeleUSE > stderr: USER=root > stderr: _=/usr/sbin/pcs > stderr: __OCF_TRC_DEST= > stderr: __OCF_TRC_MANAGE= > stderr: + ocf_is_true > stderr: + false > stderr: + . /usr/lib/ocf/lib/heartbeat/apache-conf.sh > stderr: + . /usr/lib/ocf/lib/heartbeat/http-mon.sh > stderr: + bind_address=127.0.0.1 > stderr: + curl_ipv6_opts= > stderr: + ocf_is_true > stderr: + false > stderr: + echo > stderr: + grep -qs :: > stderr: + WGETOPTS=-O- -q -L --no-proxy --bind-address=127.0.0.1 > stderr: + CURLOPTS=-o - -Ss -L --interface lo > stderr: + HA_VARRUNDIR=/var/run > stderr: + IBMHTTPD=/opt/IBMHTTPServer/bin/httpd > stderr: + HTTPDLIST=/sbin/httpd2 /usr/sbin/httpd2 /usr/sbin/apache2 /sbin/httpd /usr/sbin/httpd /usr/sbin/apache /opt/IBMHTTPServer/bin/httpd > stderr: + MPM=/usr/share/apache2/find_mpm > stderr: + [ -x /usr/share/apache2/find_mpm ] > stderr: + LOCALHOST=http://localhost > stderr: + HTTPDOPTS=-DSTATUS > stderr: + DEFAULT_IBMCONFIG=/opt/IBMHTTPServer/conf/httpd.conf > stderr: + DEFAULT_SUSECONFIG=/etc/apache2/httpd.conf > stderr: + DEFAULT_RHELCONFIG=/etc/httpd/conf/httpd.conf > stderr: + DEFAULT_DEBIANCONFIG=/etc/apache2/apache2.conf > stderr: + basename /usr/lib/ocf/resource.d/heartbeat/apache > stderr: + CMD=apache > stderr: + OCF_REQUIRED_PARAMS= > stderr: + OCF_REQUIRED_BINARIES= > stderr: + ocf_rarun monitor > stderr: + mk_action_func > stderr: + echo apache_monitor > stderr: + tr - _ > stderr: + ACTION_FUNC=apache_monitor > stderr: + validate_args > stderr: + is_function apache_monitor > stderr: + command -v apache_monitor > stderr: + test zapache_monitor = zapache_monitor > stderr: + simple_actions > stderr: + check_required_params > stderr: + local v > stderr: + run_function apache_getconfig > stderr: + is_function apache_getconfig > stderr: + command -v apache_getconfig > stderr: + test zapache_getconfig = zapache_getconfig > stderr: + apache_getconfig > stderr: + HTTPD= > stderr: + PORT= > stderr: + STATUSURL=http://localhost/server-status > stderr: + CONFIGFILE=/etc/apache2/apache2.conf > stderr: + OPTIONS= > stderr: + CLIENT= > stderr: + TESTREGEX= > stderr: + TESTURL= > stderr: + TESTREGEX10= > stderr: + TESTCONFFILE= > stderr: + TESTNAME= > stderr: + : /etc/apache2/envvars > stderr: + source_envfiles /etc/apache2/envvars > stderr: + [ -f /etc/apache2/envvars -a -r /etc/apache2/envvars ] > stderr: + . /etc/apache2/envvars > stderr: + unset HOME > stderr: + [ != ] > stderr: + SUFFIX= > stderr: + export APACHE_RUN_USER=www-data > stderr: + export APACHE_RUN_GROUP=www-data > stderr: + export APACHE_PID_FILE=/var/run/apache2/apache2.pid > stderr: + export APACHE_RUN_DIR=/var/run/apache2 > stderr: + export APACHE_LOCK_DIR=/var/lock/apache2 > stderr: + export APACHE_LOG_DIR=/var/log/apache2 > stderr: + export LANG=C > stderr: + export LANG > stderr: + [ X = X -o ! -f -o ! -x ] > stderr: + find_httpd_prog > stderr: + HTTPD= > stderr: + [ -f /sbin/httpd2 -a -x /sbin/httpd2 ] > stderr: + [ -f /usr/sbin/httpd2 -a -x /usr/sbin/httpd2 ] > stderr: + [ -f /usr/sbin/apache2 -a -x /usr/sbin/apache2 ] > stderr: + HTTPD=/usr/sbin/apache2 > stderr: + break > stderr: + [ X != X -a X/usr/sbin/apache2 != X ] > stderr: + detect_default_config > stderr: + [ -f /etc/apache2/httpd.conf ] > stderr: + [ -f /etc/apache2/apache2.conf ] > stderr: + echo /etc/apache2/apache2.conf > stderr: + DefaultConfig=/etc/apache2/apache2.conf > stderr: + CONFIGFILE=/etc/apache2/apache2.conf > stderr: + [ -n /usr/sbin/apache2 ] > stderr: + basename /usr/sbin/apache2 > stderr: + httpd_basename=apache2 > stderr: + GetParams /etc/apache2/apache2.conf > stderr: + ConfigFile=/etc/apache2/apache2.conf > stderr: + [ ! -f /etc/apache2/apache2.conf ] > stderr: + get_apache_params /etc/apache2/apache2.conf ServerRoot PidFile Port Listen > stderr: + configfile=/etc/apache2/apache2.conf > stderr: + shift 1 > stderr: + echo ServerRoot PidFile Port Listen > stderr: + sed s/ /,/g > stderr: + vars=ServerRoot,PidFile,Port,Listen > stderr: + apachecat /etc/apache2/apache2.conf > stderr: + awk -v vars=ServerRoot,PidFile,Port,Listen > stderr: BEGIN{ > stderr: split(vars,v,","); > stderr: for( i in v ) > stderr: vl[i]=tolower(v[i]); > stderr: } > stderr: { > stderr: for( i in v ) > stderr: if( tolower($1)==vl[i] ) { > stderr: print v[i]"="$2 > stderr: delete vl[i] > stderr: break > stderr: } > stderr: } > stderr: > stderr: + awk > stderr: function procline() { > stderr: split($0,a); > stderr: if( a~/^[Ii]nclude$/ ) { > stderr: includedir=a; > stderr: gsub("\"","",includedir); > stderr: procinclude(includedir); > stderr: } else { > stderr: if( a=="ServerRoot" ) { > stderr: rootdir=a; > stderr: gsub("\"","",rootdir); > stderr: } > stderr: print; > stderr: } > stderr: } > stderr: function printfile(infile, a) { > stderr: while( (getline 0 ) { > stderr: procline(); > stderr: } > stderr: close(infile); > stderr: } > stderr: function allfiles(dir, cmd,f) { > stderr: cmd="find -L "dir" -type f"; > stderr: while( ( cmd | getline f ) > 0 ) { > stderr: printfile(f); > stderr: } > stderr: close(cmd); > stderr: } > stderr: function listfiles(pattern, cmd,f) { > stderr: cmd="ls "pattern" 2>/dev/null"; > stderr: while( ( cmd | getline f ) > 0 ) { > stderr: printfile(f); > stderr: } > stderr: close(cmd); > stderr: } > stderr: function procinclude(spec) { > stderr: if( rootdir!="" && spec!~/^\// ) { > stderr: spec=rootdir"/"spec; > stderr: } > stderr: if( isdir(spec) ) { > stderr: allfiles(spec); # read all files in a directory (and subdirs) > stderr: } else { > stderr: listfiles(spec); # there could be jokers > stderr: } > stderr: } > stderr: function isdir(s) { > stderr: return !system("test -d \""s"\""); > stderr: } > stderr: { procline(); } > stderr: /etc/apache2/apache2.conf > stderr: + sed s/#.*//;s/[[:blank:]]*$//;s/^[[:blank:]]*// > stderr: + grep -v ^$ > stderr: + eval PidFile=${APACHE_PID_FILE} > stderr: + PidFile=/var/run/apache2/apache2.pid > stderr: + CheckPort > stderr: + ocf_is_decimal > stderr: + false > stderr: + CheckPort > stderr: + ocfError performing operation: Operation not permitted _is_decimal > stderr: + false > stderr: + CheckPort 80 > stderr: + ocf_is_decimal 80 > stderr: + true > stderr: + [ 80 -gt 0 ] > stderr: + PORT=80 > stderr: + break > stderr: + echo > stderr: + grep : > stderr: + Listen=localhost: > stderr: + [ Xhttp://localhost/server-status = X ] > stderr: + test /var/run/apache2/apache2.pid > stderr: + return 0 > stderr: + validate_env > stderr: + check_required_binaries > stderr: + local v > stderr: + is_function apache_validate_all > stderr: + command -v apache_validate_all > stderr: + test zapache_validate_all = zapache_validate_all > stderr: + local rc > stderr: + LSB_STATUS_STOPPED=3 > stderr: + apache_validate_all > stderr: + [ -z /usr/sbin/apache2 ] > stderr: + [ ! -x /usr/sbin/apache2 ] > stderr: + [ ! -f /etc/apache2/apache2.conf ] > stderr: + [ -n ] > stderr: + [ -n ] > stderr: + dirname /var/run/apache2/apache2.pid > stderr: + local a > stderr: + local b > stderr: + [ 1 = 1 ] > stderr: + a=/var/run/apache2/apache2.pid > stderr: + [ 1 ] > stderr: + b=/var/run/apache2/apache2.pid > stderr: + [ /var/run/apache2/apache2.pid = /var/run/apache2/apache2.pid ] > stderr: + break > stderr: + b=/var/run/apache2 > stderr: + [ -z /var/run/apache2 -o /var/run/apache2/apache2.pid = /var/run/apache2 ] > stderr: + echo /var/run/apache2 > stderr: + return 0 > stderr: + ocf_mkstatedir root 755 /var/run/apache2 > stderr: + local owner > stderr: + local perms > stderr: + local path > stderr: + owner=root > stderr: + perms=755 > stderr: + path=/var/run/apache2 > stderr: + test -d /var/run/apache2 > stderr: + return 0 > stderr: + return 0 > stderr: + rc=0 > stderr: + [ 0 -ne 0 ] > stderr: + ocf_is_probe > stderr: + [ monitor = monitor -a 0 = 0 ] > stderr: + run_probe > stderr: + is_function apache_probe > stderr: + command -v apache_probe > stderr: + test z = zapache_probe > stderr: + shift 1 > stderr: + apache_monitor > stderr: + silent_status > stderr: + local pid > stderr: + get_pid > stderr: + [ -f /var/run/apache2/apache2.pid ] > stderr: + cat /var/run/apache2/apache2.pid > stderr: + pid=17552 > stderr: + [ -n 17552 ] > stderr: + ProcessRunning 17552 > stderr: + local pid=17552 > stderr: + [ -d /proc -a -d /proc/1 ] > stderr: + [ -d /proc/17552 ] > stderr: + [ 0 -ne 0 ] > stderr: + findhttpclient > stderr: + [ x != x ] > stderr: + which wget > stderr: + echo wget > stderr: + ourhttpclient=wget > stderr: + [ -z wget ] > stderr: + ocf_check_level 10 > stderr: + local lvl prev > stderr: + lvl=0 > stderr: + prev=0 > stderr: + ocf_is_decimal 0 > stderr: + true > stderr: + [ 10 -eq 0 ] > stderr: + [ 10 -gt 0 ] > stderr: + lvl=0 > stderr: + break > stderr: + echo 0 > stderr: + apache_monitor_basic > stderr: + wget_func http://localhost/server-status > stderr: + auth= > stderr: + cl_opts=-O- -q -L --no-proxy --bind-address=127.0.0.1 > stderr: + [ x !=+ x ] > stderr: grep+ wget -Ei -O- -q > stderr: -L --no-proxy --bind-address=127.0.0.1 http://localhost/server-status > stderr: + attempt_index_monitor_request > stderr: + local indexpage= > stderr: + [ -n ] > stderr: + [ -n ] > stderr: + [ -n ] > stderr: + [ -n http://localhost/server-status ] > stderr: + return 1 > stderr: + [ 1 -eq 0 ] > stderr: + ocf_is_probe > stderr: + [ monitor = monitor -a 0 = 0 ] > stderr: + return 1 **pcs config** Resource: MasterVip (class=ocf provider=heartbeat type=IPaddr2) Attributes: ip=10.226.***.*** nic=lo cidr_netmask=32 iflabel=pgrepvip Meta Attrs: target-role=Started Operations: start interval=0s timeout=20s (MasterVip-start-interval-0s) stop interval=0s timeout=20s (MasterVip-stop-interval-0s) monitor interval=90s (MasterVip-monitor-interval-90s) Resource: Apache (class=ocf provider=heartbeat type=apache) Attributes: configfile=/etc/apache2/apache2.conf statusurl=http://localhost/server-status Operations: start interval=0s timeout=40s (Apache-start-interval-0s) stop interval=0s timeout=60s (Apache-stop-interval-0s) monitor interval=1min (Apache-monitor-interval-1min) I don't know how to fix this. if anyone knows please help me.

I have ELK server, and sending Heartbeats to server. I added an monitor in monitors.d directory, added setup.dashboards.enabled: true line in heartbeat.yml file, but after restarting heartbeat service, service working 2-3 secand and then sending me this output message > Ping remote services for availability and log results to Elasticsearch or send to Logstash I deleted my added line setup.dashboards.enabled: true and have not get that messages, hearthbeats worked, sent all beats to ELK server, but didn't send monitor's index Can anybody help my to solve this issue

I'm trying to set up a Heartbeat client on a linux machine (CentOS Linux release 7.3.1611) which is sending ICMP echo requests to roughly 1300 hosts. However, in future this number will be higher. The messages generated by heartbeat are being shipped to a logstash instance on another server. These requests go out every 3 minutes, all at once. After the first round of requests, a large number of these messages contain an error: "write ip4 0.0.0.0->x.x.x.x: sendto: no buffer space available" I've tried multiple ways of mitigating this issue, including staggering the requests so there are less being sent out in 1 second. This caused issues with data synchronisation at the end of this pipeline (as well as not completely resolving the issue) so it's not really a feasible option for me. Researching online led me to increasing the memory allocated to TCP/IP connections by the system, however these are ICMP requests which as far as I'm aware should be separate. I'm also unable to bounce the network interface as the machine is remote and I don't have a way of getting back in to bring it back up. I also tested out increasing (and setting to 0) the icmp_ratelimit variable in /proc/sys/net/ipv4 but this hasn't helped either. My question is pretty general, what could be causing this issue? Is there some sort of variable I need to tweak on the system that will allow these requests to all be sent out at once? I can't really deduce what buffer the error message is referencing. Any help would be greatly appreciated... P.S. if further clarification is needed I'd be happy to provide it. **EDIT** ---------- Still unsure if there is a fault elsewhere however increasing the size of the socket buffer determined by the wmem_max, wmem_default, rem_max, rem_default variables in /proc/sys/net/core has fixed the issue. It's likely that the total size of the data for all the ICMP echo requests was too large to fit under the previous maximum of 208kb and so a large number of them would be dropped. Doesn't really explain why the number of dropped requests would vary each time, maybe there is an underlying issue... Now my only problem is that each time I reboot the system, these variables reset to 208kb each. How do I make these changes persistent?

So my current environment is centos7 on a pair of identical bare metal servers. I've installed drbd and carved up 3 x 100G block devices for DRBD to mirror. Then I installed a Centos7 image on one of those block devices. I wanted Pacemaker to manage DRBD as well as the xen virtual images, so I followed the directions specified in clusterlabs.org's "Clusters From Scratch" doc:

pcs cluster cib drbd_cfg

pcs -f drbd_cfg resource create XenHost01_Storage ocf:linbit:drbd drbd_resource=XenHost01 op monitor interval=60s

pcs -f drbd_cfg resource master XenHost01_StorageClone XenHost01_Storage master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

pcs cluster cib-push drbd_cfg

Then I added in the Xen vm to be managed as well.

pcs cluster cib vmstate_cfg
pcs -f vmstate_cfg resource create XenHosts --group vminit systemd:vmstate op monitor interval=10s

pcs cluster cib-push vmstate_cfg

So my cluster resources would look like this:

Full list of resources:

 Master/Slave Set: XenHost01_StorageClone [XenHost01_Storage]
     Masters: [ ha1 ]
     Slaves: [ ha2 ]
 Resource Group: vminit
     XenHosts	(systemd:vmstate):	Started ha1

PCSD Status:
  ha1: Online
  ha2: Online

What I'm trying to figure out, and the reason for my post, is how to stick that DRBD master/slave pair under the vminit group, so that the Xen virtual machines only get started on the primary DRBD host. Without them in a group, if the other host shows up as primary in DRBD, the cluster doesn't care and still tries to start up the virtual images and will fail. Anybody know how to get that master/slave drbd resource under a group? I've been tearing my hair out.

It's been about a month where I've been working with "High Availability" on Elastix based on DRBD and Heartbeat. As you maybe be familiar with it, there is a file in Heartbeat called haresources that's used for assigning the primary server, replicated partition, starting services and finally creating a Float/Virtual IP address. In haresources virtual IP address define with a format like IPaddr::192.168.116.10/24/eth0/192.168.116.255 As this virtual IP address created it will be assigned with a MAC address exactly the same as real interface. Due to some problems on Voice Gateways, I think it's better to change the MAC address of virtual interface; but this changing should be placed on haresources and its format. What should I do?

The scenario is, I have two web servers both are running Nginx on it. Also I have configured HAProxy after the LB (load balancer) for both the servers. I've checked with Heartbeat, but no luck to make downtime count "zero". I want some application which will switch to secondary server instantly when primary server goes down, in other words it should not give any downtime.

I have set up a heartbeat HA two node cluster. And both of server have same configuration of ha.cf #cat /etc/ha.cf logfile /var/log/ha-log autojoin none bcast eth1 warntime 20ms deadtime 40ms initdead 30 keepalive 5ms node server1.com node server2.com I set keepalive and deadtime in milliseconds hoping failover will take place in less than second or deadtime value. But expected behaviour is not met. With this configuration if I stop heartbeat in one server another server will take only after about 20 seconds(rough estimate by a stopwatch). I was wondering is what heartbeat can only do or am I missing something?

I followed these steps correctly, everything worked well before I have revived heartbeat on node01 "at the last steps of the tutorial"
when I hit /etc/init.d/heartbeat start it started correctly but the webpage wouldn't open, in other words httpd service haven't been started by heartbeat (which I guess the reason is that the node02 still the primary node it hasn't returned to secondary yet)
so I searched for the STONITH but i didn't find anything that helps.
this is the ha.cf file:
logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 30 initdead 120 bcast eth0 udpport 694 auto_failback on node node01 node node02 haresources file: node01 192.168.1.82 httpd authkeys file: auth 2 2 sha1 test-ha

I was successful in configuring a simple Heartbeat setup on Ubuntu 12.04, which will fail-over two VIPs to a second server when BOTH Ethernet links fail. What I can't figure out is how to fail-over two VIPs when either of the two NICs loses a link connection. Heartbeat will detect the link is down, but until both go down, resources will not fail-over. I'm not using any cluster management like Pacemaker. I would appreciate a point in the right direction.