Oracle 11g+ RAC CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.

0.確認CRSS daemon是處於Observer mode or Active mode (如果config ntpd 就會變成observer mode)

[grid@db1 ~]$  crsctl check ctss

CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

[grid@db2 ~]$  crsctl check ctss

CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

1. 從db1 的crs alert log 有看到以下訊息, 兩台主機的時間應該還是不太一致, 光是有時間差就會造成RAC的單台DB被重啟

2018-06-xx xx:xx:xx.xx [OCTSSD(9974)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc. 2018-06-xx xx:xx:xx.xx [OCTSSD(9974)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.

2.建議確認DB1, DB2主機上的時間, 確定是否有時間差

#At Both DB Servers

date

3.如有時間差, 須先disable chronyd服務, 再把ntpd 服務帶起, 

3.1 Disable chronyd service
#chrony 是 RHEL/CentOS 7 預設的校時 dameon, 如果您的作業系統是RHEL6 請忽略相關步驟
#In order to use ntpd the default user space daemon, chronyd, must be stopped and disabled
systemctl stop chronyd
systemctl disable chronyd
systemctl status chronyd

3.2 NTP Client Configuration
yum install ntp
-------------------------------------------------------------------------
## cat /etc/ntp.conf
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
restrict #ntp_ip_address mask 255.255.255.0 nomodify notrap
driftfile /var/lib/ntp/drift
keys /etc/ntp/keys
-------------------------------------------------------------------------
#systemctl list-unit-files --state=enabled
systemctl enable ntpd.service
systemctl start ntpd.service

3.3 Configure firewall to enable NTP port 123
# firewall-cmd --zone=public --add-port=123/udp --permanent
success

# firewall-cmd --reload
success

不可使用ntpdate 的指令放到cron table去更新時間

#手動更新較快

ntpdate -d xxx.xxx.xxx.xxx #這裡為實際ntp server的ip address

3.4 Verify ntpd service

#At Both DB Servers, verify /etc/ntp.conf寫法是否正確

service ntpd status

service ntpd start

#Verify

ntpq -p

3.5 開啟"微調模式"
[root@ ~]# vi /etc/sysconfig/ntpd
#The following item added by Robinson
#Set to 'yes' to sycn hw clock after successful ntpdate
SYNC_HWCLOCK=yes #開啟同步更新硬體時間
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid -g" #加-x 參數，開啟微調模式

4.手動把時間從系統回寫到bios

#update date to bios

hwclock --systohc

#ref: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sect-configuring_the_date_and_time-hwclock

5.確認是否關閉numa, transparent_hugepages  (Disable Transparent_Hugepages以避免node eviction問題)

[root@oraracx1 ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=ol/root rd.lvm.lv=ol/swap rhgb quiet numa=off transparent_hugepage=never intel_idle.max_cstate=0 processor.max_cstate=0" GRUB_DISABLE_RECOVERY="true" Ref: ALERT: Disable Transparent HugePages on SLES11, RHEL6, RHEL7, OL6, OL7, and UEK2 and above (Doc ID 1557478.1)

ref:
http://www.voidcn.com/article/p-rwdsexfj-bcp.html
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sect-configuring_the_date_and_time-hwclock

FWD: Disable and Remove Unwanted Services on RHEL/CentOS 7 Minimal Installation
https://jaychu649.blogspot.com/2018/05/fwd-disable-and-remove-unwanted.html