HP multipath software download (for Linux)

http://h20000.www2.hp.com/bizsupport/TechSupport/DriverDownload.jsp?lang=en&cc=us&prodNameId=3559652&taskId=135&prodTypeId=12169&prodSeriesId=3559651&lang=en&cc=us

 設定文件

 http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=115&prodSeriesId=3559651&prodTypeId=18964&objectID=c01476873

disable HP UX/Linux root ssh 連線之方法

# cd /opt/ssh/etc

# vi sshd_config

Change the line:

PermitRootLogin yes

change into like below

PermitRootLogin no

Then stop and restart SSH.

# /sbin/init.d/secsh stop

# /sbin/init.d/secsh start

--------------------------------------

如果在linux 則類似以下設定方式

vi /etc/ssh/sshd_config

#restart SSH.

service sshd restart
--------------------------------------
P.S. 如果重啟SSH daemon , 對於已建立的SSH連線是不受影響的,所以可放心執行

Linux maximum sockets tuning

紀錄一下, Linux maximum sockets tuning的方法,

如果linux 當作NAS server 時有可能會用到

Linux sockets tuning (for concurrent RPC services (nfs , cifs server) )

http://stackoverflow.com/questions/410616/increasing-the-maximum-number-of-tcp-ip-connections-in-linux

·                  透過以下調整 可以加大到 470 個connections , 如果要再更多 , 可以放大port range : net.ipv4.ip_local_port_range = 15000 61000  , 縮小net.ipv4.tcp_fin_timeout

·                  net.ipv4.ip_local_port_range = 32768 61000

·                  net.ipv4.tcp_fin_timeout = 60 This basically means your system cannot guarantee more than (61000 - 32768) / 60 = 470 sockets at any given time. If you are not happy with that, you could begin with increasing the port_range. Setting the range to 15000 61000 is pretty common these days. You could further increase the availability by decreasing the fin_timeout. Suppose you do both, you should see over 1500 outbound connections, more readily.

Linux OS buffer tuning for dirty blocks (for Databases/Application Server)

環境 Redhat 5.5 , memory 64GB. 跑的是timesten 資料庫

客戶反映系統告警 ( free memory 不足 , 只剩2GB )

使用以下free -m 指令發現OS buffer使用過多(21GB) , 與其他設備比較相對占用較多緩存

[root@db1 log]# free -m

total used free shared buffers cached

Mem: 64429 61744 2685 0 752 21638

-/+ buffers/cache: 39352 25077

Swap: 65538 0 65538

[root@db1 log]# echo 1 > /proc/sys/vm/drop_caches ==> 要求OS kernel捨棄沒在使用的 cache

[root@db1 log]# free -m

total used free shared buffers cached

Mem: 64429 44140 20289 0 13 5272

-/+ buffers/cache: 38854 25575

Swap: 65538 0 65538

[root@db1 log]#

後來發現以上的做法只是暫解, 如果要一勞永逸須調整Linux kernel vm.dirty_ratio 參數, 相關檔案在 /etc/sysctl.conf

參考資料：

http://blog.chrisgreenough.com/2012/01/oracle-redhat-vm-swappiness-and-huge-pages/

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/s3-proc-sys-vm.html

--

Made Kernel changes so dirty pages in memory are written out more frequently

vm.dirty_background_ratio = 3  ==>使用背景daemon pdflush 去 flush回硬碟  (default 10%)

vm.dirty_ratio = 15    ==>使用總系統多少比率的dirty,就會flush回硬碟 (default  40%)

vm.dirty_expire_centisecs = 500  ==>資料可以保持為dirty狀態的最大毫秒數(ms), default (3000ms , 30s)

vm.dirty_writeback_centisecs = 100  ==>資料多久寫回(default 500ms)

Changed the kernel parameter that determines how aggressive swapping is done. Set to 0 so swapping is only done when needed.

vm.swappiness = 1  ==> default 60 , Range from 0 to 100. Lower values mean pages remain longer in memory before swapping them out to disk. Larger values swap out infrequently used memory pages faster. The default value is “60″.

Configured hugepages for Oracle which makes the memory pagesize larger for Oracle processes (this memory is not swappable)

vm.nr_hugepages = 6656  ==> number of huge pages

vm.hugetlb_shm_group = 501  ==> contains group id that is allowed to create SysV shared memory segment using hugetlb page , 在此範例 group id 為 501

如果沒設定 可能遇到如下問題
 : http://www.eygle.com/archives/2011/12/hugepageshugetl.html

客戶解決一個Linux上資料庫無法啟動的問題。
客戶的Linux 5.6 x86-64環境，安裝資料庫後，啟動資料庫報錯：ORA-27125。
Oracle文檔上關於ORA-27125錯誤的描述為：

ORA-27125: unable to create shared memory segment
Cause: shmget() call failed
Action: contact Oracle support

ulimit settings for Oracle:

oracle soft memlock 13631488

oracle hard hemlock 13631488

心得:
如果使用大記憶體的linux  , vm.dirty_ratio 預設40% 是相當不合理的, 且會搶走Application/DB的記憶體.意思是說 OS 會用OS buffer , DB 也用DB BUFFER 造成無謂浪費~

Linux tcp session 處於 ESTABLISHED狀態 且無法釋放

今天在某環境看到

TCP 由webserver(Linux)連到 AP的連線(透過F5) 呈現不對等關係

由webserver端(Linux)看到 連線數正常 為30 - 40個連線ESTABLISHED

由AP 端(Aix)看到 與webserver(Linux)連線數為200多個ESTABLISHED session...

由以下指令來算ESTABLISHED 連線數
 #netstat -an | grep ".5555" | awk '{print $5}'| cut -d '.' -f 1-4 |sort | uniq -c | sort -nr
 291 192.168.xx.1
 281 192.168.xx.2
 255 192.168.xx.3
 214 192.168.xx.4
  43 192.168.xx.5
  27 192.168.xx.6
  10 192.168.xx.7
   8 192.168.xx.8
   6 192.168.xx.9

看了一下網路文章 , 敘述發生在有NAT的環境 , 看起來蠻符合的 , 好像需調整以下TCP timeout參數:

vi /etc/sysctl.conf 

加入以下參數, 修改linux timeout

net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000   ==>預設為五天

to

net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 1200

再打sysctl -p 即生效

---

Linux NAT 的 ip_conntrack 模組會記錄 tcp 通訊協定的 established connection 記錄, 而且預設 timeout 時間長達五天 (432,000 秒)

其他相關指令:  (from http://hi.baidu.com/cwbdde/blog/item/0887bcfbf8649b1d6c22ebf2.html )

查看目前 ip_conntrack buffer 使用狀況指令: 

grep conn /proc/slabinfo

結果實例: ip_conntrack 3024 4090 384 409 409 1 (各值說明如下)

ip_conntrack the cache name

3024 the number of currently active objects

4090 the total number of available objects

384 the size of each object in bytes

409 the number of pages with at least one active object

409 the total number of allocated pages

1 the number of pages per slab are given

man slabinfo 可查詢詳細說明.

查出目前 ip_conntrack 記錄最多的前五名 IP

指令: cat /proc/net/ip_conntrack | cut -d ' ' -f 10 | cut -d '=' -f 2 | sort | uniq -c | sort -nr | head -n 5

結果實例:

    295 192.168.xx.2

     52 210.202.xx.xx

     14 125.231.xx.xx

     13 61.60.xx.xx

     11 61.60.xx.xx

由此可知, 192.168.xx.xx 佔用了絕大多數的 buffer, 推斷這個 IP 的 AP 可能request大量連線.

--
最後回到AP Server :

找出使用5555 port 的socket
#netstat -Aan|grep ".5555"|more
f10000f0076f3358 tcp4       0      0  192.168.xx.xx.5555 192.168.xx.xx.4077 ESTA
BLISHED

用Aix 的kdb 指令 分析有使用到的socket.
#kdb

(0)> sockinfo f10000f0076f3358 tcpcb |grep -E "KEEP|opts"
    t_timer....... 00000000 (TCPT_KEEP)  ==> keep-alive對應的timer(TCPT_KEEP)數值為0，表示沒有設置timer時間
    opts........ 0004 (REUSEADDR)
(0)>

#rmsock f10000f0076f3358 tcpcb
 The socket 0x7946000 is being held by proccess 376864 (xxxx_srv).

#在Aix 下以下指令,找出所有被使用的port 的socket , 反查AP pid分析

接著使用shell scripts 找到所有的ESTABLISHED 連線, check socket 到底被誰占用
netstat -Aan |grep ".5555" |awk '{print $1}' > a.lst
for a in `cat a.lst`
do
rmsock $a tcpcb  >>a.log
done
cat a.log

發現所有的socket 都被AP server占用 ( pid 376864 )

The socket 0x7946000 is being held by proccess 376864 (xxxx_srv).
The socket 0x7946000 is being held by proccess 376864 (xxxx_srv).
.
.
這只是結果現象, 但究竟成因為何?

挑了另一台AP (Solaris) , 使用snoop指令可以撈取TCP neogiation.

正常連線如下

192.168.xx.xx -> server4         TCP D=5555 S=51760 Syn Seq=117012684 Len=0 Win=5840 Options=

        server4          -> 192.168. xx.xx  TCP D=51760 S=5555 Syn Ack=117012685 Seq=3082196059 Len=0 Win=49248 Options=2834143267 3777258114,mss 1460,nop,wscale 0,nop,nop,sackOK>

192.168.xx.xx  ->  server4          TCP D=5555 S=51760 Ack=3082196060 Seq=117012685 Len=0 Win=46 Options=2834143267>

192.168.xx.xx  ->  server4          TCP D=5555 S=51760 Push Ack=3082196060 Seq=117012685 Len=251 Win=46 Options=2834143267>

         server4          -> 192.168.xx.xx  TCP D=51760 S=5555 Ack=117012936 Seq=3082196060 Len=0 Win=48997 Options=2834143267 3777258116>

         server4          -> 192.168.xx.xx  TCP D=51760 S=5555 Push Ack=117012936 Seq=3082196060 Len=28 Win=48997 Options=2834143268 3777258116>

192.168.xx.xx  ->  server4          TCP D=5555 S=51760 Ack=3082196088 Seq=117012936 Len=0 Win=46 Options=2834143268>

而出問題的TCP連線 , 主要是DB 與Server之間...沒有收到 FIN ACK (goodbye) 封包...跟客戶聯繫後...請他們改AP程式要能夠主動closing釋放socket...觀察兩個禮拜後 就正常了....結案...原來DBA也要會分析網路...XD

192.168.xx.xx -> server4         TCP D=5555 S=54587 Ack=3375666357 Seq=1217496740 Len=0 Win=46 Options=2834248185>

192.168.xx.xx -> server4         TCP D=5555 S=54587 Fin Ack=3375666357 Seq=1217496740 Len=0 Win=46 Options=2834248185>

         server4          -> 192.168.xx.xx TCP D=54587 S=5555 Ack=1217496741 Seq=3375666357 Len=0 Win=48628 Options=2834248185 3778307603>

         server4          -> 192.168.xx.xx TCP D=54587 S=5555 Fin Ack=1217496741 Seq=3375666357 Len=0 Win=48628 Options=2834248185 3778307603>

Configuring TCP KeepAlive Parameters[轉載]

Configuring TCP KeepAlive Parameters

http://www.starquest.com/Supportdocs/techStarLicense/SL002_TCPKeepAlive.shtml

Last Update: 19 June 2008
Product: StarLicense for Windows/UNIX 
Version: 1.07 or later 
Article ID: SQV00SL002

Abstract

The StarLicense server uses the settings for the TCP KeepAlive parameter to clean up connections from clients that have disconnected without releasing a license that has been checked out. You can change the TCP KeepAlive parameter if you need the abandoned license checkouts to be cleaned up more frequently.

Solution

The StarLicense server uses the system TCP/IP KeepAlive parameter settings to clean up connections from clients that have disconnected from the network without releasing the StarQuest product license that was checked out by the computer. The default values for the KeepAlive parameter should be sufficient for most users, but you can change them if necessary, such as to clean up abandoned license connections more frequently. The KeepAlive parameter names vary by operating system but generally specify:

• the interval to wait before probing the idle connection (on most platforms the default is 2 hours)
• the interval to wait before retrying the probe after an initial failure to respond
• the maximum number of times to retry the probe

Modifying the KeepAlive parameter settings may affect the operation of other applications on the computer. You also may want to set additional TCP/IP parameters that affect TCP connections, depending on the overall impact to other TCP/IP parameters after you modify the KeepAlive settings.

The method of changing the KeepAlive parameter settings varies by operating system, so refer to your system documentation for details. This technical document describes the general steps for displaying and modifying the KeepAlive parameter for the platforms on which the StarLicense server supports.

The following table shows the names of the KeepAlive parameters for each supported operating system.

Table 1: Keepalive Parameters by Operating System

Operating System	Parameter wait time before probing the connection	Parameter interval between retry probes	Parameter maximum retry probes	Unit of measure
AIX	tcp_keepidle	tcp_keepintvl	n/a	half-seconds
FreeBSD	TCPTV_KEEP_IDLE	TCPTV_KEEPINTVL	TCPTV_KEEPCNT	seconds
HP-UX 11i	tcp_time_wait_interval	tcp_keepalive_interval	tcp_keepalives_kill (1)	milliseconds
Linux	tcp_keepalive_time	tcp_keepalive_intvl	tcp_keepalive_probes	seconds
Solaris	tcp_time_wait_interval	tcp_keepalive_interval	n/a	milliseconds
Windows	KeepAliveTime	KeepAlive Interval	TcpMaxDataRetransmission	milliseconds

(1): tcp_keepalives_kill cannot be modified on HP. It is set to 1.

Displaying and Modifying KeepAlive Values

Follow these steps to modify the values.

1.       Log in as root on a UNIX-based computer or as an Administrator on a Windows computer.

2.       Refer to the information for the operating system you are using, as suggested in the following table, to display information about the network tuning parameters.

Platform	Operating System Command
AIX	no -a
HP-UX 11i	ndd -h supported
FreeBSD	Examine /usr/src/sys/netinet/tcp_timer.h and tcp_timer.c
Linux	sysctl -h
Sun Solaris	ndd /dev/tcp \?
Windows	Refer to http://technet.microsoft.com/en-us/library/bb726981.aspx#EDAA

3.       Display the current KeepAlive settings, replacing <tcp_parameter> with the names shown in Table 1 for each platform.

Platform	Operating System Command
AIX	no -o<tcp_parameter>
FreeBSD	Examine /usr/src/sys/netinet/tcp_timer.h and tcp_timer.c
HP-UX 11i	ndd -get /dev/tcp <tcp_parameter>
Linux	sysctl net.ipv4.<tcp_parameter>
Sun Solaris	ndd -get /dev/tcp <tcp_parameter>
Windows	From the Start menu, choose Run and enter regedit to view the parameter located in the Registry file HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters

4.       Display the range of available settings.

Platform	Operating System Command
AIX	This option is not available.
FreeBSD	This option is not available.
HP-UX 11i	ndd -h <tcp_parameter>
Linux	This option is not available.
Sun Solaris	This option is not available.
Windows	Refer to http://technet.microsoft.com/en-us/library/bb726981.aspx#EDAA.

5.       Modify the KeepAlive values as desired.

Platform	Operating System Command
AIX	no -o <tcp_parameter>= <tcp _value>
FreeBSD	Modify /usr/src/sys/netinet/tcp_timer.h and rebuild the kernel.
HP-UX 11i	ndd -set /dev/tcp <tcp_parameter> <tcp_value>
Linux	To set the value temporarily until the computer is restarted:  sysctl -w net.ipv4.<tcp_parameter> = <tcp_value>  To make the change permanently: Update /etc/sysctl.conf with net.ipv4.<tcp_parameter> = <tcp_value> and issue one of the following commands, depending on which version of Linux you are running: Red Hat: chkconfig sysctl on Suse: chkconfig boot.sysctl on
Sun Solaris	ndd -set /dev/tcp <tcp_parameter> <tcp_value>
Windows	Run regedit to edit the Windows Registry key located in the pathHKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters. After you change the Registry value you must restart the computer for the change to take affect.

---

DISCLAIMER

The information in technical documents comes without any warranty or applicability for a specific purpose. The author(s) or distributor(s) will not accept responsibility for any damage incurred directly or indirectly through use of the information contained in these documents. The instructions may need to be modified to be appropriate for the hardware and software that has been installed and configured within a particular organization.  The information in technical documents should be considered only as an example and may include information from various sources, including IBM, Microsoft, and other organizations.