23 Dec 2016
在rac集群中，所有的节点都必须有唯一的hostname
hostname db-oracle-node1 cat /etc/sysconfig/network NETWORKING=yes HOSTNAME=db-oracle-node1
节点网卡名称：eth1-4，对应架构图中的em1-4
规划的网段：
网络配置主要关注正确设置public和private网络接口以及Single Client Access Name(SCAN)的DNS配置。 public网络bond网卡来提供高可用性。 High Availability Internet Protocol (HAIP)为private网络接口提供故障迁移和负载均衡。 SCAN为Oracle RAC database 11g r2提供了一个唯一的名称，可供集群中的任一客户端访问Oracle数据库时使用。
step 1 由于测试机器有限，dns在node1上安装，安装dns
# node1上执行此命令 yum install bind-libs bind bind-utils
step 2 配置dns
# 主配文件 vim /etc/named.conf ****************************************************** options { # 配置dns服务监听的端口和ip listen-on port 53 { 127.0.0.1; 192.168.33.103; }; ... # zone配置所在目录(保持默认) directory "/var/named"; ... # 允许使用dns服务的ip及ip段 allow-query { localhost; 127.0.0.1; 192.168.33.0/24; }; ... # 此dns无法解析的，去哪里继续查询，这里指定局域网的公用dns forwarders {192.168.33.1; 10.0.0.4;}; }; # 增加正向解析规则，localdomain是我的测试环境，需要更新到正式环境所在的域 zone "localdomain." IN { type master; file "localdomain.zone"; allow-update { none; }; }; # 增加反向解析规则 zone "33.168.192.in-addr.arpa." IN { type master; file "33.168.192.in-addr.arpa"; allow-update { none; }; }; ****************************************************** # 创建和上面配置域名称对应的zone文件，放在directory的配置目录中 vim /var/named/localdomain.zone ****************************************************** $TTL 86400 @ IN SOA localhost root.localhost ( 42 ; serial (d. adams) 3H ; refresh 15M ; retry 1W ; expiry 1D ) ; minimum IN NS localhost localhost IN A 127.0.0.1 db-oracle-node1 IN A 192.168.33.103 db-oracle-node2 IN A 192.168.33.104 db-oracle-node1-vip IN A 192.168.33.111 db-oracle-node2-vip IN A 192.168.33.112 db-oracle-scan IN A 192.168.33.120 db-oracle-scan IN A 192.168.33.121 db-oracle-scan IN A 192.168.33.122 ****************************************************** vim /var/named/33.168.192.in-addr.arpa ****************************************************** $ORIGIN 33.168.192.in-addr.arpa. $TTL 1H @ IN SOA db-oracle-node1.localdomain. root.db-oracle-node1.localdomain. ( 2 3H 1H 1W 1H ) 33.168.192.in-addr.arpa. IN NS db-oracle-node1.localdomain. 103 IN PTR db-oracle-node1.localdomain. 104 IN PTR db-oracle-node2.localdomain. 111 IN PTR db-oracle-node1-vip.localdomain. 112 IN PTR db-oracle-node2-vip.localdomain. 120 IN PTR db-oracle-node-scan.localdomain. 121 IN PTR db-oracle-node-scan.localdomain. 122 IN PTR db-oracle-node-scan.localdomain. ****************************************************** ## A记录和PTR记录 # A记录是将domain解析成ip # PTR是将ip反解析成domain
step 3 启动dns服务
service named start chkconfig named on
step 4 配置节点的dns
# 节点上配置dns
vim /etc/resolv.conf
******************************************************
search localdomain
nameserver 192.168.33.103
******************************************************
step 5 检测oracle SCAN的解析
## host命令解析db-oracle-scan host db-oracle-scan db-oracle-scan.localdomain has address 192.168.33.120 db-oracle-scan.localdomain has address 192.168.33.121 db-oracle-scan.localdomain has address 192.168.33.122 ## 检测SCAN的三个ip轮询 # 第一次请求 nslookup db-oracle-scan Server: 192.168.33.103 Address: 192.168.33.103#53 Name: db-oracle-scan.localdomain Address: 192.168.33.120 Name: db-oracle-scan.localdomain Address: 192.168.33.121 Name: db-oracle-scan.localdomain Address: 192.168.33.122 # 第二次请求 nslookup db-oracle-scan Server: 192.168.33.103 Address: 192.168.33.103#53 Name: db-oracle-scan.localdomain Address: 192.168.33.121 Name: db-oracle-scan.localdomain Address: 192.168.33.122 Name: db-oracle-scan.localdomain Address: 192.168.33.120
step 6 检测每个节点的vip
# 检测node1的vip host db-oracle-node1-vip db-oracle-node1-vip.localdomain has address 192.168.33.111 nslookup db-oracle-node1-vip Server: 192.168.33.103 Address: 192.168.33.103#53 Name: db-oracle-node1-vip.localdomain Address: 192.168.33.111 # 检测node2的vip host db-oracle-node2-vip db-oracle-node2-vip.localdomain has address 192.168.33.112 nslookup db-oracle-node2-vip Server: 192.168.33.103 Address: 192.168.33.103#53 Name: db-oracle-node2-vip.localdomain Address: 192.168.33.112
step 7 检测反向解析
## 检测反向解析 dig -x 192.168.33.120 ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.47.rc1.el6_8.3 <<>> -x 192.168.33.120 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48957 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1 ;; QUESTION SECTION: ;120.33.168.192.in-addr.arpa. IN PTR ;; ANSWER SECTION: 120.33.168.192.in-addr.arpa. 3600 IN PTR db-oracle-node-scan.localdomain. ;; AUTHORITY SECTION: 33.168.192.in-addr.arpa. 3600 IN NS db-oracle-node1.localdomain. ;; ADDITIONAL SECTION: db-oracle-node1.localdomain. 86400 IN A 192.168.33.103 ;; Query time: 0 msec ;; SERVER: 192.168.33.103#53(192.168.33.103) ;; WHEN: Fri Dec 23 02:39:47 2016 ;; MSG SIZE rcvd: 136
至此，我们提供了以下项目的ip解析
| ip | hostname |
|---|---|
| 192.168.33.120 | db-oracle-node-scan |
| 192.168.33.121 | db-oracle-node-scan |
| 192.168.33.122 | db-oracle-node-scan |
| 192.168.33.103 | db-oracle-node1 |
| 192.168.33.104 | db-oracle-node2 |
| 192.168.33.111 | db-oracle-node1-vip |
| 192.168.33.112 | db-oracle-node2-vip |
public网络包含两个网卡eth1和eth2，这两个网卡会做绑定配置以达到高可用的效果。
step 1 关闭NetworkManager(各节点)
service NetworkManager stop chkconfig NetworkManager off
step 2 配置eth1和eth2的bond
echo "alias bond0 bonding" > /etc/modprobe.d/bonding.conf cp /etc/sysconfig/network-scripts/ifcfg-eth1 /etc/sysconfig/network-scripts/ifcfg-eth1.bak cp /etc/sysconfig/network-scripts/ifcfg-eth2 /etc/sysconfig/network-scripts/ifcfg-eth2.bak vim /etc/sysconfig/network-scripts/ifcfg-bond0 ****************************************************** DEVICE="bond0" BONDING_OPTS="mode=1 miimon=100 primary=eth1" NM_CONTROLLED="no" IPADDR="192.168.33.103" NETMASK="255.255.255.0" ONBOOT="yes" ****************************************************** vim /etc/sysconfig/network-scripts/ifcfg-eth1 ****************************************************** DEVICE=eth1 NM_CONTROLLED=no IPV6INIT=no BOOTPROTO=none ONBOOT=yes SLAVE=yes MASTER=bond0 ****************************************************** vim /etc/sysconfig/network-scripts/ifcfg-eth2 ****************************************************** DEVICE=eth2 NM_CONTROLLED=no IPV6INIT=no BOOTPROTO=none ONBOOT=yes SLAVE=yes MASTER=bond0 ******************************************************
step 3 重启network服务
service network restart
step 4 检查bond0状态
ifconfig bond0
bond0 Link encap:Ethernet HWaddr 08:00:27:C6:FD:B7
inet addr:192.168.33.103 Bcast:192.168.33.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fec6:fdb7/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:1009 errors:0 dropped:0 overruns:0 frame:0
TX packets:464 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:95221 (92.9 KiB) TX bytes:74759 (73.0 KiB)
ifconfig eth1
eth1 Link encap:Ethernet HWaddr 08:00:27:C6:FD:B7
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:5243 errors:0 dropped:0 overruns:0 frame:0
TX packets:2573 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:514986 (502.9 KiB) TX bytes:406781 (397.2 KiB)
ifconfig eth2
eth2 Link encap:Ethernet HWaddr 08:00:27:C6:FD:B7
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1177 errors:0 dropped:0 overruns:0 frame:0
TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:142004 (138.6 KiB) TX bytes:1276 (1.2 KiB)
step 5 检查public状态
# db-oracle-node1上 ping db-oracle-node2 PING db-oracle-node2.localdomain (192.168.33.104) 56(84) bytes of data. 64 bytes from db-oracle-node2.localdomain (192.168.33.104): icmp_seq=1 ttl=64 time=1.53 ms 64 bytes from db-oracle-node2.localdomain (192.168.33.104): icmp_seq=2 ttl=64 time=0.619 ms ... # db-oracle-node2上 ping db-oracle-node2 PING db-oracle-node2.localdomain (192.168.33.104) 56(84) bytes of data. 64 bytes from db-oracle-node2.localdomain (192.168.33.104): icmp_seq=1 ttl=64 time=0.011 ms 64 bytes from db-oracle-node2.localdomain (192.168.33.104): icmp_seq=2 ttl=64 time=0.038 ms ...
| ip | hostname |
|---|---|
| 192.168.33.103 | db-oracle-node1 |
| 193.168.33.104 | db-oracle-node2 |
private网络包含两个网卡eth3和eth4，提供集群节点的内部通信。通过oracle的Highly Available Internet Protocol (HAIP)，我们可以使得Oracle Grid Infrastructure可以在多达4个网卡上进行负载均衡。 step 1 配置文件修改
cp /etc/sysconfig/network-scripts/ifcfg-eth3 /etc/sysconfig/network-scripts/ifcfg-eth3.bak cp /etc/sysconfig/network-scripts/ifcfg-eth4 /etc/sysconfig/network-scripts/ifcfg-eth4.bak vim /etc/sysconfig/network-scripts/ifcfg-eth3 ****************************************************** DEVICE=eth3 NM_CONTROLLED=no IPV6INIT=no BOOTPROTO=none ONBOOT=yes IPADDR=172.16.44.103 NETMASK=255.255.255.0 PEERDNS=no MTU=9000 ****************************************************** vim /etc/sysconfig/network-scripts/ifcfg-eth4 ****************************************************** DEVICE=eth4 NM_CONTROLLED=no IPV6INIT=no BOOTPROTO=none ONBOOT=yes IPADDR=172.16.48.103 NETMASK=255.255.255.0 PEERDNS=no MTU=9000 ******************************************************
要确保同一个节点的eth3和eth4不在同一个subnet中，否则若连接丢失的情况下会导致节点重启
step 2 检查private网络状态
# db-oracle-node1 ping 172.16.44.104 PING 172.16.44.104 (172.16.44.104) 56(84) bytes of data. 64 bytes from 172.16.44.104: icmp_seq=1 ttl=64 time=0.343 ms 64 bytes from 172.16.44.104: icmp_seq=2 ttl=64 time=0.577 ms ... ping 172.16.48.104 PING 172.16.48.104 (172.16.48.104) 56(84) bytes of data. 64 bytes from 172.16.48.104: icmp_seq=1 ttl=64 time=0.207 ms 64 bytes from 172.16.48.104: icmp_seq=2 ttl=64 time=0.588 ms ... # db-oracle-node2 ping 172.16.44.103 PING 172.16.44.103 (172.16.44.103) 56(84) bytes of data. 64 bytes from 172.16.44.103: icmp_seq=1 ttl=64 time=1.65 ms 64 bytes from 172.16.44.103: icmp_seq=2 ttl=64 time=0.431 ms ping 172.16.48.103 PING 172.16.48.103 (172.16.48.103) 56(84) bytes of data. 64 bytes from 172.16.48.103: icmp_seq=1 ttl=64 time=0.721 ms 64 bytes from 172.16.48.103: icmp_seq=2 ttl=64 time=0.556 ms ...
| IP | Ethernet Interface | Host |
|---|---|---|
| 172.16.44.103 | eth3 | db-oracle-node1 |
| 172.16.48.103 | eth4 | db-oracle-node1 |
| 172.16.44.104 | eth3 | db-oracle-node2 |
| 172.16.48.104 | eth4 | db-oracle-node2 |