NFS+Heartbeat+Drbd高可用架构(2)

三、部署 heartbeat 服务
此处仅演示 M1 服务端的安装，M2 的不做复述。
1、安装heartbeat软件

[root@M1 ~]# cd /etc/yum.repos.d/ 
[root@M1 yum.repos.d]# wget http://mirrors.163.com/.help/CentOS6-Base-163.repo
[root@M1 yum.repos.d]# rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@M1 yum.repos.d]# sed -i 's@#baseurl@baseurl@g' *
[root@M1 yum.repos.d]# sed -i 's@mirrorlist@#mirrorlist@g' *
[root@M1 yum.repos.d]# yum install heartbeat -y     # 该命令有时可能需要执行2次

2、配置heartbeat服务

[root@M1  yum.repos.d]# cd /usr/share/doc/heartbeat-3.0.4/ 
[root@M1 heartbeat-3.0.4]# ll |egrep 'ha.cf|authkeys|haresources' 
-rw-r--r--. 1 root root 645 Dec 3 2013 authkeys              # heartbeat服务的认证文件
-rw-r--r--. 1 root root 10502 Dec 3 2013 ha.cf                 # heartbeat服务主配置文件
-rw-r--r--. 1 root root 5905 Dec 3 2013 haresources       # heartbeat资源文件
[root@M1 heartbeat-3.0.4]# cp ha.cf authkeys haresources /etc/ha.d/ 
[root@M1 heartbeat-3.0.4]# cd /etc/ha.d/
[root@M1 ha.d]# ls 
authkeys ha.cf harc haresources rc.d README.config resource.d shellfuncs

注意：主备节点两端的配置文件(ha.cf,authkeys,haresource)完全相同，下面是各个节点的文件内容
针对heartbeat的配置，主要就是修改ha.cf、authkeys、haresources这三个文件，下面我列出这三个文件的配置信息，大家仅作参考！

a、ha.cf 文件

[root@M1 ~]# cat /etc/ha.d/ha.cf 
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility     local0
keepalive 2
deadtime 10
warntime 6
#initdead 120
udpport 694
#bcast em2
mcast em2 225.0.0.192 694 1 0
auto_failback on
respawn hacluster /usr/lib64/heartbeat/ipfail
node    M1.redhat.sx
node    M2.redhat.sx
ping 192.168.0.1

b、authkeys 文件

[root@M1 ha.d]# cat authkeys 
auth 1                    # 采用何种加密方式
1 crc                       # 无加密
#2 sha1 HI!            # 启用sha1的加密方式
#3 md5 Hello!       # 采用md5的加密方式
[root@M1 ha.d]# chmod 600 authkeys    # 该文件必须设置为600权限，不然heartbeat启动会报错

c、haresources 文件

[root@M1 ha.d]# cat haresources 
M1.redhat.sx IPaddr::192.168.0.219/24/em1
#NFS IPaddr::192.168.0.219/24/em1 drbddisk::data Filesystem::/dev/drbd0::/data::ext4 rpcbind nfsd

注意：这个里的nfsd并不是heartbeat自带的，需要自己编写。

针对该脚本的编写需要满足一下需求：
           1、有可执行权限
           2、必须存放在/etc/ha.d/resource.d或/etc/init.d目录下
           3、必须有start、stop这两个功能
           具体脚本信息，下文会写。

4、启动heartbeat

[root@M1 ha.d]# /etc/init.d/heartbeat start 
Starting High-Availability services: INFO: Resource is stopped 
Done.
[root@M1 ha.d]# chkconfig heartbeat off

说明：关闭开机自启动。当服务重启时，需要人工去启动。

5、测试heartbeat
在此步测试之前，请先在 M2 上操作如上步骤！
a、正常状态

[root@M1 ha.d]# ip a|grep em1 
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 
inet 192.168.0.210/24 brd 192.168.0.255 scope global em1 
inet 192.168.0.219/24 brd 192.168.0.255 scope global secondary em1    #  之前在heartbeat资源文件中定义的 VIP
[root@M2 ha.d]# ip a|grep em1 
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 
inet 192.168.0.211/24 brd 192.168.0.255 scope global em1

说明：M1主节点拥有vip地址，M2节点没有。

b、模拟主节点宕机后的状态

[root@M1 ha.d]# /etc/init.d/heartbeat stop 
Stopping High-Availability services: Done. 
[root@M1 ha.d]# ip a|grep em1 
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 
inet 192.168.0.210/24 brd 192.168.0.255 scope global em1
[root@M2 ha.d]# ip a|grep em1 
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 
inet 192.168.0.211/24 brd 192.168.0.255 scope global em1 
inet 192.168.0.219/24 brd 192.168.0.255 scope global secondary em1

说明：M1宕机后，VIP地址漂移到M2节点上，M2节点成为主节点

c、模拟主节点故障恢复后的状态

[root@M1 ha.d]# /etc/init.d/heartbeat start 
Starting High-Availability services: INFO: Resource is stopped 
Done. 
[root@M1 ha.d]# ip a|grep em1 
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 
inet 192.168.0.210/24 brd 192.168.0.255 scope global em1 
inet 192.168.0.219/24 brd 192.168.0.255 scope global secondary em1

说明：M1节点恢复之后，又抢占回了VIP资源

四、DRBD安装部署
1、新添加（初始）硬盘
过程略

2、安装drbd
针对drbd的安装，我们不仅可以使用yum的方式，还可以使用编译安装的方式。由于我在操作的时候，无法从当前yum源取得drbd的rpm包，因此我就采用了编译的安装方式。

[root@M1 ~]# yum -y install gcc gcc-c++ kernel-devel kernel-headers flex make
[root@M1 ~]# cd /usr/local/src
[root@M1 src]# wget http://oss.linbit.com/drbd/8.4/drbd-8.4.3.tar.gz 
[root@M1 src]# tar zxf drbd-8.4.3.tar.gz 
[root@M1 src]# cd drbd-8.4.3 
[root@M1 ha.d]# ./configure --prefix=/usr/local/drbd --with-km --with-heartbeat 
[root@M1 ha.d]# make KDIR=/usr/src/kernels/2.6.32-504.el6.x86_64/
[root@M1 ha.d]# make install
[root@M1 ha.d]# mkdir -p /usr/local/drbd/var/run/drbd 
[root@M1 ha.d]# cp /usr/local/drbd/etc/rc.d/init.d/drbd /etc/init.d/ 
[root@M1 ha.d]# chmod +x /etc/init.d/drbd 
[root@M1 ha.d]# modprobe drbd      # 执行命令加载drbd模块到内核
[root@M1 ha.d]# lsmod|grep drbd     # 检查drbd是否被正确的加载到内核
drbd 310236 3 
libcrc32c 1246 1 drbd

3、配置DRBD
有关DRBD涉及到的配置文件主要是global_common.conf和用户自定义的资源文件（当然，该资源文件可以写到global_common.conf中）。
注意：M1和M2这两个主备节点的以下配置文件完全一样

[root@M1 ~]# cat /usr/local/drbd/etc/drbd.d/global_common.conf 
global {
        usage-count no; 
} 
common { 
        protocol C; 
        disk { 
                on-io-error detach;        # 配置I/O错误处理策略为分离
                no-disk-flushes; 
                no-md-flushes; 
        } 
        net { 
                cram-hmac-alg "sha1";         # 设置加密算法 
                shared-secret "allendrbd";    # 设置加密密钥
                sndbuf-size 512k;
                max-buffers 8000;
                unplug-watermark 1024;
                max-epoch-size 8000;
                after-sb-0pri disconnect;
                after-sb-1pri disconnect;
                after-sb-2pri disconnect;
                rr-conflict disconnect;
        } 
        syncer { 
                rate 1024M;             # 设置主备节点同步时的网络速率
                al-extents 517;
        }
}
[root@M1 ~]# cat /usr/local/drbd/etc/drbd.d/drbd.res
resource drbd {              # 定义一个drbd的资源名
        on M1.redhat.sx {                                 # 主机说明以on开头，后面跟主机名称
                device /dev/drbd0;                      # drbd设备名称
                disk /dev/mapper/VolGroup-lv_drbd; # drbd0 使用的是逻辑卷/dev/mapper/VolGroup-lv_drbd
                address 172.16.100.210:7789;      # 设置DRBD监听地址与端口
                meta-disk internal;                      # 设置元数据盘为内部模式
        } 
        on M2.redhat.sx { 
                device /dev/drbd0; 
                disk /dev/mapper/VolGroup-lv_drbd; 
                address 172.16.100.211:7789; 
                meta-disk internal; 
        } 
}

4、初始化meta分区

[root@M1 drbd]# drbdadm create-md drbd 
Writing meta data... 
initializing activity log 
NOT initializing bitmap 
New drbd meta data block successfully created.

5、启动drbd服务
此处，我们可以看下M1 和M2 启动drbd服务前后，drbd设备发生的变化
M1端：

[root@M1 drbd]# cat /proc/drbd       # 启动前 drbd 设备信息
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M1.redhat.sx, 2014-11-11 16:20:26 
[root@M1 drbd]# drbdadm up all      # 启动drbd，这里也可以使用脚本去启动
[root@M1 drbd]# cat /proc/drbd        # 启动后 drbd 设备信息
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M1.redhat.sx, 2014-11-11 16:20:26 
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- 
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:133615596

M2端：

[root@M2 ~]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M2.redhat.sx, 2014-11-11 16:25:08 
[root@M2 ~]# drbdadm up all 
[root@M2 ~]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M2.redhat.sx, 2014-11-11 16:25:08 
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- 
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:133615596

6、初始化设备同步,并确立主节点（覆盖备节点，保持数据一致）
M1端：

[root@M1 drbd]# drbdadm -- --overwrite-data-of-peer primary drbd 
[root@M1 drbd]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M1.redhat.sx, 2014-11-11 16:20:26 
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n- 
ns:140132 nr:0 dw:0 dr:144024 al:0 bm:8 lo:0 pe:17 ua:26 ap:0 ep:1 wo:d oos:133477612 
[>....................] sync'ed: 0.2% (130348/130480)M 
finish: 0:16:07 speed: 137,984 (137,984) K/sec

M2端：

[root@M2 ~]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M2.redhat.sx, 2014-11-11 16:25:08 
0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----- 
ns:0 nr:461440 dw:461312 dr:0 al:0 bm:28 lo:2 pe:75 ua:1 ap:0 ep:1 wo:d oos:133154284 
[>....................] sync'ed: 0.4% (130032/130480)M 
finish: 0:19:13 speed: 115,328 (115,328) want: 102,400 K/sec

同步完毕之后状态：
M1端：

[root@M1 ~]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M1.redhat.sx, 2014-11-11 16:20:26 
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- 
ns:133615596 nr:0 dw:0 dr:133616260 al:0 bm:8156 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0

M2端：

[root@M2 ~]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M2.redhat.sx, 2014-11-11 16:25:08 
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- 
ns:0 nr:133615596 dw:133615596 dr:0 al:0 bm:8156 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0

7、挂载drbd分区到data数据目录

[root@M1 drbd]# mkfs.ext4 /dev/drbd0 
mke2fs 1.41.12 (17-May-2010) 
Filesystem label= 
OS type: Linux 
Block size=4096 (log=2) 
Fragment size=4096 (log=2) 
Stride=0 blocks, Stripe width=0 blocks 
8355840 inodes, 33403899 blocks 
1670194 blocks (5.00%) reserved for the super user 
First data block=0 
Maximum filesystem blocks=4294967296 
1020 block groups 
32768 blocks per group, 32768 fragments per group 
8192 inodes per group 
Superblock backups stored on blocks: 
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
4096000, 7962624, 11239424, 20480000, 23887872 
 
Writing inode tables: done 
Creating journal (32768 blocks): done 
Writing superblocks and filesystem accounting information: done 
 
This filesystem will be automatically checked every 21 mounts or 
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@M1 drbd]# mount /dev/drbd0 /data/
[root@M1 drbd]# df -h
Filesystem Size Used Avail Use% Mounted on 
/dev/mapper/VolGroup-lv_root 
50G 5.6G 42G 12% / 
tmpfs 7.8G 0 7.8G 0% /dev/shm 
/dev/sda1 477M 46M 406M 11% /boot 
/dev/drbd0 126G 60M 119G 1% /data

8、测试主节点写入，备节点是否能同步
M1端：

[root@M1 drbd]# dd if=/dev/zero of=/data/test bs=1G count=1 
1+0 records in 
1+0 records out 
1073741824 bytes (1.1 GB) copied, 1.26333 s, 850 MB/s 
[root@M1 drbd]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M1.redhat.sx, 2014-11-11 16:20:26 
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- 
ns:135840788 nr:0 dw:2225192 dr:133617369 al:619 bm:8156 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
[root@M1 drbd]# umount /data/
[root@M1 drbd]# drbdadm down drbd    # 关闭名字为drbd的资源

M2端：

[root@M2 ~]# cat /proc/drbd                   # 主节点关闭资源之后，查看备节点的信息，可以看到主节点的角色已经变为UnKnown
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M2.redhat.sx, 2014-11-11 16:25:08 
0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- 
ns:0 nr:136889524 dw:136889524 dr:0 al:0 bm:8156 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
[root@M2 ~]# drbdadm primary drbd       # 确立自己的角色为primary，即主节点
[root@M2 ~]# mount /dev/drbd0 /data
[root@M2 ~]# cd /data 
[root@M2 data]# ls                                     # 发现数据还在
lost+found test 
[root@M2 data]# du -sh test 
1.1G test
[root@M2 data]# cat /proc/drbd                # 查看当前 drbd 设备信息
version: 8.4.3 (api:1/proto:86-101) 
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@M2.redhat.sx, 2014-11-11 16:25:08 
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----- 
ns:0 nr:136889524 dw:136889548 dr:1045 al:3 bm:8156 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:24

转载请保留固定链接： https://linuxeye.com/architecture/2045.html

热门标签:

NFS+Heartbeat+Drbd高可用架构(2)