LinuxEye - Linux系统教程
TAG标签RSS订阅[设为首页] [加入收藏]

LinuxEye - Linux系统教程

热门标签:

当前位置: 主页 > Linux教程 >

mfs分布式存储+master端高可用

时间:2015-08-03 09:42来源:未知 编辑:linuxeye 点击: 次
master采用keepalived+drbd实现高可用 mfsmaster 172.25.254.10 mfsbackup 172.25.254.11 VIP 是172.25.254.12 mfsmaster 和mfsbackup一定要主机名解析,同时主机名也必须和主机名解析一致。 uname -n查看你的名字~ dr
master采用keepalived+drbd实现高可用
mfsmaster 172.25.254.10
mfsbackup 172.25.254.11

VIP 是172.25.254.12


mfsmaster 和mfsbackup一定要主机名解析,同时主机名也必须和主机名解析一致。
uname -n查看你的名字~


drbd配置:
Mfsmaster mfsbackup 都做以下操作,首先安装包，这是我自己打的rpm包。
 

安装即可
vim /etc/drbd.d/mfs.res 
resource mfs {                                                                                              
        meta-disk internal;
        device /dev/drbd1;
        syncer {
                verify-alg sha1;
        }
on mfsmaster {
        disk /dev/vda1;
        address 172.25.254.10:7789;
}
on mfsbackup {
        disk /dev/vda1;
        address 172.25.254.11:7789;
}
}
安装mfs master 

[root@mfsmaster etc]# cd  /etc
[root@mfsmaster etc]# mv mfsmaster.cfg.dist mfsmaster.cfg
[root@mfsmaster etc]# mv mfsexports.cfg.dist mfsexports.cfg
[root@mfsmaster etc]# mv mfsmetalogger.cfg.dist mfsmetalogger.cfg
[root@mfsmaster etc]# mv mfstopology.cfg.dist mfstopology.cfg
vim mfsmaster.cfg 
WORKING_USER = nobody
WORKING_GROUP = nobody
SYSLOG_IDENT = mfsmaster
LOCK_MEMORY = 0
NICE_LEVEL = -19

EXPORTS_FILENAME = /etc/mfsexports.cfg

TOPOLOGY_FILENAME = /etc/mfstopology.cfg

DATA_PATH = /data1/drbd

BACK_LOGS = 50
BACK_META_KEEP_PREVIOUS = 1

REPLICATIONS_DELAY_INIT = 300
REPLICATIONS_DELAY_DISCONNECT = 3600

MATOML_LISTEN_HOST = *
MATOML_LISTEN_PORT = 9419

MATOCS_LISTEN_HOST = *
MATOCS_LISTEN_PORT = 9420

MATOCL_LISTEN_HOST = *
MATOCL_LISTEN_PORT = 9421

CHUNKS_LOOP_CPS = 100000
CHUNKS_LOOP_TIME = 300

CHUNKS_SOFT_DEL_LIMIT = 10
CHUNKS_HARD_DEL_LIMIT = 25
CHUNKS_WRITE_REP_LIMIT = 2
CHUNKS_READ_REP_LIMIT = 10

REJECT_OLD_CLIENTS = 0
vim mfsmetalogger.cfg 
WORKING_USER = nobody                                                                                      
WORKING_GROUP = nobody 
SYSLOG_IDENT = mfsmetalogger
LOCK_MEMORY = 0
NICE_LEVEL = -19

DATA_PATH = /var/lib/mfs

BACK_LOGS = 50
BACK_META_KEEP_PREVIOUS = 3
META_DOWNLOAD_FREQ = 24

MASTER_RECONNECTION_DELAY = 5

MASTER_HOST = mfsmaster
MASTER_PORT = 9419

MASTER_TIMEOUT = 60
ok 为了方便，直接把这几个文件复制到mfsbackup上就可以 
[root@mfsmaster etc]# scp mfs* 172.25.254.11:/etc/
root@172.25.254.11's password: 
mfsexports.cfg                                                                100% 4060     4.0KB/s   00:00    
mfsmaster.cfg                                                                 100%  849     0.8KB/s   00:00    
mfsmetalogger.cfg                                                             100%  401     0.4KB/s   00:00    
mfstopology.cfg                                                               100% 1123     1.1KB/s   00:00
master上执行 
[root@mfsmaster etc]# drbdsetup /dev/drbd1 primary --force
[root@mfsmaster etc]# mkfs.ext4 /dev/drbd1
都执行 
mkdir -p /data1/drbdchown -R nobody.nobody  /data1/drbd/
[root@mfsmaster mfs]# cp /var/lib/mfs/metadata.mfs.empty /data1/drbd/
[root@mfsmaster drbd]# mv metadata.mfs.empty metadata.mfs
[root@mfsmaster drbd]# mfsmaster
ok master 已经可以正常运行了

,为了保证无误我们在backup上也测试一次。

[root@mfsmaster drbd]# mfsmaster stop
[root@mfsmaster drbd]# cd[root@mfsmaster ~]
# umount /dev/drbd1
[root@mfsmaster ~]# drbdadm secondary mfs
[root@mfsbackup drbd+mfs-rpm]# drbdadm primary mfs
[root@mfsbackup drbd+mfs-rpm]# mount /dev/drbd1 /data1/drbd/
[root@mfsbackup drbd+mfs-rpm]# mfsmaster

这个时候我们的master端已经全线部署完成，接下来我们要实现他的高可用，这个很重要

首先说明，我使用自己的方法很简单的实现主断备启，但是是一次性的，也就是说，只支持一次主的mfsmaster断了，backup可以接替，并不能来回。需要手工操作。

脚本写的简单后续有时间修改，网上有很多成品的，虽然很菜还是写自己的。。

同样是keepalived +drbd大神们都是全自动的

我这其实是一次性的但是轻量简单呗。

安装keepalived 两端执行： 
! Configuration File for keepalived

global_defs {
   notification_email {
     576786031@qq.com
   }
   notification_email_from zabbix@server2.com
   smtp_server 172.25.254.2
   smtp_connect_timeout 30
   router_id mfs_master
}
vrrp_script check_drbd {
        script "/etc/keepalived/check_drbd.sh"
            interval 15     
    }
vrrp_instance mfs {
    state MASTER
    interface eth0
    virtual_router_id 51                                                                                                                                                 
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.12
    }
    track_script {
        check_drbd
        }
notify_master /etc/keepalived/master.sh
notify_backup /etc/keepalived/backup.sh
}
[root@mfsmaster keepalived]# vim check_drbd.sh 
/bin/bash: Configuration: command not found
# File Name: check_drbd.sh
# Author: dty
# mail: 576786031@qq.com
# Created Time: Sun 02 Aug 2015 01:32:43 PM CST
#########################################################################
#!/bin/bash    
A=`ps -C mfsmaster --no-header |wc -l`
if [ $A -eq 0 ];then
    umount /data1/drbd/
    drbdadm secondary mfs
    killall keepalived     
fi
mfsbackup 上      cd /etc/keepalived
                写两个脚本 master.sh backup.sh
[root@mfsbackup keepalived]# cat backup.sh master.sh 
#!/bin/bash
mfsmaster stop
umount /dev/drbd1
drbdadm secondary mfs
#########################################################################
# File Name: master.sh
# Author: dty
# mail: 576786031@qq.com
# Created Time: Sun 02 Aug 2015 01:38:12 PM CST
#########################################################################
#!/bin/bash
drbdadm primary mfs
mount /dev/drbd1 /data1/drbd
mfsmaster start
vim /etc/keepalived.conf 
! Configuration File for keepalived
  
global_defs { 
   notification_email { 
     576786031@qq.com
   }      
   notification_email_from zabbix@server2.com
   smtp_server 172.25.254.2 
   smtp_connect_timeout 30
   router_id mfs_master
}       
vrrp_instance mfs {
    state BACKUP   
    interface eth0
    virtual_router_id 51
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.12
    }
notify_master /etc/keepalived/master.sh
notify_backup /etc/keepalived/backup.sh
}

此时我们可以实验了

开启mfsmaster，

开启keepalived

                      

mfsbackup上也启动 keepalived

[root@mfsbackup keepalived]# /etc/init.d/keepalived start

我们把主上的mfsmaster关掉

可以看到vip已经没了，转移到了备机上

 

服务已经完美转移了 。

这个时候我们来搞定客户端和chunkserver的安装

  yum localinstall mfs-chunkserver-1.6.26-1.x86_64.rpm -y

 mv mfschunkserver.cfg.dist mfschunkserver

mkdir -p /data1/mfs //设置个目录存放数据

chown nobody.nobody /data1/mfs

 

 vim /etc/mfschunkserver

WORKING_USER = nobody                                                      
WORKING_GROUP = nobody 
SYSLOG_IDENT = mfschunkserver
LOCK_MEMORY = 0
NICE_LEVEL = -19

DATA_PATH = /data1/mfs

MASTER_RECONNECTION_DELAY = 5

BIND_HOST = *
MASTER_HOST = 172.25.254.12
MASTER_PORT = 9420

MASTER_TIMEOUT = 60

CSSERV_LISTEN_HOST = *
CSSERV_LISTEN_PORT = 9422

HDD_CONF_FILENAME = /etc/mfshdd.cfg
HDD_TEST_FREQ = 10
设置chunk存放路径 
[root@server7 ~]# cat /etc/mfshdd.cfg
# mount points of HDD drives
#
#/mnt/hd1
#/mnt/hd2
#etc.
/data1/mfs/
chunkserver已经部署完成了有多个以此类推就好了.

下来就是客户端的安装及使用了

# yum localinstall -y mfs-client-1.6.26-1.x86_64.rpm
# cd /etc# cp mfsmount.cfg.dist mfsmount.cfg
# vi mfsmount.cfg定义客户端默认挂载mfsmaster=mfsmaster/mnt/mfs# mfsmount
[root@server4 ~]# df -h Filesystem                    Size  Used Avail Use% Mounted on/dev/mapper/VolGroup-lv_root  6.7G  978M  5.4G  16% /tmpfs
 inodes with permission denied:          0 
[root@server4 mfs]# cp /etc/passwd leaf1
[root@server4 mfs]# cp /etc/passwd leaf2
[root@server4 mfs]#  mfsfileinfo leaf1/passwd
leaf1/passwd:
    chunk 0: 0000000000000008_00000001 / (id:8 ver:1)
        copy 1: 172.25.254.7:9422
[root@server4 mfs]#  mfsfileinfo leaf2/passwd
leaf2/passwd:
    chunk 0: 0000000000000009_00000001 / (id:9 ver:1)
        copy 1: 172.25.254.8:9422
        copy 2: 172.25.254.9:9422
ok 效果就这样粗来了

其他的mfs的使用这里不做赘述，主要为了验证我们的高可用机制是否可行，就到这里了

ps：当出现问题调转到backup后，如果主的正常了，我们手动切换必须遵守以下步骤

  1. 停掉mfsmaster服务

     mfsmaster stop

          umount /dev/drbd1

         drbdadm secondary mfs

  1. 主机：

    1.drbdadm  primary mfs

        2.mount /dev/drbd1 /data1/drbd

        3.mfsmaster start

        4.keepalived start

还有要提到的就是可能会出现脑裂，这一般时操作不按顺序，也可能有其他原因，下面是网上大神写的，

脑裂的处理方法

drbd1主,drbd2辅

1，正常情况下状态：
 [root@drbd1 ~]# cat /proc/drbd 
 version: 8.3.8 (api:88/proto:86-94)
 : 299AFE04D7AFD98B3CA0AF9 
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
  ns:2144476 nr:0 dw:36468 dr:2115769 al:14 bm:129 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
  
[root@drbd2 ~]# cat /proc/drbd 
  
version: 8.3.8 (api:88/proto:86-94)
  
srcversion: 299AFE04D7AFD98B3CA0AF9 
  
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
  
    ns:0 nr:2141684 dw:2141684 dr:0 al:0 bm:130 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
  
2，drbd1故障后
  
drbd1状态：
  
[root@drbd1 ~]# cat /proc/drbd 
  
version: 8.3.8 (api:88/proto:86-94)
  
srcversion: 299AFE04D7AFD98B3CA0AF9 
  
 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r----
  
    ns:4 nr:102664 dw:102668 dr:157 al:1 bm:8 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
  
drbd2的状态：
  
[root@drbd2 ~]# cat /proc/drbd 
  
version: 8.3.8 (api:88/proto:86-94)
  
srcversion: 299AFE04D7AFD98B3CA0AF9 
  
 0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----
  
    ns:0 nr:2141684 dw:2141684 dr:0 al:0 bm:130 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
  
3，处理方法：
  
a，将secondary配置成primary角色
  
[root@drbd2 ~]# drbdsetup /dev/drbd0 primary -o
  
[root@drbd2 ~]# cat /proc/drbd 
  
version: 8.3.8 (api:88/proto:86-94)
  
srcversion: 299AFE04D7AFD98B3CA0AF9 
  
 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/Outdated C r----
  
    ns:0 nr:2141684 dw:2141684 dr:0 al:0 bm:130 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
  
挂载：
  
[root@drbd2 /]# mount /dev/drbd0 /data1
  
[root@drbd2 data1]# ll
  
total 10272
  
-rw-r--r-- 1 root root 10485760 Feb 13 11:26 aa.img
  
drwx------ 2 root root    16384 Feb 13 11:25 lost+found
  
这个时候drbd2开始提供服务，开始写数据
  
drbd1主恢复正常后：
  
[root@drbd1 ~]# cat /proc/drbd
  
version: 8.3.8 (api:88/proto:86-94)
  
srcversion: 299AFE04D7AFD98B3CA0AF9 
  
 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r----
  
    ns:2144476 nr:0 dw:36484 dr:2115769 al:14 bm:129 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:8
  
drbd1状态是:StandAlone，此时，drbd1是不会和drbd2互相联系的
  
我们来查看下日志：
  
[root@drbd1 ~]# tailf /var/log/messages
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: conn( WFReportParams -> Disconnecting ) 
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: error receiving ReportState, l: 4!
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: asender terminated
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: Terminating drbd0_asender
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: Connection closed
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: conn( Disconnecting -> StandAlone ) 
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: receiver terminated
  
Feb 13 16:14:27 drbd1 kernel: block drbd0: Terminating drbd0_receiver
  
脑裂出现！
  
解决方法：
  
1>,我们需要将现在的drbd1角色修改为secondary
  
[root@drbd1 ~]# drbdadm secondary r0
  
[root@drbd1 ~]# drbdadm -- --discard-my-data connect r0  ##该命令告诉drbd，secondary上的数据不正确，以primary上的数据为准。
  
2>，我们还需要在drbd2上执行下面操作
  
[root@drbd2 /]# drbdadm connect r0
  
这样drbd1就能和drbd2开始连接上了，并且保证数据不会丢失：
  
[root@drbd1 ~]# cat /proc/drbd      
  
version: 8.3.8 (api:88/proto:86-94)
  
srcversion: 299AFE04D7AFD98B3CA0AF9 
  
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
  
    ns:0 nr:20592 dw:20592 dr:0 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

转载请保留固定链接： https://linuxeye.com/Linux/2760.html
------分隔线----------------------------
标签：MFSmaster
栏目列表
推荐内容
热点内容

Copyright © 2021 LINUXEYE
沪ICP备2021018617号-2