CentOS7下Solr6.6+ZooKeeper 集群安装部署

CentOS7下ZooKeeper 集群安装部署

安装JDK

yum install -y java-1.8.0-openjdk

安装ZooKeeper

下载

wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz
tar -zxvf zookeeper-3.4.10.tar.gz

官方地址：http://www.apache.org/dyn/closer.cgi/zookeeper/

配置zoo.cfg

cp conf/zoo_sample.cfg conf/zoo.cfg
vi conf/zoo.cfg

修改(本)

# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.
dataDir=/data/solr/zookeeper
# the port at which the clients will connect
clientPort=2181
server.211=dt211.corp:2888:3888
server.212=0.0.0.0:2888:3888
server.214=dt214.corp:2888:3888

创建数据目录和myid

mkdir /data/solr/zookeepervi /data/solr/zookeeper/myid

myid 文件中只有一个数字，与server id对应

212

防火墙配置

firewall-cmd--permanent--add-port=2181/tcpfirewall-cmd--permanent--add-port=2888/tcpfirewall-cmd--permanent--add-port=3888/tcpfirewall-cmd--reload

同步到其它机器

CentOS7 NFS共享设置

安装nfs工具

yum install -y nfs-utils

配置共享目录

创建共享目录

mkdir /home/share

目录授权

chmod766 /home/share

编辑exports

vi /etc/exports

添加内容

/home/share 192.168.1.0/24(rw,sync)

先启动rpcbind

systemctl start rpcbind

在启动nfs

systemctl start nfs

查看本机共享目录

exportfs

防火墙设置

firewall-cmd--permanent--add-service=nfsfirewall-cmd--reload

挂载共享目录

安装nfs，同上
创建挂载目录

mkdir /mnt/share

添加挂载

mount 192.168.1.212:/home/share /mnt/share

异常处理

1.mount.nfs: requested NFS version or transport protocol is not supported
启动顺序不对，先启动rpcbind，在启动nfs

2.mount.nfs: access denied by server while mounting

修改zoo.cfg，本机IP地址：0.0.0.0

启动集群

启动

bin/zkServer.sh start

查看状态

bin/zkServer.sh status

停止

bin/zkServer.sh stop

异常处理

1.java.net.ConnectException: 拒绝连接 (Connection refused)
本机IP要改成0.0.0.0，无法识别别名，127.0.0.1也只能有一台

CentOS7下Solr6.6 集群安装部署

Solr

添加zk地址

vi bin/solr.in.sh

# Set the ZooKeeper connection string ifusing an external ZooKeeper ensemble # e.g. host1:2181,host2:2181/chroot # Leave empty ifnotusing SolrCloud ZK_HOST="hbase201:2181,hbase202:2181,hbase203:2181/solr" # Set the ZooKeeper client timeout (for SolrCloud mode) ZK_CLIENT_TIMEOUT="15000"

上传配置文件

./zkcli.sh -zkhost hbase201,hbase202,hbase203:2181/solr -cmd upconfig -confdir /usr/solr/solr-6.6.0/server/solr/configsets/data_driven_schema_configs/conf -confname solr

同步到其它服务器

scp -r /usr/solr hbase201:/usr/
scp -r /usr/solr hbase202:/usr/

分别启动solr服务

bin/solr start -force

集群操作

查看状态

http://hbase203:8983/solr/#/~cloud

创建集群

http://hbase203:8983/solr/admin/collections?action=CREATE&name=test&numShards=3&replicationFactor=3&maxShardsPerNode=3&collection.configName=solr

删除集群

http://hbase203:8983/solr/admin/collections?action=DELETE&name=test

重新加载

http://hbase203:8983/solr/admin/collections?action=RELOAD&name=test

更新配置文件，添加DIH

修改solrconfig.xml

vi solrconfig.xml

<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />  

<requestHandlername="/dataimport"class="solr.DataImportHandler"><lstname="defaults"><strname="config">data-config.xml</str></lst></requestHandler>

创建data-config.xml

vi data-config.xml

<dataConfig><dataSourcetype="JdbcDataSource"driver="com.mysql.jdbc.Driver"convertType="true"url="jdbc:mysql://127.0.0.1:3306/db_name"user="sa"password="123456"/><document><entityname="query_news"query="SELECT id, title, content, tags FROM news" ></entity></document></dataConfig>

创建schema.xml

vi schema.xml

<schemaname="query_news"><uniqueKey>id</uniqueKey><fieldname="id"type="int"indexed="true"stored="true"required="true"multiValued="false" /><fieldname="title"type="string"indexed="true"stored="true"required="true"multiValued="false" /><fieldname="content"type="string"indexed="true"stored="true"required="false"multiValued="false" /><fieldname="tags"type="string"indexed="true"stored="false"required="false"multiValued="false" /><fieldname="_version_"type="long"indexed="true"stored="true"/><fieldname="info_text"type="text_general"indexed="true"stored="false"multiValued="true" /><copyFieldsource="title"dest="info_text" /><copyFieldsource="content"dest="info_text" /><copyFieldsource="tags"dest="info_text" /><fieldTypename="string"class="solr.StrField"sortMissingLast="true" /><fieldTypename="int"class="solr.TrieIntField"precisionStep="0"positionIncrementGap="0"/><fieldTypename="text_general"class="solr.TextField"positionIncrementGap="100"><analyzertype="index"><tokenizerclass="solr.StandardTokenizerFactory"/><filterclass="solr.LowerCaseFilterFactory"/></analyzer><analyzertype="query"><tokenizerclass="solr.StandardTokenizerFactory"/><filterclass="solr.LowerCaseFilterFactory"/></analyzer></fieldType></schema>

上传配置文件

./zkcli.sh -zkhost hbase201,hbase202,hbase203:2181/solr -cmd putfile /configs/solr/solrconfig.xml /usr/solr/solr-6.6.0/server/solr/configsets/data_driven_schema_configs/conf/solrconfig.xml./zkcli.sh -zkhost hbase201,hbase202,hbase203:2181/solr -cmd putfile /configs/solr/data-config.xml /usr/solr/solr-6.6.0/server/solr/configsets/data_driven_schema_configs/conf/data-config.xml./zkcli.sh -zkhost hbase201,hbase202,hbase203:2181/solr -cmd putfile /configs/solr/schema.xml /usr/solr/solr-6.6.0/server/solr/configsets/data_driven_schema_configs/conf/schema.xml

更新配置文件

http://hbase201:8983/solr/admin/collections?action=RELOAD&name=test

备份和恢复

备份

http://hbase201:8983/solr/test/replication?command=backup&location=/home/test_bak

恢复

1. 停止Solr服务
2. Copy备份文件到CoreIndex目录
3. 启动Solr服务

Index存储为HDFS

修改solrconfig.xml

vi solrconfig.xml

<directoryFactory name="DirectoryFactory"class="solr.HdfsDirectoryFactory">
  <str name="solr.hdfs.home">hdfs://hbase201:9000/solr</str>
  <bool name="solr.hdfs.blockcache.enabled">true</bool>
  <int name="solr.hdfs.blockcache.slab.count">1</int>
  <bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
  <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
  <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
  <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
  <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
  <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>

 

<lockType>${solr.lock.type:hdfs}</lockType>

上传配置文件到zookeeper

./zkcli.sh -zkhost hbase201,hbase202,hbase203:2181/solr -cmd putfile /configs/solr/data-config.xml /usr/solr/solr-6.6.0/server/solr/configsets/data_driven_schema_configs/conf/data-config.xml

更新Hadoop处理jar包（solr自带版本不一致）

rm -f server/solr-webapp/webapp/WEB-INF/lib/hadoop-*
rm -f server/solr-webapp/webapp/WEB-INF/lib/protobuf-java-*

cd /usr/hadoop/hadoop-2.7.4/share

cp hadoop/common/hadoop-common-2.7.4.jar /usr/solr/solr-6.6.0/server/solr-webapp/webapp/WEB-INF/lib/
cp hadoop/common/lib/hadoop-* /usr/solr/solr-6.6.0/server/solr-webapp/webapp/WEB-INF/lib/
cp hadoop/common/lib/protobuf-java-2.5.0.jar /usr/solr/solr-6.6.0/server/solr-webapp/webapp/WEB-INF/lib/
cp hadoop/hdfs/hadoop-hdfs-2.7.4.jar /usr/solr/solr-6.6.0/server/solr-webapp/webapp/WEB-INF/lib/