ELK_DAY02_各类插件使用

1 案例1:导入数据
1.1 问题

本案例要求批量导入数据:
批量导入数据并查看
1.2 步骤

实现此案例需要按照如下步骤进行。
步骤一:导入数据

使用POST方式批量导入数据,数据格式为json,url 编码使用data-binary导入含有index配置的json文件
[[email protected] ~]# scp /var/ftp/elk/*.gz 192.168.1.66:/root/
[[email protected] ~]# gzip -d logs.jsonl.gz
[[email protected] ~]# gzip -d accounts.json.gz
[[email protected] ~]# gzip -d shakespeare.json.gz
[[email protected] ~]# curl -X POST “http://192.168.1.61:9200/_bulk” \
–data-binary @shakespeare.json
[[email protected] ~]# curl -X POST “http://192.168.1.61:9200/xixi/haha/_bulk” \
–data-binary @accounts.json
//索引是xixi,类型是haha,必须导入索引和类型,没有索引,要加上
[[email protected] ~]# curl -X POST “http://192.168.1.61:9200/_bulk” \
–data-binary @logs.jsonl
2)使用GET查询结果
[[email protected] ~]# curl -XGET ‘http://192.168.1.61:9200/_mget?pretty’ -d ‘{
“docs”:[
{
“_index”:”shakespeare”,
“_type:”:”act”,
“_id”:0
},
{
“_index”:”shakespeare”,
“_type:”:”line”,
“_id”:0
},
{
“_index”:”xixi”,
“_type:”:”haha”,
“_id”:25
}
]
}’
{ //查询的结果
“docs” : [ {
“_index” : “shakespeare”,
“_type” : “act”,
“_id” : “0”,
“_version” : 1,
“found” : true,
“_source” : {
“line_id” : 1,
“play_name” : “Henry IV”,
“speech_number” : “”,
“line_number” : “”,
“speaker” : “”,
“text_entry” : “ACT I”
}
}, {
“_index” : “shakespeare”,
“_type” : “act”,
“_id” : “0”,
“_version” : 1,
“found” : true,
“_source” : {
“line_id” : 1,
“play_name” : “Henry IV”,
“speech_number” : “”,
“line_number” : “”,
“speaker” : “”,
“text_entry” : “ACT I”
}
}, {
“_index” : “xixi”,
“_type” : “haha”,
“_id” : “25”,
“_version” : 1,
“found” : true,
“_source” : {
“account_number” : 25,
“balance” : 40540,
“firstname” : “Virginia”,
“lastname” : “Ayala”,
“age” : 39,
“gender” : “F”,
“address” : “171 Putnam Avenue”,
“employer” : “Filodyne”,
“email” : “[email protected]”,
“city” : “Nicholson”,
“state” : “PA”
}
} ]
}
代码如下
第一步,导入数据,代码太长,不贴了
[[email protected] ~]# gzip -d logs.jsonl.gz
[[email protected] ~]# gzip -d accounts.json.gz
[[email protected] ~]# gzip -d shakespeare.json.gz
[[email protected] ~]# curl -X POST “http://192.168.1.61:9200/_bulk” \
–data-binary @shakespeare.json
[[email protected] ~]# curl -X POST “http://192.168.1.61:9200/xixi/haha/_bulk” \
–data-binary @accounts.json
//索引是xixi,类型是haha,必须导入索引和类型,没有索引,要加上
[[email protected] ~]# curl -X POST “http://192.168.1.61:9200/_bulk” \
–data-binary @logs.jsonl

第二步,使用GET查询结果
[[email protected] ~]# curl -XGET ‘http://192.168.1.61:9200/_mget?pretty’ -d ‘{
> “docs”:[
> {
> “_index”:”shakespeare”,
> “_type:”:”act”,
> “_id”:0
> },
> {
> “_index”:”shakespeare”,
> “_type:”:”line”,
> “_id”:0
> },
> {
> “_index”:”xixi”,
> “_type:”:”haha”,
> “_id”:25
> }
> ]
> }’
输出的查询结果如下
{
“docs” : [ {
“_index” : “shakespeare”,
“_type” : “act”,
“_id” : “0”,
“_version” : 1,
“found” : true,
“_source” : {
“line_id” : 1,
“play_name” : “Henry IV”,
“speech_number” : “”,
“line_number” : “”,
“speaker” : “”,
“text_entry” : “ACT I”
}
}, {
“_index” : “shakespeare”,
“_type” : “act”,
“_id” : “0”,
“_version” : 1,
“found” : true,
“_source” : {
“line_id” : 1,
“play_name” : “Henry IV”,
“speech_number” : “”,
“line_number” : “”,
“speaker” : “”,
“text_entry” : “ACT I”
}
}, {
“_index” : “xixi”,
“_type” : “haha”,
“_id” : “25”,
“_version” : 1,
“found” : true,
“_source” : {
“account_number” : 25,
“balance” : 40540,
“firstname” : “Virginia”,
“lastname” : “Ayala”,
“age” : 39,
“gender” : “F”,
“address” : “171 Putnam Avenue”,
“employer” : “Filodyne”,
“email” : “[email protected]”,
“city” : “Nicholson”,
“state” : “PA”
}
} ]
}

步骤二:使用kibana查看数据是否导入成功

1)数据导入以后查看logs是否导入成功,如图-1所示:
[[email protected] ~]# firefox http://192.168.1.65:9200/_plugin/head/

图-1
2)kibana导入数据,如图-2所示:
[[email protected] ~]# firefox http://192.168.1.66:5601

图-2
3)成功创建会有logstash-*,如图-3所示:
/
图-3
4)导入成功之后选择Discover,如图-4所示:

图-4
注意: 这里没有数据的原因是导入日志的时间段不对,默认配置是最近15分钟,在这可以修改一下时间来显示
5)kibana修改时间,选择Lsat 15 miuntes,如图-5所示:

图-5
6)选择Absolute,如图-6所示:

图-6
7)选择时间2015-5-15到2015-5-22,如图-7所示:

图-7
8)查看结果,如图-8所示:

图-8
9)除了柱状图,Kibana还支持很多种展示方式 ,如图-9所示:

图-9
10)做一个饼图,选择Pie chart,如图-10所示:

图-10
11)选择from a new serach,如图-11所示:

图-11
12)选择Spilt Slices,如图-12所示:

图-12
13)选择Trems,Memary(也可以选择其他的,这个不固定),如图-13所示:

图-13
14)结果,如图-14所示:

图-14
15)保存后可以在Dashboard查看,如图-15所示:

代码如下:
(没有太多代码,需要去网页上面去操作)

2 案例2:综合练习
2.1 问题

本案例要求:
练习插件
安装一台Apache服务并配置
使用filebeat收集Apache服务器的日志
使用grok处理filebeat发送过来的日志
存入elasticsearch
2.2 步骤

实现此案例需要按照如下步骤进行。
步骤一:安装logstash

1)配置主机名,ip和yum源,配置/etc/hosts(请把se1-se5和kibana主机配置和logstash一样的/etc/hosts)
[[email protected] ~]# vim /etc/hosts
192.168.1.61 se1
192.168.1.62 se2
192.168.1.63 se3
192.168.1.64 se4
192.168.1.65 se5
192.168.1.66 kibana
192.168.1.67 logstash
2)安装java-1.8.0-openjdk和logstash
[[email protected] ~]# yum -y install java-1.8.0-openjdk
[[email protected] ~]# yum -y install logstash
[[email protected] ~]# java -version
openjdk version “1.8.0_131”
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
[[email protected] ~]# touch /etc/logstash/logstash.conf
[[email protected] ~]# /opt/logstash/bin/logstash –version
logstash 2.3.4
[[email protected] ~]# /opt/logstash/bin/logstash-plugin list //查看插件

logstash-input-stdin //标准输入插件
logstash-output-stdout //标准输出插件

[[email protected] ~]# vim /etc/logstash/logstash.conf
input{
stdin{
}
}
filter{
}
output{
stdout{
}
}
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//启动并测试
Settings: Default pipeline workers: 2
Pipeline main started
aa //logstash 配置从标准输入读取输入源,然后从标准输出输出到屏幕
2018-09-15T06:19:28.724Z logstash aa
备注:若不会写配置文件可以找帮助,插件文档的位置:
https://github.com/logstash-plugins
3)codec类插件
[[email protected] ~]# vim /etc/logstash/logstash.conf
input{
stdin{
codec => “json” //输入设置为编码json
}
}
filter{
}
output{
stdout{
codec => “rubydebug” //输出设置为rubydebug
}
}
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{“a”:1}
{
“a” => 1,
“@version” => “1”,
“@timestamp” => “2018-09-15T06:34:14.538Z”,
“host” => “logstash”
}
4)file模块插件
[[email protected] ~]# vim /etc/logstash/logstash.conf
input{
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb” //记录读取文件的位置
start_position => “beginning” //配置第一次读取文件从什么地方开始
type => “testlog” //类型名称
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[[email protected] ~]# touch /tmp/a.log
[[email protected] ~]# touch /var/tmp/b.log
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
另开一个终端:写入数据
[[email protected] ~]# echo a1 > /tmp/a.log
[[email protected] ~]# echo b1 > /var/tmp/b.log
之前终端查看:
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “a1”,
“@version” => “1”,
“@timestamp” => “2018-09-15T06:44:30.671Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b1”,
“@version” => “1”,
“@timestamp” => “2018-09-15T06:45:04.725Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}

5)tcp、udp模块插件
[[email protected] ~]# vim /etc/logstash/logstash.conf
input{
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//启动
另开一个终端查看,可以看到端口
[[email protected] tmp]# netstat -antup | grep 8888
tcp6 0 0 :::8888 :::* LISTEN 22191/java
[[email protected] tmp]# netstat -antup | grep 9999
udp6 0 0 :::9999 :::* 22191/java
在另一台主机上写一个脚本,发送数据,使启动的logstash可以接收到数据
[[email protected] ~]# vim tcp.sh
function sendmsg(){
if [[ “$1” == “tcp” ]];then
exec 9<>/dev/tcp/192.168.1.67/8888
else
exec 9<>/dev/udp/192.168.1.67/9999
fi
echo “$2” >&9
exec 9<&-
}
[[email protected] ~]# . tcp.sh //重新载入一下
[[email protected] ~]# sendmsg udp “is tcp test”
[[email protected] ~]# sendmsg udp “is tcp ss”
logstash主机查看结果
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “is tcp test\n”,
“@version” => “1”,
“@timestamp” => “2018-09-15T07:45:00.638Z”,
“type” => “udplog”,
“host” => “192.168.1.65”
}
{
“message” => “is tcp ss\n”,
“@version” => “1”,
“@timestamp” => “2018-09-15T07:45:08.897Z”,
“type” => “udplog”,
“host” => “192.168.1.65”
}
6)syslog插件练习
[[email protected] ~]# systemctl list-unit-files | grep syslog
rsyslog.service enabled
syslog.socket static
[[email protected] ~]# vim /etc/logstash/logstash.conf
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
另一个终端查看是否检测到514
[[email protected] ~]# netstat -antup | grep 514
tcp6 0 0 :::514 :::* LISTEN 22728/java
udp6 0 0 :::514 :::* 22728/java
另一台主机上面操作,本地写的日志本地可以查看
[[email protected] ~]# vim /etc/rsyslog.conf
local0.info /var/log/mylog //自己添加这一行
[[email protected] ~]# systemctl restart rsyslog //重启rsyslog
[[email protected] ~]# ll /var/log/mylog //提示没有那个文件或目录
ls: cannot access /var/log/mylog: No such file or directory
[[email protected] ~]# logger -p local0.info -t nsd “elk” //写日志
[[email protected] ~]# ll /var/log/mylog //再次查看,有文件
-rw——- 1 root root 29 Sep 15 16:23 /var/log/mylog
[[email protected] ~]# tail /var/log/mylog //可以查看到写的日志
Sep 15 16:23:25 se5 nsd: elk
[[email protected] ~]# tail /var/log/messages
//可以查看到写的日志,因为配置文件里有写以.info结尾的可以收到

Sep 15 16:23:25 se5 nsd: elk
把本地的日志发送给远程1.67
[[email protected] ~]# vim /etc/rsyslog.conf
local0.info @192.168.1.67:514
//写一个@或两个@@都可以,一个@代表udp,两个@@代表tcp
[[email protected] ~]# systemctl restart rsyslog
[[email protected] ~]# logger -p local0.info -t nds “001 elk”
[[email protected] bin]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//检测到写的日志
{
“message” => “001 elk”,
“@version” => “1”,
“@timestamp” => “2018-09-05T09:15:47.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 134,
“timestamp” => “Jun 5 17:15:47”,
“logsource” => “kibana”,
“program” => “nds1801”,
“severity” => 6,
“facility” => 16,
“facility_label” => “local0”,
“severity_label” => “Informational”
}
rsyslog.conf配置向远程发送数据,远程登陆1.65的时侯,把登陆日志的信息(/var/log/secure)转发给logstash即1.67这台机器
[[email protected] ~]# vim /etc/rsyslog.conf
57 authpriv.* @@192.168.1.67:514
//57行的/var/log/secure改为@@192.168.1.67:514
[[email protected] ~]# systemctl restart rsyslog
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//找一台主机登录1.65,logstash主机会有数据
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “Accepted password for root from 192.168.1.254 port 33780 ssh2\n”,
“@version” => “1”,
“@timestamp” => “2018-09-15T08:40:57.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 86,
“timestamp” => “Sep 15 16:40:57”,
“logsource” => “se5”,
“program” => “sshd”,
“pid” => “26133”,
“severity” => 6,
“facility” => 10,
“facility_label” => “security/authorization”,
“severity_label” => “Informational”
}
{
“message” => “pam_unix(sshd:session): session opened for user root by (uid=0)\n”,
“@version” => “1”,
“@timestamp” => “2018-09-15T08:40:57.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 86,
“timestamp” => “Sep 15 16:40:57”,
“logsource” => “se5”,
“program” => “sshd”,
“pid” => “26133”,
“severity” => 6,
“facility” => 10,
“facility_label” => “security/authorization”,
“severity_label” => “Informational”
7)filter grok插件
grok插件:
解析各种非结构化的日志数据插件
grok使用正则表达式把飞结构化的数据结构化
在分组匹配,正则表达式需要根据具体数据结构编写
虽然编写困难,但适用性极广
[[email protected] ~]# vim /etc/logstash/logstash.conf
input{
stdin{ codec => “json” }
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
grok{
match => [“message”, “(?<key>reg)”]
}
}
output{
stdout{
codec => “rubydebug”
}
}
[[email protected] ~]# yum -y install httpd
[[email protected] ~]# systemctl restart httpd
[[email protected] ~]# vim /var/log/httpd/access_log
192.168.1.254 – – [15/Sep/2018:18:25:46 +0800] “GET / HTTP/1.1” 403 4897 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0”
复制/var/log/httpd/access_log的日志到logstash下的/tmp/a.log
[[email protected] ~]# vim /tmp/a.log
192.168.1.254 – – [15/Sep/2018:18:25:46 +0800] “GET / HTTP/1.1” 403 4897 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0”
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//出现message的日志,但是没有解析是什么意思
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “.168.1.254 – – [15/Sep/2018:18:25:46 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\””,
“@version” => “1”,
“@timestamp” => “2018-09-15T10:26:51.335Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”,
“tags” => [
[0] “_grokparsefailure”
]
}
若要解决没有解析的问题,同样的方法把日志复制到/tmp/a.log,logstash.conf配置文件里面修改grok
查找正则宏路径
[[email protected] ~]# cd /opt/logstash/vendor/bundle/ \
jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/
[[email protected] ~]# vim grok-patterns //查找COMBINEDAPACHELOG
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
[[email protected] ~]# vim /etc/logstash/logstash.conf

filter{
grok{
match => [“message”, “%{COMBINEDAPACHELOG}”]
}
}

解析出的结果
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “192.168.1.254 – – [15/Sep/2018:18:25:46 +0800] \”GET /noindex/css/open-sans.css HTTP/1.1\” 200 5081 \”http://192.168.1.65/\” \”Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\””,
“@version” => “1”,
“@timestamp” => “2018-09-15T10:55:57.743Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”,
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “15/Sep/2018:18:25:46 +0800”,
“verb” => “GET”,
“request” => “/noindex/css/open-sans.css”,
“httpversion” => “1.1”,
“response” => “200”,
“bytes” => “5081”,
“referrer” => “\”http://192.168.1.65/\””,
“agent” => “\”Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\””
}
步骤二:安装Apache服务,用filebeat收集Apache服务器的日志,存入elasticsearch

1)在之前安装了Apache的主机上面安装filebeat
[[email protected] ~]# yum -y install filebeat
[[email protected] ~]# vim/etc/filebeat/filebeat.yml
paths:
– /var/log/httpd/access_log //日志的路径,短横线加空格代表yml格式
document_type: apachelog //文档类型
elasticsearch: //加上注释
hosts: [“localhost:9200”] //加上注释
logstash: //去掉注释
hosts: [“192.168.1.67:5044”] //去掉注释,logstash那台主机的ip
[[email protected] ~]# systemctl start filebeat
[[email protected] ~]# vim /etc/logstash/logstash.conf
input{
stdin{ codec => “json” }
beats{
port => 5044
}
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/dev/null”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
if [type] == “apachelog”{
grok{
match => [“message”, “%{COMBINEDAPACHELOG}”]
}}
}
output{
stdout{ codec => “rubydebug” }
if [type] == “filelog”{
elasticsearch {
hosts => [“192.168.1.61:9200”, “192.168.1.62:9200”]
index => “filelog”
flush_size => 2000
idle_flush_time => 10
}}
}
[[email protected] logstash]# /opt/logstash/bin/logstash \
-f /etc/logstash/logstash.conf
打开另一终端查看5044是否成功启动
[[email protected] ~]# netstat -antup | grep 5044
tcp6 0 0 :::5044 :::* LISTEN 23776/java
[[email protected] ~]# firefox 192.168.1.65 //ip为安装filebeat的那台机器
回到原来的终端,有数据
2)修改logstash.conf文件
[[email protected] logstash]# vim logstash.conf

output{
stdout{ codec => “rubydebug” }
if [type] == “apachelog”{
elasticsearch {
hosts => [“192.168.1.61:9200”, “192.168.1.62:9200”]
index => “apachelog”
flush_size => 2000
idle_flush_time => 10
}}
}
浏览器访问Elasticsearch,有apachelog,如图-16所示:

代码如下
HOSTS修改
现在NODE1上面改好HOSTS,然后全部同步过去
[[email protected] ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.61 se1
192.168.1.62 se2
192.168.1.63 se3
192.168.1.64 se4
192.168.1.65 se5
192.168.1.66 kibana
192.168.1.67 logstash
[[email protected] ~]#
[[email protected] ~]# for i in {62..67} ; do scp /etc/hosts 192.168.1.$i:/etc/hosts ; done
[email protected]’s password:
hosts 100% 285 613.5KB/s 00:00
[email protected]’s password:
hosts 100% 285 683.4KB/s 00:00
[email protected]’s password:
hosts 100% 285 292.7KB/s 00:00
[email protected]’s password:
hosts 100% 285 158.4KB/s 00:00
The authenticity of host ‘192.168.1.66 (192.168.1.66)’ can’t be established.
ECDSA key fingerprint is SHA256:9eETwPzeFux5NrrknGudd0gBIRJYNtFs2RefL2HtTvY.
ECDSA key fingerprint is MD5:0a:a0:3e:e0:4d:ea:fa:4b:b8:6b:c4:df:4e:cd:0d:e7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.168.1.66’ (ECDSA) to the list of known hosts.
[email protected]’s password:
hosts 100% 285 5.0KB/s 00:00
The authenticity of host ‘192.168.1.67 (192.168.1.67)’ can’t be established.
ECDSA key fingerprint is SHA256:JbQOqJPIrX0AR2+zx/zH0l5PgTqG+/iI18msTH6TeYE.
ECDSA key fingerprint is MD5:88:61:43:fa:37:f4:ab:a4:bd:67:de:9d:51:81:94:13.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.168.1.67’ (ECDSA) to the list of known hosts.
[email protected]’s password:
hosts 100% 285 421.1KB/s 00:00

到logstash机器上面去操作,安装基本的环境
[[email protected] ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.61 se1
192.168.1.62 se2
192.168.1.63 se3
192.168.1.64 se4
192.168.1.65 se5
192.168.1.66 kibana
192.168.1.67 logstash
[[email protected] ~]# yum install -y java-1.8.0-openjdk
已加载插件:fastestmirror
base | 3.6 kB 00:00:00
elk | 2.9 kB 00:00:00
extras | 3.4 kB 00:00:00
local_repo | 3.6 kB 00:00:00
updates | 3.4 kB 00:00:00
Loading mirror speeds from cached hostfile
* base: mirrors.cqu.edu.cn
* extras: mirrors.163.com
* updates: mirrors.163.com
正在解决依赖关系
–> 正在检查事务
—> 软件包 java-1.8.0-openjdk.x86_64.1.1.8.0.181-3.b13.el7_5 将被 安装
–> 正在处理依赖关系 java-1.8.0-openjdk-headless(x86-64) = 1:1.8.0.181-3.b13.el7_5,它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 xorg-x11-fonts-Type1,它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libpng15.so.15(PNG15_0)(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjvm.so(SUNWprivate_1.1)(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjpeg.so.62(LIBJPEG_6.2)(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjava.so(SUNWprivate_1.1)(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 fontconfig(x86-64),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libpng15.so.15()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjvm.so()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjpeg.so.62()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjava.so()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libgif.so.4()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXtst.so.6()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXrender.so.1()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXi.so.6()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXext.so.6()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXcomposite.so.1()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libX11.so.6()(64bit),它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在检查事务
—> 软件包 fontconfig.x86_64.0.2.10.95-11.el7 将被 安装
–> 正在处理依赖关系 fontpackages-filesystem,它被软件包 fontconfig-2.10.95-11.el7.x86_64 需要
–> 正在处理依赖关系 font(:lang=en),它被软件包 fontconfig-2.10.95-11.el7.x86_64 需要
—> 软件包 giflib.x86_64.0.4.1.6-9.el7 将被 安装
–> 正在处理依赖关系 libSM.so.6()(64bit),它被软件包 giflib-4.1.6-9.el7.x86_64 需要
–> 正在处理依赖关系 libICE.so.6()(64bit),它被软件包 giflib-4.1.6-9.el7.x86_64 需要
—> 软件包 java-1.8.0-openjdk-headless.x86_64.1.1.8.0.181-3.b13.el7_5 将被 安装
–> 正在处理依赖关系 tzdata-java >= 2015d,它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 nss-softokn(x86-64) >= 3.36.0,它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 nss(x86-64) >= 3.36.0,它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 copy-jdk-configs >= 2.2,它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 lksctp-tools(x86-64),它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 jpackage-utils,它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
—> 软件包 libX11.x86_64.0.1.6.5-1.el7 将被 安装
–> 正在处理依赖关系 libX11-common >= 1.6.5-1.el7,它被软件包 libX11-1.6.5-1.el7.x86_64 需要
–> 正在处理依赖关系 libxcb.so.1()(64bit),它被软件包 libX11-1.6.5-1.el7.x86_64 需要
—> 软件包 libXcomposite.x86_64.0.0.4.4-4.1.el7 将被 安装
—> 软件包 libXext.x86_64.0.1.3.3-3.el7 将被 安装
—> 软件包 libXi.x86_64.0.1.7.9-1.el7 将被 安装
—> 软件包 libXrender.x86_64.0.0.9.10-1.el7 将被 安装
—> 软件包 libXtst.x86_64.0.1.2.3-1.el7 将被 安装
—> 软件包 libjpeg-turbo.x86_64.0.1.2.90-5.el7 将被 安装
—> 软件包 libpng.x86_64.2.1.5.13-7.el7_2 将被 安装
—> 软件包 xorg-x11-fonts-Type1.noarch.0.7.5-9.el7 将被 安装
–> 正在处理依赖关系 ttmkfdir,它被软件包 xorg-x11-fonts-Type1-7.5-9.el7.noarch 需要
–> 正在处理依赖关系 ttmkfdir,它被软件包 xorg-x11-fonts-Type1-7.5-9.el7.noarch 需要
–> 正在处理依赖关系 mkfontdir,它被软件包 xorg-x11-fonts-Type1-7.5-9.el7.noarch 需要
–> 正在处理依赖关系 mkfontdir,它被软件包 xorg-x11-fonts-Type1-7.5-9.el7.noarch 需要
–> 正在检查事务
—> 软件包 copy-jdk-configs.noarch.0.3.3-10.el7_5 将被 安装
—> 软件包 fontpackages-filesystem.noarch.0.1.44-8.el7 将被 安装
—> 软件包 javapackages-tools.noarch.0.3.4.1-11.el7 将被 安装
–> 正在处理依赖关系 python-javapackages = 3.4.1-11.el7,它被软件包 javapackages-tools-3.4.1-11.el7.noarch 需要
–> 正在处理依赖关系 libxslt,它被软件包 javapackages-tools-3.4.1-11.el7.noarch 需要
—> 软件包 libICE.x86_64.0.1.0.9-9.el7 将被 安装
—> 软件包 libSM.x86_64.0.1.2.2-2.el7 将被 安装
—> 软件包 libX11-common.noarch.0.1.6.5-1.el7 将被 安装
—> 软件包 libxcb.x86_64.0.1.12-1.el7 将被 安装
–> 正在处理依赖关系 libXau.so.6()(64bit),它被软件包 libxcb-1.12-1.el7.x86_64 需要
—> 软件包 lksctp-tools.x86_64.0.1.0.17-2.el7 将被 安装
—> 软件包 nss.x86_64.0.3.28.4-8.el7 将被 升级
–> 正在处理依赖关系 nss = 3.28.4-8.el7,它被软件包 nss-sysinit-3.28.4-8.el7.x86_64 需要
–> 正在处理依赖关系 nss(x86-64) = 3.28.4-8.el7,它被软件包 nss-tools-3.28.4-8.el7.x86_64 需要
—> 软件包 nss.x86_64.0.3.36.0-7.el7_5 将被 更新
–> 正在处理依赖关系 nss-util >= 3.36.0-1,它被软件包 nss-3.36.0-7.el7_5.x86_64 需要
–> 正在处理依赖关系 nspr >= 4.19.0,它被软件包 nss-3.36.0-7.el7_5.x86_64 需要
–> 正在处理依赖关系 libnssutil3.so(NSSUTIL_3.31)(64bit),它被软件包 nss-3.36.0-7.el7_5.x86_64 需要
—> 软件包 nss-softokn.x86_64.0.3.28.3-6.el7 将被 升级
—> 软件包 nss-softokn.x86_64.0.3.36.0-5.el7_5 将被 更新
–> 正在处理依赖关系 nss-softokn-freebl(x86-64) >= 3.36.0-5.el7_5,它被软件包 nss-softokn-3.36.0-5.el7_5.x86_64 需要
—> 软件包 stix-fonts.noarch.0.1.1.0-5.el7 将被 安装
—> 软件包 ttmkfdir.x86_64.0.3.0.9-42.el7 将被 安装
—> 软件包 tzdata-java.noarch.0.2018e-3.el7 将被 安装
—> 软件包 xorg-x11-font-utils.x86_64.1.7.5-20.el7 将被 安装
–> 正在处理依赖关系 libfontenc.so.1()(64bit),它被软件包 1:xorg-x11-font-utils-7.5-20.el7.x86_64 需要
–> 正在处理依赖关系 libXfont.so.1()(64bit),它被软件包 1:xorg-x11-font-utils-7.5-20.el7.x86_64 需要
–> 正在检查事务
—> 软件包 libXau.x86_64.0.1.0.8-2.1.el7 将被 安装
—> 软件包 libXfont.x86_64.0.1.5.2-1.el7 将被 安装
—> 软件包 libfontenc.x86_64.0.1.1.3-3.el7 将被 安装
—> 软件包 libxslt.x86_64.0.1.1.28-5.el7 将被 安装
—> 软件包 nspr.x86_64.0.4.13.1-1.0.el7_3 将被 升级
—> 软件包 nspr.x86_64.0.4.19.0-1.el7_5 将被 更新
—> 软件包 nss-softokn-freebl.x86_64.0.3.28.3-6.el7 将被 升级
—> 软件包 nss-softokn-freebl.x86_64.0.3.36.0-5.el7_5 将被 更新
—> 软件包 nss-sysinit.x86_64.0.3.28.4-8.el7 将被 升级
—> 软件包 nss-sysinit.x86_64.0.3.36.0-7.el7_5 将被 更新
—> 软件包 nss-tools.x86_64.0.3.28.4-8.el7 将被 升级
—> 软件包 nss-tools.x86_64.0.3.36.0-7.el7_5 将被 更新
—> 软件包 nss-util.x86_64.0.3.28.4-3.el7 将被 升级
—> 软件包 nss-util.x86_64.0.3.36.0-1.el7_5 将被 更新
—> 软件包 python-javapackages.noarch.0.3.4.1-11.el7 将被 安装
–> 正在处理依赖关系 python-lxml,它被软件包 python-javapackages-3.4.1-11.el7.noarch 需要
–> 正在检查事务
—> 软件包 python-lxml.x86_64.0.3.2.1-4.el7 将被 安装
–> 解决依赖关系完成

依赖关系解决

=====================================================================================================================
Package 架构 版本 源 大小
=====================================================================================================================
正在安装:
java-1.8.0-openjdk x86_64 1:1.8.0.181-3.b13.el7_5 updates 250 k
为依赖而安装:
copy-jdk-configs noarch 3.3-10.el7_5 updates 21 k
fontconfig x86_64 2.10.95-11.el7 base 229 k
fontpackages-filesystem noarch 1.44-8.el7 base 9.9 k
giflib x86_64 4.1.6-9.el7 base 40 k
java-1.8.0-openjdk-headless x86_64 1:1.8.0.181-3.b13.el7_5 updates 32 M
javapackages-tools noarch 3.4.1-11.el7 base 73 k
libICE x86_64 1.0.9-9.el7 base 66 k
libSM x86_64 1.2.2-2.el7 base 39 k
libX11 x86_64 1.6.5-1.el7 base 606 k
libX11-common noarch 1.6.5-1.el7 base 164 k
libXau x86_64 1.0.8-2.1.el7 base 29 k
libXcomposite x86_64 0.4.4-4.1.el7 base 22 k
libXext x86_64 1.3.3-3.el7 base 39 k
libXfont x86_64 1.5.2-1.el7 base 152 k
libXi x86_64 1.7.9-1.el7 base 40 k
libXrender x86_64 0.9.10-1.el7 base 26 k
libXtst x86_64 1.2.3-1.el7 base 20 k
libfontenc x86_64 1.1.3-3.el7 base 31 k
libjpeg-turbo x86_64 1.2.90-5.el7 base 134 k
libpng x86_64 2:1.5.13-7.el7_2 base 213 k
libxcb x86_64 1.12-1.el7 base 211 k
libxslt x86_64 1.1.28-5.el7 base 242 k
lksctp-tools x86_64 1.0.17-2.el7 base 88 k
python-javapackages noarch 3.4.1-11.el7 base 31 k
python-lxml x86_64 3.2.1-4.el7 base 758 k
stix-fonts noarch 1.1.0-5.el7 base 1.3 M
ttmkfdir x86_64 3.0.9-42.el7 base 48 k
tzdata-java noarch 2018e-3.el7 updates 185 k
xorg-x11-font-utils x86_64 1:7.5-20.el7 base 87 k
xorg-x11-fonts-Type1 noarch 7.5-9.el7 base 521 k
为依赖而更新:
nspr x86_64 4.19.0-1.el7_5 updates 127 k
nss x86_64 3.36.0-7.el7_5 updates 835 k
nss-softokn x86_64 3.36.0-5.el7_5 updates 315 k
nss-softokn-freebl x86_64 3.36.0-5.el7_5 updates 222 k
nss-sysinit x86_64 3.36.0-7.el7_5 updates 62 k
nss-tools x86_64 3.36.0-7.el7_5 updates 515 k
nss-util x86_64 3.36.0-1.el7_5 updates 78 k

事务概要
=====================================================================================================================
安装 1 软件包 (+30 依赖软件包)
升级 ( 7 依赖软件包)

总下载量:39 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/38): copy-jdk-configs-3.3-10.el7_5.noarch.rpm | 21 kB 00:00:00
(2/38): fontpackages-filesystem-1.44-8.el7.noarch.rpm | 9.9 kB 00:00:00
(3/38): giflib-4.1.6-9.el7.x86_64.rpm | 40 kB 00:00:00
(4/38): fontconfig-2.10.95-11.el7.x86_64.rpm | 229 kB 00:00:00
(5/38): libICE-1.0.9-9.el7.x86_64.rpm | 66 kB 00:00:00
(6/38): javapackages-tools-3.4.1-11.el7.noarch.rpm | 73 kB 00:00:00
(7/38): libSM-1.2.2-2.el7.x86_64.rpm | 39 kB 00:00:00
(8/38): libX11-common-1.6.5-1.el7.noarch.rpm | 164 kB 00:00:00
(9/38): java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64.rpm | 250 kB 00:00:01
(10/38): libX11-1.6.5-1.el7.x86_64.rpm | 606 kB 00:00:00
(11/38): libXau-1.0.8-2.1.el7.x86_64.rpm | 29 kB 00:00:00
(12/38): libXcomposite-0.4.4-4.1.el7.x86_64.rpm | 22 kB 00:00:00
(13/38): libXext-1.3.3-3.el7.x86_64.rpm | 39 kB 00:00:00
(14/38): libXfont-1.5.2-1.el7.x86_64.rpm | 152 kB 00:00:00
(15/38): libXi-1.7.9-1.el7.x86_64.rpm | 40 kB 00:00:00
(16/38): libXrender-0.9.10-1.el7.x86_64.rpm | 26 kB 00:00:00
(17/38): libXtst-1.2.3-1.el7.x86_64.rpm | 20 kB 00:00:00
(18/38): libfontenc-1.1.3-3.el7.x86_64.rpm | 31 kB 00:00:00
(19/38): libjpeg-turbo-1.2.90-5.el7.x86_64.rpm | 134 kB 00:00:00
(20/38): libpng-1.5.13-7.el7_2.x86_64.rpm | 213 kB 00:00:00
(21/38): libxcb-1.12-1.el7.x86_64.rpm | 211 kB 00:00:00
(22/38): libxslt-1.1.28-5.el7.x86_64.rpm | 242 kB 00:00:00
(23/38): lksctp-tools-1.0.17-2.el7.x86_64.rpm | 88 kB 00:00:00
(24/38): nspr-4.19.0-1.el7_5.x86_64.rpm | 127 kB 00:00:01
(25/38): nss-3.36.0-7.el7_5.x86_64.rpm | 835 kB 00:00:00
(26/38): nss-softokn-3.36.0-5.el7_5.x86_64.rpm | 315 kB 00:00:00
(27/38): nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm | 222 kB 00:00:00
(28/38): nss-sysinit-3.36.0-7.el7_5.x86_64.rpm | 62 kB 00:00:00
(29/38): nss-tools-3.36.0-7.el7_5.x86_64.rpm | 515 kB 00:00:00
(30/38): java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64.rpm | 32 MB 00:00:05
(31/38): nss-util-3.36.0-1.el7_5.x86_64.rpm | 78 kB 00:00:00
(32/38): python-javapackages-3.4.1-11.el7.noarch.rpm | 31 kB 00:00:00
(33/38): python-lxml-3.2.1-4.el7.x86_64.rpm | 758 kB 00:00:01
(34/38): ttmkfdir-3.0.9-42.el7.x86_64.rpm | 48 kB 00:00:00
(35/38): stix-fonts-1.1.0-5.el7.noarch.rpm | 1.3 MB 00:00:00
(36/38): xorg-x11-font-utils-7.5-20.el7.x86_64.rpm | 87 kB 00:00:00
(37/38): xorg-x11-fonts-Type1-7.5-9.el7.noarch.rpm | 521 kB 00:00:00
(38/38): tzdata-java-2018e-3.el7.noarch.rpm | 185 kB 00:00:01
———————————————————————————————————————
总计 4.7 MB/s | 39 MB 00:00:08
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在更新 : nspr-4.19.0-1.el7_5.x86_64 1/45
正在更新 : nss-util-3.36.0-1.el7_5.x86_64 2/45
正在安装 : libfontenc-1.1.3-3.el7.x86_64 3/45
正在安装 : libxslt-1.1.28-5.el7.x86_64 4/45
正在安装 : fontpackages-filesystem-1.44-8.el7.noarch 5/45
正在安装 : libjpeg-turbo-1.2.90-5.el7.x86_64 6/45
正在安装 : libICE-1.0.9-9.el7.x86_64 7/45
正在安装 : libSM-1.2.2-2.el7.x86_64 8/45
正在安装 : stix-fonts-1.1.0-5.el7.noarch 9/45
正在安装 : fontconfig-2.10.95-11.el7.x86_64 10/45
正在安装 : python-lxml-3.2.1-4.el7.x86_64 11/45
正在安装 : python-javapackages-3.4.1-11.el7.noarch 12/45
正在安装 : javapackages-tools-3.4.1-11.el7.noarch 13/45
正在安装 : libXfont-1.5.2-1.el7.x86_64 14/45
正在安装 : 1:xorg-x11-font-utils-7.5-20.el7.x86_64 15/45
正在更新 : nss-softokn-freebl-3.36.0-5.el7_5.x86_64 16/45
正在更新 : nss-softokn-3.36.0-5.el7_5.x86_64 17/45
正在更新 : nss-sysinit-3.36.0-7.el7_5.x86_64 18/45
正在更新 : nss-3.36.0-7.el7_5.x86_64 19/45
正在安装 : tzdata-java-2018e-3.el7.noarch 20/45
正在安装 : libXau-1.0.8-2.1.el7.x86_64 21/45
正在安装 : libxcb-1.12-1.el7.x86_64 22/45
正在安装 : libX11-common-1.6.5-1.el7.noarch 23/45
正在安装 : libX11-1.6.5-1.el7.x86_64 24/45
正在安装 : libXext-1.3.3-3.el7.x86_64 25/45
正在安装 : libXi-1.7.9-1.el7.x86_64 26/45
正在安装 : libXtst-1.2.3-1.el7.x86_64 27/45
正在安装 : giflib-4.1.6-9.el7.x86_64 28/45
正在安装 : libXrender-0.9.10-1.el7.x86_64 29/45
正在安装 : libXcomposite-0.4.4-4.1.el7.x86_64 30/45
正在安装 : lksctp-tools-1.0.17-2.el7.x86_64 31/45
正在安装 : copy-jdk-configs-3.3-10.el7_5.noarch 32/45
正在安装 : 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 33/45
正在安装 : ttmkfdir-3.0.9-42.el7.x86_64 34/45
正在安装 : xorg-x11-fonts-Type1-7.5-9.el7.noarch 35/45
正在安装 : 2:libpng-1.5.13-7.el7_2.x86_64 36/45
正在安装 : 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 37/45
正在更新 : nss-tools-3.36.0-7.el7_5.x86_64 38/45
清理 : nss-tools-3.28.4-8.el7.x86_64 39/45
清理 : nss-sysinit-3.28.4-8.el7.x86_64 40/45
清理 : nss-3.28.4-8.el7.x86_64 41/45
清理 : nss-softokn-3.28.3-6.el7.x86_64 42/45
清理 : nss-util-3.28.4-3.el7.x86_64 43/45
清理 : nspr-4.13.1-1.0.el7_3.x86_64 44/45
清理 : nss-softokn-freebl-3.28.3-6.el7.x86_64 45/45
验证中 : libXext-1.3.3-3.el7.x86_64 1/45
验证中 : libX11-1.6.5-1.el7.x86_64 2/45
验证中 : libXi-1.7.9-1.el7.x86_64 3/45
验证中 : python-javapackages-3.4.1-11.el7.noarch 4/45
验证中 : libICE-1.0.9-9.el7.x86_64 5/45
验证中 : giflib-4.1.6-9.el7.x86_64 6/45
验证中 : libjpeg-turbo-1.2.90-5.el7.x86_64 7/45
验证中 : nspr-4.19.0-1.el7_5.x86_64 8/45
验证中 : libXrender-0.9.10-1.el7.x86_64 9/45
验证中 : libXfont-1.5.2-1.el7.x86_64 10/45
验证中 : python-lxml-3.2.1-4.el7.x86_64 11/45
验证中 : 2:libpng-1.5.13-7.el7_2.x86_64 12/45
验证中 : nss-3.36.0-7.el7_5.x86_64 13/45
验证中 : fontpackages-filesystem-1.44-8.el7.noarch 14/45
验证中 : ttmkfdir-3.0.9-42.el7.x86_64 15/45
验证中 : nss-util-3.36.0-1.el7_5.x86_64 16/45
验证中 : copy-jdk-configs-3.3-10.el7_5.noarch 17/45
验证中 : stix-fonts-1.1.0-5.el7.noarch 18/45
验证中 : libXcomposite-0.4.4-4.1.el7.x86_64 19/45
验证中 : libXtst-1.2.3-1.el7.x86_64 20/45
验证中 : lksctp-tools-1.0.17-2.el7.x86_64 21/45
验证中 : 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 22/45
验证中 : 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 23/45
验证中 : libxcb-1.12-1.el7.x86_64 24/45
验证中 : xorg-x11-fonts-Type1-7.5-9.el7.noarch 25/45
验证中 : libxslt-1.1.28-5.el7.x86_64 26/45
验证中 : nss-tools-3.36.0-7.el7_5.x86_64 27/45
验证中 : javapackages-tools-3.4.1-11.el7.noarch 28/45
验证中 : libX11-common-1.6.5-1.el7.noarch 29/45
验证中 : nss-sysinit-3.36.0-7.el7_5.x86_64 30/45
验证中 : libXau-1.0.8-2.1.el7.x86_64 31/45
验证中 : nss-softokn-freebl-3.36.0-5.el7_5.x86_64 32/45
验证中 : libSM-1.2.2-2.el7.x86_64 33/45
验证中 : fontconfig-2.10.95-11.el7.x86_64 34/45
验证中 : tzdata-java-2018e-3.el7.noarch 35/45
验证中 : libfontenc-1.1.3-3.el7.x86_64 36/45
验证中 : nss-softokn-3.36.0-5.el7_5.x86_64 37/45
验证中 : 1:xorg-x11-font-utils-7.5-20.el7.x86_64 38/45
验证中 : nspr-4.13.1-1.0.el7_3.x86_64 39/45
验证中 : nss-softokn-3.28.3-6.el7.x86_64 40/45
验证中 : nss-util-3.28.4-3.el7.x86_64 41/45
验证中 : nss-softokn-freebl-3.28.3-6.el7.x86_64 42/45
验证中 : nss-3.28.4-8.el7.x86_64 43/45
验证中 : nss-sysinit-3.28.4-8.el7.x86_64 44/45
验证中 : nss-tools-3.28.4-8.el7.x86_64 45/45

已安装:
java-1.8.0-openjdk.x86_64 1:1.8.0.181-3.b13.el7_5

作为依赖被安装:
copy-jdk-configs.noarch 0:3.3-10.el7_5 fontconfig.x86_64 0:2.10.95-11.el7
fontpackages-filesystem.noarch 0:1.44-8.el7 giflib.x86_64 0:4.1.6-9.el7
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.181-3.b13.el7_5 javapackages-tools.noarch 0:3.4.1-11.el7
libICE.x86_64 0:1.0.9-9.el7 libSM.x86_64 0:1.2.2-2.el7
libX11.x86_64 0:1.6.5-1.el7 libX11-common.noarch 0:1.6.5-1.el7
libXau.x86_64 0:1.0.8-2.1.el7 libXcomposite.x86_64 0:0.4.4-4.1.el7
libXext.x86_64 0:1.3.3-3.el7 libXfont.x86_64 0:1.5.2-1.el7
libXi.x86_64 0:1.7.9-1.el7 libXrender.x86_64 0:0.9.10-1.el7
libXtst.x86_64 0:1.2.3-1.el7 libfontenc.x86_64 0:1.1.3-3.el7
libjpeg-turbo.x86_64 0:1.2.90-5.el7 libpng.x86_64 2:1.5.13-7.el7_2
libxcb.x86_64 0:1.12-1.el7 libxslt.x86_64 0:1.1.28-5.el7
lksctp-tools.x86_64 0:1.0.17-2.el7 python-javapackages.noarch 0:3.4.1-11.el7
python-lxml.x86_64 0:3.2.1-4.el7 stix-fonts.noarch 0:1.1.0-5.el7
ttmkfdir.x86_64 0:3.0.9-42.el7 tzdata-java.noarch 0:2018e-3.el7
xorg-x11-font-utils.x86_64 1:7.5-20.el7 xorg-x11-fonts-Type1.noarch 0:7.5-9.el7

作为依赖被升级:
nspr.x86_64 0:4.19.0-1.el7_5 nss.x86_64 0:3.36.0-7.el7_5 nss-softokn.x86_64 0:3.36.0-5.el7_5
nss-softokn-freebl.x86_64 0:3.36.0-5.el7_5 nss-sysinit.x86_64 0:3.36.0-7.el7_5 nss-tools.x86_64 0:3.36.0-7.el7_5
nss-util.x86_64 0:3.36.0-1.el7_5

完毕!
[[email protected] ~]# yum install -y logstash
已加载插件:fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cqu.edu.cn
* extras: mirrors.163.com
* updates: mirrors.163.com
正在解决依赖关系
–> 正在检查事务
—> 软件包 logstash.noarch.1.2.3.4-1 将被 安装
–> 解决依赖关系完成

依赖关系解决

=====================================================================================================================
Package 架构 版本 源 大小
=====================================================================================================================
正在安装:
logstash noarch 1:2.3.4-1 elk 77 M

事务概要
=====================================================================================================================
安装 1 软件包

总下载量:77 M
安装大小:136 M
Downloading packages:
logstash-2.3.4-1.noarch.rpm | 77 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : 1:logstash-2.3.4-1.noarch 1/1
验证中 : 1:logstash-2.3.4-1.noarch 1/1

已安装:
logstash.noarch 1:2.3.4-1

完毕!
[[email protected] ~]# java -version
openjdk version “1.8.0_181”
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
[[email protected] ~]#
[[email protected] ~]# touch /etc/logstash/logstash.conf
[[email protected] ~]# /opt/logstash/bin/logstash –version
logstash 2.3.4
[[email protected] ~]# /opt/logstash/bin/logstash-plugin list
Ignoring ffi-1.9.13 because its extensions are not built. Try: gem pristine ffi –version 1.9.13
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-oldlogstashjson
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-anonymize
logstash-filter-checksum
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-dns
logstash-filter-drop
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-multiline
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-beats
logstash-input-couchdb_changes
logstash-input-elasticsearch
logstash-input-eventlog
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-irc
logstash-input-jdbc
logstash-input-kafka
logstash-input-log4j
logstash-input-lumberjack
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-input-xmpp
logstash-input-zeromq
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-exec
logstash-output-file
logstash-output-ganglia
logstash-output-gelf
logstash-output-graphite
logstash-output-hipchat
logstash-output-http
logstash-output-irc
logstash-output-juggernaut
logstash-output-kafka
logstash-output-lumberjack
logstash-output-nagios
logstash-output-nagios_nsca
logstash-output-null
logstash-output-opentsdb
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-statsd
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-xmpp
logstash-output-zeromq
logstash-patterns-core
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]# /opt/logstash/bin/logstash-plugin list | grep std
Ignoring ffi-1.9.13 because its extensions are not built. Try: gem pristine ffi –version 1.9.13
logstash-input-stdin
logstash-output-stdout
[[email protected] ~]#
[[email protected] ~]#

配置配置文件
[[email protected] ~]# vim /etc/logstash/logstash.conf
[[email protected] ~]# cat /etc/logstash/logstash.conf
input{
stdin{
}
}
filter{
}
output{
stdout{
}
}
启动,测试一下,logstash 配置从标准输入读取输入源,然后从标准输出输出到屏幕
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
aa
2018-10-21T08:29:43.307Z logstash aa
test
2018-10-21T08:30:10.145Z logstash test
备注:若不会写配置文件可以找帮助,插件文档的位置:
https://github.com/logstash-plugins

下面是各类插件的使用
注意,我起的名字和上面的案例不一样
1、CODEC类插件
[[email protected] ~]# vim /etc/logstash/codec.conf
//输入设置为编码json
//输出设置为rubydebug
[[email protected] ~]# cat /etc/logstash/codec.conf
input{
stdin{
codec => “json”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/codec.conf
Settings: Default pipeline workers: 2
Pipeline main started
{“a”:1}
{
“a” => 1,
“@version” => “1”,
“@timestamp” => “2018-10-21T08:34:07.627Z”,
“host” => “logstash”
}
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}

Pipeline main has been shutdown
[[email protected] ~]#

file模块插件
[[email protected] ~]# vim /etc/logstash/file.conf
[[email protected] ~]# touch /tmp/a.log
[[email protected] ~]# touch /var/tmp/b.log
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/file.conf

另外起一个终端去写点东西
[[email protected] ~]# echo a1 > /tmp/a.log
[[email protected] ~]# echo b1 > /var/tmp/b.log

回到刚才的终端查看一下
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/file.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “a1”,
“@version” => “1”,
“@timestamp” => “2018-10-21T08:56:55.540Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b1”,
“@version” => “1”,
“@timestamp” => “2018-10-21T08:56:55.628Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
^CSIGINT received. Shutting down the agent. {:level=>:warn}

注意,如果已经存在结果,即刚才写的 /var/lib/logstash/sincedb 那么是不会有输出的,需要删掉这个文件再来

TCP/UDP模块
现在logstash主机上写好配置文件,并启动
[[email protected] ~]# vim /etc/logstash/tcpudp.conf
[[email protected] ~]# cat /etc/logstash/tcpudp.conf
input{
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/tcpudp.conf
Settings: Default pipeline workers: 2
Pipeline main started

再开一个终端,可以看到端口已经起来了
[[email protected] ~]# netstat -antup | grep 8888
tcp6 0 0 :::8888 :::* LISTEN 1836/java
[[email protected] ~]# netstat -antup | grep 9999
udp6 0 0 :::9999 :::* 1836/java
[[email protected] ~]#

随便找个主机,写个脚本发点数据
[[email protected] ~]# vim tcp.sh
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]# cat tcp.sh
function sendmsg(){
if [[ “$1” == “tcp” ]];then
exec 9<>/dev/tcp/192.168.1.67/8888
else
exec 9<>/dev/udp/192.168.1.67/9999
fi
echo “$2” >&9
exec 9<&-
}
[[email protected] ~]# . tcp.sh
[[email protected] ~]# sendmsg tcp “is tcp test”
[[email protected] ~]# sendmsg udp “is udp ssss”

再回到logstash主机,可以看到结果
Pipeline main started
{
“message” => “is tcp test”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:26:11.369Z”,
“host” => “192.168.1.65”,
“port” => 39894,
“type” => “tcplog”
}
{
“message” => “is udp ssss\n”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:26:27.501Z”,
“type” => “udplog”,
“host” => “192.168.1.65”

然后退出
}
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>[“org/jruby/RubyIO.java:3682:in `select'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:77:in `udp_listener'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:50:in `run'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:342:in `inputworker'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:336:in `start_input'”], :level=>:warn}
Pipeline main has been shutdown

syslog插件练习
1、写好配置文件并启动
(注意:刚才写的配置文件懒得删除了,将就看吧)
[[email protected] ~]# systemctl list-unit-files | grep syslog
rsyslog.service enabled
syslog.socket static
[[email protected] ~]# cat /etc/logstash/syslog.conf
input{
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/syslog.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “a2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:49:00.756Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:49:00.997Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}

另外起一个终端,确认514端口已经起来
[[email protected] ~]# netstat -antup | grep 514
tcp6 0 0 :::514 :::* LISTEN 2024/java
udp6 0 0 :::514 :::* 2024/java
[[email protected] ~]#

关键的测试步骤,找台机器se5,把配置文件改下
//写一个@或两个@@都可以,一个@代表udp,两个@@代表tcp
[[email protected] ~]# vim /etc/rsyslog.conf
[[email protected] ~]# tail /etc/rsyslog.conf
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
#

local0.info @192.168.1.67:514

重启服务,然后写点东西发出去
[[email protected] ~]# systemctl restart rsyslog.service
[[email protected] ~]# logger -p local0.info -t nds “001 elk”

回到logstash确认一下,可以看到已经收到结果了,注意第1个和第2个是上面的实验
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/syslog.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “a2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:49:00.756Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:49:00.997Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “001 elk”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:53:03.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 134,
“timestamp” => “Oct 21 17:53:03”,
“logsource” => “se5”,
“program” => “nds”,
“severity” => 6,
“facility” => 16,
“facility_label” => “local0”,
“severity_label” => “Informational”
}

再搞一个实验,就是ES5登陆的时候,把验证过程也发过去给els主机去记录,先改配置文件
[[email protected] ~]# grep authpriv /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
#authpriv.* /var/log/secure
authpriv.* @@192.168.1.67:514
[[email protected] ~]# systemctl restart rsyslog.service

为了方便看实验结果,先清个屏测试一下
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
Errno::EBADF: Bad file descriptor – Bad file descriptor
each at org/jruby/RubyIO.java:3542
tcp_receiver at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:173
tcp_listener at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:159
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/syslog.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:57:21.892Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}

重新启个终端,登陆65
[[email protected] ~]# ssh 192.168.1.65
[email protected]’s password:
Last login: Sun Oct 21 10:13:00 2018 from 192.168.1.254
[[email protected] ~]#

回到logstash看一眼,你会看到信息已经来了
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/syslog.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:57:21.892Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “Accepted password for root from 192.168.1.254 port 48914 ssh2\n”,
“@version” => “1”,
“@timestamp” => “2018-10-22T01:20:07.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 86,
“timestamp” => “Oct 22 09:20:07”,
“logsource” => “se5”,
“program” => “sshd”,
“pid” => “2340”,
“severity” => 6,
“facility” => 10,
“facility_label” => “security/authorization”,
“severity_label” => “Informational”
}
{
“message” => “pam_unix(sshd:session): session opened for user root by (uid=0)\n”,
“@version” => “1”,
“@timestamp” => “2018-10-22T01:20:07.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 86,
“timestamp” => “Oct 22 09:20:07”,
“logsource” => “se5”,
“program” => “sshd”,
“pid” => “2340”,
“severity” => 6,
“facility” => 10,
“facility_label” => “security/authorization”,
“severity_label” => “Informational”
}

^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>[“org/jruby/RubyIO.java:3682:in `select'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:77:in `udp_listener'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:50:in `run'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:342:in `inputworker'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:336:in `start_input'”], :level=>:warn}
Errno::EBADF: Bad file descriptor – Bad file descriptor
each at org/jruby/RubyIO.java:3542
tcp_receiver at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:173
tcp_listener at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:159

7)filter grok插件
grok插件:
解析各种非结构化的日志数据插件
grok使用正则表达式把飞结构化的数据结构化
在分组匹配,正则表达式需要根据具体数据结构编写
虽然编写困难,但适用性极广
先在logstash编写配置文件
[[email protected] ~]# vim /etc/logstash/grok.conf
[[email protected] ~]# cat /etc/logstash/grok.conf
input{
stdin{ codec => “json” }
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
grok{
match => [“message”, “(?<key>reg)”]
}
}
output{
stdout{
codec => “rubydebug”
}
}

然后在测试主机se5去装个APACHE,用真机访问一下,确认有日志
[[email protected] ~]# yum install -y httpd
Loaded plugins: fastestmirror
base | 3.6 kB 00:00:00
elk | 2.9 kB 00:00:00
extras | 3.4 kB 00:00:00
local_repo | 3.6 kB 00:00:00
updates | 3.4 kB 00:00:00
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
* extras: mirrors.163.com
* updates: mirrors.163.com
Resolving Dependencies
–> Running transaction check
—> Package httpd.x86_64 0:2.4.6-80.el7.centos.1 will be installed
–> Processing Dependency: httpd-tools = 2.4.6-80.el7.centos.1 for package: httpd-2.4.6-80.el7.centos.1.x86_64
–> Processing Dependency: /etc/mime.types for package: httpd-2.4.6-80.el7.centos.1.x86_64
–> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-80.el7.centos.1.x86_64
–> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-80.el7.centos.1.x86_64
–> Running transaction check
—> Package apr.x86_64 0:1.4.8-3.el7_4.1 will be installed
—> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed
—> Package httpd-tools.x86_64 0:2.4.6-80.el7.centos.1 will be installed
—> Package mailcap.noarch 0:2.1.41-2.el7 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================
Package Arch Version Repository Size
====================================================================================================================
Installing:
httpd x86_64 2.4.6-80.el7.centos.1 updates 2.7 M
Installing for dependencies:
apr x86_64 1.4.8-3.el7_4.1 base 103 k
apr-util x86_64 1.5.2-6.el7 base 92 k
httpd-tools x86_64 2.4.6-80.el7.centos.1 updates 90 k
mailcap noarch 2.1.41-2.el7 base 31 k

Transaction Summary
====================================================================================================================
Install 1 Package (+4 Dependent packages)

Total download size: 3.0 M
Installed size: 10 M
Downloading packages:
(1/5): apr-util-1.5.2-6.el7.x86_64.rpm | 92 kB 00:00:00
(2/5): apr-1.4.8-3.el7_4.1.x86_64.rpm | 103 kB 00:00:00
(3/5): httpd-tools-2.4.6-80.el7.centos.1.x86_64.rpm | 90 kB 00:00:00
(4/5): httpd-2.4.6-80.el7.centos.1.x86_64.rpm | 2.7 MB 00:00:01
mailcap-2.1.41-2.el7.noarch.rp FAILED ==-] 443 B/s | 3.0 MB 00:00:51 ETA
http://mirrors.zju.edu.cn/centos/7.5.1804/os/x86_64/Packages/mailcap-2.1.41-2.el7.noarch.rpm: [Errno 12] Timeout on http://mirrors.zju.edu.cn/centos/7.5.1804/os/x86_64/Packages/mailcap-2.1.41-2.el7.noarch.rpm: (28, ‘Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds’)
Trying other mirror.
(5/5): mailcap-2.1.41-2.el7.noarch.rpm | 31 kB 00:00:00
——————————————————————————————————————–
Total 85 kB/s | 3.0 MB 00:00:36
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : apr-1.4.8-3.el7_4.1.x86_64 1/5
Installing : apr-util-1.5.2-6.el7.x86_64 2/5
Installing : httpd-tools-2.4.6-80.el7.centos.1.x86_64 3/5
Installing : mailcap-2.1.41-2.el7.noarch 4/5
Installing : httpd-2.4.6-80.el7.centos.1.x86_64 5/5
Verifying : mailcap-2.1.41-2.el7.noarch 1/5
Verifying : httpd-tools-2.4.6-80.el7.centos.1.x86_64 2/5
Verifying : apr-util-1.5.2-6.el7.x86_64 3/5
Verifying : apr-1.4.8-3.el7_4.1.x86_64 4/5
Verifying : httpd-2.4.6-80.el7.centos.1.x86_64 5/5

Installed:
httpd.x86_64 0:2.4.6-80.el7.centos.1

Dependency Installed:
apr.x86_64 0:1.4.8-3.el7_4.1 apr-util.x86_64 0:1.5.2-6.el7 httpd-tools.x86_64 0:2.4.6-80.el7.centos.1
mailcap.noarch 0:2.1.41-2.el7

Complete!
[[email protected] ~]# systemctl restart httpd
[[email protected] ~]# vim /var/log/httpd/access_log
[[email protected] ~]# cat /var/log/httpd/access_log
192.168.1.254 – – [22/Oct/2018:10:07:23 +0800] “GET / HTTP/1.1” 403 4897 “-” “curl/7.29.0”

测试一下,先手动创建测试的文件,也就是把刚才在SE5查到的日志复制到logstash主机的测试目录下面,运行,并测试
[[email protected] ~]# vim /tmp/a.log
[[email protected] ~]# cat /tmp/a.log
192.168.1.254 – – [22/Oct/2018:10:07:23 +0800] “GET / HTTP/1.1” 403 4897 “-” “curl/7.29.0”
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/grok.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “.168.1.254 – – [22/Oct/2018:10:07:23 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”curl/7.29.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:18:54.916Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”,
“tags” => [
[0] “_grokparsefailure”
]
}
//出现message的日志,但是没有解析是什么意思

为了实现解析,根据消息的规律,我们匹配COMBINEDAPACHELOG这个关键词,先修改配置文件
[[email protected] ~]# vim /etc/logstash/grok.conf
[[email protected] ~]# cat /etc/logstash/grok.conf
input{
stdin{ codec => “json” }
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
grok{
match => [“message”, “%{COMBINEDAPACHELOG}”]
}
}
output{
stdout{
codec => “rubydebug”
}
}

然后重新启动一下,注意,之前说过,因为配置文件第一个file里面,写了解析结果路径,因为上面已经运行过了,所以要把之前的结果删除掉
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/grok.conf
Settings: Default pipeline workers: 2
Pipeline main started
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>[“org/jruby/RubyIO.java:3682:in `select'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:77:in `udp_listener'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:50:in `run'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:342:in `inputworker'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:336:in `start_input'”], :level=>:warn}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}
^CSIGINT received. Terminating immediately.. {:level=>:fatal}

删除刚才出来的结果
[[email protected] ~]# rm -rf /var/lib/logstash/sincedb

再次运行一下,现在就有结果了,而且相比较上面的详细多了(忽略下面那个b2)
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/grok.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:07:23 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”curl/7.29.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:29:34.705Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”,
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “22/Oct/2018:10:07:23 +0800”,
“verb” => “GET”,
“request” => “/”,
“httpversion” => “1.1”,
“response” => “403”,
“bytes” => “4897”,
“referrer” => “\”-\””,
“agent” => “\”curl/7.29.0\””
}
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:29:35.001Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”,
“tags” => [
[0] “_grokparsefailure”
]
}

步骤二,在apache主机上,安装filebeat手机apache服务器的日志,存入elasticsearch
[[email protected] ~]# yum install -y filebeat
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
* extras: mirrors.163.com
* updates: mirrors.163.com
Resolving Dependencies
–> Running transaction check
—> Package filebeat.x86_64 0:1.2.3-1 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================
Package Arch Version Repository Size
====================================================================================================================
Installing:
filebeat x86_64 1.2.3-1 elk 3.9 M

Transaction Summary
====================================================================================================================
Install 1 Package

Total download size: 3.9 M
Installed size: 13 M
Downloading packages:
filebeat-1.2.3-x86_64.rpm | 3.9 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : filebeat-1.2.3-1.x86_64 1/1
Verifying : filebeat-1.2.3-1.x86_64 1/1

Installed:
filebeat.x86_64 0:1.2.3-1

Complete!
[[email protected] ~]# vim /etc/filebeat/filebeat.yml

[[email protected] ~]# vim/etc/filebeat/filebeat.yml
paths:
– /var/log/httpd/access_log //日志的路径,短横线加空格代表yml格式
document_type: apachelog //文档类型
elasticsearch: //加上注释
hosts: [“localhost:9200”] //加上注释
logstash: //去掉注释
hosts: [“192.168.1.67:5044”] //去掉注释,logstash那台主机的ip

启动filebeat服务
[[email protected] ~]# systemctl start filebeat
[[email protected] ~]# systemctl status filebeat
● filebeat.service – filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
Active: active (running) since 一 2018-10-22 10:42:39 CST; 6ms ago
Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Main PID: 2690 (filebeat)
CGroup: /system.slice/filebeat.service
└─2690 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml

10月 22 10:42:39 se5 systemd[1]: Started filebeat.
10月 22 10:42:39 se5 systemd[1]: Starting filebeat…

注意,如果是以下提示,就只能等会再启动
[[email protected] ~]# systemctl status filebeat
● filebeat.service – filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since 一 2018-10-22 10:41:05 CST; 1min 31s ago
Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Process: 2651 ExecStart=/usr/bin/filebeat -c /etc/filebeat/filebeat.yml (code=exited, status=1/FAILURE)
Main PID: 2651 (code=exited, status=1/FAILURE)

10月 22 10:41:05 se5 systemd[1]: Unit filebeat.service entered failed state.
10月 22 10:41:05 se5 systemd[1]: filebeat.service failed.
10月 22 10:41:05 se5 systemd[1]: filebeat.service holdoff time over, scheduling restart.
10月 22 10:41:05 se5 systemd[1]: start request repeated too quickly for filebeat.service
10月 22 10:41:05 se5 systemd[1]: Failed to start filebeat.
10月 22 10:41:05 se5 systemd[1]: Unit filebeat.service entered failed state.
10月 22 10:41:05 se5 systemd[1]: filebeat.service failed.

到logstash主机上面去修改配置文件,然后启动服务
[[email protected] ~]# cat /etc/logstash/filebeat.conf
input{
stdin{ codec => “json” }
beats{
port => 5044
}
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/dev/null”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
if [type] == “apachelog”{
grok{
match => [“message”, “%{COMBINEDAPACHELOG}”]
}}
}
output{
stdout{ codec => “rubydebug” }
if [type] == “filelog”{
elasticsearch {
hosts => [“192.168.1.61:9200”, “192.168.1.62:9200”]
index => “filelog”
flush_size => 2000
idle_flush_time => 10
}}
}
启动服务
[[email protected] ~]# /opt/logstash/bin/logstash -f /etc/logstash/filebeat.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:07:23 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”curl/7.29.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:44:24.922Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:44:26.320Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
重启启个终端,确认5044端口已经启动成功
[[email protected] ~]# netstat -antup | grep 5044
tcp6 0 0 :::5044 :::* LISTEN 3177/java

测试,用curl和真机上的谷歌、火狐浏览器都试一下
[[email protected] ~]# curl 192.168.1.65

回到原来的logstash主机,可以看到数据都过来了,我就随便贴几个
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:48:26 +0800] \”GET /favicon.ico HTTP/1.1\” 404 209 \”http://192.168.1.65/\” \”Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:58:02.568Z”,
“fields” => nil,
“beat” => {
“hostname” => “se5”,
“name” => “se5”
},
“offset” => 2265,
“input_type” => “log”,
“count” => 1,
“source” => “/var/log/httpd/access_log”,
“type” => “apachelog”,
“host” => “se5”,
“tags” => [
[0] “beats_input_codec_plain_applied”
],
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “22/Oct/2018:10:48:26 +0800”,
“verb” => “GET”,
“request” => “/favicon.ico”,
“httpversion” => “1.1”,
“response” => “404”,
“bytes” => “209”,
“referrer” => “\”http://192.168.1.65/\””,
“agent” => “\”Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\””
}
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:48:47 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”curl/7.29.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:58:02.568Z”,
“beat” => {
“hostname” => “se5”,
“name” => “se5”
},
“type” => “apachelog”,
“source” => “/var/log/httpd/access_log”,
“offset” => 2479,
“input_type” => “log”,
“count” => 1,
“fields” => nil,
“host” => “se5”,
“tags” => [
[0] “beats_input_codec_plain_applied”
],
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “22/Oct/2018:10:48:47 +0800”,
“verb” => “GET”,
“request” => “/”,
“httpversion” => “1.1”,
“response” => “403”,
“bytes” => “4897”,
“referrer” => “\”-\””,
“agent” => “\”curl/7.29.0\””
}
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:59:24 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:59:27.572Z”,
“count” => 1,
“source” => “/var/log/httpd/access_log”,
“type” => “apachelog”,
“fields” => nil,
“beat” => {
“hostname” => “se5”,
“name” => “se5”
},
“offset” => 2570,
“input_type” => “log”,
“host” => “se5”,
“tags” => [
[0] “beats_input_codec_plain_applied”
],
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “22/Oct/2018:10:59:24 +0800”,
“verb” => “GET”,
“request” => “/”,
“httpversion” => “1.1”,
“response” => “403”,
“bytes” => “4897”,
“referrer” => “\”-\””,
“agent” => “\”Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0\””
}

拓展以下,把刚才说的主是elasticsearch加上注释取消掉,然后elasticsearch就有数据了,去浏览器访问elasticsearch访问确认以下就可以看到
elasticsearch: //加上注释
hosts: [“localhost:9200”] //加上注释
实验到此结束

此条目发表在ELK分类目录,贴了标签。将固定链接加入收藏夹。

发表评论

您的电子邮箱地址不会被公开。 必填项已用*标注