1 案例1：导入数据
1.1 问题
本案例要求批量导入数据：
批量导入数据并查看
1.2 步骤
实现此案例需要按照如下步骤进行。
步骤一：导入数据
使用POST方式批量导入数据，数据格式为json，url 编码使用data-binary导入含有index配置的json文件
[root@room9pc01 ~]# scp /var/ftp/elk/*.gz 192.168.1.66:/root/
[root@kibana ~]# gzip -d logs.jsonl.gz
[root@kibana ~]# gzip -d accounts.json.gz
[root@kibana ~]# gzip -d shakespeare.json.gz
[root@kibana ~]# curl -X POST “http://192.168.1.61:9200/_bulk” \
–data-binary @shakespeare.json
[root@kibana ~]# curl -X POST “http://192.168.1.61:9200/xixi/haha/_bulk” \
–data-binary @accounts.json
//索引是xixi，类型是haha，必须导入索引和类型，没有索引，要加上
[root@kibana ~]# curl -X POST “http://192.168.1.61:9200/_bulk” \
–data-binary @logs.jsonl
2）使用GET查询结果
[root@kibana ~]# curl -XGET ‘http://192.168.1.61:9200/_mget?pretty’ -d ‘{
“docs”:[
{
“_index”:”shakespeare”,
“_type:”:”act”,
“_id”:0
},
{
“_index”:”shakespeare”,
“_type:”:”line”,
“_id”:0
},
{
“_index”:”xixi”,
“_type:”:”haha”,
“_id”:25
}
]
}’
{ //查询的结果
“docs” : [ {
“_index” : “shakespeare”,
“_type” : “act”,
“_id” : “0”,
“_version” : 1,
“found” : true,
“_source” : {
“line_id” : 1,
“play_name” : “Henry IV”,
“speech_number” : “”,
“line_number” : “”,
“speaker” : “”,
“text_entry” : “ACT I”
}
}, {
“_index” : “shakespeare”,
“_type” : “act”,
“_id” : “0”,
“_version” : 1,
“found” : true,
“_source” : {
“line_id” : 1,
“play_name” : “Henry IV”,
“speech_number” : “”,
“line_number” : “”,
“speaker” : “”,
“text_entry” : “ACT I”
}
}, {
“_index” : “xixi”,
“_type” : “haha”,
“_id” : “25”,
“_version” : 1,
“found” : true,
“_source” : {
“account_number” : 25,
“balance” : 40540,
“firstname” : “Virginia”,
“lastname” : “Ayala”,
“age” : 39,
“gender” : “F”,
“address” : “171 Putnam Avenue”,
“employer” : “Filodyne”,
“email” : “[email protected]”,
“city” : “Nicholson”,
“state” : “PA”
}
} ]
}
代码如下
第一步，导入数据，代码太长，不贴了
[root@kibana ~]# gzip -d logs.jsonl.gz
[root@kibana ~]# gzip -d accounts.json.gz
[root@kibana ~]# gzip -d shakespeare.json.gz
[root@kibana ~]# curl -X POST “http://192.168.1.61:9200/_bulk” \
–data-binary @shakespeare.json
[root@kibana ~]# curl -X POST “http://192.168.1.61:9200/xixi/haha/_bulk” \
–data-binary @accounts.json
//索引是xixi，类型是haha，必须导入索引和类型，没有索引，要加上
[root@kibana ~]# curl -X POST “http://192.168.1.61:9200/_bulk” \
–data-binary @logs.jsonl
第二步，使用GET查询结果
[root@kibana ~]# curl -XGET ‘http://192.168.1.61:9200/_mget?pretty’ -d ‘{
> “docs”:[
> {
> “_index”:”shakespeare”,
> “_type:”:”act”,
> “_id”:0
> },
> {
> “_index”:”shakespeare”,
> “_type:”:”line”,
> “_id”:0
> },
> {
> “_index”:”xixi”,
> “_type:”:”haha”,
> “_id”:25
> }
> ]
> }’
输出的查询结果如下
{
“docs” : [ {
“_index” : “shakespeare”,
“_type” : “act”,
“_id” : “0”,
“_version” : 1,
“found” : true,
“_source” : {
“line_id” : 1,
“play_name” : “Henry IV”,
“speech_number” : “”,
“line_number” : “”,
“speaker” : “”,
“text_entry” : “ACT I”
}
}, {
“_index” : “shakespeare”,
“_type” : “act”,
“_id” : “0”,
“_version” : 1,
“found” : true,
“_source” : {
“line_id” : 1,
“play_name” : “Henry IV”,
“speech_number” : “”,
“line_number” : “”,
“speaker” : “”,
“text_entry” : “ACT I”
}
}, {
“_index” : “xixi”,
“_type” : “haha”,
“_id” : “25”,
“_version” : 1,
“found” : true,
“_source” : {
“account_number” : 25,
“balance” : 40540,
“firstname” : “Virginia”,
“lastname” : “Ayala”,
“age” : 39,
“gender” : “F”,
“address” : “171 Putnam Avenue”,
“employer” : “Filodyne”,
“email” : “[email protected]”,
“city” : “Nicholson”,
“state” : “PA”
}
} ]
}
步骤二：使用kibana查看数据是否导入成功
1）数据导入以后查看logs是否导入成功，如图-1所示：
[root@se5 ~]# firefox http://192.168.1.65:9200/_plugin/head/
图-1
2）kibana导入数据，如图-2所示：
[root@kibana ~]# firefox http://192.168.1.66:5601
图-2
3）成功创建会有logstash-*，如图-3所示：
/
图-3
4）导入成功之后选择Discover，如图-4所示：
图-4
注意： 这里没有数据的原因是导入日志的时间段不对，默认配置是最近15分钟，在这可以修改一下时间来显示
5）kibana修改时间，选择Lsat 15 miuntes，如图-5所示：
图-5
6）选择Absolute，如图-6所示：
图-6
7）选择时间2015-5-15到2015-5-22，如图-7所示：
图-7
8）查看结果，如图-8所示：
图-8
9）除了柱状图，Kibana还支持很多种展示方式 ，如图-9所示：
图-9
10）做一个饼图，选择Pie chart，如图-10所示：
图-10
11）选择from a new serach，如图-11所示：
图-11
12）选择Spilt Slices，如图-12所示：
图-12
13）选择Trems,Memary(也可以选择其他的，这个不固定)，如图-13所示：
图-13
14）结果，如图-14所示：
图-14
15）保存后可以在Dashboard查看，如图-15所示：
代码如下：
（没有太多代码，需要去网页上面去操作）
2 案例2：综合练习
2.1 问题
本案例要求：
练习插件
安装一台Apache服务并配置
使用filebeat收集Apache服务器的日志
使用grok处理filebeat发送过来的日志
存入elasticsearch
2.2 步骤
实现此案例需要按照如下步骤进行。
步骤一：安装logstash
1）配置主机名，ip和yum源，配置/etc/hosts（请把se1-se5和kibana主机配置和logstash一样的/etc/hosts）
[root@logstash ~]# vim /etc/hosts
192.168.1.61 se1
192.168.1.62 se2
192.168.1.63 se3
192.168.1.64 se4
192.168.1.65 se5
192.168.1.66 kibana
192.168.1.67 logstash
2）安装java-1.8.0-openjdk和logstash
[root@logstash ~]# yum -y install java-1.8.0-openjdk
[root@logstash ~]# yum -y install logstash
[root@logstash ~]# java -version
openjdk version “1.8.0_131”
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
[root@logstash ~]# touch /etc/logstash/logstash.conf
[root@logstash ~]# /opt/logstash/bin/logstash –version
logstash 2.3.4
[root@logstash ~]# /opt/logstash/bin/logstash-plugin list //查看插件
…
logstash-input-stdin //标准输入插件
logstash-output-stdout //标准输出插件
…
[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
stdin{
}
}
filter{
}
output{
stdout{
}
}
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//启动并测试
Settings: Default pipeline workers: 2
Pipeline main started
aa //logstash 配置从标准输入读取输入源,然后从标准输出输出到屏幕
2018-09-15T06:19:28.724Z logstash aa
备注：若不会写配置文件可以找帮助，插件文档的位置：
https://github.com/logstash-plugins
3）codec类插件
[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
stdin{
codec => “json” //输入设置为编码json
}
}
filter{
}
output{
stdout{
codec => “rubydebug” //输出设置为rubydebug
}
}
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{“a”:1}
{
“a” => 1,
“@version” => “1”,
“@timestamp” => “2018-09-15T06:34:14.538Z”,
“host” => “logstash”
}
4）file模块插件
[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb” //记录读取文件的位置
start_position => “beginning” //配置第一次读取文件从什么地方开始
type => “testlog” //类型名称
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[root@logstash ~]# touch /tmp/a.log
[root@logstash ~]# touch /var/tmp/b.log
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
另开一个终端：写入数据
[root@logstash ~]# echo a1 > /tmp/a.log
[root@logstash ~]# echo b1 > /var/tmp/b.log
之前终端查看：
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “a1”,
“@version” => “1”,
“@timestamp” => “2018-09-15T06:44:30.671Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b1”,
“@version” => “1”,
“@timestamp” => “2018-09-15T06:45:04.725Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
5）tcp、udp模块插件
[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//启动
另开一个终端查看，可以看到端口
[root@logstash tmp]# netstat -antup | grep 8888
tcp6 0 0 :::8888 :::* LISTEN 22191/java
[root@logstash tmp]# netstat -antup | grep 9999
udp6 0 0 :::9999 :::* 22191/java
在另一台主机上写一个脚本，发送数据，使启动的logstash可以接收到数据
[root@se5 ~]# vim tcp.sh
function sendmsg(){
if [[ “$1” == “tcp” ]];then
exec 9<>/dev/tcp/192.168.1.67/8888
else
exec 9<>/dev/udp/192.168.1.67/9999
fi
echo “$2” >&9
exec 9<&-
}
[root@se5 ~]# . tcp.sh //重新载入一下
[root@se5 ~]# sendmsg udp “is tcp test”
[root@se5 ~]# sendmsg udp “is tcp ss”
logstash主机查看结果
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “is tcp test\n”,
“@version” => “1”,
“@timestamp” => “2018-09-15T07:45:00.638Z”,
“type” => “udplog”,
“host” => “192.168.1.65”
}
{
“message” => “is tcp ss\n”,
“@version” => “1”,
“@timestamp” => “2018-09-15T07:45:08.897Z”,
“type” => “udplog”,
“host” => “192.168.1.65”
}
6）syslog插件练习
[root@logstash ~]# systemctl list-unit-files | grep syslog
rsyslog.service enabled
syslog.socket static
[root@logstash ~]# vim /etc/logstash/logstash.conf
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
另一个终端查看是否检测到514
[root@logstash ~]# netstat -antup | grep 514
tcp6 0 0 :::514 :::* LISTEN 22728/java
udp6 0 0 :::514 :::* 22728/java
另一台主机上面操作,本地写的日志本地可以查看
[root@se5 ~]# vim /etc/rsyslog.conf
local0.info /var/log/mylog //自己添加这一行
[root@se5 ~]# systemctl restart rsyslog //重启rsyslog
[root@se5 ~]# ll /var/log/mylog //提示没有那个文件或目录
ls: cannot access /var/log/mylog: No such file or directory
[root@se5 ~]# logger -p local0.info -t nsd “elk” //写日志
[root@se5 ~]# ll /var/log/mylog //再次查看，有文件
-rw——- 1 root root 29 Sep 15 16:23 /var/log/mylog
[root@se5 ~]# tail /var/log/mylog //可以查看到写的日志
Sep 15 16:23:25 se5 nsd: elk
[root@se5 ~]# tail /var/log/messages
//可以查看到写的日志，因为配置文件里有写以.info结尾的可以收到
…
Sep 15 16:23:25 se5 nsd: elk
把本地的日志发送给远程1.67
[root@se5 ~]# vim /etc/rsyslog.conf
local0.info @192.168.1.67:514
//写一个@或两个@@都可以，一个@代表udp，两个@@代表tcp
[root@se5 ~]# systemctl restart rsyslog
[root@se5 ~]# logger -p local0.info -t nds “001 elk”
[root@logstash bin]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//检测到写的日志
{
“message” => “001 elk”,
“@version” => “1”,
“@timestamp” => “2018-09-05T09:15:47.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 134,
“timestamp” => “Jun 5 17:15:47”,
“logsource” => “kibana”,
“program” => “nds1801”,
“severity” => 6,
“facility” => 16,
“facility_label” => “local0”,
“severity_label” => “Informational”
}
rsyslog.conf配置向远程发送数据，远程登陆1.65的时侯，把登陆日志的信息（/var/log/secure）转发给logstash即1.67这台机器
[root@se5 ~]# vim /etc/rsyslog.conf
57 authpriv.* @@192.168.1.67:514
//57行的/var/log/secure改为@@192.168.1.67:514
[root@se5 ~]# systemctl restart rsyslog
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//找一台主机登录1.65，logstash主机会有数据
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “Accepted password for root from 192.168.1.254 port 33780 ssh2\n”,
“@version” => “1”,
“@timestamp” => “2018-09-15T08:40:57.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 86,
“timestamp” => “Sep 15 16:40:57”,
“logsource” => “se5”,
“program” => “sshd”,
“pid” => “26133”,
“severity” => 6,
“facility” => 10,
“facility_label” => “security/authorization”,
“severity_label” => “Informational”
}
{
“message” => “pam_unix(sshd:session): session opened for user root by (uid=0)\n”,
“@version” => “1”,
“@timestamp” => “2018-09-15T08:40:57.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 86,
“timestamp” => “Sep 15 16:40:57”,
“logsource” => “se5”,
“program” => “sshd”,
“pid” => “26133”,
“severity” => 6,
“facility” => 10,
“facility_label” => “security/authorization”,
“severity_label” => “Informational”
7）filter grok插件
grok插件：
解析各种非结构化的日志数据插件
grok使用正则表达式把飞结构化的数据结构化
在分组匹配，正则表达式需要根据具体数据结构编写
虽然编写困难，但适用性极广
[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
stdin{ codec => “json” }
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
grok{
match => [“message”, “(?<key>reg)”]
}
}
output{
stdout{
codec => “rubydebug”
}
}
[root@se5 ~]# yum -y install httpd
[root@se5 ~]# systemctl restart httpd
[root@se5 ~]# vim /var/log/httpd/access_log
192.168.1.254 – – [15/Sep/2018:18:25:46 +0800] “GET / HTTP/1.1” 403 4897 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0”
复制/var/log/httpd/access_log的日志到logstash下的/tmp/a.log
[root@logstash ~]# vim /tmp/a.log
192.168.1.254 – – [15/Sep/2018:18:25:46 +0800] “GET / HTTP/1.1” 403 4897 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0”
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
//出现message的日志，但是没有解析是什么意思
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “.168.1.254 – – [15/Sep/2018:18:25:46 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\””,
“@version” => “1”,
“@timestamp” => “2018-09-15T10:26:51.335Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”,
“tags” => [
[0] “_grokparsefailure”
]
}
若要解决没有解析的问题，同样的方法把日志复制到/tmp/a.log，logstash.conf配置文件里面修改grok
查找正则宏路径
[root@logstash ~]# cd /opt/logstash/vendor/bundle/ \
jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/
[root@logstash ~]# vim grok-patterns //查找COMBINEDAPACHELOG
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
[root@logstash ~]# vim /etc/logstash/logstash.conf
…
filter{
grok{
match => [“message”, “%{COMBINEDAPACHELOG}”]
}
}
…
解析出的结果
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “192.168.1.254 – – [15/Sep/2018:18:25:46 +0800] \”GET /noindex/css/open-sans.css HTTP/1.1\” 200 5081 \”http://192.168.1.65/\” \”Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\””,
“@version” => “1”,
“@timestamp” => “2018-09-15T10:55:57.743Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”,
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “15/Sep/2018:18:25:46 +0800”,
“verb” => “GET”,
“request” => “/noindex/css/open-sans.css”,
“httpversion” => “1.1”,
“response” => “200”,
“bytes” => “5081”,
“referrer” => “\”http://192.168.1.65/\””,
“agent” => “\”Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\””
}
步骤二：安装Apache服务，用filebeat收集Apache服务器的日志，存入elasticsearch
1）在之前安装了Apache的主机上面安装filebeat
[root@se5 ~]# yum -y install filebeat
[root@se5 ~]# vim/etc/filebeat/filebeat.yml
paths:
– /var/log/httpd/access_log //日志的路径，短横线加空格代表yml格式
document_type: apachelog //文档类型
elasticsearch: //加上注释
hosts: [“localhost:9200”] //加上注释
logstash: //去掉注释
hosts: [“192.168.1.67:5044”] //去掉注释,logstash那台主机的ip
[root@se5 ~]# systemctl start filebeat
[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
stdin{ codec => “json” }
beats{
port => 5044
}
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/dev/null”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
if [type] == “apachelog”{
grok{
match => [“message”, “%{COMBINEDAPACHELOG}”]
}}
}
output{
stdout{ codec => “rubydebug” }
if [type] == “filelog”{
elasticsearch {
hosts => [“192.168.1.61:9200”, “192.168.1.62:9200”]
index => “filelog”
flush_size => 2000
idle_flush_time => 10
}}
}
[root@logstash logstash]# /opt/logstash/bin/logstash \
-f /etc/logstash/logstash.conf
打开另一终端查看5044是否成功启动
[root@logstash ~]# netstat -antup | grep 5044
tcp6 0 0 :::5044 :::* LISTEN 23776/java
[root@se5 ~]# firefox 192.168.1.65 //ip为安装filebeat的那台机器
回到原来的终端，有数据
2）修改logstash.conf文件
[root@logstash logstash]# vim logstash.conf
…
output{
stdout{ codec => “rubydebug” }
if [type] == “apachelog”{
elasticsearch {
hosts => [“192.168.1.61:9200”, “192.168.1.62:9200”]
index => “apachelog”
flush_size => 2000
idle_flush_time => 10
}}
}
浏览器访问Elasticsearch，有apachelog，如图-16所示：
代码如下
HOSTS修改
现在NODE1上面改好HOSTS，然后全部同步过去
[root@se1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.61 se1
192.168.1.62 se2
192.168.1.63 se3
192.168.1.64 se4
192.168.1.65 se5
192.168.1.66 kibana
192.168.1.67 logstash
[root@se1 ~]#
[root@se1 ~]# for i in {62..67} ; do scp /etc/hosts 192.168.1.$i:/etc/hosts ; done
[email protected]’s password:
hosts 100% 285 613.5KB/s 00:00
[email protected]’s password:
hosts 100% 285 683.4KB/s 00:00
[email protected]’s password:
hosts 100% 285 292.7KB/s 00:00
[email protected]’s password:
hosts 100% 285 158.4KB/s 00:00
The authenticity of host ‘192.168.1.66 (192.168.1.66)’ can’t be established.
ECDSA key fingerprint is SHA256:9eETwPzeFux5NrrknGudd0gBIRJYNtFs2RefL2HtTvY.
ECDSA key fingerprint is MD5:0a:a0:3e:e0:4d:ea:fa:4b:b8:6b:c4:df:4e:cd:0d:e7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.168.1.66’ (ECDSA) to the list of known hosts.
[email protected]’s password:
hosts 100% 285 5.0KB/s 00:00
The authenticity of host ‘192.168.1.67 (192.168.1.67)’ can’t be established.
ECDSA key fingerprint is SHA256:JbQOqJPIrX0AR2+zx/zH0l5PgTqG+/iI18msTH6TeYE.
ECDSA key fingerprint is MD5:88:61:43:fa:37:f4:ab:a4:bd:67:de:9d:51:81:94:13.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.168.1.67’ (ECDSA) to the list of known hosts.
[email protected]’s password:
hosts 100% 285 421.1KB/s 00:00
到logstash机器上面去操作，安装基本的环境
[root@logstash ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.61 se1
192.168.1.62 se2
192.168.1.63 se3
192.168.1.64 se4
192.168.1.65 se5
192.168.1.66 kibana
192.168.1.67 logstash
[root@logstash ~]# yum install -y java-1.8.0-openjdk
已加载插件：fastestmirror
base | 3.6 kB 00:00:00
elk | 2.9 kB 00:00:00
extras | 3.4 kB 00:00:00
local_repo | 3.6 kB 00:00:00
updates | 3.4 kB 00:00:00
Loading mirror speeds from cached hostfile
* base: mirrors.cqu.edu.cn
* extras: mirrors.163.com
* updates: mirrors.163.com
正在解决依赖关系
–> 正在检查事务
—> 软件包 java-1.8.0-openjdk.x86_64.1.1.8.0.181-3.b13.el7_5 将被 安装
–> 正在处理依赖关系 java-1.8.0-openjdk-headless(x86-64) = 1:1.8.0.181-3.b13.el7_5，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 xorg-x11-fonts-Type1，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libpng15.so.15(PNG15_0)(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjvm.so(SUNWprivate_1.1)(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjpeg.so.62(LIBJPEG_6.2)(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjava.so(SUNWprivate_1.1)(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 fontconfig(x86-64)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libpng15.so.15()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjvm.so()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjpeg.so.62()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libjava.so()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libgif.so.4()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXtst.so.6()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXrender.so.1()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXi.so.6()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXext.so.6()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libXcomposite.so.1()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 libX11.so.6()(64bit)，它被软件包 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在检查事务
—> 软件包 fontconfig.x86_64.0.2.10.95-11.el7 将被 安装
–> 正在处理依赖关系 fontpackages-filesystem，它被软件包 fontconfig-2.10.95-11.el7.x86_64 需要
–> 正在处理依赖关系 font(:lang=en)，它被软件包 fontconfig-2.10.95-11.el7.x86_64 需要
—> 软件包 giflib.x86_64.0.4.1.6-9.el7 将被 安装
–> 正在处理依赖关系 libSM.so.6()(64bit)，它被软件包 giflib-4.1.6-9.el7.x86_64 需要
–> 正在处理依赖关系 libICE.so.6()(64bit)，它被软件包 giflib-4.1.6-9.el7.x86_64 需要
—> 软件包 java-1.8.0-openjdk-headless.x86_64.1.1.8.0.181-3.b13.el7_5 将被 安装
–> 正在处理依赖关系 tzdata-java >= 2015d，它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 nss-softokn(x86-64) >= 3.36.0，它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 nss(x86-64) >= 3.36.0，它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 copy-jdk-configs >= 2.2，它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 lksctp-tools(x86-64)，它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
–> 正在处理依赖关系 jpackage-utils，它被软件包 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 需要
—> 软件包 libX11.x86_64.0.1.6.5-1.el7 将被 安装
–> 正在处理依赖关系 libX11-common >= 1.6.5-1.el7，它被软件包 libX11-1.6.5-1.el7.x86_64 需要
–> 正在处理依赖关系 libxcb.so.1()(64bit)，它被软件包 libX11-1.6.5-1.el7.x86_64 需要
—> 软件包 libXcomposite.x86_64.0.0.4.4-4.1.el7 将被 安装
—> 软件包 libXext.x86_64.0.1.3.3-3.el7 将被 安装
—> 软件包 libXi.x86_64.0.1.7.9-1.el7 将被 安装
—> 软件包 libXrender.x86_64.0.0.9.10-1.el7 将被 安装
—> 软件包 libXtst.x86_64.0.1.2.3-1.el7 将被 安装
—> 软件包 libjpeg-turbo.x86_64.0.1.2.90-5.el7 将被 安装
—> 软件包 libpng.x86_64.2.1.5.13-7.el7_2 将被 安装
—> 软件包 xorg-x11-fonts-Type1.noarch.0.7.5-9.el7 将被 安装
–> 正在处理依赖关系 ttmkfdir，它被软件包 xorg-x11-fonts-Type1-7.5-9.el7.noarch 需要
–> 正在处理依赖关系 ttmkfdir，它被软件包 xorg-x11-fonts-Type1-7.5-9.el7.noarch 需要
–> 正在处理依赖关系 mkfontdir，它被软件包 xorg-x11-fonts-Type1-7.5-9.el7.noarch 需要
–> 正在处理依赖关系 mkfontdir，它被软件包 xorg-x11-fonts-Type1-7.5-9.el7.noarch 需要
–> 正在检查事务
—> 软件包 copy-jdk-configs.noarch.0.3.3-10.el7_5 将被 安装
—> 软件包 fontpackages-filesystem.noarch.0.1.44-8.el7 将被 安装
—> 软件包 javapackages-tools.noarch.0.3.4.1-11.el7 将被 安装
–> 正在处理依赖关系 python-javapackages = 3.4.1-11.el7，它被软件包 javapackages-tools-3.4.1-11.el7.noarch 需要
–> 正在处理依赖关系 libxslt，它被软件包 javapackages-tools-3.4.1-11.el7.noarch 需要
—> 软件包 libICE.x86_64.0.1.0.9-9.el7 将被 安装
—> 软件包 libSM.x86_64.0.1.2.2-2.el7 将被 安装
—> 软件包 libX11-common.noarch.0.1.6.5-1.el7 将被 安装
—> 软件包 libxcb.x86_64.0.1.12-1.el7 将被 安装
–> 正在处理依赖关系 libXau.so.6()(64bit)，它被软件包 libxcb-1.12-1.el7.x86_64 需要
—> 软件包 lksctp-tools.x86_64.0.1.0.17-2.el7 将被 安装
—> 软件包 nss.x86_64.0.3.28.4-8.el7 将被 升级
–> 正在处理依赖关系 nss = 3.28.4-8.el7，它被软件包 nss-sysinit-3.28.4-8.el7.x86_64 需要
–> 正在处理依赖关系 nss(x86-64) = 3.28.4-8.el7，它被软件包 nss-tools-3.28.4-8.el7.x86_64 需要
—> 软件包 nss.x86_64.0.3.36.0-7.el7_5 将被 更新
–> 正在处理依赖关系 nss-util >= 3.36.0-1，它被软件包 nss-3.36.0-7.el7_5.x86_64 需要
–> 正在处理依赖关系 nspr >= 4.19.0，它被软件包 nss-3.36.0-7.el7_5.x86_64 需要
–> 正在处理依赖关系 libnssutil3.so(NSSUTIL_3.31)(64bit)，它被软件包 nss-3.36.0-7.el7_5.x86_64 需要
—> 软件包 nss-softokn.x86_64.0.3.28.3-6.el7 将被 升级
—> 软件包 nss-softokn.x86_64.0.3.36.0-5.el7_5 将被 更新
–> 正在处理依赖关系 nss-softokn-freebl(x86-64) >= 3.36.0-5.el7_5，它被软件包 nss-softokn-3.36.0-5.el7_5.x86_64 需要
—> 软件包 stix-fonts.noarch.0.1.1.0-5.el7 将被 安装
—> 软件包 ttmkfdir.x86_64.0.3.0.9-42.el7 将被 安装
—> 软件包 tzdata-java.noarch.0.2018e-3.el7 将被 安装
—> 软件包 xorg-x11-font-utils.x86_64.1.7.5-20.el7 将被 安装
–> 正在处理依赖关系 libfontenc.so.1()(64bit)，它被软件包 1:xorg-x11-font-utils-7.5-20.el7.x86_64 需要
–> 正在处理依赖关系 libXfont.so.1()(64bit)，它被软件包 1:xorg-x11-font-utils-7.5-20.el7.x86_64 需要
–> 正在检查事务
—> 软件包 libXau.x86_64.0.1.0.8-2.1.el7 将被 安装
—> 软件包 libXfont.x86_64.0.1.5.2-1.el7 将被 安装
—> 软件包 libfontenc.x86_64.0.1.1.3-3.el7 将被 安装
—> 软件包 libxslt.x86_64.0.1.1.28-5.el7 将被 安装
—> 软件包 nspr.x86_64.0.4.13.1-1.0.el7_3 将被 升级
—> 软件包 nspr.x86_64.0.4.19.0-1.el7_5 将被 更新
—> 软件包 nss-softokn-freebl.x86_64.0.3.28.3-6.el7 将被 升级
—> 软件包 nss-softokn-freebl.x86_64.0.3.36.0-5.el7_5 将被 更新
—> 软件包 nss-sysinit.x86_64.0.3.28.4-8.el7 将被 升级
—> 软件包 nss-sysinit.x86_64.0.3.36.0-7.el7_5 将被 更新
—> 软件包 nss-tools.x86_64.0.3.28.4-8.el7 将被 升级
—> 软件包 nss-tools.x86_64.0.3.36.0-7.el7_5 将被 更新
—> 软件包 nss-util.x86_64.0.3.28.4-3.el7 将被 升级
—> 软件包 nss-util.x86_64.0.3.36.0-1.el7_5 将被 更新
—> 软件包 python-javapackages.noarch.0.3.4.1-11.el7 将被 安装
–> 正在处理依赖关系 python-lxml，它被软件包 python-javapackages-3.4.1-11.el7.noarch 需要
–> 正在检查事务
—> 软件包 python-lxml.x86_64.0.3.2.1-4.el7 将被 安装
–> 解决依赖关系完成
依赖关系解决
=====================================================================================================================
Package 架构 版本 源 大小
=====================================================================================================================
正在安装:
java-1.8.0-openjdk x86_64 1:1.8.0.181-3.b13.el7_5 updates 250 k
为依赖而安装:
copy-jdk-configs noarch 3.3-10.el7_5 updates 21 k
fontconfig x86_64 2.10.95-11.el7 base 229 k
fontpackages-filesystem noarch 1.44-8.el7 base 9.9 k
giflib x86_64 4.1.6-9.el7 base 40 k
java-1.8.0-openjdk-headless x86_64 1:1.8.0.181-3.b13.el7_5 updates 32 M
javapackages-tools noarch 3.4.1-11.el7 base 73 k
libICE x86_64 1.0.9-9.el7 base 66 k
libSM x86_64 1.2.2-2.el7 base 39 k
libX11 x86_64 1.6.5-1.el7 base 606 k
libX11-common noarch 1.6.5-1.el7 base 164 k
libXau x86_64 1.0.8-2.1.el7 base 29 k
libXcomposite x86_64 0.4.4-4.1.el7 base 22 k
libXext x86_64 1.3.3-3.el7 base 39 k
libXfont x86_64 1.5.2-1.el7 base 152 k
libXi x86_64 1.7.9-1.el7 base 40 k
libXrender x86_64 0.9.10-1.el7 base 26 k
libXtst x86_64 1.2.3-1.el7 base 20 k
libfontenc x86_64 1.1.3-3.el7 base 31 k
libjpeg-turbo x86_64 1.2.90-5.el7 base 134 k
libpng x86_64 2:1.5.13-7.el7_2 base 213 k
libxcb x86_64 1.12-1.el7 base 211 k
libxslt x86_64 1.1.28-5.el7 base 242 k
lksctp-tools x86_64 1.0.17-2.el7 base 88 k
python-javapackages noarch 3.4.1-11.el7 base 31 k
python-lxml x86_64 3.2.1-4.el7 base 758 k
stix-fonts noarch 1.1.0-5.el7 base 1.3 M
ttmkfdir x86_64 3.0.9-42.el7 base 48 k
tzdata-java noarch 2018e-3.el7 updates 185 k
xorg-x11-font-utils x86_64 1:7.5-20.el7 base 87 k
xorg-x11-fonts-Type1 noarch 7.5-9.el7 base 521 k
为依赖而更新:
nspr x86_64 4.19.0-1.el7_5 updates 127 k
nss x86_64 3.36.0-7.el7_5 updates 835 k
nss-softokn x86_64 3.36.0-5.el7_5 updates 315 k
nss-softokn-freebl x86_64 3.36.0-5.el7_5 updates 222 k
nss-sysinit x86_64 3.36.0-7.el7_5 updates 62 k
nss-tools x86_64 3.36.0-7.el7_5 updates 515 k
nss-util x86_64 3.36.0-1.el7_5 updates 78 k
事务概要
=====================================================================================================================
安装 1 软件包 (+30 依赖软件包)
升级 ( 7 依赖软件包)
总下载量：39 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/38): copy-jdk-configs-3.3-10.el7_5.noarch.rpm | 21 kB 00:00:00
(2/38): fontpackages-filesystem-1.44-8.el7.noarch.rpm | 9.9 kB 00:00:00
(3/38): giflib-4.1.6-9.el7.x86_64.rpm | 40 kB 00:00:00
(4/38): fontconfig-2.10.95-11.el7.x86_64.rpm | 229 kB 00:00:00
(5/38): libICE-1.0.9-9.el7.x86_64.rpm | 66 kB 00:00:00
(6/38): javapackages-tools-3.4.1-11.el7.noarch.rpm | 73 kB 00:00:00
(7/38): libSM-1.2.2-2.el7.x86_64.rpm | 39 kB 00:00:00
(8/38): libX11-common-1.6.5-1.el7.noarch.rpm | 164 kB 00:00:00
(9/38): java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64.rpm | 250 kB 00:00:01
(10/38): libX11-1.6.5-1.el7.x86_64.rpm | 606 kB 00:00:00
(11/38): libXau-1.0.8-2.1.el7.x86_64.rpm | 29 kB 00:00:00
(12/38): libXcomposite-0.4.4-4.1.el7.x86_64.rpm | 22 kB 00:00:00
(13/38): libXext-1.3.3-3.el7.x86_64.rpm | 39 kB 00:00:00
(14/38): libXfont-1.5.2-1.el7.x86_64.rpm | 152 kB 00:00:00
(15/38): libXi-1.7.9-1.el7.x86_64.rpm | 40 kB 00:00:00
(16/38): libXrender-0.9.10-1.el7.x86_64.rpm | 26 kB 00:00:00
(17/38): libXtst-1.2.3-1.el7.x86_64.rpm | 20 kB 00:00:00
(18/38): libfontenc-1.1.3-3.el7.x86_64.rpm | 31 kB 00:00:00
(19/38): libjpeg-turbo-1.2.90-5.el7.x86_64.rpm | 134 kB 00:00:00
(20/38): libpng-1.5.13-7.el7_2.x86_64.rpm | 213 kB 00:00:00
(21/38): libxcb-1.12-1.el7.x86_64.rpm | 211 kB 00:00:00
(22/38): libxslt-1.1.28-5.el7.x86_64.rpm | 242 kB 00:00:00
(23/38): lksctp-tools-1.0.17-2.el7.x86_64.rpm | 88 kB 00:00:00
(24/38): nspr-4.19.0-1.el7_5.x86_64.rpm | 127 kB 00:00:01
(25/38): nss-3.36.0-7.el7_5.x86_64.rpm | 835 kB 00:00:00
(26/38): nss-softokn-3.36.0-5.el7_5.x86_64.rpm | 315 kB 00:00:00
(27/38): nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm | 222 kB 00:00:00
(28/38): nss-sysinit-3.36.0-7.el7_5.x86_64.rpm | 62 kB 00:00:00
(29/38): nss-tools-3.36.0-7.el7_5.x86_64.rpm | 515 kB 00:00:00
(30/38): java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64.rpm | 32 MB 00:00:05
(31/38): nss-util-3.36.0-1.el7_5.x86_64.rpm | 78 kB 00:00:00
(32/38): python-javapackages-3.4.1-11.el7.noarch.rpm | 31 kB 00:00:00
(33/38): python-lxml-3.2.1-4.el7.x86_64.rpm | 758 kB 00:00:01
(34/38): ttmkfdir-3.0.9-42.el7.x86_64.rpm | 48 kB 00:00:00
(35/38): stix-fonts-1.1.0-5.el7.noarch.rpm | 1.3 MB 00:00:00
(36/38): xorg-x11-font-utils-7.5-20.el7.x86_64.rpm | 87 kB 00:00:00
(37/38): xorg-x11-fonts-Type1-7.5-9.el7.noarch.rpm | 521 kB 00:00:00
(38/38): tzdata-java-2018e-3.el7.noarch.rpm | 185 kB 00:00:01
———————————————————————————————————————
总计 4.7 MB/s | 39 MB 00:00:08
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在更新 : nspr-4.19.0-1.el7_5.x86_64 1/45
正在更新 : nss-util-3.36.0-1.el7_5.x86_64 2/45
正在安装 : libfontenc-1.1.3-3.el7.x86_64 3/45
正在安装 : libxslt-1.1.28-5.el7.x86_64 4/45
正在安装 : fontpackages-filesystem-1.44-8.el7.noarch 5/45
正在安装 : libjpeg-turbo-1.2.90-5.el7.x86_64 6/45
正在安装 : libICE-1.0.9-9.el7.x86_64 7/45
正在安装 : libSM-1.2.2-2.el7.x86_64 8/45
正在安装 : stix-fonts-1.1.0-5.el7.noarch 9/45
正在安装 : fontconfig-2.10.95-11.el7.x86_64 10/45
正在安装 : python-lxml-3.2.1-4.el7.x86_64 11/45
正在安装 : python-javapackages-3.4.1-11.el7.noarch 12/45
正在安装 : javapackages-tools-3.4.1-11.el7.noarch 13/45
正在安装 : libXfont-1.5.2-1.el7.x86_64 14/45
正在安装 : 1:xorg-x11-font-utils-7.5-20.el7.x86_64 15/45
正在更新 : nss-softokn-freebl-3.36.0-5.el7_5.x86_64 16/45
正在更新 : nss-softokn-3.36.0-5.el7_5.x86_64 17/45
正在更新 : nss-sysinit-3.36.0-7.el7_5.x86_64 18/45
正在更新 : nss-3.36.0-7.el7_5.x86_64 19/45
正在安装 : tzdata-java-2018e-3.el7.noarch 20/45
正在安装 : libXau-1.0.8-2.1.el7.x86_64 21/45
正在安装 : libxcb-1.12-1.el7.x86_64 22/45
正在安装 : libX11-common-1.6.5-1.el7.noarch 23/45
正在安装 : libX11-1.6.5-1.el7.x86_64 24/45
正在安装 : libXext-1.3.3-3.el7.x86_64 25/45
正在安装 : libXi-1.7.9-1.el7.x86_64 26/45
正在安装 : libXtst-1.2.3-1.el7.x86_64 27/45
正在安装 : giflib-4.1.6-9.el7.x86_64 28/45
正在安装 : libXrender-0.9.10-1.el7.x86_64 29/45
正在安装 : libXcomposite-0.4.4-4.1.el7.x86_64 30/45
正在安装 : lksctp-tools-1.0.17-2.el7.x86_64 31/45
正在安装 : copy-jdk-configs-3.3-10.el7_5.noarch 32/45
正在安装 : 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 33/45
正在安装 : ttmkfdir-3.0.9-42.el7.x86_64 34/45
正在安装 : xorg-x11-fonts-Type1-7.5-9.el7.noarch 35/45
正在安装 : 2:libpng-1.5.13-7.el7_2.x86_64 36/45
正在安装 : 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 37/45
正在更新 : nss-tools-3.36.0-7.el7_5.x86_64 38/45
清理 : nss-tools-3.28.4-8.el7.x86_64 39/45
清理 : nss-sysinit-3.28.4-8.el7.x86_64 40/45
清理 : nss-3.28.4-8.el7.x86_64 41/45
清理 : nss-softokn-3.28.3-6.el7.x86_64 42/45
清理 : nss-util-3.28.4-3.el7.x86_64 43/45
清理 : nspr-4.13.1-1.0.el7_3.x86_64 44/45
清理 : nss-softokn-freebl-3.28.3-6.el7.x86_64 45/45
验证中 : libXext-1.3.3-3.el7.x86_64 1/45
验证中 : libX11-1.6.5-1.el7.x86_64 2/45
验证中 : libXi-1.7.9-1.el7.x86_64 3/45
验证中 : python-javapackages-3.4.1-11.el7.noarch 4/45
验证中 : libICE-1.0.9-9.el7.x86_64 5/45
验证中 : giflib-4.1.6-9.el7.x86_64 6/45
验证中 : libjpeg-turbo-1.2.90-5.el7.x86_64 7/45
验证中 : nspr-4.19.0-1.el7_5.x86_64 8/45
验证中 : libXrender-0.9.10-1.el7.x86_64 9/45
验证中 : libXfont-1.5.2-1.el7.x86_64 10/45
验证中 : python-lxml-3.2.1-4.el7.x86_64 11/45
验证中 : 2:libpng-1.5.13-7.el7_2.x86_64 12/45
验证中 : nss-3.36.0-7.el7_5.x86_64 13/45
验证中 : fontpackages-filesystem-1.44-8.el7.noarch 14/45
验证中 : ttmkfdir-3.0.9-42.el7.x86_64 15/45
验证中 : nss-util-3.36.0-1.el7_5.x86_64 16/45
验证中 : copy-jdk-configs-3.3-10.el7_5.noarch 17/45
验证中 : stix-fonts-1.1.0-5.el7.noarch 18/45
验证中 : libXcomposite-0.4.4-4.1.el7.x86_64 19/45
验证中 : libXtst-1.2.3-1.el7.x86_64 20/45
验证中 : lksctp-tools-1.0.17-2.el7.x86_64 21/45
验证中 : 1:java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 22/45
验证中 : 1:java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 23/45
验证中 : libxcb-1.12-1.el7.x86_64 24/45
验证中 : xorg-x11-fonts-Type1-7.5-9.el7.noarch 25/45
验证中 : libxslt-1.1.28-5.el7.x86_64 26/45
验证中 : nss-tools-3.36.0-7.el7_5.x86_64 27/45
验证中 : javapackages-tools-3.4.1-11.el7.noarch 28/45
验证中 : libX11-common-1.6.5-1.el7.noarch 29/45
验证中 : nss-sysinit-3.36.0-7.el7_5.x86_64 30/45
验证中 : libXau-1.0.8-2.1.el7.x86_64 31/45
验证中 : nss-softokn-freebl-3.36.0-5.el7_5.x86_64 32/45
验证中 : libSM-1.2.2-2.el7.x86_64 33/45
验证中 : fontconfig-2.10.95-11.el7.x86_64 34/45
验证中 : tzdata-java-2018e-3.el7.noarch 35/45
验证中 : libfontenc-1.1.3-3.el7.x86_64 36/45
验证中 : nss-softokn-3.36.0-5.el7_5.x86_64 37/45
验证中 : 1:xorg-x11-font-utils-7.5-20.el7.x86_64 38/45
验证中 : nspr-4.13.1-1.0.el7_3.x86_64 39/45
验证中 : nss-softokn-3.28.3-6.el7.x86_64 40/45
验证中 : nss-util-3.28.4-3.el7.x86_64 41/45
验证中 : nss-softokn-freebl-3.28.3-6.el7.x86_64 42/45
验证中 : nss-3.28.4-8.el7.x86_64 43/45
验证中 : nss-sysinit-3.28.4-8.el7.x86_64 44/45
验证中 : nss-tools-3.28.4-8.el7.x86_64 45/45
已安装:
java-1.8.0-openjdk.x86_64 1:1.8.0.181-3.b13.el7_5
作为依赖被安装:
copy-jdk-configs.noarch 0:3.3-10.el7_5 fontconfig.x86_64 0:2.10.95-11.el7
fontpackages-filesystem.noarch 0:1.44-8.el7 giflib.x86_64 0:4.1.6-9.el7
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.181-3.b13.el7_5 javapackages-tools.noarch 0:3.4.1-11.el7
libICE.x86_64 0:1.0.9-9.el7 libSM.x86_64 0:1.2.2-2.el7
libX11.x86_64 0:1.6.5-1.el7 libX11-common.noarch 0:1.6.5-1.el7
libXau.x86_64 0:1.0.8-2.1.el7 libXcomposite.x86_64 0:0.4.4-4.1.el7
libXext.x86_64 0:1.3.3-3.el7 libXfont.x86_64 0:1.5.2-1.el7
libXi.x86_64 0:1.7.9-1.el7 libXrender.x86_64 0:0.9.10-1.el7
libXtst.x86_64 0:1.2.3-1.el7 libfontenc.x86_64 0:1.1.3-3.el7
libjpeg-turbo.x86_64 0:1.2.90-5.el7 libpng.x86_64 2:1.5.13-7.el7_2
libxcb.x86_64 0:1.12-1.el7 libxslt.x86_64 0:1.1.28-5.el7
lksctp-tools.x86_64 0:1.0.17-2.el7 python-javapackages.noarch 0:3.4.1-11.el7
python-lxml.x86_64 0:3.2.1-4.el7 stix-fonts.noarch 0:1.1.0-5.el7
ttmkfdir.x86_64 0:3.0.9-42.el7 tzdata-java.noarch 0:2018e-3.el7
xorg-x11-font-utils.x86_64 1:7.5-20.el7 xorg-x11-fonts-Type1.noarch 0:7.5-9.el7
作为依赖被升级:
nspr.x86_64 0:4.19.0-1.el7_5 nss.x86_64 0:3.36.0-7.el7_5 nss-softokn.x86_64 0:3.36.0-5.el7_5
nss-softokn-freebl.x86_64 0:3.36.0-5.el7_5 nss-sysinit.x86_64 0:3.36.0-7.el7_5 nss-tools.x86_64 0:3.36.0-7.el7_5
nss-util.x86_64 0:3.36.0-1.el7_5
完毕！
[root@logstash ~]# yum install -y logstash
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cqu.edu.cn
* extras: mirrors.163.com
* updates: mirrors.163.com
正在解决依赖关系
–> 正在检查事务
—> 软件包 logstash.noarch.1.2.3.4-1 将被 安装
–> 解决依赖关系完成
依赖关系解决
=====================================================================================================================
Package 架构 版本 源 大小
=====================================================================================================================
正在安装:
logstash noarch 1:2.3.4-1 elk 77 M
事务概要
=====================================================================================================================
安装 1 软件包
总下载量：77 M
安装大小：136 M
Downloading packages:
logstash-2.3.4-1.noarch.rpm | 77 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : 1:logstash-2.3.4-1.noarch 1/1
验证中 : 1:logstash-2.3.4-1.noarch 1/1
已安装:
logstash.noarch 1:2.3.4-1
完毕！
[root@logstash ~]# java -version
openjdk version “1.8.0_181”
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
[root@logstash ~]#
[root@logstash ~]# touch /etc/logstash/logstash.conf
[root@logstash ~]# /opt/logstash/bin/logstash –version
logstash 2.3.4
[root@logstash ~]# /opt/logstash/bin/logstash-plugin list
Ignoring ffi-1.9.13 because its extensions are not built. Try: gem pristine ffi –version 1.9.13
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-oldlogstashjson
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-anonymize
logstash-filter-checksum
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-dns
logstash-filter-drop
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-multiline
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-beats
logstash-input-couchdb_changes
logstash-input-elasticsearch
logstash-input-eventlog
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-irc
logstash-input-jdbc
logstash-input-kafka
logstash-input-log4j
logstash-input-lumberjack
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-input-xmpp
logstash-input-zeromq
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-exec
logstash-output-file
logstash-output-ganglia
logstash-output-gelf
logstash-output-graphite
logstash-output-hipchat
logstash-output-http
logstash-output-irc
logstash-output-juggernaut
logstash-output-kafka
logstash-output-lumberjack
logstash-output-nagios
logstash-output-nagios_nsca
logstash-output-null
logstash-output-opentsdb
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-statsd
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-xmpp
logstash-output-zeromq
logstash-patterns-core
[root@logstash ~]#
[root@logstash ~]#
[root@logstash ~]# /opt/logstash/bin/logstash-plugin list | grep std
Ignoring ffi-1.9.13 because its extensions are not built. Try: gem pristine ffi –version 1.9.13
logstash-input-stdin
logstash-output-stdout
[root@logstash ~]#
[root@logstash ~]#
配置配置文件
[root@logstash ~]# vim /etc/logstash/logstash.conf
[root@logstash ~]# cat /etc/logstash/logstash.conf
input{
stdin{
}
}
filter{
}
output{
stdout{
}
}
启动，测试一下，logstash 配置从标准输入读取输入源,然后从标准输出输出到屏幕
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
aa
2018-10-21T08:29:43.307Z logstash aa
test
2018-10-21T08:30:10.145Z logstash test
备注：若不会写配置文件可以找帮助，插件文档的位置：
https://github.com/logstash-plugins
下面是各类插件的使用
注意，我起的名字和上面的案例不一样
1、CODEC类插件
[root@logstash ~]# vim /etc/logstash/codec.conf
//输入设置为编码json
//输出设置为rubydebug
[root@logstash ~]# cat /etc/logstash/codec.conf
input{
stdin{
codec => “json”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/codec.conf
Settings: Default pipeline workers: 2
Pipeline main started
{“a”:1}
{
“a” => 1,
“@version” => “1”,
“@timestamp” => “2018-10-21T08:34:07.627Z”,
“host” => “logstash”
}
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}
Pipeline main has been shutdown
[root@logstash ~]#
file模块插件
[root@logstash ~]# vim /etc/logstash/file.conf
[root@logstash ~]# touch /tmp/a.log
[root@logstash ~]# touch /var/tmp/b.log
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/file.conf
另外起一个终端去写点东西
[root@logstash ~]# echo a1 > /tmp/a.log
[root@logstash ~]# echo b1 > /var/tmp/b.log
回到刚才的终端查看一下
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/file.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “a1”,
“@version” => “1”,
“@timestamp” => “2018-10-21T08:56:55.540Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b1”,
“@version” => “1”,
“@timestamp” => “2018-10-21T08:56:55.628Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
^CSIGINT received. Shutting down the agent. {:level=>:warn}
注意，如果已经存在结果，即刚才写的 /var/lib/logstash/sincedb 那么是不会有输出的，需要删掉这个文件再来
TCP/UDP模块
现在logstash主机上写好配置文件，并启动
[root@logstash ~]# vim /etc/logstash/tcpudp.conf
[root@logstash ~]# cat /etc/logstash/tcpudp.conf
input{
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/tcpudp.conf
Settings: Default pipeline workers: 2
Pipeline main started
再开一个终端，可以看到端口已经起来了
[root@logstash ~]# netstat -antup | grep 8888
tcp6 0 0 :::8888 :::* LISTEN 1836/java
[root@logstash ~]# netstat -antup | grep 9999
udp6 0 0 :::9999 :::* 1836/java
[root@logstash ~]#
随便找个主机，写个脚本发点数据
[root@se5 ~]# vim tcp.sh
[root@se5 ~]#
[root@se5 ~]#
[root@se5 ~]# cat tcp.sh
function sendmsg(){
if [[ “$1” == “tcp” ]];then
exec 9<>/dev/tcp/192.168.1.67/8888
else
exec 9<>/dev/udp/192.168.1.67/9999
fi
echo “$2” >&9
exec 9<&-
}
[root@se5 ~]# . tcp.sh
[root@se5 ~]# sendmsg tcp “is tcp test”
[root@se5 ~]# sendmsg udp “is udp ssss”
再回到logstash主机，可以看到结果
Pipeline main started
{
“message” => “is tcp test”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:26:11.369Z”,
“host” => “192.168.1.65”,
“port” => 39894,
“type” => “tcplog”
}
{
“message” => “is udp ssss\n”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:26:27.501Z”,
“type” => “udplog”,
“host” => “192.168.1.65”
然后退出
}
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>[“org/jruby/RubyIO.java:3682:in `select'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:77:in `udp_listener'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:50:in `run'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:342:in `inputworker'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:336:in `start_input'”], :level=>:warn}
Pipeline main has been shutdown
syslog插件练习
1、写好配置文件并启动
（注意：刚才写的配置文件懒得删除了，将就看吧）
[root@logstash ~]# systemctl list-unit-files | grep syslog
rsyslog.service enabled
syslog.socket static
[root@logstash ~]# cat /etc/logstash/syslog.conf
input{
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/syslog.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “a2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:49:00.756Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:49:00.997Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
另外起一个终端，确认514端口已经起来
[root@logstash ~]# netstat -antup | grep 514
tcp6 0 0 :::514 :::* LISTEN 2024/java
udp6 0 0 :::514 :::* 2024/java
[root@logstash ~]#
关键的测试步骤，找台机器se5，把配置文件改下
//写一个@或两个@@都可以，一个@代表udp，两个@@代表tcp
[root@se5 ~]# vim /etc/rsyslog.conf
[root@se5 ~]# tail /etc/rsyslog.conf
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
#
local0.info @192.168.1.67:514
重启服务，然后写点东西发出去
[root@se5 ~]# systemctl restart rsyslog.service
[root@se5 ~]# logger -p local0.info -t nds “001 elk”
回到logstash确认一下，可以看到已经收到结果了，注意第1个和第2个是上面的实验
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/syslog.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “a2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:49:00.756Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:49:00.997Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “001 elk”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:53:03.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 134,
“timestamp” => “Oct 21 17:53:03”,
“logsource” => “se5”,
“program” => “nds”,
“severity” => 6,
“facility” => 16,
“facility_label” => “local0”,
“severity_label” => “Informational”
}
再搞一个实验，就是ES5登陆的时候，把验证过程也发过去给els主机去记录，先改配置文件
[root@se5 ~]# grep authpriv /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
#authpriv.* /var/log/secure
authpriv.* @@192.168.1.67:514
[root@se5 ~]# systemctl restart rsyslog.service
为了方便看实验结果，先清个屏测试一下
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
Errno::EBADF: Bad file descriptor – Bad file descriptor
each at org/jruby/RubyIO.java:3542
tcp_receiver at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:173
tcp_listener at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:159
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/syslog.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:57:21.892Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
重新启个终端，登陆65
[root@room9pc52 ~]# ssh 192.168.1.65
[email protected]’s password:
Last login: Sun Oct 21 10:13:00 2018 from 192.168.1.254
[root@se5 ~]#
回到logstash看一眼，你会看到信息已经来了
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/syslog.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-21T09:57:21.892Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “Accepted password for root from 192.168.1.254 port 48914 ssh2\n”,
“@version” => “1”,
“@timestamp” => “2018-10-22T01:20:07.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 86,
“timestamp” => “Oct 22 09:20:07”,
“logsource” => “se5”,
“program” => “sshd”,
“pid” => “2340”,
“severity” => 6,
“facility” => 10,
“facility_label” => “security/authorization”,
“severity_label” => “Informational”
}
{
“message” => “pam_unix(sshd:session): session opened for user root by (uid=0)\n”,
“@version” => “1”,
“@timestamp” => “2018-10-22T01:20:07.000Z”,
“type” => “syslog”,
“host” => “192.168.1.65”,
“priority” => 86,
“timestamp” => “Oct 22 09:20:07”,
“logsource” => “se5”,
“program” => “sshd”,
“pid” => “2340”,
“severity” => 6,
“facility” => 10,
“facility_label” => “security/authorization”,
“severity_label” => “Informational”
}
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>[“org/jruby/RubyIO.java:3682:in `select'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:77:in `udp_listener'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:50:in `run'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:342:in `inputworker'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:336:in `start_input'”], :level=>:warn}
Errno::EBADF: Bad file descriptor – Bad file descriptor
each at org/jruby/RubyIO.java:3542
tcp_receiver at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:173
tcp_listener at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0.5/lib/logstash/inputs/syslog.rb:159
7）filter grok插件
grok插件：
解析各种非结构化的日志数据插件
grok使用正则表达式把飞结构化的数据结构化
在分组匹配，正则表达式需要根据具体数据结构编写
虽然编写困难，但适用性极广
先在logstash编写配置文件
[root@logstash ~]# vim /etc/logstash/grok.conf
[root@logstash ~]# cat /etc/logstash/grok.conf
input{
stdin{ codec => “json” }
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
grok{
match => [“message”, “(?<key>reg)”]
}
}
output{
stdout{
codec => “rubydebug”
}
}
然后在测试主机se5去装个APACHE，用真机访问一下，确认有日志
[root@se5 ~]# yum install -y httpd
Loaded plugins: fastestmirror
base | 3.6 kB 00:00:00
elk | 2.9 kB 00:00:00
extras | 3.4 kB 00:00:00
local_repo | 3.6 kB 00:00:00
updates | 3.4 kB 00:00:00
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
* extras: mirrors.163.com
* updates: mirrors.163.com
Resolving Dependencies
–> Running transaction check
—> Package httpd.x86_64 0:2.4.6-80.el7.centos.1 will be installed
–> Processing Dependency: httpd-tools = 2.4.6-80.el7.centos.1 for package: httpd-2.4.6-80.el7.centos.1.x86_64
–> Processing Dependency: /etc/mime.types for package: httpd-2.4.6-80.el7.centos.1.x86_64
–> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-80.el7.centos.1.x86_64
–> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-80.el7.centos.1.x86_64
–> Running transaction check
—> Package apr.x86_64 0:1.4.8-3.el7_4.1 will be installed
—> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed
—> Package httpd-tools.x86_64 0:2.4.6-80.el7.centos.1 will be installed
—> Package mailcap.noarch 0:2.1.41-2.el7 will be installed
–> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================================
Package Arch Version Repository Size
====================================================================================================================
Installing:
httpd x86_64 2.4.6-80.el7.centos.1 updates 2.7 M
Installing for dependencies:
apr x86_64 1.4.8-3.el7_4.1 base 103 k
apr-util x86_64 1.5.2-6.el7 base 92 k
httpd-tools x86_64 2.4.6-80.el7.centos.1 updates 90 k
mailcap noarch 2.1.41-2.el7 base 31 k
Transaction Summary
====================================================================================================================
Install 1 Package (+4 Dependent packages)
Total download size: 3.0 M
Installed size: 10 M
Downloading packages:
(1/5): apr-util-1.5.2-6.el7.x86_64.rpm | 92 kB 00:00:00
(2/5): apr-1.4.8-3.el7_4.1.x86_64.rpm | 103 kB 00:00:00
(3/5): httpd-tools-2.4.6-80.el7.centos.1.x86_64.rpm | 90 kB 00:00:00
(4/5): httpd-2.4.6-80.el7.centos.1.x86_64.rpm | 2.7 MB 00:00:01
mailcap-2.1.41-2.el7.noarch.rp FAILED ==-] 443 B/s | 3.0 MB 00:00:51 ETA
http://mirrors.zju.edu.cn/centos/7.5.1804/os/x86_64/Packages/mailcap-2.1.41-2.el7.noarch.rpm: [Errno 12] Timeout on http://mirrors.zju.edu.cn/centos/7.5.1804/os/x86_64/Packages/mailcap-2.1.41-2.el7.noarch.rpm: (28, ‘Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds’)
Trying other mirror.
(5/5): mailcap-2.1.41-2.el7.noarch.rpm | 31 kB 00:00:00
——————————————————————————————————————–
Total 85 kB/s | 3.0 MB 00:00:36
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : apr-1.4.8-3.el7_4.1.x86_64 1/5
Installing : apr-util-1.5.2-6.el7.x86_64 2/5
Installing : httpd-tools-2.4.6-80.el7.centos.1.x86_64 3/5
Installing : mailcap-2.1.41-2.el7.noarch 4/5
Installing : httpd-2.4.6-80.el7.centos.1.x86_64 5/5
Verifying : mailcap-2.1.41-2.el7.noarch 1/5
Verifying : httpd-tools-2.4.6-80.el7.centos.1.x86_64 2/5
Verifying : apr-util-1.5.2-6.el7.x86_64 3/5
Verifying : apr-1.4.8-3.el7_4.1.x86_64 4/5
Verifying : httpd-2.4.6-80.el7.centos.1.x86_64 5/5
Installed:
httpd.x86_64 0:2.4.6-80.el7.centos.1
Dependency Installed:
apr.x86_64 0:1.4.8-3.el7_4.1 apr-util.x86_64 0:1.5.2-6.el7 httpd-tools.x86_64 0:2.4.6-80.el7.centos.1
mailcap.noarch 0:2.1.41-2.el7
Complete!
[root@se5 ~]# systemctl restart httpd
[root@se5 ~]# vim /var/log/httpd/access_log
[root@se5 ~]# cat /var/log/httpd/access_log
192.168.1.254 – – [22/Oct/2018:10:07:23 +0800] “GET / HTTP/1.1” 403 4897 “-” “curl/7.29.0”
测试一下，先手动创建测试的文件，也就是把刚才在SE5查到的日志复制到logstash主机的测试目录下面，运行，并测试
[root@logstash ~]# vim /tmp/a.log
[root@logstash ~]# cat /tmp/a.log
192.168.1.254 – – [22/Oct/2018:10:07:23 +0800] “GET / HTTP/1.1” 403 4897 “-” “curl/7.29.0”
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/grok.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “.168.1.254 – – [22/Oct/2018:10:07:23 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”curl/7.29.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:18:54.916Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”,
“tags” => [
[0] “_grokparsefailure”
]
}
//出现message的日志，但是没有解析是什么意思
为了实现解析，根据消息的规律，我们匹配COMBINEDAPACHELOG这个关键词,先修改配置文件
[root@logstash ~]# vim /etc/logstash/grok.conf
[root@logstash ~]# cat /etc/logstash/grok.conf
input{
stdin{ codec => “json” }
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
grok{
match => [“message”, “%{COMBINEDAPACHELOG}”]
}
}
output{
stdout{
codec => “rubydebug”
}
}
然后重新启动一下，注意，之前说过，因为配置文件第一个file里面，写了解析结果路径，因为上面已经运行过了，所以要把之前的结果删除掉
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/grok.conf
Settings: Default pipeline workers: 2
Pipeline main started
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>”main”}
UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>[“org/jruby/RubyIO.java:3682:in `select'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:77:in `udp_listener'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:50:in `run'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:342:in `inputworker'”, “/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:336:in `start_input'”], :level=>:warn}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}
^CSIGINT received. Terminating immediately.. {:level=>:fatal}
删除刚才出来的结果
[root@logstash ~]# rm -rf /var/lib/logstash/sincedb
再次运行一下，现在就有结果了，而且相比较上面的详细多了（忽略下面那个b2）
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/grok.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:07:23 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”curl/7.29.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:29:34.705Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”,
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “22/Oct/2018:10:07:23 +0800”,
“verb” => “GET”,
“request” => “/”,
“httpversion” => “1.1”,
“response” => “403”,
“bytes” => “4897”,
“referrer” => “\”-\””,
“agent” => “\”curl/7.29.0\””
}
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:29:35.001Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”,
“tags” => [
[0] “_grokparsefailure”
]
}
步骤二，在apache主机上，安装filebeat手机apache服务器的日志，存入elasticsearch
[root@se5 ~]# yum install -y filebeat
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
* extras: mirrors.163.com
* updates: mirrors.163.com
Resolving Dependencies
–> Running transaction check
—> Package filebeat.x86_64 0:1.2.3-1 will be installed
–> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================================
Package Arch Version Repository Size
====================================================================================================================
Installing:
filebeat x86_64 1.2.3-1 elk 3.9 M
Transaction Summary
====================================================================================================================
Install 1 Package
Total download size: 3.9 M
Installed size: 13 M
Downloading packages:
filebeat-1.2.3-x86_64.rpm | 3.9 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : filebeat-1.2.3-1.x86_64 1/1
Verifying : filebeat-1.2.3-1.x86_64 1/1
Installed:
filebeat.x86_64 0:1.2.3-1
Complete!
[root@se5 ~]# vim /etc/filebeat/filebeat.yml
[root@se5 ~]# vim/etc/filebeat/filebeat.yml
paths:
– /var/log/httpd/access_log //日志的路径，短横线加空格代表yml格式
document_type: apachelog //文档类型
elasticsearch: //加上注释
hosts: [“localhost:9200”] //加上注释
logstash: //去掉注释
hosts: [“192.168.1.67:5044”] //去掉注释,logstash那台主机的ip
启动filebeat服务
[root@se5 ~]# systemctl start filebeat
[root@se5 ~]# systemctl status filebeat
● filebeat.service – filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
Active: active (running) since 一 2018-10-22 10:42:39 CST; 6ms ago
Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Main PID: 2690 (filebeat)
CGroup: /system.slice/filebeat.service
└─2690 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
10月 22 10:42:39 se5 systemd[1]: Started filebeat.
10月 22 10:42:39 se5 systemd[1]: Starting filebeat…
注意，如果是以下提示，就只能等会再启动
[root@se5 ~]# systemctl status filebeat
● filebeat.service – filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since 一 2018-10-22 10:41:05 CST; 1min 31s ago
Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Process: 2651 ExecStart=/usr/bin/filebeat -c /etc/filebeat/filebeat.yml (code=exited, status=1/FAILURE)
Main PID: 2651 (code=exited, status=1/FAILURE)
10月 22 10:41:05 se5 systemd[1]: Unit filebeat.service entered failed state.
10月 22 10:41:05 se5 systemd[1]: filebeat.service failed.
10月 22 10:41:05 se5 systemd[1]: filebeat.service holdoff time over, scheduling restart.
10月 22 10:41:05 se5 systemd[1]: start request repeated too quickly for filebeat.service
10月 22 10:41:05 se5 systemd[1]: Failed to start filebeat.
10月 22 10:41:05 se5 systemd[1]: Unit filebeat.service entered failed state.
10月 22 10:41:05 se5 systemd[1]: filebeat.service failed.
到logstash主机上面去修改配置文件，然后启动服务
[root@logstash ~]# cat /etc/logstash/filebeat.conf
input{
stdin{ codec => “json” }
beats{
port => 5044
}
file {
path => [ “/tmp/a.log”, “/var/tmp/b.log” ]
sincedb_path => “/dev/null”
start_position => “beginning”
type => “testlog”
}
tcp {
host => “0.0.0.0”
port => “8888”
type => “tcplog”
}
udp {
host => “0.0.0.0”
port => “9999”
type => “udplog”
}
syslog {
port => “514”
type => “syslog”
}
}
filter{
if [type] == “apachelog”{
grok{
match => [“message”, “%{COMBINEDAPACHELOG}”]
}}
}
output{
stdout{ codec => “rubydebug” }
if [type] == “filelog”{
elasticsearch {
hosts => [“192.168.1.61:9200”, “192.168.1.62:9200”]
index => “filelog”
flush_size => 2000
idle_flush_time => 10
}}
}
启动服务
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/filebeat.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:07:23 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”curl/7.29.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:44:24.922Z”,
“path” => “/tmp/a.log”,
“host” => “logstash”,
“type” => “testlog”
}
{
“message” => “b2”,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:44:26.320Z”,
“path” => “/var/tmp/b.log”,
“host” => “logstash”,
“type” => “testlog”
}
重启启个终端，确认5044端口已经启动成功
[root@logstash ~]# netstat -antup | grep 5044
tcp6 0 0 :::5044 :::* LISTEN 3177/java
测试，用curl和真机上的谷歌、火狐浏览器都试一下
[root@room9pc52 ~]# curl 192.168.1.65
回到原来的logstash主机，可以看到数据都过来了，我就随便贴几个
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:48:26 +0800] \”GET /favicon.ico HTTP/1.1\” 404 209 \”http://192.168.1.65/\” \”Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:58:02.568Z”,
“fields” => nil,
“beat” => {
“hostname” => “se5”,
“name” => “se5”
},
“offset” => 2265,
“input_type” => “log”,
“count” => 1,
“source” => “/var/log/httpd/access_log”,
“type” => “apachelog”,
“host” => “se5”,
“tags” => [
[0] “beats_input_codec_plain_applied”
],
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “22/Oct/2018:10:48:26 +0800”,
“verb” => “GET”,
“request” => “/favicon.ico”,
“httpversion” => “1.1”,
“response” => “404”,
“bytes” => “209”,
“referrer” => “\”http://192.168.1.65/\””,
“agent” => “\”Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\””
}
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:48:47 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”curl/7.29.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:58:02.568Z”,
“beat” => {
“hostname” => “se5”,
“name” => “se5”
},
“type” => “apachelog”,
“source” => “/var/log/httpd/access_log”,
“offset” => 2479,
“input_type” => “log”,
“count” => 1,
“fields” => nil,
“host” => “se5”,
“tags” => [
[0] “beats_input_codec_plain_applied”
],
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “22/Oct/2018:10:48:47 +0800”,
“verb” => “GET”,
“request” => “/”,
“httpversion” => “1.1”,
“response” => “403”,
“bytes” => “4897”,
“referrer” => “\”-\””,
“agent” => “\”curl/7.29.0\””
}
{
“message” => “192.168.1.254 – – [22/Oct/2018:10:59:24 +0800] \”GET / HTTP/1.1\” 403 4897 \”-\” \”Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0\””,
“@version” => “1”,
“@timestamp” => “2018-10-22T02:59:27.572Z”,
“count” => 1,
“source” => “/var/log/httpd/access_log”,
“type” => “apachelog”,
“fields” => nil,
“beat” => {
“hostname” => “se5”,
“name” => “se5”
},
“offset” => 2570,
“input_type” => “log”,
“host” => “se5”,
“tags” => [
[0] “beats_input_codec_plain_applied”
],
“clientip” => “192.168.1.254”,
“ident” => “-“,
“auth” => “-“,
“timestamp” => “22/Oct/2018:10:59:24 +0800”,
“verb” => “GET”,
“request” => “/”,
“httpversion” => “1.1”,
“response” => “403”,
“bytes” => “4897”,
“referrer” => “\”-\””,
“agent” => “\”Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0\””
}
拓展以下，把刚才说的主是elasticsearch加上注释取消掉，然后elasticsearch就有数据了，去浏览器访问elasticsearch访问确认以下就可以看到
elasticsearch: //加上注释
hosts: [“localhost:9200”] //加上注释
实验到此结束