5.1、更改nginx日志输出格式
5.1.1、定义日志格式
cat /etc/nginx/nginx.conf
log_format main '$remote_addr - $remote_user [$time_local]"$request" '
'$status$body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
5.1.2、加载日志格式到对应域名配置中
cat /etc/nginx/conf.d/vhost/api.mingongge.com.cn.conf
server {
listen 80;
server_name newtest-msp-api.mingongge.com.cn;
access_log /var/log/nginx/api.mingongge.com.cn.log main;
}5.1.3、reload生效
nginx -s reload
5.1.4、清空原输出文件，并查看输出的日志格式
:> /var/log/nginx/api.mingongge.com.cn.log
tailf /var/log/nginx/api.mingongger.com.cn.log
1xx.2xx.72.175 - - [18/Mar/2019:13:51:17 +0800] "GET/user/fund/113 HTTP/1.1" 200 673 "-" "Mozilla/5.0 (WindowsNT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) sun/1.5.6 Chrome/69.0.3497.106Electron/4.0.3 Safari/537.36" "-"
5.3.1、安装filebeat
cd /opt/ && wget http://download.mingongge.com.cn/download/software/filebeat-6.3.2-x86_64.rpm
yum localinstall filebeat-6.3.2-x86_64.rpm -y
5.3.2、编辑配置文件
cat /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
enabled: true
paths:
- /var/log/nginx/api.mingongge.com.cn.log#收集日志路径
fields:
log_topics: nginxlog #kafka中topic名称
json.keys_under_root: true
json.overwrite_keys: true
output.kafka:
enabled: true
hosts:["192.168.0.53:9091"] #kafka地址
topic:'%{[fields][log_topics]}' #kafka中topic名称
partition.round_robin:
reachable_only: false
compression: gzip
max_message_bytes: 1000000
required_acks: 15.3.3、启动filebeat& 开机启动
systemctl start filebeat
systemctl enable filebeat
5.4、配置logstash
5.4.1 编辑配置
cat /usr/local/logstash/config/nginx.conf
input {
kafka {
type =>"nginxlog"
topics =>["nginxlog"]
bootstrap_servers=> ["192.168.0.53:9091"]
group_id =>"nginxlog"
auto_offset_reset=> latest
codec =>"json"
}
}
filter {
if [type] == "nginxlog"{
grok {
match => {"message" => "%{COMBINEDAPACHELOG}" }
remove_field =>"message"
}
date {
match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
geoip {
source =>"clientip"
target =>"geoip"
database =>"/usr/local/logstash/config/GeoLite2-City.mmdb"
add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}" ] #添加字段coordinates，值为经度
add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}" ] #添加字段coordinates，值为纬度
}
mutate {
convert => ["[geoip][coordinates]", "float"]
}
useragent {
source =>"agent"
target =>"userAgent"
}
}
}
output {
if [type] == 'nginxlog' {
elasticsearch {
hosts =>["http://192.168.0.48:9200"]
index =>"logstash-nginxlog-%{+YYYY.MM.dd}"
}
stdout {codec =>rubydebug}
}
}5.4.2、使用配置文件启动logstash服务，观察输出
/usr/local/logstash/bin/logstash -f nginx.conf
{
"httpversion"=> "1.1",
"verb" =>"GET",
"auth"=> "-",
"@timestamp"=> 2019-03-18T06:41:27.000Z,
"type"=> "nginxlog",
"json"=> {},
"source"=> "/var/log/nginx/newtest-msp-api.mingongge.com.cn-80.log",
"fields" =>{
"log_topics"=> "nginxlog"
},
"response"=> "200",
"offset"=> 957434,
"host"=> {
"name" =>"test-kafka-web"
},
"beat"=> {
"hostname"=> "test-kafka-web",
"version"=> "6.3.2",
"name"=> "test-kafka-web"
},
"bytes"=> "673",
"request"=> "/user/fund/113",
"timestamp"=> "18/Mar/2019:14:41:27 +0800",
"referrer"=> "\"-\"",
"userAgent"=> {
"os"=> "Windows",
"major" => "4",
"patch"=> "3",
"build"=> "",
"minor"=> "0",
"os_name"=> "Windows",
"device"=> "Other",
"name"=> "Electron"
},
"geoip"=> {
"ip" => "1xx.2xx.72.175",
"country_name" => "China",
"coordinates" => [
[0] 121.4012,
[1] 31.0449
],
"region_name" => "Shanghai",
"location" => {
"lat"=> 31.0449,
"lon"=> 121.4012
},
"continent_code" => "AS",
"timezone" => "Asia/Shanghai",
"longitude" => 121.4012,
"city_name" => "Shanghai",
"country_code2" => "CN",
"region_code" => "SH",
"latitude" => 31.0449,
"country_code3" => "CN"
},
"@version"=> "1",
"clientip"=> "1xx.2xx.72.175",
"ident"=> "-",
"agent"=> "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36(KHTML, like Gecko) sun/1.5.6 Chrome/69.0.3497.106 Electron/4.0.3Safari/537.36\""
}5.4.3、后台启动logstash
确认出现以上输出后，将logstash分离出当前shell,并放在后台运行
nohup /usr/local/logstash/bin/logstash -f nginx.conf &>/dev/null &
5.5、kibana配置
5.5.1、修改kibana配置
/usr/local/kibana-6.5.4-linux-x86_64/config/kibana.yml #增加高德地图
tilemap.url:'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'5.5.2、创建Index Pattern: logstash-nginxlog
5.5.3、IP访问TOP5
选择柱形图 ---> 添加X轴，以geoip.ip为order by字段
5.5.4 、PV
选择metric ---> 默认统计总日志条数,即为PV数
5.5.5、全球访问地图
选择map ---> Field选择geoip.location ---> 选择添加高德地图
5.5.6、实时流量
选择线条图
5.5.7、操作系统
选择饼图
5.5.8、登陆次数
过滤login关键字,并做count统计
5.5.9、访问地区
5.5.10、Dashboard展示
- IP访问Top5:每日客户端IP请求数最多的前五个(可分析出攻击者IP)
- PV:每日页面访问量
- 全球访问图:直观的展示用户来自哪个国家哪个地区
- 实时流量:根据@timestamp字段来展示单位时间的请求数(可根据异常峰值判断是否遭遇攻击)
- 操作系统:展示客户端所用设备所占比重
- 登陆次数:通过过滤request中login的访问记录,粗略估算出进行过登陆的次数
- 访问地区:展示访问量最多的国家或地区
- 需展示其他指标，可进行自由发挥
