pylogsparser 解析apache的日志

斜淳

2023-12-01

安装依赖模块

pytz-2014.2(https://pypi.python.org/pypi/pytz/2014.2)

lxml(lxml-3.3.5.win32-py2.7.exe)

设置环境变量

NORMALIZERS_PATH=E:\workspaces\pylogsparser\normalizers\

找一个apache的access_log日志：

192.168.94.21 - - [24/Apr/2014:16:16:36 +0800] "GET / HTTP/1.1" 200 44
192.168.94.21 - - [24/Apr/2014:16:16:36 +0800] "GET /favicon.ico HTTP/1.1" 404 209
192.168.94.60 - - [24/Apr/2014:16:22:28 +0800] "GET / HTTP/1.1" 200 44
192.168.94.60 - - [24/Apr/2014:16:22:28 +0800] "GET /favicon.ico HTTP/1.1" 404 209
192.168.94.60 - - [24/Apr/2014:16:22:35 +0800] "GET / HTTP/1.1" 200 44
192.168.94.60 - - [24/Apr/2014:16:22:35 +0800] "GET /favicon.ico HTTP/1.1" 404 209
192.168.94.21 - - [24/Apr/2014:16:23:01 +0800] "GET / HTTP/1.1" 200 44
192.168.94.21 - - [24/Apr/2014:16:23:02 +0800] "GET /favicon.ico HTTP/1.1" 404 209
192.168.94.22 - - [24/Apr/2014:16:26:15 +0800] "GET / HTTP/1.1" 200 44
192.168.94.22 - - [24/Apr/2014:16:26:15 +0800] "GET /favicon.ico HTTP/1.1" 404 209
192.168.94.22 - - [24/Apr/2014:16:26:28 +0800] "GET / HTTP/1.1" 200 44
192.168.94.22 - - [24/Apr/2014:16:26:28 +0800] "GET /favicon.ico HTTP/1.1" 404 209
192.168.94.22 - - [24/Apr/2014:16:26:39 +0800] "GET / HTTP/1.1" 200 44

测试代码：

from logsparser.lognormalizer import LogNormalizer as LN
import os
 
normalizer_path = os.environ['NORMALIZERS_PATH']
normalizer = LN(normalizer_path)
access_log = open('E:/workspaces/pylogsparser/logs/access_log', 'r')
 
for log in access_log:
    l = {'body' : log } 
    lg = normalizer.normalize(l)
    print lg

输出结果：

{'body': '192.168.94.21 - - [24/Apr/2014:16:16:36 +0800] "GET / HTTP/1.1" 200 44\n', 'status': '200', 'source_ip': '192.168.94.21', 'len': '44', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 16, 36), 'url_path': '/', 'source_logname': '-', 'request': 'GET / HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.21 - - [24/Apr/2014:16:16:36 +0800] "GET /favicon.ico HTTP/1.1" 404 209\n', 'status': '404', 'source_ip': '192.168.94.21', 'len': '209', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 16, 36), 'url_path': '/favicon.ico', 'source_logname': '-', 'request': 'GET /favicon.ico HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.60 - - [24/Apr/2014:16:22:28 +0800] "GET / HTTP/1.1" 200 44\n', 'status': '200', 'source_ip': '192.168.94.60', 'len': '44', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 22, 28), 'url_path': '/', 'source_logname': '-', 'request': 'GET / HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.60 - - [24/Apr/2014:16:22:28 +0800] "GET /favicon.ico HTTP/1.1" 404 209\n', 'status': '404', 'source_ip': '192.168.94.60', 'len': '209', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 22, 28), 'url_path': '/favicon.ico', 'source_logname': '-', 'request': 'GET /favicon.ico HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.60 - - [24/Apr/2014:16:22:35 +0800] "GET / HTTP/1.1" 200 44\n', 'status': '200', 'source_ip': '192.168.94.60', 'len': '44', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 22, 35), 'url_path': '/', 'source_logname': '-', 'request': 'GET / HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.60 - - [24/Apr/2014:16:22:35 +0800] "GET /favicon.ico HTTP/1.1" 404 209\n', 'status': '404', 'source_ip': '192.168.94.60', 'len': '209', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 22, 35), 'url_path': '/favicon.ico', 'source_logname': '-', 'request': 'GET /favicon.ico HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.21 - - [24/Apr/2014:16:23:01 +0800] "GET / HTTP/1.1" 200 44\n', 'status': '200', 'source_ip': '192.168.94.21', 'len': '44', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 23, 1), 'url_path': '/', 'source_logname': '-', 'request': 'GET / HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.21 - - [24/Apr/2014:16:23:02 +0800] "GET /favicon.ico HTTP/1.1" 404 209\n', 'status': '404', 'source_ip': '192.168.94.21', 'len': '209', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 23, 2), 'url_path': '/favicon.ico', 'source_logname': '-', 'request': 'GET /favicon.ico HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.22 - - [24/Apr/2014:16:26:15 +0800] "GET / HTTP/1.1" 200 44\n', 'status': '200', 'source_ip': '192.168.94.22', 'len': '44', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 26, 15), 'url_path': '/', 'source_logname': '-', 'request': 'GET / HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.22 - - [24/Apr/2014:16:26:15 +0800] "GET /favicon.ico HTTP/1.1" 404 209\n', 'status': '404', 'source_ip': '192.168.94.22', 'len': '209', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 26, 15), 'url_path': '/favicon.ico', 'source_logname': '-', 'request': 'GET /favicon.ico HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.22 - - [24/Apr/2014:16:26:28 +0800] "GET / HTTP/1.1" 200 44\n', 'status': '200', 'source_ip': '192.168.94.22', 'len': '44', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 26, 28), 'url_path': '/', 'source_logname': '-', 'request': 'GET / HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.22 - - [24/Apr/2014:16:26:28 +0800] "GET /favicon.ico HTTP/1.1" 404 209\n', 'status': '404', 'source_ip': '192.168.94.22', 'len': '209', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 26, 28), 'url_path': '/favicon.ico', 'source_logname': '-', 'request': 'GET /favicon.ico HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}
{'body': '192.168.94.22 - - [24/Apr/2014:16:26:39 +0800] "GET / HTTP/1.1" 200 44\n', 'status': '200', 'source_ip': '192.168.94.22', 'len': '44', 'user': '-', 'date': datetime.datetime(2014, 4, 24, 16, 26, 39), 'url_path': '/', 'source_logname': '-', 'request': 'GET / HTTP/1.1', 'program': 'apache', 'taxonomy': 'web server', 'method': 'GET'}

pylogsparser 解析apache的日志

相关阅读

相关文章

相关问答

相关文档