如何在Logstash过滤器中删除所有值为NULL的字段
郑安晏
问题内容：
问题答案：
我正在读取带有logstash的csv格式的检查点日志文件，并且某些字段的值为空。
我想删除所有值为空的字段。
我无法确切预测哪些字段（键）将具有空值，因为我在csv文件中有150列，而且我不想检查它们中的每一个。
是否有可能在logstash中进行动态过滤，以删除具有空值的所有字段？
我的logstash配置文件如下所示：
input {
stdin { tags => "checkpoint" }
file {
type => "file-input"
path => "D:\Browser Downloads\logstash\logstash-1.4.2\bin\checkpoint.csv"
sincedb_path => "D:\Browser Downloads\logstash\logstash-1.4.2\bin\sincedb-access2"
start_position => "beginning"
tags => ["checkpoint","offline"]
}
}
filter {
if "checkpoint" in [tags] {
csv {
columns => ["num","date","time","orig","type","action","alert","i/f_name","i/f_dir","product","Internal_CA:","serial_num:","dn:","sys_message:","inzone","outzone","rule","rule_uid","rule_name","service_id","src","dst","proto","service","s_port","dynamic object","change type","message_info","StormAgentName","StormAgentAction","TCP packet out of state","tcp_flags","xlatesrc","xlatedst","NAT_rulenum","NAT_addtnl_rulenum","xlatedport","xlatesport","fw_message","ICMP","ICMP Type","ICMP Code","DCE-RPC Interface UUID","rpc_prog","log_sys_message","scheme:","Validation log:","Reason:","Serial num:","Instruction:","fw_subproduct","vpn_feature_name","srckeyid","dstkeyid","user","methods:","peer gateway","IKE:","CookieI","CookieR","msgid","IKE notification:","Certificate DN:","IKE IDs:","partner","community","Session:","L2TP:","PPP:","MAC:","OM:","om_method:","assigned_IP:","machine:","reject_category","message:","VPN internal source IP","start_time","connection_uid","encryption failure:","vpn_user","Log ID","message","old IP","old port","new IP","new port","elapsed","connectivity_state","ctrl_category","description","description ","severity","auth_status","identity_src","snid","src_user_name","endpoint_ip","src_machine_name","src_user_group","src_machine_group","auth_method","identity_type","Authentication trial","roles","dst_user_name","dst_machine_name","spi","encryption fail reason:","information","error_description","domain_name","termination_reason","duration"]
# remove_field => [ any fields with null value] how to do it please
separator => "|"
}
# drop csv header
if [num] == "num" and [date] == "date" and [time] == "time" and [orig] == "orig" {
drop { }
}
}
}
}
output {
stdout {
codec => rubydebug
}
file {
path => "output.txt"
}
在此附上一些日志示例：
num|date|time|orig|type|action|alert|i/f_name|i/f_dir|product|Internal_CA:|serial_num:|dn:|sys_message:|inzone|outzone|rule|rule_uid|rule_name|service_id|src|dst|proto|service|s_port|dynamic object|change type|message_info|StormAgentName|StormAgentAction|TCP packet out of state|tcp_flags|xlatesrc|xlatedst|NAT_rulenum|NAT_addtnl_rulenum|xlatedport|xlatesport|fw_message|ICMP|ICMP Type|ICMP Code|DCE-RPC Interface UUID|rpc_prog|log_sys_message|scheme:|Validation log:|Reason:|Serial num:|Instruction:|fw_subproduct|vpn_feature_name|srckeyid|dstkeyid|user|methods:|peer gateway|IKE:|CookieI|CookieR|msgid|IKE notification:|Certificate DN:|IKE IDs:|partner|community|Session:|L2TP:|PPP:|MAC:|OM:|om_method:|assigned_IP:|machine:|reject_category|message:|VPN internal source IP|start_time|connection_uid|encryption failure:|vpn_user|Log ID|message|old IP|old port|new IP|new port|elapsed|connectivity_state|ctrl_category|description|description |severity|auth_status|identity_src|snid|src_user_name|endpoint_ip|src_machine_name|src_user_group|src_machine_group|auth_method|identity_type|Authentication trial|roles|dst_user_name|dst_machine_name|spi|encryption fail reason:|information|error_description|domain_name|termination_reason|duration
0|8Jun2012|16:33:35|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|started|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
1|8Jun2012|16:36:34|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|started|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
2|8Jun2012|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|86232|CN=fw-KO,O=sc-KO.KO.dc.obn8cx|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
3|8Jun2012|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Initiated certificate is now valid|86232|CN=fw-KO,O=sc-KO.KO.dc.obn8cx|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
4|8Jun2012|16:55:44|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Issued empty CRL 1|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
20|8Jun2012|16:58:28|10.0.0.1|log|accept||eth1|inbound|VPN-1 & FireWall-1|||||Internal|External|1|{2A42C8CD-148D-4809-A480-3171108AD6C7}||domain-udp|192.168.100.1|198.32.64.12|udp|53|1036|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
问题答案：
Ruby 过滤器可以满足您的要求。
input {
stdin {
}
}
filter {
csv {
columns => ["num","date","time","orig","type","action","alert","i/f_name","i/f_dir","product","Internal_CA:","serial_num:","dn:","sys_message:","inzone","outzone","rule","rule_uid","rule_name","service_id","src","dst","proto","service","s_port","dynamic object","change type","message_info","StormAgentName","StormAgentAction","TCP packet out of state","tcp_flags","xlatesrc","xlatedst","NAT_rulenum","NAT_addtnl_rulenum","xlatedport","xlatesport","fw_message","ICMP","ICMP Type","ICMP Code","DCE-RPC Interface UUID","rpc_prog","log_sys_message","scheme:","Validation log:","Reason:","Serial num:","Instruction:","fw_subproduct","vpn_feature_name","srckeyid","dstkeyid","user","methods:","peer gateway","IKE:","CookieI","CookieR","msgid","IKE notification:","Certificate DN:","IKE IDs:","partner","community","Session:","L2TP:","PPP:","MAC:","OM:","om_method:","assigned_IP:","machine:","reject_category","message:","VPN internal source IP","start_time","connection_uid","encryption failure:","vpn_user","Log ID","message","old IP","old port","new IP","new port","elapsed","connectivity_state","ctrl_category","description","description ","severity","auth_status","identity_src","snid","src_user_name","endpoint_ip","src_machine_name","src_user_group","src_machine_group","auth_method","identity_type","Authentication trial","roles","dst_user_name","dst_machine_name","spi","encryption fail reason:","information","error_description","domain_name","termination_reason","duration"]
separator => "|"
}
ruby {
code => "
hash = event.to_hash
hash.each do |k,v|
if v == nil
event.remove(k)
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
您可以使用ruby插件来过滤所有带有nil值的字段（Ruby中为null）
更新：
这是我的环境：Windows Server 2008和Logstash 1.4.1。您的日志样本对我有用！我已经更新了配置，输入和输出。
输入项
2|8Jun2012|16:52:39|10.0.0.1|log|keyinst||daemon|inbound|VPN-1 & FireWall-1|Certificate initialized|86232|CN=fw-KO,O=sc-KO.KO.dc.obn8cx|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
输出：
{
"@version" => "1",
"@timestamp" => "2015-03-12T00:30:34.123Z",
"host" => "BENLIM",
"num" => "2",
"date" => "8Jun2012",
"time" => "16:52:39",
"orig" => "10.0.0.1",
"type" => "log",
"action" => "keyinst",
"i/f_name" => "daemon",
"i/f_dir" => "inbound",
"product" => "VPN-1 & FireWall-1",
"Internal_CA:" => "Certificate initialized",
"serial_num:" => "86232",
"dn:" => "CN=fw-KO,O=sc-KO.KO.dc.obn8cx"
}
类似资料：
-
问题内容： 您能告诉我如何从json对象中删除所有null和空字符串值吗？删除密钥时出现错误。 到目前为止，这是我所拥有的，但是不能正常工作： 问题答案： 您实际上是要删除。您需要使用数组访问符号： 但是，这也将删除value等于0的地方，因为您没有使用严格的比较。使用来代替： 但是，这只会使对象浅走。要深入地做，可以使用递归： 请注意，如果您愿意使用lodash / underscore.js之
-
我有以下矩阵： 我只想删除id为3的行。请注意，当我使用时，它仍然保留，因为id本身是唯一的。
-
问题内容： 我必须从Spring Security堆栈中排除一个默认过滤器。因此，所有过滤器都应照常工作。看来我找到了解决方法，请创建自定义FilterChainProxy： 如您所见，它具有获取过滤器列表的构造函数，因此我将能够根据需要从链中删除一个过滤器，其余所有过滤器将照常工作。但是我不能在这样的构造函数的安全配置中制作bean。如果我用 当然，使用默认构造函数构建对象。好的，我尝试用一
-
我想清除所有筛选规则，但保留筛选本身。 有没有直接快速的方法去做呢？ 我找到的代码是： 它获取筛选器对象，我对筛选器#的选项数量是有限的。 注意：删除此筛选器，但我需要保留它。
-
问题内容： 我有一个很大的数据集，我想删除包含值的列并返回一个新的数据框。我怎样才能做到这一点？ 以下内容仅删除包含的单个列或行。 例如 在上述情况下，它将丢弃整个列，因为其值之一为空。 问题答案： 这是删除所有具有NULL值的所有列的一种可能的方法，以获取每列NULL值计数代码的源代码。 之前： 后： 希望这可以帮助！
-
问题内容： 基本上，我正在做一些数据分析。我以numpy.ndarray的形式读取数据集，但缺少了某些值（要么只是不在那里，要么就是被写为“ ”的字符串）。 我想清除包含这样任何条目的所有行。我该如何用一个numpy的ndarray？ 问题答案： 并将其重新分配给。 说明：返回一个相似的阵列，其中，在其他地方。降低了阵列与逻辑对整个行，操作反相并从原始数组只选择行，其具有括号内。