syslog-ng文档阅读

https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/chapter-quickstart.html
5.3 内联配置对象，只能使用一次
如

source s_local {
    system();
    internal();
};
destination d_local {
    file("/var/log/messages");
};
log {
    source(s_local);
    destination(d_local);
};
#两者等价
log {
    source {
        system();
        internal();
    };
    destination {
        file("/var/log/messages");
    };
};

5.4在配置块中使用channels来嵌入语句
e.g. 使用channels对source进行预处理

source s_apache {
    channel {
        source { file("/var/log/apache/error.log"); };
        parser(p_apache_parser);
    };
};
log { source(s_apache); ... };

e.g. 使用junction

parser demo-parser() {
    channel {
        junction {
        channel { ... };
        channel { ... };
        };
    };
};

5.6 定义全局变量
@define name “value”
e.g.

@define mypath "/opt/myapp/logs"
    source s_myapp_1 { file("`mypath`/access.log" follow-freq(1)); };
    source s_myapp_2 { file("`mypath`/error.log" follow-freq(1)); };
    source s_myapp_3 { file("`mypath`/debug.log" follow-freq(1)); };
#The syslog-ng OSE application will interpret this as:
@define mypath "/opt/myapp/logs"
    source s_myapp_1 { file("/opt/myapp/logs/access.log" follow-freq(1)); };
    source s_myapp_2 { file("/opt/myapp/logs/error.log" follow-freq(1)); };
    source s_myapp_3 { file("/opt/myapp/logs/debug.log" follow-freq(1)); };

若想包括其他文件，在文件上方

@include "<filename>"

source

定义

source <identifier> { source-driver(params); source-driver(params); ... };

The following source statement collects the following log messages:
■ internal(): Messages generated by syslog-ng.
■ network(transport("udp")): Messages arriving to the 514/UDP port of any interface of the host.
■ unix-dgram("/dev/log");: Messages arriving to the /dev/log socket.
source s_demo {
    internal();
    network(transport("udp"));
    unix-dgram("/dev/log"); };

6.3.2 file() source options
file() filename with path
encoding() 编码
flags() 具体说明日志解析的选项
follow-freq() 指明源应该被周期性检查，如果大于0，表示每一次 follow-freq()间隔过去，文件是否改变
keep-timestamp() 是否要有时间印章，默认yes
log-fetch-limit() 一个单轮查询循环中去读的最大消息数
当这个值过高的时候，destination序列可能在flow-control（流量控制）停止读入之前，被填满
log-iw-size() 初始窗口的大小，流量控制中使用，类似缓存，Make sure that log-iw-size() is larger than the value of
log-msg-size() 最大传入消息长度，默认8192
log-prefix() (DEPRECATED） string类型，可以在所有log messages 前加前缀
multi-line-garbage() 处理多线消息，将消息中不需要的部分去除。
If the multi-line-garbage() option is set, syslog-ng OSE ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix().

destination

A destination is where a log message is sent if the filtering rules match.、

destination <identifier> {
    destination-driver(params); destination-driver(params); ... };

注意，不要重复定义相同destination ，不要在不同log中使用相同destination，只有当被用于log时，destination和source才会初始化。
以文件为例。
file(filename options());
create-dirs() 修改文件路径属性，默认为Use the global settings
dir-owner()
dir-perm() 创建的路径的权限掩码，若要保存原有路径的属性，用该option不给出具体属性，若在新建路径时未给出具体permission mask ，则默认权限由parent process决定
flags()
no-multi-line 禁止换行
syslog-protocol 使用RFC5424协议而不需 frame header
threaded 可以开启多线程、
flush-lines() 说明在一次批处理中发给destination 多少行，该属性增加，即会以一次批处理中发出更多消息的形式增多吞吐量，也多增大消息延迟，确保性能最优，应该使其小于 log-iw-size() 选项设定的窗口大小。
flush-timeout() (DEPRECATED)不建议用
frac-digits()
fsync() 每次对信息写入都进行目的文件的同步，会降低性能
group() 设定创建文件的group
local-time-zone() 时区
log-fifo-size() 消息队列能存储的消息数量
overwrite-if-older() 不保存全部日志，设定时间覆盖
suppress() 多个相同消息连续发送，设定此option可以实现只发过来一次
template()指明日志格式，但是注意对于network destinations可能不合适，需确保接收端可以应对自定义格式。
template-escape()
ts-format()修改时间戳格式。

destination d_file {
    file("/var/log/${YEAR}.${MONTH}.${DAY}/messages"
        template("${HOUR}:${MIN}:${SEC} ${TZ} ${HOST} [${LEVEL}] ${MSG} ${MSG}\n")
        template-escape(no));
};

使用py或者java写自己的destination
https://syslog-ng.gitbooks.io/getting-started/content/chapters/chapter_4/section_1.html
注意编写自己的des类之后，需要添加到路径之中例如
export PYTHONPATH=$PYTHONPATH:/etc/syslog-ng/mydes.py

log paths and filters

embedded log statements
使用上阶段处理过的数据结果作为数据

log {source(s1);filter(f1);destination(d1);
    log{filter(f2);destination(d2);};
};

Junctions and channels

Log path flags
catchall 忽略消息来源，匹配信息时候只考虑filters
fallback 处理那些未被 ‘non-fallback’ 处理的信息
flow-control 当destinations在要求的速度未能处理完消息，sources将停止发送，如果设为disable，那么destination队列满了的时候会丢失信息，若设置为enabled，那么只有queues/window sizes大小设置不正确的时候才会丢失信息。
final 他后面的 log statements 失效。

8.2 Managing incoming and outgoing messages with flow-control

当在conf文件中定义的所监控的sources中产生消息，这个消息以fifo的形式存储在syslog-ng 的buffer里，从这个buffer中，syslog-ng向destination发消息，log-fetch-limit() 描述了一次查询循环中从一个单独的source中读取的消息数，每个destination有自己的输出buffer，考虑到其不能瞬间接受所有消息，这个buffer是很有必要的log-fifo-size() 这个设定了buffer 的大小，这个缓冲器必须大于source的log-fetch-limit()，确保每条消息都能读入buffer中，如果destination接收多个源所发的消息，那么buffer必须足够大来存储每一个source源发来的消息。
TCP and unix-stream sources可能从多个不同的连接来获得message，对于这类源，syslog_ng从多个连接读取消息，因此
log-fetch-limit() 对每一个连接单独起作用。
实际上这里是定义了一个控制窗，这个控制syslog-ng能从源报错多少条消息，每一条从source读取的message都会使这个window size减一，每次成功发送到输出buffer都会使其加一，如果为0则停止接收。
原始的窗口大小是默认的1000，the log-fifo-size()这个必须比其大。如果一个source从多个连接中获取消息，这些链接共享同一个control window
如果一个destination从多个source接收msg，那么要求这些对这些sources

log-fifo-size() >sum(log-iw-size())

总结一下对于msg流量的控制
1.source
log-iw-size() 定义窗口大小，默认1000，大于log-fetch-limit()

log-iw-size()
Type: number
Default: 1000
Description: The size of the initial window, this value is used during flow control. Make sure that
log-iw-size() is larger than the value of log-fetch-limit()

log-fetch-limit() 一次消息查询轮从一个source中读取的msg数目

log-fetch-limit()
Type: number
Default: 10
Description: The maximum number of messages fetched from a source during a single poll loop. The destination
queues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

2.destination
log-fifo-size() 缓冲区大小，大于所接受所有source的sum(log-iw-size())

log-fifo-size()
Type: number
Default: Use global setting.
Description: The number of messages that the output queue can store.

flush-lines() 一次发给该destination的消息行数，确保小于log-iw-size()。syslog-ng当发送了flush-lines()条msg，或者队列为空的时候刷新msg

flush-lines()
Type: number
Default: Use global setting.
Description: Specifies how many lines are flushed to a destination at a time. The syslog-ng OSE application
waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number
increases throughput as more messages are sent in a single batch, but also increases message latency.
The syslog-ng OSE application flushes the messages if it has sent flush-lines() number of messages, or
the queue became empty. If you stop or reload syslog-ng OSE or in case of network sources, the connection
with the client is closed, syslog-ng OSE automatically sends the unsent messages to the destination.
For optimal performance when sending messages to an syslog-ng OSE server, make sure that the
flush-lines() is smaller than the window size set using the log-iw-size() option in the source of your
server.

当然，在log中别忘了设置flow-control

8.3 Filters

过滤信息，为true时才会进入destination，可以使用布尔运算符

filter demo_filter1 { host("example1"); };
filter demo_filter2 { host("example2"); };
log {
    source(s1); source(s2);
    filter(demo_filter1); filter(demo_filter2);
    destination(d1); destination(d2); };

To select the messages that come from either host example1 or example2, use a single filter expression:

filter demo_filter { host("example1") or host("example2"); };
log {
    source(s1); source(s2);
    filter(demo_filter);
    destination(d1); destination(d2); };

12.Parsing

filters和default macros 作用于 msg的headers and metainformation，但是无法解析其内容，这里我们就需要Parser了，它可以分割mscg成name-value pairs而且这些name可以用作用户定义的宏，从而可以在后面的filter中。

12.5 The JSON parser

syslog-ng可以将传入msg转化为 name-value pairs，可以将json中的key作为宏使用，通过“.”访问下一层，例如
{“KEY1”: {“KEY2”: “VALUE”}}, use the
${KEY1.KEY2} macro.
当然，如果key名与syslog-ng中宏重复，则会覆盖该宏，所以，尽量使用前缀属性。

parser parser_name {
    json-parser(
        marker()
        prefix()
    );
}
parser p_json {
    json-parser (prefix(".json.")); 
};

json_options
marker
防止混合msgs

Synopsis: marker()
Description: Use a marker in case of mixed log messages, to identify JSON encoded messages for the parser.
Some logging implementations require a marker to be set before the JSON payload. The JSON parser is able
to find these markers and parse the message only if it is present.

prefix()

Synopsis: prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing.
For example:
■ To insert the my-parsed-data. prefix, use the prefix(.my-parsed-data.) option.
■ If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed
data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
■ To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example,
${my-parsed-data.name} .

给出自己的配置文件

@version: 3.7
@include "scl.conf"
@include "`scl-root`/system/tty10.conf"

# Syslog-ng configuration file, compatible with default Debian syslogd
# installation.

# First, set some global options.
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);
      owner("root"); group("adm"); perm(0640); stats_freq(0);
      bad_hostname("^gconfd$");
};
parser p_json {json-parser (prefix(".json."));};
source filea{
        channel{
            source{
                file("/etc/syslog-ng/temp.txt" log-iw-size(1000) follow-freq(1));
                    };
                    parser(p_json);
        };
};

source fileb {
       file ("/etc/syslog-ng/temp.txt"  follow-freq(1));
};

destination d_testa { 
        file("/etc/syslog-ng/tempa.txt" create_dirs(yes) flush-lines(1) log-fifo-size(1000));  
        };
destination d_testb {
        python(
            class("mydes.TextDestination")
            on-error("fallback-to-string")
            value-pairs(scope(everything))
            );
};
log { source(filea); destination(d_testa); };
log { source(fileb); destination(d_testb); };
@include "/etc/syslog-ng/conf.d/*.conf"

python类使用

#coding=utf-8
import sys
import time
reload(sys)  
sys.setdefaultencoding('utf8')  
class LogDestination(object):

    def open(self):
        """Open a connection to the target service"""
        return True

    def close(self):
        """Close the connection to the target service"""
        pass

    def is_opened(self):
        """Check if the connection to the target is able to receive messages"""
        return True

    def init(self):
        """This method is called at initialization time"""
        return True

    def deinit(self):
        """This method is called at deinitialization time"""
        pass

    def send(self, msg):
        """Send a message to the target service

        It should return True to indicate success, False will suspend the
        destination for a period specified by the time-reopen() option."""
        pass
class TextDestination(LogDestination):
    def __init__(self):
        self.outfile = None

    def init(self,what):
        self.outfile = open('/etc/syslog-ng/python_parsed.txt', 'w+')
        self.outfile.write("initialized\n")
        self.outfile.write("\n")
        self.outfile.flush()
        return True

    def open(self):
        self.outfile.write("opened\n")
        self.outfile.flush()
        return True

    def close(self):
        self.outfile.write("closed\n")
        self.outfile.flush()
        return True

    def deinit(self):
        self.outfile.write("deinit\n")
        self.outfile.flush()
        self.outfile.close();
        return True

    def send(self, msg):
        self.outfile.write("Name Value Pairs are \n")
        self.json_rec={}
        for key,v in msg.items():
            self.json_rec[str(key)]=str(v)
        self.outfile.write(self.json_rec['MSG']+"\n");
        self.outfile.flush()
        return True