WebSphere MQ与水槽集成

Question

我正在尝试将 Websphere （IBM） MQ 与 flume 集成。我有几个来自MQ的xml文件

我正在AWS EC2实例上进行此集成，其中也安装了我的Hadoop。以下是我遵循的集成步骤。

1. 创建队列管理器：https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.con.doc/q015210_.htm

./crtmqm-q-dMY.DEFAULT.BTS.QUEUE-uSYSTEM.DEAD.LETTER.QUEUECSBTS.QUEUE.MANAGER

启动队列管理器：

./strmqmCSBTS.QUEUE.MANAGER

./runmqsc CSBTS.队列。经理

定义服务器通道

定义CHANNEL（CHAN2）CHLTYPE（SVRCONN）TRPTYPE（TCP）DESR（'服务器连接到Client_2'）

定义通道(chan 2)CHL type(CLNTCONN)Trp type(TCP)CONNAME(10 . 0 . 0 . 238)QMNAME(cs BTS。QUEUE.MANAGER) DESCR('客户端-到Server_2的连接')

ALTER QMGR CHLAUTH(禁用)

定义本地队列

定义QLOCAL (CSBTS。成交)

添加消息

./amqsput CSBTS.交易CSBTS。队列。经理

创建连接工厂

DEF CF（CF4） QMGR（CSBTS.队列。管理器） 传输（客户端） 主机名（10.0.0.238） 通道（CHAN2）

启动监听器

定义监听器(CSBTS_LIST) TRPTYPE (TCP)端口(1414)控制(QMGR)

显示监听器(CSBTS_LIST)

显示 LSSTATUS （CSBTS_LIST）

（重启qm）

运行水槽

/usr/bin/flume-ng agent-c/etc/flume-ng/conf-f/etc/flume-ng/conf/flume . conf-n tier 1

Flume通过linux用户“hduser”执行。

下面是我们遇到的身份验证问题。

20 Jul 2016 09:17:33,236 ERROR [PollableSourceRunner-JMSSource-source1] (org.apache.flume.source.PollableSourceRunner$PollingRunner.run:156)  - Unhandled exception, logging and sleeping for 5000ms
org.apache.flume.FlumeException: Source had error configuring or starting
        at org.apache.flume.source.AbstractPollableSource.process(AbstractPollableSource.java:48)
        at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.FlumeException: Could not create connection to broker
        at org.apache.flume.source.jms.JMSMessageConsumer.<init>(JMSMessageConsumer.java:76)
        at org.apache.flume.source.jms.JMSMessageConsumerFactory.create(JMSMessageConsumerFactory.java:32)
        at org.apache.flume.source.jms.JMSSource.createConsumer(JMSSource.java:317)
        at org.apache.flume.source.jms.JMSSource.doStart(JMSSource.java:299)
        at org.apache.flume.source.BasicSourceSemantics.start(BasicSourceSemantics.java:83)
        at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:74)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'CSBTS.QUEUE.MANAGER' with connection mode 'Client' and host name '10.0.0.238(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.
        at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:531)
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
        at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:413)
        at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8475)
        at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7814)
        at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:299)
        at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)
        at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6024)
        at com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6073)
        at org.apache.flume.source.jms.JMSMessageConsumer.<init>(JMSMessageConsumer.java:69)
        ... 13 more
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
        ... 21 more

在过去的 3 周里，我被困住了这个问题，如果有人面对过这个朋友，那么请帮助我们。

AMQERR01的错误日志。原木

----- cmqxrsrv.c : 2321 -------------------------------------------------------
07/21/2016 10:08:17 AM - Process(16169.50) User(mqm) Program(amqzlaa0)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ5540: Application 'flume.node.Application' did not supply a user ID and
password

EXPLANATION:
The queue manager is configured to require a user ID and password, but none was
supplied.
ACTION:
Ensure that the application provides a valid user ID and password, or change
the queue manager configuration to OPTIONAL to allow applications to connect
which have not supplied a user ID and password.
----- amqzfuca.c : 4287 -------------------------------------------------------
07/21/2016 10:08:17 AM - Process(16169.50) User(mqm) Program(amqzlaa0)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ5541: The failed authentication check was caused by the queue manager
CONNAUTH CHCKCLNT(REQDADM) configuration.

EXPLANATION:
The user ID 'mqm' and its password were checked because the user ID is
privileged and the queue manager connection authority (CONNAUTH) configuration
refers to an authentication information (AUTHINFO) object named
'SYSTEM.DEFAULT.AUTHINFO.IDPWOS' with CHCKCLNT(REQDADM). 

This message accompanies a previous error to clarify the reason for the user ID
and password check.
ACTION:
Refer to the previous error for more information. 

Ensure that a password is specified by the client application and that the
password is correct for the user ID. The authentication configuration of the
queue manager connection determines the user ID repository. For example, the
local operating system user database or an LDAP server. 

To avoid the authentication check, you can either use an unprivileged user ID
or amend the authentication configuration of the queue manager. You can amend
the CHCKCLNT attribute in the CHLAUTH record, but you should generally not
allow unauthenticated remote access.
-------------------------------------------------------------------------------
07/21/2016 10:08:18 AM - Process(18280.13) User(mqm) Program(amqrmppa)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ9557: Queue Manager User ID initialization failed for 'mqm'.

EXPLANATION:
The call to initialize the User ID 'mqm' failed with CompCode 2 and Reason
2035.
ACTION:
Correct the error and try again.
----- cmqxrsrv.c : 2321 -------------------------------------------------------
07/21/2016 10:10:58 AM - Process(16169.51) User(mqm) Program(amqzlaa0)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ8077: Entity 'hduser' has insufficient authority to access object
'CSBTS.QUEUE.MANAGER'.

EXPLANATION:
The specified entity is not authorized to access the required object. The
following requested permissions are unauthorized: connect
ACTION:
Ensure that the correct level of authority has been set for this entity against
the required object, or ensure that the entity is a member of a privileged
group.
----- amqzfubx.c : 687 --------------------------------------------------------
07/21/2016 10:10:58 AM - Process(18280.14) User(mqm) Program(amqrmppa)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ9557: Queue Manager User ID initialization failed for 'hduser'.

EXPLANATION:
The call to initialize the User ID 'hduser' failed with CompCode 2 and Reason
2035.
ACTION:
Correct the error and try again.
----- cmqxrsrv.c : 2321 -------------------------------------------------------

Answer 1

谢谢@MoragHughson，这是基于你的建议，寻找AMQERR01.LOG，以找出更多的问题。这有助于克服我与Flume互动MQ的问题。

我所做的是将队列管理器配置更改为 OPTIONAL，以允许未提供用户标识和密码的应用程序连接。就我而言，我没有在我的flume.conf文件中传递密码。

我关注了这个链接-http://www-01.ibm.com/support/docview.wss?uid=swg21680930并使用选项B。

下面是我用来使身份验证可选的命令-

ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) CHCKCLNT(OPTIONAL)

REFRESH SECURITY TYPE(CONNAUTH)

我希望这对任何试图将MQ与FLume/Kakfa集成的人有所帮助。