Unfortunately there is not a software without any mistakes, bugs or exceptions. Mistakes happen every day. I think it’s not a problem, but the problem is how to find and analyze these exceptions. Usually developers use the system log or special log file for tracking exceptions. This is reliable way to store information about bugs but not so user friendly. How to make these files more visual, more friendly? Solution is ELK stack: Elastisearch, Logstash, Kibana. This is a small example how do we use it at our work:
It’s easy to understand what’s going on with your applications or server in real time from these dashboards. Developer or support team can find a full stack trace of exception.
This is a high level diagram of architecture:
There are the next main components:
- Elasticsearch and Kibana instances which work inside the dockers’ containers. Elasticsearch database stores data, Kibana visualizes it. Docker is one more interesting technology. I think each developer has to know how to use it. This is really powerful tool.
- User defined node “Catch Exception”. The main role of this node is gathering information about exception, environment and message. After that the node puts this aggregated message into special queue. This queue will be processed by “Exception Processing Flow”
- “Exception Processing Flow”. The role of this application is to convert XML message into JSON structure and send it to elastic search database.
Let’s consider these components more detailed.
Elasticsearch and Kibana
I use Docker for instances of Kibana and Elasticsearch. It’s better to split a database depends on application or your requirements. On one server you can have many isolated instances. By default elastic search and Kibana don’t have any authentication. Of course you can buy X-Pack extension. But in my case a simple basic authentication is enough. For this I use nginx proxy server.
I know that my configuration is not the best, but it is enough for not high loaded projects.
This is github repository with configuration files: https://github.com/sharavara/elk
”IIBLogger” — user-defined node
This is the main part of exception processing architecture. Each flow should have a logic for mistakes processing. Sometimes this logic is more complicated even then a master flow. In our POC project the flow is simple:
This is a “GatherData” ESQL code. As you can see this node modifies just MQRFH2 header and does not change the original message.
--Settings SET OutputRoot.MQRFH2.usr.set.Emails = Emails; SET OutputRoot.MQRFH2.usr.set.BackupQueue = BackupQueue; SET OutputRoot.MQRFH2.usr.set.DSN = DSN; SET OutputRoot.MQRFH2.usr.set.Table = Table; SET OutputRoot.MQRFH2.usr.set.Service_URL = Service_URL; SET OutputRoot.MQRFH2.usr.set.Service_Auth = Service_Auth; --Exception & Environment Information SET OutputRoot.MQRFH2.usr.vs.BrokerName = SQL.BrokerName; SET OutputRoot.MQRFH2.usr.vs.QueueManagerName = SQL.QueueManagerName; SET OutputRoot.MQRFH2.usr.vs.ExecutionGroupLabel = SQL.ExecutionGroupLabel; SET OutputRoot.MQRFH2.usr.vs.ApplicationLabel = ApplicationLabel; SET OutputRoot.MQRFH2.usr.vs.MessageFlowLabel = SQL.MessageFlowLabel; SET OutputRoot.MQRFH2.usr.vs.MsgID = SUBSTRING(CAST(InputRoot.MQMD.MsgId AS CHARACTER) from 3 for 48); SET OutputRoot.MQRFH2.usr.vs.CorrelId = SUBSTRING(CAST(InputRoot.MQMD.CorrelId AS CHARACTER) from 3 for 48); SET OutputRoot.MQRFH2.usr.vs.Timestamp = CAST(CURRENT_TIMESTAMP AS CHARACTER FORMAT 'I'); SET OutputRoot.MQRFH2.usr.vs.Exception = serializeReferenceAsString(InputExceptionList, 'Exception'); Call getLastExceptionDetail(InputExceptionList, OutputRoot.MQRFH2.usr.vs.ExceptionNumber, OutputRoot.MQRFH2.usr.vs.ExceptionText); SET OutputRoot.MQRFH2.usr.vs.LocalEnvironment = serializeReferenceAsString(InputLocalEnvironment, 'LocalEnvironment'); SET OutputRoot.MQRFH2.usr.vs.Environment = serializeReferenceAsString(Environment, 'Environment'); SET OutputRoot.MQRFH2.usr.vs.Properties = serializeReferenceAsString(InputRoot.Properties , 'Properties'); SET OutputRoot.MQRFH2.usr.vs.MQMD = serializeReferenceAsString(InputRoot.MQMD, 'MQMD'); IF EXISTS(InputRoot.HTTPInputHeader) THEN SET OutputRoot.MQRFH2.usr.vs.HTTPInputHeader = serializeReferenceAsString(InputRoot.HTTPInputHeader , 'HTTPInputHeader'); END IF; IF EXISTS(InputRoot.HTTPReplyHeader) THEN SET OutputRoot.MQRFH2.usr.vs.HTTPReplyHeader = serializeReferenceAsString(InputRoot.HTTPReplyHeader , 'HTTPReplyHeader'); END IF; IF EXISTS(InputRoot.HTTPRequestHeader) THEN SET OutputRoot.MQRFH2.usr.vs.HTTPRequestHeader = serializeReferenceAsString(InputRoot.HTTPRequestHeader, 'HTTPRequestHeader'); END IF; IF EXISTS(InputRoot.HTTPResponseHeader) THEN SET OutputRoot.MQRFH2.usr.vs.HTTPResponseHeader = serializeReferenceAsString(InputRoot.HTTPResponseHeader, 'HTTPResponseHeader'); END IF; --Log Queue SET OutputLocalEnvironment.Destination.MQDestinationList.DestinationData.queueName = LogQueue;
After that flow puts the modified message into the queue IIB.LOGGER.IN
You can install this custom node from this site: https://raw.githubusercontent.com/sharavara/iiblogger.package.site/master/site.xml
After installation you can use this node in your projects. “IIB Logger” node is in the “Construction” section.
Note: unfortunately the custom nodes don’t work in linux toolkit.
Settings of node:
Currently only Basic tab is implemented. The idea was to use this node with the different instances of Elasticsearch and save exceptions into relation database as well. But during exploitation we realized that existing functionality is covered all our cases.
This one has one disadvantage. Technically this node is a sub flow, as you know there is not possible to use the sub flows inside other sub flows. But Java implementation has to resolve this issue.
Exception Processing Flow
This flow listens the queue IIB.LOGGER.IN:
It translates all input messages into JSON and through REST api sends this data to Elasticsearch database. If message has an email the flow will send a link to exception to this email.
This is repository with a source code of flow: https://github.com/sharavara/IIBLogger_Processor
Actually that is all. Now we can monitor all our exception via Kibana: