logstash and Apache log4j or how to manage all of your events and logs from all over your infrastructure in one place

Posted by on October 8, 2013

Having deployed our applications all over the world on clients server, usually force us to monitor and support those applications.

Few years ago, the normal operation was: log into server clients and discover what is happening based on our application logs.

Yes, watching the logs. The system or application health was controlled with some monitoring system.

But the application logs, always remained at clients servers.

This is a problem, because of time consuming just trying to get remote access to servers.

Why a problem? Because the IT departments of those clients usually sucks.

So the idea now is to externalize all those logs to a central server.

In this way we will be able to detect problems faster and prepare fixes quickly.

This is nothing new, there are a lot of solutions available for this.

In this case we are going to play with logstash and log4j for sending out logs to the “central” server.

Some description before continue:

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs. For more info http://logstash.net/

Apache log4j is a logging library for Java. The log4j package is designed so that these statements can remain in shipped code without incurring a heavy performance cost. Logging behavior can be controlled by editing a configuration file, without touching the application binary. For more info http://logging.apache.org/log4j

Now, with that background, lets start.

First, we need to configure logstash to accept all logs coming from our applications or servers (everything using log4j can talk to logstash)

logstash.conf content:

input {
 log4j {
     mode => "server"
     host  => ""
    port => 56445
    debug => true
    type => "log4j"

output {                                       
  stdout { debug => true }                     
  elasticsearch { embedded => true }           

We are telling to logstash that there will be and input of type log4j that will listen on port 56445, in this way, the info will be correctly handled. Also, we are telling to start it with elasticseach embedded (a powerful open source search and analytics engine that makes data easy to explore)

We that wonderful configuration is time to run logstath:

java -jar logstash-1.2.2.dev-flatjar.jar agent -f logstash.conf -- web 

We are running logstash with the agent mode and web interface using the config file logstash.conf The web interface is available at port 9292.

Time for the java app.

First, the log4j config file, usually log4j.properties

# output messages to stdout and over the network

# stdout
log4j.appender.stdout.layout.ConversionPattern=%5p %d{HH:mm:ss,SSS} %m%n

# logging over the network. Set port and IP for your network

Now a very simple sample:

package jtest;

import org.apache.log4j.Logger;

import java.io.IOException;

public class JTest {

    static Logger log = Logger.getLogger(JTest.class.getName());

    public static void main(String[] args) throws IOException, InterruptedException {

        log.debug(String.format("Hello this is an debug message %1$,.2f", Math.random() ));
        log.info(String.format("Hello this is an info message %1$,.2f", Math.random() ));
        log.error(String.format("Hello this is an ERORRRRRRRRRRRRR ERRORRRRR ERRORRRRRR %1$,.2f", Math.random() ));


And thats all folks, just hit F5 on the browser (because you are at http://localhost:9292 rigth? ) and that info should appear in logstash.