Question

We have an enterprise application (deployed as an ear) is a batch processor which creates multiple threads to work on the batch items concurrently (to a max number of concurrent threads). As part of the processing, it makes calls to a couple of RESTful web services deployed in the same Glassfish, but across two domains. Each application logs to its own application log file on the server and all logs are also being ingested by a Splunk server.

We’ve been asked to find a way to “tag” each log statement so that we can do a search on all the logs and get all the statements that deal with the processing of one batch item.

Each item being processed has a unique ID so adding this to the processor log statements is easy. The question is how do we tag the logs from the web services?

One suggestion has been to add a parameter to each webservice call that takes this ID and propogates this down into each method of those services. I think it’s a crazy idea to make this type of change to code to pass a value simply for logging.

Has anybody done anything like this? Any suggestions on how to identify log statements that “belong together”?

BTW, we are using slf4j with log4j, but are considering using logback.

UPDATE

I’ve been working on passing the transaction ID as an HTTP header, but can’t seem to quite make it work. I’m using Jersey for my and here’s what I’ve got:

My Processor class which puts its a value in the logging MDC as a transactionId and uses a REST service client for some processing:

public Processor
{
    RESTClient myRESTClient = new RESTClient("http://path/to/restService");

    public void process(final Object object)
    {
        //Put the object ID in the logging MDC
        log.debug("Putting '{}' in the MDC as the {} header value.",  object.getObjectID(), "transactionID");
        MDC.put("transactionID", object.getObjectID());

    //Do some stuff

    Object anotherObject = myRESTClient.doQuery(object.getValue());

    //Do more some stuff
    }
}

My RESTClient to access the REST service. It’s here that I’m pulling the transactionId out of the MDC and adding it as a header to the request:

public RESTClient
{
    public Object doQuery(String value)
    {
        Object object = null;

        try
        {
            Builder builder = myRestService.queryParam(PARAM_KEY_VALUE, value)
                    .accept(MediaType.APPLICATION_XML);

            String transactionId = (String) MDC.get("transactionID");

            logger.debug("Retrieved '{}' from MDC for key {}",
                transactionId,
                "transactionID");

            if (this.getTransactionID() != null)
            {
                builder = builder.header("transactionID", transactionId);
            }

            object = builder.get(Object.class);
        }
        catch (Throwable ex)
        {
            //Do error handling
        }
    }
}

My REST service Resource class which should have the transactionId in its request headers and places it in its logging MDC:

@Path("/myPath")
public class MyResource
{
    @Context
    private HttpContext httpContext;

    @GET
    @Produces(MediaType.APPLICATION_XML)
    public Object doQuery( @QueryParam("value") String value)
    {
        putTransactionIdInMDC();

        SubscriberAccount account = null;

        try
        {
            //Do query stuff
        }
        catch (Exception ex)
        {
            throw new WebApplicationException(Response.Status.INTERNAL_SERVER_ERROR);
        }

        return account;
    }

    private void putTransactionIdInMDC()
    {
        if (httpContext != null)
        {
            String transactionID = httpContext.getRequest()
                    .getHeaderValue("transactionID");

            if (transactionID != null && transactionID.isEmpty())
            {
                /*
                 * It's not likely, but possible that two headers with the same
                 * header key were put in the request. If so, the value will be
                 * a comma separated list. We're using the first one.
                 */
                String[] strings = transactionID.split(",");

                logger.debug("Header '{}' value(s): {}",
                        "transactionID",
                        strings);

                MDC.put("transactionID", strings[0]);
            }
            else
            {
                logger.debug("The header '{}' was not included in the request.",
                        "transactionID");
            }
        }
        else
        {
            logger.info("Could not get an HttpContext for the request");
        }
    }
}

Based on my logging, I know that the transactionId is being put in the Processor MDC and being pulled from it by the RESTClient class. However, it’s not getting passed as an http header to the REST service. Can anyone tell me why not?

Processor Log File:

2012-04-13T17:30:36.541 MDT  INFO [Ejb-Async-Thread-2] DEBUG my.package.Processor - Putting '12311497-2279-4516-af7d-cf9716f7748a' in the MDC as the transactionId header value.

2012-04-13T17:30:36.541 MDT  INFO [Ejb-Async-Thread-2] DEBUG my.package.RESTClient- Retrieved '12311497-2279-4516-af7d-cf9716f7748a' from MDC for key transactionId

REST Service Log File:

2012 Apr 13 17:30:36,337 MDT [http-thread-pool-80(3)] DEBUG my.package.MyResource - The header 'transactionId' was not included in the request.

UPDATE TWO

Found the logic error in my above code:

if (transactionID != null && transactionID.isEmpty())

should have been:

if (transactionID != null !&& transactionID.isEmpty())

Answer 1

Just a vague idea: Perhaps you can use the SOAP envelope to “hide” that pesky job-id from the business data. And using some kind of interceptor or JAX-WS handler on the client side it could be set into the requests without touching the business code. On the server side another handler or interceptor could extract id from the envelope and stuff it into the Log4J stuff using either a NDC (nested Diagnostic context) or MDC (mapped diagnostic context). Then the log formats have to be adjusted to actualy log the NDC/MDC values.