Extending Jolokia

In Jolokia 1.x, the necessary functions were provided by services directly instantiated by these agent implementations:

org.jolokia.http.AgentServlet (WAR)
org.jolokia.osgi.servlet.JolokiaServlet (OSGi)
org.jolokia.jvmagent.JolokiaServer (JVM)

Only implementations of detectors and simplifiers were detected respectively from:

META-INF/detectors[-default] (implementations of org.jolokia.detector.ServerDetector)
META-INF/simplifiers[-default] (implementations of org.jolokia.converter.json.Extractor)

In Jolokia 2 there’s another resource checked - services[-default] and additionally the declaration files were moved to more specific location:

META-INF/jolokia/detectors[-default] (implementations of org.jolokia.server.core.detector.ServerDetector)
META-INF/jolokia/simplifiers[-default] (implementations of org.jolokia.converter.json.ObjectAccessor)
META-INF/jolokia/services[-default] (implementations of org.jolokia.server.core.service.api.JolokiaService)

For example, what was a dedicated org.jolokia.history.HistoryStore in Jolokia 1, used directly by org.jolokia.backend.BackendManager is now used by org.jolokia.service.history.HistoryMBeanRequestInterceptor Jolokia service declared in META-INF/jolokia/services-default resource in org.jolokia:jolokia-service-history module.

A note about Service Loader approach

META-INF/jolokia/services is a bit similar approach to Java standard java.util.ServiceLoader API, where the services are declared in META-INF/services/interface-name.

The most important aspect is that the instantiation of such services is performed by the JDK (or Jolokia) itself and if 3rd party applications use more sophisticated service registries (like CDI or Spring DI), then we can’t avoid static fields…

So if a Jolokia service extension needs to use other dependencies, it’s up to the application to provide necessary integration. Static fields is the easiest (but not a super-clean) option.

Extension points in Jolokia 2

Jolokia, when starting, uses java.lang.ClassLoader.getResources call to find various locations of the above service declaration resources. This allows 3rd party libraries to simply add a classpath library containing relevant resource and declare class names to be instantiated and used by Jolokia.

The format of the extension file META-INF/jolokia/services[-default] is:

# comment
[!]fully.qualified.class.name[,order]

if line is not a comment, it is treated as service entry. There are two types of entries:

remove entry - if a line starts with ! (exclamation mark), we can declare a fully qualified name of a service declared in another (most probably Jolokia’s own) service file, so we can disable usage and instantiation of given service. For example, we can remove history service with !org.jolokia.service.history.HistoryMBeanRequestInterceptor even if org.jolokia:jolokia-service-history is available on classpath
service entry - if a line doesn’t start with !, a class is instantiated using one of two possible constructors:
- empty constructor
- a constructor accepting integer value which is treated as "order" of the service. The value to pass is read after the comma in service entry.

Service order is used to prioritize the services of the same interface - the lower the number, the higher the priority (preference). Default order is 100.

In the following sections we describe various services that may be declared by 3rd party libraries.

MBeanInfo cache

Jolokia is first and foremost a JMX protocol adaptor (see Architecture). This means that whatever is registered in local (or remote) MBeanServer, is accessible using "JSON over HTTP".

While this is a straightforward statement, the reality may be harsh sometimes. For example in Apache ActiveMQ Artemis broker, if you create, say, 10,000 queues, JVM ends up with additional 20,000 MBeans registered - a pair if these MBeans for single queue:

address="queue-name",broker="0.0.0.0",component=addresses
address="queue-name",broker="0.0.0.0",component=addresses,queue="queue-name",routing-type="anycast",subcomponent=queues

The first Mbean is for org.apache.activemq.artemis.core.management.impl.AddressControlImpl and second one is for org.apache.activemq.artemis.core.management.impl.QueueControlImpl. What’s more, the JSON map for javax.management.MBeanInfo of these MBeans is huge (over hundred attributes and operations for each pair). Multiplying it by 10K, Jolokia has to return 234MB of JSON data.

To address this problem, Jolokia 2.1.0 introduces an optimized list operation, where instead of simple (but big) structure of:

domain:
  mbean:
    op:
    attr:
    notif:
    class:
    desc:
  ...
...

we provide a smarter (a bit more complex, but much smaller) structure:

"domains":
  domain:
    mbean: cache-key
    ...
  ...
"cache":
  cache-key:
    op:
    attr:
    notif:
    class:
    desc:
  ...

Basically instead of duplicating the same JSON representation of javax.management.MBeanInfo over and over again for the same MBeans (2x10,000 times for 10,000 Artemis queues), we do some optimization here:

Jolokia calls javax.management.MBeanServerConnection.queryMBeans() normally, getting a set of javax.management.ObjectInstance objects.
Each ObjectInstance carries an ObjectName and className.
Without any optimization, ObjectName is used in javax.management.MBeanServerConnection.getMBeanInfo() call and result is the serialized using org.jolokia.server.core.service.api.DataUpdater services.
However, both ObjectName and ObjectInstance's class can be used to obtain a key (or a hint) which determines that some MBeans may share its javax.management.MBeanInfo with other MBeans.
When a non-null key for an MBean is obtained, its MBeanInfo is cached under "cache" field and related MBean under "domains" field simply points to cached JSON data of javax.management.MBeanInfo.

In order to use such variant of list() operation, a new processing request parameter has to be specified. This is done using listCache=true parameter (defaults to false for compatibility reasons).

When optimization is enabled, Jolokia uses org.jolokia.service.jmx.api.CacheKeyProvider services. if determineKey(ObjectInstance) method returns non-null key, it is used to share common JSON MBean information with other MBeans that produce the same cache key.

Jolokia itself provides optimization only for few fundamental MBeans:

java.lang:type=MemoryPool,name=* (class is sun.management.MemoryPoolImpl) - there are 8 instances by default in standard JVM (without any sophisticated memory settings)
java.lang:type=MemoryManager,name=* (class is sun.management.MemoryManagerImpl) - there are 2 instances by default in standard JVM
java.nio:type=BufferPool,name=* (class is sun.management.ManagementFactoryHelper$1) - there are 3 instances by default in standard JVM

With these 3 optimizations, instead of this response:

"value": {
  "java.lang": {
    "name=G1 Survivor Space,type=MemoryPool": {
      "op": {
        "resetPeakUsage": {
          "args": [],
          "ret": "void",
          "desc": "resetPeakUsage"
        }
      },
      "attr": {
        "Usage": {
          "rw": false,
          "type": "javax.management.openmbean.CompositeData",
          "desc": "Usage"
        },
        ...
    "name=Metaspace,type=MemoryPool": {
      "op": {
        "resetPeakUsage": {
          "args": [],
          "ret": "void",
          "desc": "resetPeakUsage"
        }
      },
      "attr": {
        "Usage": {
          "rw": false,
          "type": "javax.management.openmbean.CompositeData",
          "desc": "Usage"
        },
        ...
    "name=G1 Eden Space,type=MemoryPool": {
      "op": {
        "resetPeakUsage": {
          "args": [],
          "ret": "void",
          "desc": "resetPeakUsage"
        }
      },
      ...

We get this:

"value": {
  "cache": {
    "java.lang:MemoryPool": {
      "op": {
        "resetPeakUsage": {
          "args": [],
          "ret": "void",
          "desc": "resetPeakUsage"
        }
      },
      "attr": {
        "Usage": {
          "rw": false,
          "type": "javax.management.openmbean.CompositeData",
          "desc": "Usage"
        },
        ...
    "java.nio:BufferPool": {
      "attr": {
        "TotalCapacity": {
          "rw": false,
          "type": "long",
          "desc": "TotalCapacity"
        },
        ...
      }
    }
  }
  "domains": {
    "java.lang": {
      "name=G1 Survivor Space,type=MemoryPool": "java.lang:MemoryPool",
      "name=Metaspace,type=MemoryPool": "java.lang:MemoryPool",
      "name=G1 Eden Space,type=MemoryPool": "java.lang:MemoryPool",
      "name=CodeCacheManager,type=MemoryManager": "java.lang:MemoryManager",
      "name=CodeHeap 'non-nmethods',type=MemoryPool": "java.lang:MemoryPool",
      "name=G1 Old Gen,type=MemoryPool": "java.lang:MemoryPool",
      "name=Compressed Class Space,type=MemoryPool": "java.lang:MemoryPool",
      "name=CodeHeap 'non-profiled nmethods',type=MemoryPool": "java.lang:MemoryPool",
      "name=CodeHeap 'profiled nmethods',type=MemoryPool": "java.lang:MemoryPool",
      "name=Metaspace Manager,type=MemoryManager": "java.lang:MemoryManager",
      ...
    },
    "java.nio": {
      "name=direct,type=BufferPool": "java.nio:BufferPool",
      "name=mapped,type=BufferPool": "java.nio:BufferPool",
      "name=mapped - 'non-volatile memory',type=BufferPool": "java.nio:BufferPool"
    },
    ...

NOTE: The MBeans for which we don’t determine any cache key are included under "domains"/<domain>/<key-list-of-MBean> normally.

We can imagine Artemis adding a cache key provider for org.apache.activemq.artemis domain and MBeans with component=addresses key. There’s a lot of optimization to be declared in Apache Camel too.

See jolokia/jolokia#705 for the rationale behind new Jolokia protocol version.

Data updaters

Previous section described services that may affect the structure of entire list() response. This section is about services that affect single MBeanInfo of any MBean.

org.jolokia.server.core.service.api.DataUpdater is an interface used by the list() operation. Normally when invoking list operation, we get a JSON tree with a structure like this:

{
  "<domain of ObjectName>": {
    "<prop list from ObjectName>": {
      "attr": ...,
      "op": ...,
      "notif": ...,
      "class": ...,
      "desc": ...
    },
    ...
  },
  ...
}

Such JSON contains information obtained from javax.management.MBeanInfo objects and describes what is known about the registered MBean from the perspective of javax.management.MBeanServer.

Built-in implementations of org.jolokia.server.core.service.api.DataUpdater interface simply add these fields (attr, op, …) to the JSON data.

However, there may be scenarios where for each MBean we need some additional data - for example the security roles assigned to the given MBean (to implement a form of Role-Based Access Control (RBAC)). Jolokia isn’t aware of the way an application implements security mechanisms, but should allow for any extension of the basic MBean data.

We can provide an extension module, add it to the CLASSPATH and declare additional data updaters in META-INF/jolokia/services. org.jolokia.server.core.service.api.DataUpdater is already an implementation of org.jolokia.server.core.service.api.JolokiaService and it’s an abstract class we can extend. For example:

package com.example;

public class MyUpdater extends org.jolokia.server.core.service.api.DataUpdater {

    public MyUpdater() {
        super(100);
    }

    @Override
    public String getKey() {
        return "time";
    }

    @Override
    public JSONObject extractData(ObjectName pObjectName, MBeanInfo pMBeanInfo, String pFilter) {
        JSONObject json = new JSONObject();
        json.put("now", System.currentTimeMillis());
        return json;
    }

}

This updater adds time field to the JSON info for an MBean with current timestamp, but we can imagine any other kind of updater. It is declared using the below line in META-INF/jolokia/services resource:

com.example.MyUpdater

NOTE: It is recommended for advanced 3rd party extensions to implement both cache key provider and data updater. We can imagine for Artemis broker that some queues are more restricted than others, so their serialized MBean info may contain different custom data then other queues, so they should produce different cache key.

`ListKeysDataUpdater`

Since 2.1.0, Jolokia provides optional implementation of org.jolokia.server.core.service.api.DataUpdater service which can be enabled using listKeys processing parameter. When it is set (at request time) to true, in addition to standard attr, op, notif, class and desc fields of serialized MBeanInfo, we get another field keys.
It contains a map of keys obtained from the ObjectName itself using javax.management.ObjectName.getKeyPropertyList() method.

With this parameter we can get the below response:

{
  "request": {
    "type": "list"
  },
  "value": {
    "java.lang": {
      "name=CodeHeap 'non-nmethods',type=MemoryPool": {
        "keys": {
          "name": "CodeHeap 'non-nmethods'",
          "type": "MemoryPool"
        },
        "op": ...,
        "attr": ...
        ...

This may be used to save time parsing object key-list like name=CodeHeap 'non-nmethods',type=MemoryPool.

Jolokia Server Detectors

Jolokia used org.jolokia.server.core.detector.ServerDetector interface for a long time. Jolokia included server detectors for several products like ActiveMQ, JBoss, WebLogic, WebSphere and other. Product-specific detectors provide:

special javax.management.MBeanServerConnection instances for MBean servers not detected by java.lang.management.ManagementFactory
org.jolokia.server.core.service.request.RequestInterceptor services to handle the requests in product-specific way (there is only one Glassfish-specific interceptor)
org.jolokia.server.core.service.api.ServerHandle with product-specific information (name, version)

jolokia#258 added new responsibility to the server detector, so Jolokia JVM agent could actively wait for some conditions before proceeding. This is used to wait for some classes or services to be available. The detectors can wait and additionally return … a java.lang.ClassLoader instance:

public interface ServerDetector extends JolokiaService<ServerDetector>, Comparable<ServerDetector> {

    ...
    ClassLoader jvmAgentStartup(Instrumentation instrumentation);

}

This may sound complex and unnecessary, but the concept is quite clear:

let the detectors wait for initialization
if the detector returns a ClassLoader, this loader will then ne used by Jolokia service to perform initialization (including looking up additional services).

This is important when using Jolokia JVM Agent in containers like Tomcat or Artemis, where the application (boot) classpath usually consists of some kind of bootstrap JARs which role is only to create a full ClassLoader with JARs and resources from lib/ directory (or other).

see Tomcat’s ${catalina.home}/lib directory
see Artemis' ${artemis.instance}/lib and ${artemis.home}/lib directories

And recently in Jolokia 2.1.2, with jolokia#754 we’ve introduced the concept of a org.jolokia.server.core.service.container.ContainerLocator:

public interface ContainerLocator extends JolokiaService<ContainerLocator> {

    /**
     * Returns an instance of a <em>runtime</em> or <em>container</em> specific class to be used by
     * a dedicated Jolokia service.
     *
     * @param clazz A class guard to ensure that the returned instance is of proper class
     * @return
     * @param <T>
     */
    <T> T locate(Class<T> clazz);

}

Currently only org.jolokia.server.detector.misc.ArtemisDetector returns special ContainerLocator, but this locator is not used by Jolokia itself.

The role of this interface is to give access to server-specific services/interfaces/classes from other services - usually 3rd party libraries. See the following Jolokia Integration project section.

Jolokia itself should not depend on any special classes and interfaces from other projects. Note that classes like org.jolokia.server.detector.misc.ArtemisDetector may use reflection or other form of dynamic discovery, but there are no Maven dependencies on (in this case) Artemis.

Jolokia Integration project

Jolokia Integration is a special project built on the concepts mentioned in this chapter:

MBeanInfo cache and Data updaters may be implemented by external services
Jolokia Server Detectors may reflectively/dynamically make Jolokia aware of the environment where the Agent is running

Specifically:

Jolokia should not statically (with Maven dependencies) depend on projects like Artemis or Tomcat
Projects like Artemis or Tomcat should not depend on Jolokia (when not needed)

Currently Jolokia Integration should be treated as _technical preview project, but it is already actively used with Apache Artemis.

As of the latest version 0.8.0, Jolokia-Integration provides two services:

org.jolokia.integration.artemis.ArtemisCacheKeyProvider - optimizes the JSON size for the MBeans which are known to come in huge amounts (like queues and addresses)
org.jolokia.integration.artemis.ArtemisDataUpdater - adds canInvoke flag for each MBean, MBean operation and MBean attribute based on RBAC configuration provided by Artemis.

There are plans to include similar integration with Apache Camel.