Detecting Java Threads in Deadlock with Groovy and JMX

Unfortunately, Java applications taking advantage of multiple threads can at times run into the dreaded deadlock condition. Fortunately, the Java Platform makes deadlock detection relatively easy. In fact, the built-in (since J2SE 5) ThreadMXBean (a PlatformManagedObject exposed via JMX) makes this information available to any client that "speaks JMX" via the findDeadlockedThreads() and findMonitorDeadlockThreads() methods. General "JMX clients" such as JConsole and VisualVM use this to provide information on detected deadlocks, but custom tools and scripts can be written to provide the same details. In this post, I look at using Groovy in conjunction with the Attach API to detect deadlocked threads for a locally running JVM process.

The Java Tutorials provides a simple and fairly interesting example of Java code that will typically result in deadlock on the "Deadlock" page in the "Concurrency" lesson of the "Essential Classes" trail. I provided that example here in only the slightest adapted form as the code I will run JMX clients against to detect deadlock. (As a side note, I keep seeing posts on reddit/Java and other online forums asking for the best free online introductory resources for learning Java; I cannot think of a better answer to this than the Java Tutorials.)

Deadlock.java (Adapted from Java Tutorials)

package dustin.examples.jmx.threading;

import static java.lang.System.out;

/**
 * Example of a class that often will lead to deadlock adapted from the 
 * Java Tutorials "Concurrency" section on "Deadlock":
 * http://docs.oracle.com/javase/tutorial/essential/concurrency/deadlock.html.
 */
public class Deadlock
{
   static class Friend
   {
      /** Friend's name. */
      private final String name;

      /**
       * Parameterized constructor accepting name for Friend instance.
       * @param newName Name of new Friend instance.
       */
      public Friend(final String newName)
      {
         this.name = newName;
      }
      
      /**
       * Provide this instance's name.
       * 
       * @return Friend's name.
       */
      public String getName()
      {
         return this.name;
      }

      /**
       * Bow from friend. Synchronized for thread-safe access.
       * 
       * @param bower Friend that is bowing.
       */
      public synchronized void bow(final Friend bower)
      {
         out.format("%s: %s has bowed to me!%n", 
            this.name, bower.getName());
         bower.bowBack(this);
      }

      /**
       * Bow back to friend who bowed to me. Synchronized for thread-safe access.
       * 
       * @param bower Friend who has bowed back to me.
       */
      public synchronized void bowBack(final Friend bower)
      {
         out.format("%s: %s  has bowed back to me!%n",
            this.name, bower.getName());
      }
   }

   /**
    * Simple executable function that demonstrates deadlock when two friends
    * are waiting on each other to bow to finish bowing.
    * 
    * @param arguments Command-line arguments: none expected.
    */
   public static void main(final String[] arguments)
   {
      final Friend alphonse = new Friend("Alphonse");
      final Friend gaston = new Friend("Gaston");
      new Thread(new Runnable()
      {
         public void run() { alphonse.bow(gaston); }
      }, "Gaston Bowing").start();
      new Thread(new Runnable()
      {
         public void run() { gaston.bow(alphonse); }
      }, "Alphonse Bowing").start();
   }
}

The next screen snapshot shows that this simple application becomes deadlocked between the two threads.

Even if we had not known this application was intended to demonstrate deadlock, the fact that it never ends or changes state is a good hint that it has run into deadlock. We can use JMX to determine that for certain. Whether we use a JMX client, jstack, or some other mechanism for determining deadlock, we'll likely need the Java process's pid (process ID), so the next screen snapshot indicates doing that (pid is 3792 in this case) with both jps and jcmd (the latter is only available since JDK 7).

Knowing that the pid is 3792, I can run "jconsole 3794" at the command line to have JConsole come up loaded with information about that deadlocked Java process. In this case, I'm interested in the threads and the next image shows the "Threads" tab in JConsole for this deadlock demonstration application.

The immediately previous screen snapshot has the two threads of interest circled. In my adapted code example, I named these "Gaston Bowing" and "Alphonse Bowing" to make them easier to find in JConsole and other tools. Had I not provided these explicit names in the example (which the original example in the Java Tutorials did not do), they would have been referenced by the more generic names "Thread-0" and "Thread-1." With JConsole loaded and the "Threads" tab available, I can click on the "Detect Deadlocks" button to see the information in the next two screen snapshots (one each for the two threads being selected).

By looking at the JConsole output for each of the identified deadlocked threads, we can see that each thread is BLOCKED and waiting on the other thread. The stack trace provided with each thread also provides pretty clear documentation of where this deadlock occurs in the code.

Java VisualVM can be started with the command jvisualvm and then the Java process in question can be selected as demonstrated in the next screen snapshot. Note that VisualVM automatically detects the deadlocks on the "Threads" tab of VisualVM and provides a message in red font indicating that.

When I click the "Thread Dump" button as indicated to see which threads are deadlocked, a thread dump is provided. Toward the bottom of that thread dump, as indicated in the next screen snapshot, are the details surrounding the deadlocked threads.

As JConsole did, VisualVM also indicates in the provided thread dump that the two bowing threads are waiting upon one another and "Java stack information" is provided for each of the threads identified as participating in the deadly embrace.

JConsole and Java VisualVM make it easy to identify the threads involved in deadlock. However, when a command line tool is preferred over a GUI-based tool, jstack is an obvious choice. Again using the pid identified earlier for this particular process (3792 in my case), one can simply type jstack 3792 to see a thread dump for that application. The next two screen snapshots show a piece of this with the first snapshot showing the command being run and the beginning of the output and the second snapshot showing the portion related to deadlock.

One doesn't have to look too closely to realize that jstack's output is the same as that provided in the "thread dump" in VisualVM.

JConsole and VisualVM provide a nice GUI approach for identifying deadlock and jstack does the same from the command-line. Given these tools, one might wonder why we'd care about being able to build our own tools and scripts to do the same. Often, these tools are enough, but there are times I may want to do specific things with the information once I have it or I want the answer to be even more direct than these tools provide. If I need to programmatically do something with the information or simply want the identification of threads involved in deadlock without the other threads' details, a custom tool might be in order. The remainder of this post focuses on that.

The following code listing contains Groovy code for printing threading information related to deadlocked threads as provided by the ThreadMXBean. This code makes use of a piece of Groovy code not shown here (JmxServer using Attach API) that can be found in my last blog post.

displayDetectedDeadlock.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaThreads = new javax.management.ObjectName("java.lang:type=Threading")
def server = JmxServer.retrieveServerConnection(pid)
long[] deadlockedThreadIds = server.invoke(javaThreads, "findDeadlockedThreads", null, null)
deadlockedThreadIds.each
{ threadId ->
   Object[] parameters = [threadId, 10]
   String[] signature = [Long.TYPE, Integer.TYPE]
   def threadInfo = server.invoke(javaThreads, "getThreadInfo", parameters, signature)
   print "\nThread '${threadInfo.threadName}' [${threadInfo.threadId}] is "
   print "${threadInfo.threadState} on thread '${threadInfo.lockOwnerName}' ["
   println "${threadInfo.lockOwnerId}]:\n"
   println "Java stack information for the threads listed above:"
   println "==================================================="
   println "'${threadInfo.threadName}':"
   threadInfo.stackTrace.each
   { compositeData ->
      print "\tat ${compositeData.className}.${compositeData.methodName}("
      println "${compositeData.fileName}:${compositeData.lineNumber})"
   }
   println "\n\n"
}

The Groovy script above specifies 10 as the maximum depth for the stack trace information to be provided as it provides details similar to those provided in JConsole, VisualVM, and jstack. Of course, the advantage of this approach is that any of this data that is printed to standard output here can also be used in runtime decisions and acted upon programatically from Java or Groovy code. The next screen snapshot shows the output from running this script against the deadlocked application used previously in this post.

Java makes it easy to detect threads locked in the deadly embrace known as deadlock. Although general tools like JConsole, VisualVM, and jstack are often sufficient to identify these cases, it is nice to be able to write custom scripts and tools for doing the same thing. This allows developers the flexibility to include deadlock detecting directly in their scripts and tools rather than needing to parse jstack output or resort to other data scraping approaches.

Monitoring Key JVM Characteristics with Groovy, JMX, and RuntimeMXBean

Since J2SE 5, Platform MBeans have been available that allow some key characteristics regarding the JVM to be monitored and (even managed in some cases) via JMX. In addition, many JVM-based applications add their own JMX-enabled features for monitoring and management. In the blog post Groovy, JMX, and the Attach API, I looked at how to display many of the platform-provided MBeans using Groovy, JMX, and the Attach API. In this post, I look at more specific scripts that access a narrowly focused subset of these platform-exposed values available in the RuntimeMXBean.

My previous post demonstrated ability to view a wide spectrum of JVM details with Groovy and JMX. In most cases, I don't need all of these details all at once, but simply need one or two of the items. Although I could use a tool such as JConsole, VisualVM, or even a modern Java IDE to see these details, I sometimes wish to run a simple script to provide the exact results I'm looking for rather than using a general JMX client that I need to start up and navigate to that information. This is where simple scripts such as shown in this post are particularly handy.

All of my examples in this post assume the person running the Java process to be monitored/managed is the same person (same username) running the scripts and that they are being used to monitor Java processes running on the same local machine. This assumption allows the Attach API to be used. If a different user was running the script than ran the Java processes or if the scripts were to be run on remote Java processes, techniques for remote JMX would be used instead. Because the Attach API will be used in these examples, I have placed the code that each example uses in a Groovy script file that each example invokes. That file to be used by all of my scripts is called JmxServer.groovy and is shown next.

JmxServer.groovy

/*
 * JmxServer.groovy
 *
 * Functions meant to be called by other scripts or tools that need to use
 * the Attach API to access an MBeanServerConnection to use to manage and
 * monitor a particular JVM (and possibly the application hosted in that JVM)
 * identified by the provided Process ID (pid)
 */

import javax.management.MBeanServerConnection
import javax.management.ObjectName
import javax.management.remote.JMXConnector
import javax.management.remote.JMXConnectorFactory
import javax.management.remote.JMXServiceURL

import com.sun.tools.attach.VirtualMachine

/**
 * Provide an MBeanServerConnection based on the provided process ID (pid).
 *
 * @param pid Process ID of Java process for which MBeanServerConnection is
 *    desired.
 * @return MBeanServerConnection connecting to Java process identified by pid.
 */
def static MBeanServerConnection retrieveServerConnection(String pid)
{
   def connectorAddressStr = "com.sun.management.jmxremote.localConnectorAddress"
   def jmxUrl = retrieveUrlForPid(pid, connectorAddressStr)
   def jmxConnector = JMXConnectorFactory.connect(jmxUrl)
   return jmxConnector.getMBeanServerConnection()
}

/**
 * Provide JMX URL for attaching to the provided process ID (pid).
 *
 * @param @pid Process ID for which JMX URL is needed to connect.
 * @param @connectorAddressStr String for connecting.
 * @return JMX URL to communicating with Java process identified by pid.
 */
def static JMXServiceURL retrieveUrlForPid(String pid, String connectorAddressStr)
{
   // Attach to the target application's virtual machine
   def vm = VirtualMachine.attach(pid)

   // Obtain Connector Address
   def connectorAddress =
      vm.getAgentProperties().getProperty(connectorAddressStr)

   // Load Agent if no connector address is available
   if (connectorAddress == null)
   {
      def agent = vm.getSystemProperties().getProperty("java.home") +
          File.separator + "lib" + File.separator + "management-agent.jar"
      vm.loadAgent(agent)

      // agent is started, get the connector address
      connectorAddress =
         vm.getAgentProperties().getProperty(connectorAddressStr)
   }

   return new JMXServiceURL(connectorAddress);
}

The most important method defined in JmxServer.groovy, from a script client perspective, is the retrieveServerConnection(String) method. That methods excepts a pid as a process identifier and uses that and a call to the other method, retrieveUrlForPid(String,String) to attach to the Java process identified by that pid and provide an associated MBeanServerConnection instance. The easiest approaches for finding the appropriate Java pid are use of jps (pre-Java 7 and Java 7) or jcmd (Java 7+ only) as I briefly described in my recent post Booting AMX in GlassFish 3 with Groovy.

Having encapsulated the logic for acquiring a particular JVM (Java process's) MBeanServerConnection instance based on a provided pid, it is easy to start constructing simple Groovy scripts that use this provided MBeanServerConnection to provide specific desirable details about the JVM (Java process) in question. Note that all of these example scripts are in the same directory as the JmxServer.groovy file and so do not need to explicitly specify package or scoping details.

It is common to want to know what is on the classpath of a particular JVM for various reasons including detection of causes of ClassNotFoundException and NoClassDefFoundError. The classpath of a particular JVM is exposed by one of its platform MBean ("ClassPath" attribute of RuntimeMXBean) and can be easily discovered with the next Groovy script:

displayClasspath.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def classpath = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "ClassPath")
println "\nClasspath for Java pid (${pid}):\n${classpath}\n"

That same RuntimeMXBean also provides the boot classpath and the Java library path. The next two code listings show Groovy scripts for accessing both of those.

displayBootClasspath.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def bootClasspath = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "BootClassPath")
println "\nBoot Classpath for Java pid (${pid}):\n${bootClasspath}\n"

displayLibraryPath.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def libraryPath = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "LibraryPath")
println "\nLibrary Path for Java pid (${pid}):\n${libraryPath}\n"

The Groovy scripts shown so far for displaying details related finding classes (the classpath, boot classpath, and library path) are all essentially the same script, but with a different attribute looked up on the RuntimeMXBean in each case. All three scripts are executed against a running instance of GlassFish in the following screen snapshot. The snapshot includes commands to jps and jcmd to determine the pid (5352) for the GlassFish instance.

The RuntimeMXBean has more to offer than just information on where classes were loaded from. It also includes attributes related to the JVM vendor such as name and version, but the other three methods that I like to have scripts to use are input arguments, system properties, and JVM uptime (and start time). The next three Groovy code listings show three scripts for displaying these details and each code snippet is followed by a screen snapshot demonstrating that script in action against the same GlassFish instance used in the last examples.

displayInputArguments.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def inputArguments = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "InputArguments")
println "\nInput Arguments for Java pid (${pid}):\n${inputArguments}\n"

displaySystemProperties.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def systemProperties = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "SystemProperties")
println "\nSystem Properties for Java pid (${pid}):\n${systemProperties}\n"

displayJvmUptime.groovy

#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def server = JmxServer.retrieveServerConnection(pid)
def startTime = server.getAttribute(javaRuntime, "StartTime")
def uptime = server.getAttribute(javaRuntime, "Uptime")
println "\nJava process pid (${pid}) was started at ${new Date(startTime)} and has been up for ${uptime} ms.\n"

I have used this post to demonstrate simple Groovy scripts that acquire information from a JVM's Platform RuntimeMXBean such as the JVM's start time and uptime, the system properties and input arguments for the JVM process, and path information such as classpath, boot classpath, and library path. Although this information is available via tools such as JConsole, VisualVM, and Java IDEs (they all get it from the same source as these scripts!), the scripts can be advantageous at times (quicker to run and can be included within other scripts, for example).

"Booting AMX" in GlassFish 3 with Groovy

In my previous blog post, I looked at using JMX as one of multiple methods supported by GlassFish 3 for its administration, monitoring, and management. In this blog post, I look in more detail at monitoring and managing GlassFish 3 via JMX and Groovy. I focus on local connection to GlassFish using the Attach API in this post, but I have covered remote JMX access of GlassFish in a previous post (see also Remote Glassfish V3.1 and the mystical JMX settings). The MBeans used for administration are generally the same in either case.

Because I'm using the Attach API in this post's examples to connect to local Java processes that are started by me, I don't need to specify the JMX remote connection properties (java.rmi.server.hostname,com.sun.management.jmxremote.port, and com.sun.management.jmxremote.ssl=false, and com.sun.management.jmxremote.authenticate) for remote access. The easiest way to find Java processes meeting the standard of being on the local machine and being processes that I started is via use of jps (pre-JDK 7 and JDK 7) or jcmd (JDK 7 only), which I show in the next two screen snapshots.

As the above images indicate, GlassFish is running locally with PID (AKA Process ID or pid) 1584.

To begin, I'm going to use Groovy to access the relevant MBeanServerConnection via the Attach API. This is done as shown in the following Groovy code (next two code listings) in which the method retrieveServerConnection(String) accepts the PID and returns the MBeanServerConnection for that Java process. That method uses another method in the code listing, retrieveUrlForPid(String, String), which uses the Attach API to provide a JMXServiceURL for the Java process.

retrieveServerConnection(String)

/**
 * Provide an MBeanServerConnection based on the provided process ID (pid).
 *
 * @param pid Process ID of Java process for which MBeanServerConnection is
 *    desired.
 * @return MBeanServerConnection connecting to Java process identified by pid.
 */
def MBeanServerConnection retrieveServerConnection(String pid)
{
   def connectorAddressStr = "com.sun.management.jmxremote.localConnectorAddress"
   def jmxUrl = retrieveUrlForPid(pid, connectorAddressStr)
   def jmxConnector = JMXConnectorFactory.connect(jmxUrl)
   return jmxConnector.getMBeanServerConnection()
}

retrieveUrlForPid(String, String)

/**
 * Provide JMX URL for attaching to the provided process ID (pid).
 *
 * @param @pid Process ID for which JMX URL is needed to connect.
 * @param @connectorAddressStr String for connecting.
 * @return JMX URL to communicating with Java process identified by pid.
 */
def JMXServiceURL retrieveUrlForPid(String pid, String connectorAddressStr)
{
   // Attach to the target application's virtual machine
   def vm = VirtualMachine.attach(pid)

   // Obtain Connector Address
   def connectorAddress =
      vm.getAgentProperties().getProperty(connectorAddressStr)

   // Load Agent if no connector address is available
   if (connectorAddress == null)
   {
      def agent = vm.getSystemProperties().getProperty("java.home") +
          File.separator + "lib" + File.separator + "management-agent.jar"
      vm.loadAgent(agent)

      // agent is started, get the connector address
      connectorAddress =
         vm.getAgentProperties().getProperty(connectorAddressStr)
   }

   return new JMXServiceURL(connectorAddress);
}

With access to the MBeanServerConnection, I can start to do all types of useful things with Groovy and JMX to manage and monitor the a Java process. For example, the next code listing demonstrates how easy it is to now list the MBeans exposed by the Java process identified by a provided PID.

displayHostedMBeans(MBeanServerConnection)

/**
 * Display MBeans hosted on the provided MBeanServerConnection.
 *
 * @param mbeanServer MBeanServerConnection for which hosted MBeans are to be
 *    provided.
 */
def displayHostedMBeans(MBeanServerConnection mbeanServer)
{
   mbeanServer.queryNames(null, null).each
   {
      println it
   }
}

Running the above Groovy method against my running GlassFish instance with PID 1584 is demonstrated in the next screen snapshot.

The next code listing and associated screen snapshot demonstrate using Groovy and JMX to find not only the exposed MBeans, but to also provide the attributes and operations available on each of those exposed MBeans.

displayHostedMBeansAttributesAndOperations(MBeanServerConnection)

/**
 * Display MBeans hosted on provided MBean Server along with the attributes and
 * operations available on each MBean.
 */
def displayHostedMBeansAttributesAndOperations(MBeanServerConnection mbeanServer)
{
   mbeanServer.queryNames(null, null).each
   { mbeanObjectName ->
      def mbeanInfo = mbeanServer.getMBeanInfo(mbeanObjectName)
      println mbeanObjectName
      println "\tAttributes:"
      mbeanInfo.attributes.each
      { attribute ->
         println "\t\t${attribute.type} ${attribute.name}"
      }
      println "\tOperations:"
      mbeanInfo.operations.each
      { operation ->
         def operationStr = new StringBuilder();
         operationStr << "\t\t" << operation.name << "("
         operation.signature.each
         { parameter ->
            operationStr << parameter.type << " " << parameter.name << ", "
         }
         def operationString = operationStr.contains(",") ? operationStr.substring(0, operationStr.length()-2) : operationStr
         println "${operationString})"
      }
      println ""
   }
}

The previously displayed screen snapshot shows the bootAMX operation available on the amx-support:type=boot-amx MBean. This needs to be run to make other GlassFish-provided AMX MBean available. The next snippet of Groovy code, a method I've named bootAmx(MBeanServerConnection), invokes this bootAMX operation to instruct GlassFish to expose significantly more details via its JMX interface.

bootAmx(MBeanServerConnection)

/**
 * "Boot AMX" on GlassFish.
 *
 * @param mbeanServer MBeanServerConnection to be used to invoke bootAMX
 *    operation.
 */
def bootAmx(MBeanServerConnection mbeanServer)
{
   def amxObjectName = new ObjectName("amx-support:type=boot-amx")
   mbeanServer.invoke(amxObjectName, "bootAMX", null, null)
}

When the simple Groovy just shown is run, GlassFish exposes significant more functionality via JMX. Now that I've "booted AMX," I can rerun displayHostedMBeans(MBeanServerConnection) to see the new MBeans that are available. A portion of this is shown next in the screen snapshot and the small-font text below it contains the entire output.

amx:pp=/domain/configs/config[server-config]/network-config,type=transports
amx:pp=/domain/configs/config[default-config]/iiop-service,type=iiop-listener,name=orb-listener-1
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=encryption.key.alias
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=serverName
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-1]/http,type=file-cache
amx:pp=/domain/configs/config[server-config],type=mdb-container
amx:pp=/domain/configs/config[default-config],type=system-property,name=HTTP_LISTENER_PORT
amx:pp=/domain/configs/config[server-config]/web-container/session-config/session-manager,type=store-properties
amx:pp=/ext,type=config-tools
amx:pp=/domain/nodes,type=node,name=localhost-domain1
amx:pp=/domain/servers/server[server],type=application-ref,name=__admingui
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=sec-admin-listener
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=DownloadServlet,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[server-config]/network-config/transports,type=transport,name=tcp
amx:pp=/domain/configs/config[default-config],type=system-property,name=OSGI_SHELL_TELNET_PORT
amx:pp=/domain/configs/config[server-config],type=thread-pools
amx:pp=/domain/configs/config[server-config]/security-service,type=audit-module,name=default
amx:pp=/domain/secure-admin,type=secure-admin-principal,name="CN=localhost-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=signature.key.alias
amx:pp=/,type=logging
java.lang:type=MemoryPool,name=Code Cache
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config]/web-container,type=session-config
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[file],type=property,name=file
amx:pp=/domain/configs/config[server-config]/iiop-service,type=iiop-listener,name=SSL
amx:pp=/domain/configs/config[server-config]/security-service,type=message-security-config,name=SOAP
amx:pp=/domain/configs/config[server-config]/iiop-service/iiop-listener[SSL_MUTUALAUTH],type=ssl
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-1],type=http
amx:pp=/domain/configs/config[default-config]/network-config,type=transports
amx:pp=/domain/configs/config[server-config]/http-service,type=virtual-server,name=server
amx:pp=/domain/configs/config[default-config]/web-container/session-config/session-manager,type=store-properties
amx:pp=/domain/configs/config[default-config],type=ejb-container
amx:pp=/ext,type=system-status
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[admin-listener],type=http
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=response-policy
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=property,name=loginPage
amx:pp=/domain/servers/server[server],type=resource-ref,name=jdbc/__TimerPool
java.lang:type=Memory
amx:pp=/mon/server-mon[server],type=compilation-system-mon,name=jvm/compilation-system
amx:pp=/domain/configs/config[server-config]/security-service,type=auth-realm,name=admin-realm
amx:pp=/domain/configs/config[default-config]/log-service,type=module-log-levels
amx:pp=/mon/server-mon[server],type=server-runtime-mon
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ClientProvider
amx:pp=/domain/configs/config[default-config]/web-container/session-config,type=session-manager
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ServerProvider
amx:pp=/domain/configs/config[server-config]/thread-pools,type=thread-pool,name=http-thread-pool
amx:pp=/domain/configs/config[default-config]/admin-service,type=property,name=adminConsoleDownloadLocation
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=connectionAttributes
amx:pp=/domain/resources/jdbc-connection-pool[__TimerPool],type=property,name=connectionAttributes
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=pu-protocol
amx:pp=/domain/configs/config[server-config]/network-config,type=protocols
amx:pp=/ext,type=connector-runtime-api-provider
amx:pp=/domain,type=resources
jmxremote:type=jmx-connector-server,protocol=rmi_jrmp,name=system
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-2],type=ssl
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-2],type=http
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=signature.key.alias
amx:pp=/,type=ext
amx:pp=/domain/configs/config[default-config]/monitoring-service,type=module-monitoring-levels
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=ClientProvider
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=property,name=restAuthURL
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-1],type=http
amx:pp=/domain/configs/config[server-config]/jms-service,type=jms-host,name=default_JMS_host
amx:pp=/domain/configs/config[server-config]/web-container/session-config/session-manager,type=manager-properties
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/network-config/network-listeners,type=network-listener,name=admin-listener
amx:pp=/domain/configs/config[default-config]/security-service/jacc-provider[default],type=property,name=repository
amx:pp=/mon/server-mon[server],type=operating-system-mon,name=jvm/operating-system
amx:pp=/domain/configs/config[default-config]/security-service,type=audit-module,name=default
java.lang:type=MemoryPool,name=Survivor Space
amx:pp=/domain/configs/config[default-config]/network-config,type=protocols
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-2],type=ssl
amx:pp=/domain/configs/config[default-config]/thread-pools,type=thread-pool,name=http-thread-pool
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=jsp,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[pu-protocol],type=port-unification
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=security.config
amx:pp=/domain/configs/config[default-config],type=system-property,name=HTTP_SSL_LISTENER_PORT
amx:pp=/domain/configs/config[default-config]/http-service,type=virtual-server,name=server
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=request-policy
amx:pp=/domain/configs/config[server-config],type=iiop-service
amx:pp=/domain/configs/config[default-config]/security-service,type=jacc-provider,name=default
amx:pp=/domain/configs/config[default-config]/web-container/session-config/session-manager,type=manager-properties
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=request-policy
amx:pp=/domain/configs/config[default-config]/security-service,type=auth-realm,name=certificate
java.nio:type=BufferPool,name=mapped
java.lang:type=Compilation
amx:pp=/,type=runtime
amx:pp=/domain/configs/config[server-config]/monitoring-service,type=module-monitoring-levels
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=dynamic.username.password
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=encryption.key.alias
amx-support:type=amx-loader,name=config
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=dynamic.username.password
amx:pp=/runtime,type=server-runtime,name=server
amx:pp=/domain/configs/config[server-config]/network-config/protocols,type=protocol,name=http-listener-1
amx:pp=/domain,type=system-applications
amx:pp=/domain/configs/config[server-config]/security-service,type=jacc-provider,name=default
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__default]/JDBCDataSource[jdbc/__default],type=JDBCDriver,name=jdbc/__default,j2eeType=JDBCDriver,J2EEServer=server,JDBCResource=jdbc/__default,JDBCDataSource=jdbc/__default
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[file],type=property,name=jaas-context
amx:pp=/domain/configs/config[server-config]/ejb-container,type=ejb-timer-service
amx:pp=/domain/configs/config[default-config]/security-service,type=auth-realm,name=file
java.lang:type=Runtime
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=default,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[default-config],type=web-container
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=debug
amx:pp=/domain/system-applications/application[__admingui]/module[__admingui],type=engine,name=security
amx:pp=/domain/configs/config[server-config]/admin-service,type=jmx-connector,name=system
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=response-policy
amx:pp=/ext,type=realms
amx:pp=/,type=query
amx:pp=/domain/configs/config[default-config],type=thread-pools
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=property,name=loginErrorPage
amx:pp=/domain/configs/config[server-config],type=connector-service
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config]/network-config,type=network-listeners
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__TimerPool]/JDBCDataSource[jdbc/__TimerPool],type=JDBCDriver,name=jdbc/__TimerPool,j2eeType=JDBCDriver,J2EEServer=server,JDBCResource=jdbc/__TimerPool,JDBCDataSource=jdbc/__TimerPool
amx:pp=/domain/configs/config[default-config],type=system-property,name=JMS_PROVIDER_PORT
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=request-policy
amx:pp=/domain/configs/config[default-config],type=availability-service
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[admin-http-redirect],type=http-redirect
amx:pp=/,type=J2EEDomain,j2eeType=J2EEDomain,name=amx
amx:pp=/domain/configs/config[default-config],type=system-property,name=IIOP_SSL_LISTENER_PORT
amx:pp=/domain/configs/config[default-config]/iiop-service,type=iiop-listener,name=SSL
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config]/network-config/transports,type=transport,name=tcp
amx:pp=/domain/configs/config[default-config],type=system-property,name=JMX_SYSTEM_CONNECTOR_PORT
amx:pp=/mon/server-mon[server],type=garbage-collector-mon,name=jvm/garbage-collectors/Copy
amx:pp=/domain/configs/config[default-config]/http-service,type=virtual-server,name=__asadmin
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=request-policy
amx:pp=/,type=system-info
amx:pp=/domain/configs/config[default-config],type=group-management-service
amx:pp=/domain/configs/config[default-config]/availability-service,type=web-container-availability
amx:pp=/domain/configs/config[server-config]/group-management-service,type=failure-detection
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=response-policy
java.lang:type=GarbageCollector,name=Copy
amx:pp=/domain/configs/config[server-config],type=java-config
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=ServerProvider
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-2]/http,type=file-cache
amx:pp=/domain/configs/config[default-config],type=system-property,name=ASADMIN_LISTENER_PORT
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=request-policy
amx:pp=/domain/configs/config[server-config]/network-config/protocols,type=protocol,name=http-listener-2
amx:pp=/domain/configs/config[default-config]/jms-service,type=jms-host,name=default_JMS_host
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=ServerProvider
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=response-policy
amx:pp=/domain/configs/config[server-config],type=web-container
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[admin-realm],type=property,name=jaas-context
java.util.logging:type=Logging
amx:pp=/,type=tools
java.lang:type=MemoryPool,name=Eden Space
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=http-listener-2
com.sun.management:type=HotSpotDiagnostic
amx:pp=/domain/configs/config[default-config]/security-service,type=jacc-provider,name=simple
amx:pp=/domain/configs/config[default-config]/security-service,type=auth-realm,name=admin-realm
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=FacesServlet,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[server-config]/admin-service,type=property,name=adminConsoleContextRoot
amx:pp=/domain/configs/config[default-config],type=monitoring-service
amx:pp=/domain/configs/config[server-config]/security-service,type=auth-realm,name=file
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[admin-realm],type=property,name=jaas-context
amx:pp=/domain/configs/config[server-config]/security-service/audit-module[default],type=property,name=auditOn
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=response-policy
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[file],type=property,name=jaas-context
amx:pp=/domain/system-applications,type=application,name=__admingui
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=security.config
amx:pp=/domain/configs/config[default-config]/iiop-service/iiop-listener[SSL],type=ssl
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config],type=connector-service
amx:pp=/domain/configs/config[server-config],type=ejb-container
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=request-policy
amx:pp=/mon/server-mon[server],type=garbage-collector-mon,name=jvm/garbage-collectors/MarkSweepCompact
amx:pp=/domain/configs/config[default-config]/security-service/audit-module[default],type=property,name=auditOn
amx:pp=/domain,type=secure-admin
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=http-listener-1
amx:pp=/,type=sample
amx:pp=/domain/configs/config[default-config]/group-management-service,type=failure-detection
amx:pp=/domain/configs/config[default-config]/network-config/network-listeners,type=network-listener,name=http-listener-1
amx:pp=/domain/system-applications/application[__admingui],type=module,name=__admingui
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=ThemeServlet,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=request-policy
amx:pp=/domain/configs,type=config,name=default-config
amx:pp=/domain/system-applications/application[__admingui]/module[__admingui],type=engine,name=web
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=dynamic.username.password
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=security.config
java.nio:type=BufferPool,name=direct
amx:pp=/domain/configs/config[server-config]/network-config/network-listeners,type=network-listener,name=http-listener-2
amx:pp=/domain/configs/config[server-config]/iiop-service,type=orb
amx:pp=/domain/configs/config[default-config],type=java-config
amx:pp=/domain,type=property,name=administrative.domain.name
amx:pp=/domain,type=load-balancers
amx:pp=/domain/resources/jdbc-connection-pool[__TimerPool],type=property,name=databaseName
amx:pp=/domain/resources,type=jdbc-connection-pool,name=DerbyPool
amx:pp=/domain/configs/config[default-config]/thread-pools,type=thread-pool,name=thread-pool-1
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config],type=diagnostic-service
amx:pp=/domain/configs/config[default-config],type=security-service
amx:pp=/domain/configs/config[default-config],type=system-property,name=IIOP_LISTENER_PORT
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__TimerPool],type=JDBCDataSource,name=jdbc/__TimerPool,j2eeType=JDBCDataSource,J2EEServer=server,JDBCResource=jdbc/__TimerPool
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=PortNumber
amx:pp=/domain/configs/config[server-config],type=admin-service
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=DatabaseName
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[admin-listener]/http,type=file-cache
amx:pp=/domain/configs/config[server-config]/web-container/session-config,type=session-manager
java.lang:type=MemoryPool,name=Tenured Gen
amx:pp=/domain/configs/config[server-config]/thread-pools,type=thread-pool,name=admin-thread-pool
amx:pp=/domain/configs/config[server-config]/security-service,type=jacc-provider,name=simple
amx:pp=/domain/configs/config[server-config],type=monitoring-service
amx:pp=/domain/configs/config[server-config]/security-service,type=property,name=default-digest-algorithm
amx:pp=/domain/configs/config[default-config],type=network-config
amx:pp=/,type=mon
amx:pp=/domain/configs/config[default-config],type=mdb-container
amx-support:type=boot-amx
amx:pp=/domain/configs/config[default-config],type=transaction-service
amx:pp=/J2EEDomain/J2EEServer[server],type=JDBCResource,name=jdbc/__default,j2eeType=JDBCResource,J2EEServer=server
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[admin-listener]/http,type=file-cache
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[admin-listener],type=http
amx:pp=/domain,type=servers
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=admin-http-redirect
amx:pp=/domain/servers,type=server,name=server
amx:pp=/domain/configs/config[default-config]/availability-service,type=ejb-container-availability
amx:pp=/domain/configs/config[server-config]/admin-service,type=property,name=ipsRoot
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=response-policy
amx:pp=/domain/configs/config[server-config]/iiop-service,type=iiop-listener,name=orb-listener-1
java.lang:type=GarbageCollector,name=MarkSweepCompact
amx:pp=/domain/configs/config[server-config]/admin-service,type=das-config
amx:pp=/mon/server-mon[server],type=thread-system-mon,name=jvm/thread-system
amx:pp=/domain/resources,type=jdbc-resource,name=jdbc/__TimerPool
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__default],type=JDBCDataSource,name=jdbc/__default,j2eeType=JDBCDataSource,J2EEServer=server,JDBCResource=jdbc/__default
amx:pp=/domain/configs/config[server-config]/security-service/jacc-provider[default],type=property,name=repository
amx:pp=/domain/configs/config[server-config]/http-service,type=access-log
java.lang:type=ClassLoading
amx:pp=/domain/configs/config[server-config]/network-config/protocols,type=protocol,name=admin-listener
java.lang:type=Threading
amx-support:type=amx-loader,name=j2ee
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[admin-realm],type=property,name=file
amx:pp=/mon/server-mon[server],type=memory-mon,name=jvm/memory
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=admin-listener
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=debug
amx:pp=/domain/configs/config[server-config]/security-service,type=message-security-config,name=HttpServlet
amx:pp=/domain/configs/config[server-config]/web-container,type=session-config
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=User
amx:pp=/domain/configs/config[server-config]/thread-pools,type=thread-pool,name=thread-pool-1
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config]/http-service,type=access-log
amx:pp=/mon,type=server-mon,name=server
amx:pp=/domain/servers/server[server],type=resource-ref,name=jdbc/__default
amx:pp=/domain/configs/config[default-config],type=http-service
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=dynamic.username.password
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=Password
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[pu-protocol]/port-unification,type=protocol-finder,name=admin-http-redirect
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[file],type=property,name=file
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[admin-realm],type=property,name=file
amx:pp=/domain/configs/config[default-config]/iiop-service,type=orb
amx:pp=/domain/configs/config[default-config]/admin-service,type=das-config
amx:pp=/domain/configs,type=config,name=server-config
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=request-policy
amx:pp=/domain/configs/config[default-config],type=log-service
amx:pp=/domain/configs/config[default-config]/iiop-service,type=iiop-listener,name=SSL_MUTUALAUTH
amx:pp=/domain/configs/config[default-config],type=iiop-service
amx:pp=/domain/configs/config[server-config]/iiop-service,type=iiop-listener,name=SSL_MUTUALAUTH
amx:pp=/domain/configs/config[server-config]/security-service,type=auth-realm,name=certificate
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=ClientProvider
amx:pp=/domain/configs/config[server-config],type=security-service
amx:pp=/domain/configs/config[default-config]/security-service,type=message-security-config,name=SOAP
amx:pp=/mon/server-mon[server],type=runtime-mon,name=jvm/runtime
amx:pp=/domain,type=lb-configs
amx:pp=/J2EEDomain/J2EEServer[server],type=WebModule,name=__admingui,j2eeType=WebModule,J2EEServer=server,J2EEApplication=null
java.lang:type=MemoryPool,name=Perm Gen
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet],type=provider-config,name=GFConsoleAuthModule
amx:pp=/mon/server-mon[server],type=class-loading-system-mon,name=jvm/class-loading-system
java.lang:type=MemoryManager,name=CodeCacheManager
amx:pp=/domain/resources,type=jdbc-resource,name=jdbc/__default
amx-support:type=mbean-tracker
amx:pp=/domain/configs/config[server-config],type=network-config
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-2],type=http
amx:pp=/domain,type=configs
amx:pp=/domain/configs/config[server-config]/http-service,type=virtual-server,name=__asadmin
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ClientProvider
amx:pp=/domain/configs/config[default-config]/web-container/session-config,type=session-properties
amx-support:type=amx-loader,name=startup
amx:pp=/domain/configs/config[default-config],type=admin-service
amx:pp=/domain/configs/config[default-config]/network-config/network-listeners,type=network-listener,name=http-listener-2
amx:pp=/domain/configs/config[server-config],type=jms-service
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[sec-admin-listener]/http,type=file-cache
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[sec-admin-listener],type=ssl
amx:pp=/domain,type=clusters
amx:pp=/domain,type=nodes
amx:pp=/domain/configs/config[default-config]/thread-pools,type=thread-pool,name=admin-thread-pool
amx-support:type=AMXConfigLoader
amx:pp=/domain/configs/config[default-config]/ejb-container,type=ejb-timer-service
amx:pp=/domain/configs/config[default-config]/iiop-service/iiop-listener[SSL_MUTUALAUTH],type=ssl
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-2]/http,type=file-cache
amx:pp=/domain/resources,type=jdbc-connection-pool,name=__TimerPool
amx:pp=/domain/configs/config[default-config],type=jms-service
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=request-policy
amx:pp=/domain/configs/config[server-config]/network-config/network-listeners,type=network-listener,name=admin-listener
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config],type=system-property,name=JAVA_DEBUGGER_PORT
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[server-config]/network-config,type=network-listeners
amx:pp=/domain/secure-admin,type=secure-admin-principal,name="CN=localhost,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"
amx:pp=/domain/configs/config[server-config]/network-config/network-listeners,type=network-listener,name=http-listener-1
amx:pp=/domain/configs/config[server-config]/web-container/session-config,type=session-properties
amx:pp=/,type=bulk-access
amx:pp=/domain/configs/config[server-config],type=group-management-service
amx:pp=/domain/configs/config[default-config],type=management-rules
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=response-policy
amx:pp=/domain/configs/config[default-config]/availability-service,type=jms-availability
amx:pp=/domain/configs/config[default-config]/admin-service,type=jmx-connector,name=system
amx:pp=/domain/configs/config[default-config]/http-service/virtual-server[server],type=property,name=default-web-xml
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=response-policy
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-1]/http,type=file-cache
amx:pp=/domain/configs/config[server-config]/admin-service,type=property,name=adminConsoleDownloadLocation
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[sec-admin-listener],type=http
amx:pp=/domain/configs/config[server-config],type=transaction-service
java.lang:type=OperatingSystem
amx:pp=/J2EEDomain/J2EEServer[server],type=JVM,j2eeType=JVM,J2EEServer=server
amx:pp=/,type=pathnames
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=security.config
amx-support:type=amx-loader,name=ext
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ServerProvider
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=response-policy
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[pu-protocol]/port-unification,type=protocol-finder,name=http-finder
amx:pp=/J2EEDomain/J2EEServer[server],type=JDBCResource,name=jdbc/__TimerPool,j2eeType=JDBCResource,J2EEServer=server
amx:pp=/domain/configs/config[server-config],type=http-service
amx:pp=/,type=domain
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=encryption.key.alias
amx:pp=/domain,type=applications
amx:pp=/domain/configs/config[default-config],type=system-property,name=IIOP_SSL_MUTUALAUTH_PORT
amx:pp=/domain/configs/config[server-config]/iiop-service/iiop-listener[SSL],type=ssl
amx:pp=/J2EEDomain,type=J2EEServer,name=server,j2eeType=J2EEServer
amx:pp=,type=domain-root
JMImplementation:type=MBeanServerDelegate

As this image and the small-font text demonstrate, "booting AMX" has led to GlassFish exposing significant monitoring and management capabilities via JMX.

Conclusion

This post has demonstrated using Groovy to explore the standard JMX MBeans provided by the Java Platform as well as the AMX beans that GlassFish provides. The last Groovy code snippet invokes GlassFish's "bootAMX" operation to unleash significantly more GlassFish JMX-based monitoring and managing capabilities. Although all of this could be accomplished with a GUI-based JMX client such as JConsole or JVisualVM, there are advantages at times to running things in scripts. Using Groovy as demonstrated in this post allows scripts to easily interact with GlassFish for administration.

Multiple Methods for Monitoring and Managing GlassFish 3

GlassFish 3 supports multiple methods of monitoring and management. In this post, I look briefly at the approaches GlassFish provides for administration, monitoring, and management.

GlassFish Admin Console

GlassFish's web-based Admin Console GUI is probably the best-known interface for GlassFish administration. By default, it is accessed via the URL http://localhost:4848/ once GlassFish is running. The two screen snapshots below provide a taste of this approach, but I don't look any deeper at this option here as this is a fairly easy to understand interface that is fairly easy to learn and use once logged into the website.

 

GlassFish Admin Command Line Interface

The GlassFish Admin Console GUI offers advantages of a GUI such as ease of learning and using, but also comes with the drawbacks of a GUI (can take longer to get through the "overhead" of using the GUI approach for things that are easily done from the command line and does not work as well in scripts and headless environments). In some cases, a command-line approach is preferred and GlassFish supports command-line administration with the GlassFish Admin Command Line Interface.

Running asadmin start-domain is used to start a Domain in GlassFish. The command asadmin help can be used to learn more about the available commands. A very small snippet from the top of this help output is shown next:

Utility Commands                                      asadmin(1m)

NAME
     asadmin - utility for performing  administrative  tasks  for
     Oracle GlassFish Server

SYNOPSIS
     asadmin [--host host]
     [--port port]
     [--user admin-user]
     [--passwordfile filename]
     [--terse={true|false}]
     [--secure={false|true}]
     [--echo={true|false}]
     [--interactive={true|false}]
     [--help]
     [subcommand [options] [operands]]

DESCRIPTION
     Use the asadmin utility to perform administrative tasks  for
     Oracle GlassFish Server. You can use this utility instead of
     the Administration Console interface.

As the beginning of the asadmin help indicates, the asadmin utility is an alternative to the GUI-based "Administration Console interface."

There are numerous sub-commands available and some of those are listed here:

• list-applications to list deployed applications
• deploy and other deployment subcommands
• version to see version of GlassFish (shown in the screen snapshot below)
• list-commands (lists available commands) [portion of output shown in the screen snapshot below]

Additional information regarding the GlassFish Admin Command Line Interface is available in Learning GlassFish v3 Command Line Administration Interface (CLI).

 

GlassFish JMX/AMX

The two approaches shown in this post so far for monitoring and managing GlassFish (web-based Admin Console GUI and GlassFish Admin Command Line Interface) are specific to GlassFish. GlassFish also supports monitoring and management via Java Management Extensions (JMX), including JSR 77 ("J2EE Management") as I have blogged about before in my post Simple Remote JMX with GlassFish.

Because GlassFish supports a JMX interface, it can be easily monitored and managed with readily available tools such as JConsole and JVisualVM. Besides the MBeans that GlassFish exposes itself, the JVM has built-in MBeans since J2SE 5 that can be monitored in relation to the hosted GlassFish instances as well.

The next set of images demonstrates using JConsole to view MBeans exposed via GlassFish and the JVM. The first image shows the standard JVM Platform MBeans available and the images following that one show GlassFish-specific MBeans including the amx-support and jmxremote domains. When the bootAMX operation of the boot-amx MBean (amx-support domain) is clicked on that latter MBean, the full complement of AMX MBeans is available as shown in the remainder of the images.

 

GlassFish REST

The Oracle GlassFish Server 3.1 Administration Guide includes a section called "Using REST Interfaces to Administer GlassFish Server" that states that the "GlassFish Server provides representational state transfer (REST) interfaces to enable you to access monitoring and configuration data for GlassFish Server." It goes on to suggest that clients applications such as web browsers, cURL, and GNU Wget can be used to interact with GlassFish via the Jersey-based REST interfaces. Of course, as this page also points out, any tool written in any language that handles REST-based interfaces can be used in conjunction with GlassFish's REST support.

Not surprisingly, the GlassFish REST APIs are exposed via URLs over HTTP. The previously cited Admin Guide states that configuration/management operations are accessed via URLs of form http://host:port/management/domain/path and monitoring operations are accessed via URLs of form http://host:port/monitoring/domain/path.

One of the easiest ways to use GlassFish's REST interfaces is via web browser using the URLs mentioned earlier (http://localhost:4848/management/domain/ and http://localhost:4848/monitoring/domain/ for example). The next three screen snapshots attempt to give a taste of this style of access. The middle image shows that the monitoring needs to be enabled in GlassFish.

Using a web browser to interact with GlassFish for management and monitoring is easy, but this can be done with the Web Admin Console I covered at the beginning of this blog post. The real advantage of the REST-based interface is the ability to call it from other client tools, especially custom-built tools and scripts. For example, one can write scripts in Groovy, Python, Ruby, and other scripting languages to interact with GlassFish. Like GlassFish's JMX-exposed APIs, GlassFish's REST-exposed APIs allow custom scripts and tools to be used or even written to manage and monitor GlassFish.

Jason Lee has posted several posts on using GlassFish's REST management/monitoring APIs such as RESTful GlassFish Monitoring, Deploying Applications to GlassFish Using curl, and GlassFish Administration: The REST of the Story.

Ant Tasks

GlassFish provides several Ant tasks that allow Ant to be used for starting and stopping the GlassFish server, for deploying applications, and for performing other management tasks. A StackOverflow Thread covers this approach. The next two screen snapshots demonstrate using the GlassFish Web Admin Console's Update Tool -> Available Add-Ons feature to select the Ant Tasks for installation and the contents of the ant-tasks.jar that is made available upon this selection.

With the ant-tasks.jar JAR available, it can be placed on the Ant build's classpath to script certain GlassFish actions via an Ant build.

Conclusion

The ability to manage and monitor an application server is one of its highly important features. This post has looked at several of the most common methods GlassFish supports for its management, monitoring, and general administration.

The Big Java News Keeps Coming: Java SE 6 and OpenJDK

I recently posted Recent Java 8 News, summarizing some of the recent developments in Java 8. In less than a week since publishing that post, several more significant Java news stories have come out and I summarize some of these in this post. Common themes include different Java SE 6 implementations and OpenJDK.

Java Security Issues Impact End-of-Life for Oracle Java SE 6 Implementation

In February 2012, Oracle extended the end-of-life for its Java SE 6 implementation from July 2012 to November 2012. This was later changed to February 2013 as shown in the Java SE EOL Policy under the section "Java SE 6 End of Public Updates Notice": "After February 2013, Oracle will no longer post updates of Java SE 6 to its public download sites." Plans were slightly changed, however, due to discovered security vulnerabilities that led to two more Critical Patch Updates (CPUs): Java SE 6 Update 41 and Java SE 6 Update 43. Note that the odd numbers in the releases (41 and 43) indicate CPUs rather than limited updates. Oracle encourage developers to move to Java 7 for future updates.

RedHat to Lead OpenJDK 6

It was announced this week that RedHat will take over leadership of the OpenJDK 6 project. As stated in the Red Hat press release, this move "effectively [extends] support for [OpenJDK 6] and its users." This is especially significant, of course, given the imminent end-of-life of the Java SE 6 implementation provided by Oracle.

Java on iOS

I recently posted on the announcement of plans to put JavaFX on iOS and Android. Now, there's talk via an OpenJDK JDK Enhancement Proposal (JEP #178 - "Statically-Linked JNI Libraries") of "[enhancing] the JNI specification to support statically-linked native libraries." This becomes even more interesting than it may initially sound when one considers implications such as the potential for this to allow Java applications to be deployed to iOS.

Complimentary (Free!) Admission to JAXConf US

It was announced this week that JAXConf US 2013 is open without charge to attendees to attend two days of keynotes and conference sessions (Tutorials day is $299). The conference is being held in Santa Clara, California, in early June (June 4-5).

Other Interesting and Recent Java-Related Posts

Although not necessarily news, there are other recent Java posts of general interest. Java Version History outlines some of the features introduced with each major version of Java/JDK. Geert Bevin uses the post My three-year Java abstinence: there and back again to look at "three reasons why I came back to Java and the JVM": community, tools and technology, JVM (and Java). Another recent post that was surprisingly interesting is Edmund Kirwan's JUnit's evolving structure that looks at how JUnit (in particular version 4.3) delved into a convoluted package structure that is difficult to visualize. That post references radial encapsulation as one way to avoid this.