Friday, June 27, 2008

Remote JMX Exceptions

Perhaps the most difficult aspect of getting a freshly written JSR-160 (Remote JMX) compliant application working is ensuring a proper JMXServiceURL. In this blog entry, I cover some of the most common JMXServiceURL-related exceptions one may run into when developing the client and server portions of a management and monitoring system using Remote JMX.

The JMX 1.4 Specification combines the original JMX specification with the Remote JMX specification in a single document (the JMX Remote API Specification is part III of this document). This third part covers the required RMI connector as well as the optional JMXMP connector. In this blog entry, I'll be focusing on using the RMI connector with Java SE 6 HotSpot platform MBean server. Instead of using JConsole as a client, I'll use a simple custom client to illustrate some of the exceptions one may encounter when writing one's own client.

I am using essentially the same code in this blog for examples that I previously used in the previous blog entry Remote JMX: With and Without Spring and Proxies. I will not reproduce the code here, but it is sufficient to know that the JMX Connector server code generally establishes its JMXServiceURL as service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi. Note that the protocol is RMI, the host is localhost, the port is 1099, and the URL string is "jmxrmi". Because the JMX client needs to use the same JMXServiceURL as provided by the server, I appreciate it when third-party JMX servers provide this to me. For example, both GlassFish and the JMX Web Services Connector Reference Implementation sample provide their JMXServiceURL to the console when they are executed.

When the JMXServiceURL is known, it is relatively straightforward to write a client. The most significant lines of code for doing this are extracted from the code in the previously referenced blog and displayed next:


final String objectNameStr = "dustin:type=status,name=remoteJMX";
final String jmxRmiStr =
"service:jmx:rmi://localhost/jndi/rmi://localhost:1099/jmxrmi";
try
{
final ObjectName objectName = new ObjectName(objectNameStr);
final JMXServiceURL jmxUrl = new JMXServiceURL(jmxRmiStr);
final JMXConnector jmxConnector = JMXConnectorFactory.connect(jmxUrl);
final MBeanServerConnection mbsc = jmxConnector.getMBeanServerConnection();


As the code listing above demonstrates, it is fairly easy (and mostly boilerplate) to connect clients to a JMX Connector Server with a well-known JMXServiceURL. However, it is not uncommon for new JMX developers to struggle to get the JMXServiceURL appropriately set. In the remainder of this blog entry, I now look at some of the most common JMXServiceURL-related exceptions one may run into.


ERROR: IOException trying to connect to JMX Connector Server: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused: connect]

A possible cause of this exception includes no JMX Connector Server running on the specified host and port combination.

Also, ensure that an RMI registry has been started at the specified port.



ERROR: IOException trying to connect to JMX Connector Server: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmx-rmi

This NameNotFoundException was a result of inadvertently mistyping the client's String URL portion (jmx-rmi specified by the client rather than jmxrmi without the hyphen). This is different from the above error because a JMX Connector Server is at the specified host and port, but with a different String portion. Because multiple JMX Connector Servers can use the same host/port combination as long as they have unique String portions of their URLs, it is not surprising that no assumptions can be made about what String was intended.



Exception in thread "main" java.lang.ClassCastException: com.sun.jndi.rmi.registry.RegistryContext cannot be cast to javax.management.remote.rmi.RMIServer

The previous exception demonstrated the result of attempting to access a JMX Connector Server from a client with the correct host/port comibination, but with an incorrect String URL portion. This ClassCastException is the result of not specifying any String URL portion at all in the JMXServiceURL. In other words, the jmxrmi portion was left off the end of the client's JMXServiceURL.




java.io.IOException: Cannot bind to URL [rmi://localhost:1099/jmxrmi]: javax.naming.NameAlreadyBoundException: jmxrmi [Root exception is java.rmi.AlreadyBoundException: jmxrmi]

Until now, all of the previously covered exceptions resulted from a JMX Client using a mistyped or otherwise incorrect JMXServiceURL that did not match that used by the JMX Connector Server. This exception, however, is a JMX Connector Server exception that results from two different applications trying to use the same JMXServiceURL on the same host/port with the same URL String.



ERROR: Problem with JMXServiceURL based on rmi:///jndi/rmi://localhost:1099/jmxrmi: Service URL must start with service:jmx:

This error message is particularly descriptive and occurs when the JMXServiceURL does not begin with the specification-required service:jmx: portion. See Section 13.8 ("Connector Server Addresses") for additional details on all JMXServiceURLs beginning with this portion.



ERROR: Problem with JMXServiceURL based on service:jmx::///jndi/rmi://localhost:1099/jmxrmi: Missing or invalid protocol name: ""

As the error message indicates, the protocol is missing from the JMXServiceURL, which should be service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi, but the portion in red was omitted.



ERROR: Problem with JMXServiceURL based on service:jmx:nothing:///jndi/rmi://localhost:1099/jmxrmi: Unsupported protocol: nothing

Whereas the previous error message indicated a missing protocol designation, this error indicates an erroneous protocol specification. In this particular case, the incorrect JMXServiceURL used was service:jmx:nothing:///jndi/rmi://localhost:1099/jmxrmi where the "nothing" in red should have been "rmi".



ERROR: IOException trying to connect to JMX Connector Server: Failed to retrieve RMIServer stub: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial

This error message is displayed for the case where the second protocol listing in the JMXServiceURL is either omitted or has an incorrect value. In other words, if the highlighted portion of the correct JMXServiceURL (service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi)was omitted entirely or had something unexpected like "nothing," this error message would occur.


Other Remote JMX Exceptions/Errors Considerations

There are, of course, several other types of exceptions one can run into when remotely accessing a JMX Connector Server. These exceptions, which have not been covered here, are those that are thrown when the connection is successfully made, but the sought-after MBean cannot be found or some other problem occurs. The JMX Best Practices document outlines recommendations related to MBeans throwing standard exceptions and also recommends having MBeans intended for remote management/monitoring implement interfaces with operations that use the throws IOException clause (more granular for client than available as UndeclaredThrowableException).


JMX Exceptions in General

The focus of this blog entry has been on common exceptions and errors encountered when running a custom JMX Connector Client with a custom JMX Connector Server. Note that JMX spells out its own exception hierarchy related to the JMX Agent. Section 6.4 (JMX Exceptions) of the JMX 1.4 Specification is devoted to these JMX exceptions. The section talks about the JMException, which is the main class and parent of all descendant exception classes (exception runtime exceptions) thrown by a JMX agent implementation. The JMX runtime exceptions are represented by JMRuntimeException and its descendant exceptions.

1 comment:

Luli said...

Hi. I know your post is about using your own client but I can't make it worked with JConsole so I was hoping you could help me.

I need to monitor a weblogic server running on a remote machine using JConsole. Just for now, I'm testing using two machines with Ubuntu OS, but the real server uses Solaris. So, I'm using my pc as the server and a partner's pc to connect remotely using JConsole.

Environment: the server runningis WebLogic 10.3; the server is running with Java Sun JDK 1.6.0_26 and the client is running with Java Sun JDK 1.6.0_20

I'm able to connect Locally, running JConsole and clicking on the weblogic process.

The application deployment is pretty big, it runs a few scripts. When the domain of weblogic is created I set the following properties:

java -classpath ${CLASSPATH} ${MY_OPTS} -Dweblogic.management.username=${ADMIN_USER} \
-Dweblogic.management.password=${ADMIN_PASSWD} -Dweblogic.Domain=${MY_DOMAIN} \
-Dweblogic.Name=${MY_SERVER} -Dweblogic.ListenPort=${MY_PORT} \
-Dweblogic.RootDirectory=${MY_DOMAIN_HOME}/${MY_DOMAIN} \
-Djava.rmi.server.hostname=${ADMIN_HOST} \
-Dcom.sun.management.jmxremote.port=22222 \
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.password.file=/home/MY_USER/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/home/MY_USER/jmxremote.access \
-Dweblogic.management.GenerateDefaultConfig=true weblogic.Server &

Then, I run jconsole -debug from client's console and try to connect with the remote option using:

service:jmx:rmi:///jndi/rmi://:22222/jmxrmi

ADMIN_HOST = the server_ip, that is 172.17.209.66

And I get the following error:

22/11/2011 05:22:54 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://172.17.209.66:22222/jmxrmi] connecting...
22/11/2011 05:22:54 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://172.17.209.66:22222/jmxrmi] finding stub...
22/11/2011 05:22:54 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://172.17.209.66:22222/jmxrmi] Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 172.17.209.66; nested exception is:
java.net.ConnectException: Conexión rehusada]
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 172.17.209.65; nested exception is:
java.net.ConnectException: Conexión rehusada]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:340)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)

And in the logs of my application I see the following:

Conector JMX ready in: service:jmx:rmi:///jndi/rmi://usuario-System-Product-Name:22222/jmxrmi

The /etc/hosts shows:

172.17.209.65 usuario-System-Product-Name # Added by NetworkManager 127.0.0.1 localhost.localdomain localhost ::1 usuario-System-Product-Name localhost6.localdomain6 localhost6 127.0.1.1 usuario-System-Product-Name

is that why I see the usuario-System-Product-Name and not the ip? Even if I set as hostname the ip of my computer?

I disabled both firewals and try the command line jmx tool Alex suggested buy with no luck :-(

Any other suggestions / tips / help ????