Management and Monitoring
Because Prudence runs on the JVM, you get the benefit of its built-in support for remote management and monitoring, and array of tools such as VisualVM and JConsole. These are essential tools for administration of production deployments. There is nothing unique in how Prudence uses these tools, but this section is included to help you get started with them.
The JVM is an impressive piece of engineering, balancing code execution with memory management for optimal performance under diverse circumstances, and you get to appreciate it all that more through the lens of management and monitoring. Even if you don't need to tightly manage a deployment, you should give monitoring a spin, if only to better understand your powerful deployment platform.
JMX and VisualVM
The JVM's native management and monitoring standard is JMX (Java Management Extensions), which in turn uses RMI (Remote Method Invocation) as its network protocol. JMX lets applications and systems expose management "beans," which are simply objects with attributes and methods.
An example of a management bean is a thread pool: attributes can show you how many threads are used by the thread pool at any given time, and perhaps the number of threads in the pool can be changed live.
The "monitoring" part involves watching these attributes as they change over time. A good monitoring client can accumulate this data, display it in graphs, and trigger alerts for system administrators when bad things happen that would require human attention. "Management" involves calling methods on these beans, but also changing some attributes for beans that support it. JMX is, in essense, an API.
Most JMX clients will give you raw access to this API, but the real power comes with dedicated plugins for specific beans, which give you a specialized UI and tracking tools.
VisualVM is a good desktop client that supports many plugins that can easily be downloaded and installed from a central repository, and also supports the older JConsole plugins via a compatibility plugin. VisualVM is included with Oracle's JDK distribution, and may already be installed on your development machine. (The executable is sometimes called "visualvm" and sometimes called "jvisualvm.")
Fire it up! JMX is enabled by default for local connections, so you should see your local Prudence instance under the "Local" branch of connections, as "com.threecrickets.scripturian.Scripturian" (which is the name of the entry point JVM class). You'll even see VisualVM itself there, and any other JVMs running on your machine. Have fun exploring them! Try installing various VisualVM plugins and see what new options they give you.
You'll various management beans: the default ones that come with the JVM, beans exposed by your daemon wrapper (such as JSW), beans exposed by your database driver, etc.
Note that Hazelcast's management bean is not enabled by default. To enable it, add this option to your JVM:
-Dhazelcast.jmx=true
See Prudence clusters for more information on how Hazelcast is used by Prudence.
Remote JMX
More useful, though, is the ability to monitor and manage your remote servers.
Enabling remote JMX requires command line arguments to the JVM. Depending on how you run Prudence, you can set these in your "/bin/run.sh" or "/bin/run.bat," or in your "/configuration/wrapper.conf" if you're using JSW.
A basic setup for remote JMX would look something like this:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1650 -Djava.rmi.server.hostname=myhost.org
Setting the hostname may not be required if your operating system has a well-defined public hostname set up, but is often necessary. If it's incorrect, RMI's two-way communication cannot work.
In VisualVM, you will need to explicitly add your connection under the "Remote" branch. Make sure to add the JMX port you configured.
Securing JMX
If you've enabled remote JMX, you obviously do not want anybody to be able to access it. There are two things you can do to secure the connection.
The first is to require login credentials:
-Dcom.sun.management.jmxremote.password.file=/path/jmxremote.password
The password file is a simple text file where every row represents a user, and is made of user+whitespace+password:
monitorRole thepassword controlRole theotherpassword
These two users are by default given full read-write and full read-only access respectively. Refer to the JMX documentation if you need to create more users or more subtle security roles.
After you've enabled password security, you will need to enter the login credentials with VisualVM in order to connect.
The second way to secure your connection is to encrypt it:
-Djavax.net.ssl.keyStore=/path/my.jks -Djavax.net.ssl.keyStorePassword=mypassword
The key store is in the JVM's standard format. Instructions for managing your key store go beyond the scope of this guide, but there is a somewhat elaborated example in the section on secure servers. Note that if you do not need public https access, you may be better off using a self-signed key instead of one issued by a trusted certificate authority.
To allow VisualVM to connect to your server via encryption, you will need to install the "VisualVM-Security" plugin. Once you do that, you will have a new "Security" tab under Tools/Options. You want to set the "Trust Store" (not the "Key Store"!) to either the same key store as your server, or to a key store that contains the right key.
JMX and Firewalls
If you've enabled a firewall on your server (and you should!), you will have to open two ports for JMX to work: one for JMX (see "enabling JMX," above), and one for the RMI registry.
Unfortunately, mosts JVMs (including Oracle's) do not allow you to configure the RMI registry port. Much worse for this use case, the RMI registry port is selected at random during JVM startup. This means that you cannot configure your firewall for it!
Prudence comes with a workaround for this sticky problem, as suggested by Daniel Fuchs and implemented by Oleg Ilyenko. It involves setting a different, "firewall-friendly" agent for the JVM that can use a set port:
-javaagent:libraries/org.am.rmi.firewall.jar -Dorg.am.rmi.port=1651
So, following our example here, you would need to open ports 1650 and 1651 in your firewall for remote JMX to work.
Using the agent also requires a special connection in VisualVM, in which the RMI port is explicitly stated. Instead of adding a remote connection, choose File/Add JMX Connection, and enter a URI in this format:
service:jmx:rmi://myhost.org:1651/jndi/rmi://myhost.org:1651/jmxrmi
Note that only the RMI registry port is used in the above URI: information about the management beans, including which port to use to access them, will be read from the RMI registry.
