Thoughts on software development and other stuff

Archive for February, 2009

Java Posse Roundup (or so I’m told)

without comments

Earlier today I was pinged by Mike Levin; he wanted to pass along the message to all of my fiercely loyal followers (yes, both of you) that he’s out in Crested Butte enjoying the mountains, skiing, and the up coming Java Posse Roundup…and that you’re probably not. Of course it isn’t intended to make you jealous, but it looks like this is a pretty cool conference (small size, interesting topics, etc.) I hope that I can go one of these days! I went to their JavaOne BOF last year and had a great time (unfortunately I had to leave early to catch my flight home.) For those of you that don’t know the Java Posse, it is a podcast that covers everything that you need to know in the Java world. And it has a cool jingle. So that I can make this post relevant to Coherence, episode 7 is an interview with our dear leader.

Written by Patrick Peralta

February 24th, 2009 at 11:13 pm

Firing up the JMX Reporter in Coherence 3.4

without comments

Lately I have been helping customers set up the Coherence JMX Reporter. This is a new feature in 3.4 that will output statistics for the cluster to a CSV file that can be loaded into Excel for analysis. This feature can be enabled or disabled (it is disabled by default) in a running 3.4 cluster that has JMX for the cluster already enabled.

First, log into JConsole and take a look at the attributes for the Reporter MBean. The ConfigFile property points to an XML file that contains a list of reports to be executed (each report is also defined by an XML file.) The default value is reports/report-group.xml, however if you want to sample all of the OOTB reports, change this value to reports/report-all.xml. Alternatively, you can define your own reports based on the JMX information that you’d like to keep track of. These reports ship inside of coherence.jar.

After selecting the reports to run and (optionally) the IntervalSeconds (60 by default) and the OutputPath (defaults to the directory the JVM process is running in), flip over to the Reporter operations and select “start.”

Note: if using JConsole that ships with 1.6, you may have to specify that the IntervalSeconds is a long type. For instance, to specify a 30 second interval, enter “30L” without the quotes into the field. This is due to a bug in JConsole.

After starting up the reporter, you should see a few .txt files generated in the OutputPath. These reports are described in detail in the user guide. I opened up two of them (Cache Size and Memory Status and took a few screenshots of these files in Excel:

This graph is demonstrating the total size of the data in a cache.

The light blue line is showing the amount of heap allocated, and the red is showing the amount of heap being used.

Written by Patrick Peralta

February 19th, 2009 at 11:56 am

How to make your Coherence objects forward and backward compatible with POF

with 10 comments

One of the major challenges in distributed computing is the maintenance of the application. You need the ability to make changes to the system without breaking the current users of the application. Anyone who uses Java serialization knows that this is a difficult task, especially since the use of java.io.Serializable results in a brittle binary format.

In a previous life when I was the designer and developer of a services-style distributed system, I took a brute force approach. I determined that each version of a service would include a set of serializable objects and service interfaces. If any of the serializable objects needed to change, that would trigger a new version of the services. This meant having to keep several versions running at a time, one to support new clients, and the older ones to support older clients.

I was able to get away with this because the objects in the system didn’t live for very long. The life cycle of an object tended to start either in the DAO layer (Hibernate) and end up being serialized to a client, or a client would send a serialized object to a service which would end up being written to a database using Hibernate. Upon completion of the transaction, there were no longer any references to these objects (L2 caching was not enabled.) As a result, running multiple copies of the application in a single container was feasible as the memory requirements didn’t exceed the capacity of the heap.

An object in a typical Coherence system has a much longer life cycle; many times the objects themselves will need to outlive the specific version of the application that is running. Coherence allows you to do this by performing a rolling restart. This means that instead of taking down the entire cluster to perform an upgrade, you can take down a node, update its classes (and maybe configuration, depending on what is being modified) and restart it. This process is repeated until all nodes have been updated.

However if the objects in the cache have a different serialization format than the updated version of their classes, you can run into problems when attempting to deserialize a new data format with an old class. Consider the following scenario:

Node 1 is shut down and updated with a new version of class A. This new version of class A holds a reference to class B, which is a new class. Upon restart, node 1 begins to insert new instances of class A that contain class B. In the meantime, node 2 has not been upgraded yet and it makes a request for an instance of class A that has been inserted by node 1. Since it doesn’t know anything about class B, it throws an exception while deserializing.

The good news is that Coherence makes this problem easy to solve. Using Portable Object Format (POF), you can make your cached objects forward and backward compatible in order to deal with the scenario above.

Before Coherence 3.4, POF was used exclusively for Coherence*Extend, in particular for .NET clients. Starting with Coherence 3.4, POF is natively supported inside of the cluster and is the preferred serialization format for Coherence.

In order to accomplish this, you must implement the EvolvablePortableObject interface.

  • This means that you must write out the serialization routines (readExternal and writeExternal). Although somewhat tedious, this will pay off as POF serialized objects are smaller than their Serializable counterparts, and take less memory and CPU to serialize.
  • EvolvablePortableObject extends the Evolvable interface, so these methods must also be implemented. The easiest way to do this is to extend AbstractEvolvable. The JavaDoc for Evolvable goes into detail about how it works, but the basics are that by implementing this interface, Coherence will automatically resolve differences between class and data versions.

Here is an example implementation:

import com.tangosol.io.AbstractEvolvable;
import com.tangosol.io.pof.EvolvablePortableObject;
import com.tangosol.io.pof.PofReader;
import com.tangosol.io.pof.PofWriter;
import java.io.IOException;
public class Person
        extends AbstractEvolvable
        implements EvolvablePortableObject
    // ----- constructors ---------------------------------------------------
    public Person()
    // ----- accessors ------------------------------------------------------
    public long getId()
        return m_id;
    // ----- Evolvable interface ---------------------------------------
    public int getImplVersion()
        return VERSION;
    // ----- PortableObject interface ---------------------------------------
    public void readExternal(PofReader in)
            throws IOException
        m_id         = in.readLong(0);
        m_sFirstName = in.readString(1);
        m_sLastName  = in.readString(2);
        m_sEmail     = in.readString(3);
    public void writeExternal(PofWriter out)
            throws IOException
        out.writeLong(0, m_id);
        out.writeString(1, m_sFirstName);
        out.writeString(2, m_sLastName);
        out.writeString(3, m_sEmail);
    // ----- fields ---------------------------------------------------------
    private long   m_id;
    private String m_sFirstName;
    private String m_sLastName;
    private String m_sEmail;
    public static final int VERSION = 1;

A POF configuration for such an object may look like this:

<!DOCTYPE pof-config SYSTEM "pof-config.dtd">

To configure Coherence to use POF, the easiest method is to configure the following system properties in your startup script:


Now let’s say you have a running system and you want to add a new Address field to the Person class. The following modifications can be made to Person:

    // ----- PortableObject interface ---------------------------------------
    public void readExternal(PofReader in)
            throws IOException
        m_id         = in.readLong(0);
        m_sFirstName = in.readString(1);
        m_sLastName  = in.readString(2);
        m_sEmail     = in.readString(3);
        m_address    = (Address) in.readObject(4); // <- new Address field
    public void writeExternal(PofWriter out)
            throws IOException
        out.writeLong(0, m_id);
        out.writeString(1, m_sFirstName);
        out.writeString(2, m_sLastName);
        out.writeString(3, m_sEmail);
        out.writeObject(4, m_address);  // <- new Address field
    // ----- fields ---------------------------------------------------------
    private long   m_id;
    private String m_sFirstName;
    private String m_sLastName;
    private String m_sEmail;
    private Address m_address;  // <- new Address field
    public static final int VERSION = 2; // <- new version

And this addition to the pof-config.xml file:


There are the changes that I made:

1. Added the Address field as a member of the Person class and to the serialization routines (as index 4)
2. Incremented the version
3. Added the Address class to the POF configuration

That’s it! Now clients with an old version of the Person class can load new versions of Person data, modify it, and place it back into the cache while preserving all of the new data (for a hint on how this is done, take a look at the futureData property defined by Evolvable. The obvious caveat here is that new fields must always be added to a class, and the order must be preserved.

Written by Patrick Peralta

February 16th, 2009 at 1:30 pm

JConsole, let me introduce you to SwingWorker

without comments

Today when I was using JConsole I noticed that when I tried to connect to a process the UI was hanging. The application that it was trying to connect to was not in a good state, but I was surprised at the lack of a responsive UI. Curious, I captured a thread dump and saw this gem:

"AWT-EventQueue-0" prio=6 tid=0x0100cd00 nid=0x899e00 runnable [0xb0d13000..0xb0d14d90]
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        - locked <0x2559e880> (a java.net.SocksSocketImpl)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
        at java.net.Socket.connect(Socket.java:520)
        at java.net.Socket.connect(Socket.java:470)
        at java.net.Socket.(Socket.java:367)
        at java.net.Socket.(Socket.java:180)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:569)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:306)
        at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
        at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
        at sun.rmi.transport.DGCClient$EndpointEntry.registerRefs(DGCClient.java:275)
        at sun.rmi.transport.DGCClient.registerRefs(DGCClient.java:112)
        at sun.rmi.transport.LiveRef.read(LiveRef.java:277)
        at sun.rmi.server.UnicastRef2.readExternal(UnicastRef2.java:54)
        at java.rmi.server.RemoteObject.readObject(RemoteObject.java:438)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:946)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1809)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJRMP(RMIConnector.java:1871)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1789)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:259)
        - locked <0x25600378> (a javax.management.remote.rmi.RMIConnector)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)
        at sun.tools.jconsole.ProxyClient.(ProxyClient.java:156)
        at sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:57)
        at sun.tools.jconsole.ConnectDialog.actionPerformed(ConnectDialog.java:342)
        at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1882)
        at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2202)
        at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:420)
        at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:258)
        at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
        at java.awt.Component.processMouseEvent(Component.java:5602)
        at javax.swing.JComponent.processMouseEvent(JComponent.java:3135)
        at java.awt.Component.processEvent(Component.java:5367)
        at java.awt.Container.processEvent(Container.java:2010)
        at java.awt.Component.dispatchEventImpl(Component.java:4068)
        at java.awt.Container.dispatchEventImpl(Container.java:2068)
        at java.awt.Component.dispatchEvent(Component.java:3903)
        at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4256)
        at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3936)
        at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3866)
        at java.awt.Container.dispatchEventImpl(Container.java:2054)
        at java.awt.Window.dispatchEventImpl(Window.java:1801)
        at java.awt.Component.dispatchEvent(Component.java:3903)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:463)
        at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:180)
        at java.awt.Dialog$1.run(Dialog.java:535)
        at java.awt.Dialog$2.run(Dialog.java:565)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.awt.Dialog.show(Dialog.java:563)
        at java.awt.Component.show(Component.java:1302)
        at java.awt.Component.setVisible(Component.java:1255)
        at sun.tools.jconsole.ConnectDialog.setVisible(ConnectDialog.java:415)
        at sun.tools.jconsole.JConsole.showConnectDialog(JConsole.java:583)
        at sun.tools.jconsole.JConsole.access$100(JConsole.java:34)
        at sun.tools.jconsole.JConsole$4.run(JConsole.java:702)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:461)
        at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:184)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:176)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

Every Swing programmer that has gotten past Hello World knows that this is a no-no. AWT and Swing components are rendered by a single thread, and if that thread gets blocked then you end up with an unresponsive UI. Clearly there should never be any network code in the stack trace of the AWT event thread. This is on the 1.5 VM on OS X so I don’t know if Sun or Apple is to blame (I assume Sun), but if this isn’t fixed yet in 1.6 then they should review the Swing Tutorial:

Tasks on the event dispatch thread must finish quickly; if they don’t, unhandled events back up and the user interface becomes unresponsive.

Written by Patrick Peralta

February 10th, 2009 at 11:29 pm

Posted in Development

Ode to the 350

with one comment

O’ 350, highlight of my day
how you bring me unspeakable pain
you arrive early when I’m on time
right on time to watch you pass by
when I’m early you’re 10 minutes late
relying on you is such a cruel fate

you make the hot days hotter
and the cold days colder
this is a burden
that is too much to shoulder

some of your drivers
are certifiably insane
making NY cab drivers blush
when they see your road rage

you make me navigate through
six foot snow banks
then break down on the way
as I watch my day tank

during rush hour
you might pass by twice
while eight 77‘s
clog Mass Ave, how nice

so why do I bother
why do I try
when efforts so futile
make a grown man cry

the answer is simple
simple as can be
you’re a cheap way to get around
$1.25 is quite a low fee

so thus I am destined
to rely on mbta
to get me to work
sometime today

now I don’t mean to cuss
or complain and fuss
but can someone please tell me
where the fuck’s my fucking bus?

Edit: Maria suggested I link the last line to the video that inspired it.

Written by Patrick Peralta

February 5th, 2009 at 11:02 am

Posted in General