~pperalta

One of the major challenges in distributed computing is the maintenance of the application. You need the ability to make changes to the system without breaking the current users of the application. Anyone who uses Java serialization knows that this is a difficult task, especially since the use of java.io.Serializable results in a brittle binary format.

In a previous life when I was the designer and developer of a services-style distributed system, I took a brute force approach. I determined that each version of a service would include a set of serializable objects and service interfaces. If any of the serializable objects needed to change, that would trigger a new version of the services. This meant having to keep several versions running at a time, one to support new clients, and the older ones to support older clients.

I was able to get away with this because the objects in the system didn’t live for very long. The life cycle of an object tended to start either in the DAO layer (Hibernate) and end up being serialized to a client, or a client would send a serialized object to a service which would end up being written to a database using Hibernate. Upon completion of the transaction, there were no longer any references to these objects (L2 caching was not enabled.) As a result, running multiple copies of the application in a single container was feasible as the memory requirements didn’t exceed the capacity of the heap.

An object in a typical Coherence system has a much longer life cycle; many times the objects themselves will need to outlive the specific version of the application that is running. Coherence allows you to do this by performing a rolling restart. This means that instead of taking down the entire cluster to perform an upgrade, you can take down a node, update its classes (and maybe configuration, depending on what is being modified) and restart it. This process is repeated until all nodes have been updated.

However if the objects in the cache have a different serialization format than the updated version of their classes, you can run into problems when attempting to deserialize a new data format with an old class. Consider the following scenario:

Node 1 is shut down and updated with a new version of class A. This new version of class A holds a reference to class B, which is a new class. Upon restart, node 1 begins to insert new instances of class A that contain class B. In the meantime, node 2 has not been upgraded yet and it makes a request for an instance of class A that has been inserted by node 1. Since it doesn’t know anything about class B, it throws an exception while deserializing.

The good news is that Coherence makes this problem easy to solve. Using Portable Object Format (POF), you can make your cached objects forward and backward compatible in order to deal with the scenario above.

Before Coherence 3.4, POF was used exclusively for Coherence*Extend, in particular for .NET clients. Starting with Coherence 3.4, POF is natively supported inside of the cluster and is the preferred serialization format for Coherence.

In order to accomplish this, you must implement the EvolvablePortableObject interface.

• This means that you must write out the serialization routines (readExternal and writeExternal). Although somewhat tedious, this will pay off as POF serialized objects are smaller than their Serializable counterparts, and take less memory and CPU to serialize.
• EvolvablePortableObject extends the Evolvable interface, so these methods must also be implemented. The easiest way to do this is to extend AbstractEvolvable. The JavaDoc for Evolvable goes into detail about how it works, but the basics are that by implementing this interface, Coherence will automatically resolve differences between class and data versions.

Here is an example implementation:

import com.tangosol.io.AbstractEvolvable;
 
import com.tangosol.io.pof.EvolvablePortableObject;
import com.tangosol.io.pof.PofReader;
import com.tangosol.io.pof.PofWriter;
 
import java.io.IOException;
 
 
public class Person
        extends AbstractEvolvable
        implements EvolvablePortableObject
    {
    // ----- constructors ---------------------------------------------------
 
    public Person()
        {
        }
 
 
    // ----- accessors ------------------------------------------------------
 
 
    public long getId()
        {
        return m_id;
        }
 
    ...
 
 
    // ----- Evolvable interface ---------------------------------------
 
    public int getImplVersion()
        {
        return VERSION;
        }
 
 
    // ----- PortableObject interface ---------------------------------------
 
    public void readExternal(PofReader in)
            throws IOException
        {
        m_id         = in.readLong(0);
        m_sFirstName = in.readString(1);
        m_sLastName  = in.readString(2);
        m_sEmail     = in.readString(3);
        }
 
    public void writeExternal(PofWriter out)
            throws IOException
        {
        out.writeLong(0, m_id);
        out.writeString(1, m_sFirstName);
        out.writeString(2, m_sLastName);
        out.writeString(3, m_sEmail);
        }
 
 
    // ----- fields ---------------------------------------------------------
 
    private long   m_id;
    private String m_sFirstName;
    private String m_sLastName;
    private String m_sEmail;
 
    public static final int VERSION = 1;
    }

A POF configuration for such an object may look like this:

<!DOCTYPE pof-config SYSTEM "pof-config.dtd">
 
<pof-config>
  <user-type-list>
    <include>coherence-pof-config.xml</include>
 
    <user-type>
      <type-id>10000</type-id>
      <class-name>com.tangosol.examples.evolvable.Person</class-name>
    </user-type>
 
  </user-type-list>
</pof-config>

To configure Coherence to use POF, the easiest method is to configure the following system properties in your startup script:

-Dtangosol.pof.enabled=true 
-Dtangosol.pof.config=pof-config.xml

Now let’s say you have a running system and you want to add a new Address field to the Person class. The following modifications can be made to Person:

    // ----- PortableObject interface ---------------------------------------
 
    public void readExternal(PofReader in)
            throws IOException
        {
        m_id         = in.readLong(0);
        m_sFirstName = in.readString(1);
        m_sLastName  = in.readString(2);
        m_sEmail     = in.readString(3);
        m_address    = (Address) in.readObject(4); // <- new Address field
        }
 
    public void writeExternal(PofWriter out)
            throws IOException
        {
        out.writeLong(0, m_id);
        out.writeString(1, m_sFirstName);
        out.writeString(2, m_sLastName);
        out.writeString(3, m_sEmail);
        out.writeObject(4, m_address);  // <- new Address field
        }
 
 
    // ----- fields ---------------------------------------------------------
 
    private long   m_id;
    private String m_sFirstName;
    private String m_sLastName;
    private String m_sEmail;
    private Address m_address;  // <- new Address field
 
    public static final int VERSION = 2; // <- new version
    }

And this addition to the pof-config.xml file:

    <user-type>
      <type-id>10001</type-id>
      <class-name>com.tangosol.examples.evolvable.Address</class-name>
    </user-type>

There are the changes that I made:

1. Added the Address field as a member of the Person class and to the serialization routines (as index 4)
2. Incremented the version
3. Added the Address class to the POF configuration

That’s it! Now clients with an old version of the Person class can load new versions of Person data, modify it, and place it back into the cache while preserving all of the new data (for a hint on how this is done, take a look at the futureData property defined by Evolvable. The obvious caveat here is that new fields must always be added to a class, and the order must be preserved.

Today when I was using JConsole I noticed that when I tried to connect to a process the UI was hanging. The application that it was trying to connect to was not in a good state, but I was surprised at the lack of a responsive UI. Curious, I captured a thread dump and saw this gem:

"AWT-EventQueue-0" prio=6 tid=0x0100cd00 nid=0x899e00 runnable [0xb0d13000..0xb0d14d90]
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        - locked <0x2559e880> (a java.net.SocksSocketImpl)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
        at java.net.Socket.connect(Socket.java:520)
        at java.net.Socket.connect(Socket.java:470)
        at java.net.Socket.(Socket.java:367)
        at java.net.Socket.(Socket.java:180)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:569)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:306)
        at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
        at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
        at sun.rmi.transport.DGCClient$EndpointEntry.registerRefs(DGCClient.java:275)
        at sun.rmi.transport.DGCClient.registerRefs(DGCClient.java:112)
        at sun.rmi.transport.LiveRef.read(LiveRef.java:277)
        at sun.rmi.server.UnicastRef2.readExternal(UnicastRef2.java:54)
        at java.rmi.server.RemoteObject.readObject(RemoteObject.java:438)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:946)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1809)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJRMP(RMIConnector.java:1871)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1789)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:259)
        - locked <0x25600378> (a javax.management.remote.rmi.RMIConnector)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)
        at sun.tools.jconsole.ProxyClient.(ProxyClient.java:156)
        at sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:57)
        at sun.tools.jconsole.ConnectDialog.actionPerformed(ConnectDialog.java:342)
        at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1882)
        at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2202)
        at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:420)
        at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:258)
        at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
        at java.awt.Component.processMouseEvent(Component.java:5602)
        at javax.swing.JComponent.processMouseEvent(JComponent.java:3135)
        at java.awt.Component.processEvent(Component.java:5367)
        at java.awt.Container.processEvent(Container.java:2010)
        at java.awt.Component.dispatchEventImpl(Component.java:4068)
        at java.awt.Container.dispatchEventImpl(Container.java:2068)
        at java.awt.Component.dispatchEvent(Component.java:3903)
        at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4256)
        at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3936)
        at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3866)
        at java.awt.Container.dispatchEventImpl(Container.java:2054)
        at java.awt.Window.dispatchEventImpl(Window.java:1801)
        at java.awt.Component.dispatchEvent(Component.java:3903)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:463)
        at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:180)
        at java.awt.Dialog$1.run(Dialog.java:535)
        at java.awt.Dialog$2.run(Dialog.java:565)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.awt.Dialog.show(Dialog.java:563)
        at java.awt.Component.show(Component.java:1302)
        at java.awt.Component.setVisible(Component.java:1255)
        at sun.tools.jconsole.ConnectDialog.setVisible(ConnectDialog.java:415)
        at sun.tools.jconsole.JConsole.showConnectDialog(JConsole.java:583)
        at sun.tools.jconsole.JConsole.access$100(JConsole.java:34)
        at sun.tools.jconsole.JConsole$4.run(JConsole.java:702)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:461)
        at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:184)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:176)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

Every Swing programmer that has gotten past Hello World knows that this is a no-no. AWT and Swing components are rendered by a single thread, and if that thread gets blocked then you end up with an unresponsive UI. Clearly there should never be any network code in the stack trace of the AWT event thread. This is on the 1.5 VM on OS X so I don’t know if Sun or Apple is to blame (I assume Sun), but if this isn’t fixed yet in 1.6 then they should review the Swing Tutorial:

Tasks on the event dispatch thread must finish quickly; if they don’t, unhandled events back up and the user interface becomes unresponsive.

The NY Coherence SIG had another large turnout (at least 70+). Slides for the talks are now available for download. For those of you on the west coast, the Bay Area SIG is having its first meeting on February 19th at Oracle HQ.

There are some Coherence related activities this week that the user community may be interested in.

First, the Winter 2009 NY Coherence SIG will be on Thursday January 29th at the Oracle office on Madison Ave in midtown. Registration is required in order for the bouncers in the lobby to let you in, so be sure to check the link and register if you plan on attending. Yours truly will be talking about using Coherence in production and a peek behind the scenes at the tools and techniques used to provide support for Coherence. We’ll also have one of our customers (Anthony Casalena, Founder and CEO of Squarespace, Inc.) presenting as well as one of our top architects Brian Oliver speaking about using Coherence in a WAN. He just gave a presentation at the London SIG last week that was well received.

For those in the Boston area, you can check out the Boston Scalability User Group in Waltham run by Anthony Chaves. Here is the registration link if you’re interested in attending. This meeting will be a free format discussion on all things scalable, so if you have any questions about a new project or experiences (good or bad, the latter being more entertaining) you’d like to discuss among your peers this is a good opportunity to do so. I should be at this meeting as well.

Update: Due to the weather the Boston SUG meeting has been postponed.

In the virtual world we have a new LinkedIn Coherence Users group that has attracted over 30 members on its first day! If you’re a Coherence user or are interested in networking with Coherence users and architects, feel free to join this group.

While I’m in NY for the SIG, I’m going to try to take some time to check out Joel’s new downtown office.

By default Coherence logs to stdout, which means that Coherence applications deployed in a container will have these logs managed by said container (including rolling of the logs to keep the files from getting too large.) However, many Coherence grids contain JVMs that run the Coherence cache server process (DefaultCacheServer) which means that managing logs becomes an exercise for the developer or operations person.

An easy solution is simply to indicate the name of the file that Coherence should log to like this:

-Dtangosol.coherence.log=coherence.log

This is by far the simplest approach; unfortunately this means that the log will grow as long as the JVM is alive. For cache servers that have months (or years) of uptime, this will at best result in very large log files and at worse running out of disk space on the volume (not to mention if the volume happens to be the one the OS is running on.) To solve this problem, the most common approach is to use Log4J to manage the logs generated by Coherence.

Here is an example log4j.properties file that could be used for this purpose:

log4j.rootLogger=debug, file
 
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.File=coherence-${pid}.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%m%n
log4j.appender.file.DatePattern='.'yyyy-MM-dd
 
log4j.logger.Coherence=debug

• In my example I chose to use the DailyRollingFileAppender. The RollingFileAppender which rolls logs over by size can also be used.
• Note that the ConversionPattern is very simple; it only includes the log message and a newline character. This is because Coherence already prints out the timestamp, version, thread, and other information on each line.
• The log file name is coherence-${pid}.log. The ${pid} is a system property that will contain the process id of the cache server (see script below.)

Here is the script that I used to launch the cache server:

#!/bin/sh
 
exec java -Dtangosol.coherence.log.level=6 -Dtangosol.coherence.log=log4j \
-Dpid=$$ -cp [...] com.tangosol.net.DefaultCacheServer

Note that I am using -Dtangosol.coherence.log.level to choose the log level. The log level (debug, info, warn) can also be selected using the Log4J configuration, although using the Coherence system property provides finer grained control. The script also sets the pid system property.

Using this technique, the logs generated by Coherence are rolled, making it much easier to archive logs and (if required) to upload them to support.