New on the Coherence blog is an article on best practices for near caching written by yours truly. This article went through several reviews by PM and engineering (both internal and field engineers) and is a good summary of our experiences with near caching. The key takeaway is to understand the near cache invalidation strategies and to explicitly configure the one best suited to the application needs.

See the full Near Cache Best Practices article on the official Oracle Coherence blog.

Note that all images below are thumbnails; click on the images to expand.

You have the option of downloading Eclipse with the OEPE plugins pre-installed, but I already had a copy of Eclipse Helios 3.6; therefore I went for the plugin install. The plugin install is straight forward, similar to any other Eclipse plugin. These instructions are lifted straight from the doc:

1. Select Help > Install New Software.
2. Click Add to add a new update site.
3. In the Add Repository dialog, enter the location as http://download.oracle.com/otn_software/oepe/helios, and then click OK.
4. Select Oracle Enterprise Pack for Eclipse, verify that all of the subcomponents are selected, and then click Next.
5. Confirm information presented in the Install Details, and then click Finish.

Once the install is complete and you’ve restarted Eclipse, the next step is to install Coherence as a User Library. I’ve got the latest 3.6 bits from our source code repository, so I’ll install that as a library. Note that the plugin also supports Coherence 3.5.2.

Now, let’s create a new Java project. As part of the project creation, add Coherence as a project library.

After the project is created, we’ll have to enable Facets to use the Coherence plugin. Bring up the project properties window and search for “Facets” in the search field.

Once Facets are enabled, select Oracle Coherence.

Upon selection, a link indicating that further configuration is required will appear. Click on the link. Select the “Oracle Coherence 3.6″ library. Note how it provides the option to generate configuration files. Let’s leave all of the checkboxes selected.

Now we are ready to start a cache server. Select File | Run Configurations to bring up the Run Configurations dialog. First select “Oracle Coherence” under the list of run configurations. Next, select the “New” button on the upper left portion of the dialog to create a new run configuration.

Under the “Main” tab, enter com.tangosol.net.DefaultCacheServer as the main class. Of course you are free to create configurations with your own classes; however this example will focus on starting up a cache server.

Note the presence of a “Coherence” tab. This tab allows for operational configuration (items typically found in tangosol-coherence-override.xml) such as the cache configuration file name, multicast address configuration, management/JMX, and so on. Here I decided to leave all of the defaults as is.

After clicking on “Run”, here’s what I get:

We can see that the node started up and formed a cluster, but there are no services listed. This is because the OEPE plugin generated a cache configuration file that defaults to all caches being local. Next, let’s examine the cache configuration file (located under src and add a distributed/partitioned cache to the configuration.

One of the nice features the plugin provides is pre-configured auto complete for Coherence configuration files via the DTD.

Here’s the cache configuration file I used:

<!--?xml version="1.0"?-->
 
      *
      partitioned
 
      partitioned
 
      true

With the modified cache configuration, we now see the partitioned cache service start up:

I can see the Coherence plugin for OEPE being quite useful for Coherence developers on Eclipse not only for quickly starting up a Coherence project (since config files are generated) but also for enabling configuration validation out of the box.

An Introduction to Data Grids for Database developers (GatorJUG June 23rd)

This talk will introduce the concept of data grids to developers that have experience with Java EE and relational databases such as Oracle. The programming model will be explored (including caching patterns and similarities to NoSQL) as well as the performance & scalability improvements a data grid offers.

From IDE to Data Center – What Every Developer Should Know About Deploying Distributed Systems to Production (Orlando JUG June 24th)

Taking a distributed system from development into a working production environment is a challenge that many developers take for granted. This talk will explore these challenges, especially scenarios that are not typically seen in a development setting.

I’m especially excited about the OJUG talk as I think it will cover many topics of interest to Developers and OPS guys. It is a set of general guidelines that came about from seeing dozens of Coherence applications in production. We will cover such things as:

• What to look for when using vmstat
• Must-have production level JVM settings/flags
• Developer Do’s and Dont’s
• Crash course on thread dumps and heap dumps

We will also be giving away a copy of Oracle Coherence 3.5 at each event! If you are coming please follow the links to the events above and RSVP (you need to be a member of CodeTown to sign up, but registration is free and painless.)

On occasion I am asked about best practices for creating classes to be used as keys in Coherence. This usually comes about due to unexpected behavior that can be explained by incorrect key implementations.

First and foremost, equals and hashCode need to be implemented correctly for any type used as a key. I won’t describe how to do this – instead I’ll defer to Josh Bloch who has written the definitive guide on this topic.

There is an additional requirement that needs to be addressed. All serializable (non transient) fields in the key class must be used in the equals implementation. To understand this requirement, let’s explore how Coherence works behind the scenes.

First, let’s try the following experiment:

public class Key
        implements Serializable
    {
    public Key(int id, String zip)
        {
        m_id = id;
        m_zip = zip;
        }
 
    //...
 
    @Override
    public boolean equals(Object o)
        {
        // print stack trace
        new Throwable("equals debug").printStackTrace();
        if (this == o)
            {
            return true;
            }
        if (o == null || getClass() != o.getClass())
            {
            return false;
            }
 
        Key key = (Key) o;
 
        if (m_id != key.m_id)
            {
            return false;
            }
        if (m_zip != null ? !m_zip.equals(key.m_zip) : key.m_zip != null)
            {
            return false;
            }
        return true;
        }
 
    @Override
    public int hashCode()
        {
        // print stack trace
        new Throwable("hashCode debug").printStackTrace();        
        int result = m_id;
        result = 31 * result + (m_zip != null ? m_zip.hashCode() : 0);
        return result;
        }
 
    private int m_id;
    private String m_zip;
    }

This key prints out stack traces in equals and hashCode. Now use this key with a HashMap:

public static void testKey(Map m)
    {
    Key key = new Key(1, "12345");
 
    m.put(key, "value");
    m.get(key);
    }
 
//...
 
testKey(new HashMap());

Output is as follows:

java.lang.Throwable: hashCode debug
	at oracle.coherence.idedc.Key.hashCode(Key.java:60)
	at java.util.HashMap.put(HashMap.java:372)
	at oracle.coherence.idedc.KeyTest.testKey(KeyTest.java:46)
	at oracle.coherence.idedc.KeyTest.testKey(KeyTest.java:52)
	at oracle.coherence.idedc.KeyTest.main(KeyTest.java:18)
java.lang.Throwable: hashCode debug
	at oracle.coherence.idedc.Key.hashCode(Key.java:60)
	at java.util.HashMap.get(HashMap.java:300)
	at oracle.coherence.idedc.KeyTest.testKey(KeyTest.java:47)
	at oracle.coherence.idedc.KeyTest.testKey(KeyTest.java:52)
	at oracle.coherence.idedc.KeyTest.main(KeyTest.java:18)

Try it again with a partitioned cache this time:

testKey(CacheFactory.getCache("dist-test"));

Note the absence of stack traces this time. Does this mean Coherence is not using the key’s equals and hashCode? The short answer (for now) is yes. Here is the flow of events that occur when executing a put with a partitioned cache:

1. Invoke NamedCache.put
2. Key and value are serialized
3. Hash is executed on serialized key to determine which partition the key belongs to
4. Key and value are transferred to the storage node (likely over the network)
5. Cache entry is placed into backing map in binary form

Note that objects are not deserialized before placement into the backing map – objects are stored in their serialized binary format. As a result, this means that two keys that are equal to each other in object form must be equal to each other in binary form so that the keys can be later be used to retrieve entries from the backing map. The most common way to violate this principle is to exclude non transient fields from equals. For example:

public class BrokenKey
        implements Serializable
    {
    public BrokenKey(int id, String zip)
        {
        m_id = id;
        m_zip = zip;
        }
 
    //...
 
    @Override
    public boolean equals(Object o)
        {
        if (this == o)
            {
            return true;
            }
        if (o == null || getClass() != o.getClass())
            {
            return false;
            }
 
        BrokenKey brokenKey = (BrokenKey) o;
 
        if (m_id != brokenKey.m_id)
            {
            return false;
            }
 
        return true;
        }
 
    @Override
    public int hashCode()
        {
        int result = m_id;
        result = 31 * result;
        return result;
        }
    }

Note this key has two fields (id and zip) but it only uses id in the equals/hashCode implementation. I have the following method to test this key:

public static void testBrokenKey(Map m)
    {
    BrokenKey keyPut = new BrokenKey(1, "11111");
    BrokenKey keyGet = new BrokenKey(1, "22222");
 
    m.get(keyPut);
    m.put(keyPut, "value");
 
    System.out.println(m.get(keyPut));
    System.out.println(m.get(keyGet));
    }

Output using HashMap:

value
value

Output using partitioned cache:

value
null

This makes sense, since keyPut and keyGet will serialize to different binaries. However, things get really interesting when combining partitioned cache with a near cache. Running the example using a near cache gives the following results:

value
value

What happened? In this case, the first get resulted in a near cache miss, resulting in a read through to the backing partitioned cache. The second get resulted in a near cache hit because the object’s equals/hashCode was used (since near caches store data in object form.)

In addition to equals/hashCode, keep the following in mind:

• Keys should be immutable. Modifying a key while it is in a map generally isn’t a good idea, and it certainly won’t work in a distributed/partitioned cache.
• Key should be as small as possible. Many operations performed by Coherence assume that keys are very light weight (such as the key based listeners that are used for near cache invalidation.)
• Built in types (String, Integer, Long, etc) fit all of this criteria. If possible, consider using one of these existing classes.)

In Boston however I found the environment much more conducive to running, given that there are sidewalks and bike paths, not to mention it feels like half of the people here run. Also, the weather in the fall is very comfortable to run in (which probably explains why the NYC marathon is in November.)

Winter running, on the other hand, came with its own set of challenges. During those months I learned some lessons that I’m passing along.

At first I tried running with a thick cotton sweater. This turned out to be a bad idea, because the sweater (being a warm sweater as advertised) prevented heat from escaping my body. This, paradoxically, causes you to freeze your ass off since you now have a pool of sweat on your skin. The best practice is to wear material that wicks sweat away, including the top layer. Even on the coldest of days (my personal low was 15F) I would get comfortable after about a mile.

The biggest challenge for me was not the air temperature (at least after warming up), it was ice. On icy days my runs were much slower. On really icy days I would just skip the run. I know some hard core runners that would throw on spikes or yak tracks, but I didn’t take it to that level. A related challenge is when it isn’t quite cold enough to freeze deeper pools of water. I had the distinctly unpleasant experience of stepping into a 1 inch puddle on the bike path after wrongly assuming it was frozen over. That day my run was definitely shorter than it would have been otherwise.

For the given temperatures, I really was wearing very thin layers of clothing. This means that I had to keep moving in order to stay warm. I never slowed down to a walk because that would have been far more unpleasant than dealing with the fatigue. Running on the road in general has that advantage over the treadmill – with the treadmill you can quit anytime, whereas on the road you have to make it back home sometime, so you may as well run the distance.

Due to things being really busy lately my mileage has been decreasing. I’m hoping to turn this around soon, especially since the warm spring weather is bringing out so many runners.

So how has the running affected my fitness? It used to be that my heart and lungs were ready to break out of my chest after chasing down a bus. After taking up running, I was not only able to perform this exercise without loosing breath, but I was able to do it while carrying my 40 lbs son!

New Coherence Bloggers

We have two new bloggers sharing their experiences with Coherence!

The first is by Oracle JDBC expert Pas Apicella who recently took on Coherence. Upon his introduction to Coherence he immediately proceeded to create a CacheStore example using PL/SQL, followed by an example of using the Oracle JDBC Data Change Notification mechanism to push updates from the database to a cache.

Additionally we have Coherence architect Andy Nguyen debuting with a detailed description of a sophisticated distributed bulk loading technique he’s employed on several customer projects.

Coherence Book in March

After many months of blood, sweat, and blisters from too much typing, Aleksandar Seovic has completed the highly anticipated Coherence book published by Packt. Having worked closely with Aleks on reviews and contributions, I believe this book will be a terrific resource for developers and architects that need to write scalable applications. Both experienced users of Coherence and new users will find relevant and useful content. Aleks was recently interviewed by Cameron Purdy about the book which can be downloaded as an MP3.

User Group Meetings

The UK SIG in London is the last Coherence user group meeting for the winter, it is coming up on February 26th. The spring events are currently being planned; stay tuned for details!

Also coming up on February 24th is the first Boston SUG meeting of the year. Although the topic won’t be Coherence this time, it will be of interest for developers and architects interested in scalable systems. We’ll be meeting up for drinks and snacks at Bertucci’s afterwards. And I’ll be there if anyone wants to chat about Coherence or any other topic!

The Coherence content is available on OTN. (There’s also a PDF version that is nicer for printing out.) It includes sessions on future direction, integration with other products in the Oracle lineup, developer workshops, and customer panels describing their use of Coherence.

I’ll be talking about using Coherence to scale out external data stores (including relational databases.) This is mostly the same content that will be covered during the NY SIG.

The first talk will be by Aleksandar Seovic who has an extensive Coherence resume. In addition to implementing POF in .NET, he is also the author of an upcoming book on Coherence. In his spare time (insert tongue in cheek) he runs S4HC, a consulting company specializing in Spring, Coherence, and other technologies.

If you’re in Tampa, you can also check out his upcoming talk at Tampa JUG on September 29th.

We will also have Mark Falco, one of our rock star engineers who concentrates on our network protocol (TCMP), C++, and other areas. Mark is usually the first (and last) person I reach out to when I have questions about Coherence networking – so if you have any questions of your own be sure to bring them. He will talk to us about TCMP and how to optimize your machines and network for optimum performance.

Finally, yours truly will talk about configuring Coherence to work with an external data source (usually relational databases.) I’ll describe in detail how each of the external connectivity features work (including many features you’ve probably never heard of), best practices, and good old fashioned war stories. (Shout out to Rob for helping with the Omni Graffle diagrams!) This is the same talk that I will present in mid October at Oracle Open World in San Francisco. I’ll provide more detail on this later; for now you can check out the Application Grid lineup – which includes WebLogic Server, Coherence, JRockit, Tuxedo and Enterprise Manager.

Spring DM and OSGi

Craig Walls provided an extensive overview (and defense) of OSGi. OSGi is a framework for managing library dependencies. It enables the installation, configuration, and updating of modules in a live program without JVM restarts. Multiple versions of a class and/or libraries can coexist in a container. Each module has a defined lifecycle and dependencies between modules can be defined.

Here is a list of OSGi implementations:

Open Source

• Eclipse Equinox
• Apache Felix
• Knopflerfish
• Concierge

Commercial

• Makewave Knopflerfish Pro
• ProSyst mBedded Server
• Samsung OSGi R4 Solution
• HitachiSoft SuperJ Engine Framework

Craig provided a live demonstration of the OPS4J Pax shell running under Eclipse Equinox and Apache Felix. Unfortunately we ran out of time and we didn’t get to see much of the Spring DM server in action.

java.next

Stuart Halloway gave an overview of the latest and greatest languages available for the JVM. Like some of the other NFJS speakers he has very strong opinions, especially when it comes to the use of Java. I believe the quote was something along the lines of “every time you start a greenfield project with Java, God kills a kitten.” (Incidentally, Ted Neward believes that using Java arrays instead of collections will lead to the same fate for said kitten.)

Straight from Stuart’s slides: pros and cons for your consideration:

Clojure Pros

• Functional
• Multimethods
• Concurrency
• Lisp
• A la carte

Clojure Cons

• Youngest java.next language

Groovy Pros

• Easiest to learn
• Easiest bi-di interop
• More committed to reusing Java libs

Groovy Cons

• Worst Java baggage
• No concurrency/multicore story

JRuby Pros

• Biggest community
• Commercial support: EngineYard
• Rails
• multiple platforms

JRuby Cons

• No concurrency/multicore story

Scala Pros

• Functional
• High performance
• Pattern matching
• Actor model
• Hybrid object/functional (could also be a con)

Scala Cons

• Hardest to learn

The general theme on these new generation languages is:

• Dynamic typing (Scala is not as dynamic as the others, but offers more flexibility than Java)
• No checked exceptions
• Reasonable defaults
• Convention over configuration
• YAGNI

Garbage Collector Friendly Programming

Garbage collection is an interesting topic to me for several reasons, the main reason being that poor GC performance is very harmful in distributed environments. When you have a peer to peer system such as Coherence, any one node can directly communicate with any other node at any point in time to service a request. If a node that needs to service several requests is in a long GC pause, it isn’t just that node that is affected. Every JVM that is waiting for a response from that node also experiences high latency, thus causing a cascading effect. (More on what to do about this later.)

Brian Goetz described the evolution of GC in Java, starting with the single threaded mark and sweep algorithms up to the modern generational collector. What they each have in common is the tracking of allocation roots (which include static variables and local variables allocated on thread stacks) and the traversal of object references starting at these roots. The implementation of the generational collector is (roughly) as follows:

• New objects are allocated in the young generation space
• When a minor GC occurs, objects that have references pointing to them are copied to the survival space. The remaining objects are removed
• Eventually objects that live long enough in the survival space are moved to the old generation.
• If a minor collection fails to free up enough space in the young generation area, a full collection (which is much more expensive) will occur in the old generation space.

An implementation detail is that the JVM must track references from the old generation to the young generation in order to know which GC roots to traverse when performing a minor collection. This means that the more “old” objects there are pointing to “new” objects, the more work the collector has to do. In a practical sense, this means that allocating new objects is preferred to reference field updates.

It took a while to wrap my head around that last statement, so let me attempt to demonstrate. Let’s say I have a map that contains objects with many fields. If I wanted to update some of those fields I could do it like this:

Map map = ...
MyObject o = map.get(key);
o.setField1(f1);
o.setField3(f3);
o.setField5(f5);

Or I could do it like this:

Map map = ...
MyObject oOld = map.get(key);
MyObject oNew = new MyObject(f1, o.getField2(), f3, o.getField4(), f5);
map.put(key, oNew);

The first example updates three reference fields (assuming the object in the map is in the old generation), whereas the second is updating one (the reference held by Map.Entry) – and as an added bonus the second implementation is thread safe (assuming that MyObject is immutable and the Map implementation is also thread safe.) If anyone has a better (or more correct) example of this concept, I’d be happy to see it!

Many other concepts were covered including: why you should use finally instead of finalizers to clean up, weak references, soft references, and tracking down memory leaks. Capturing heap dumps to track memory usage is a technique I recommend to customers (this works much better than speculating/wild guesses about where unexpected memory allocation is coming from) – I especially recommend configuring the JVM to generate a heap dump when an OutOfMemoryError is thrown. My favorite tool to read heap dumps is Eclipse MAT. Heap histograms are also a nice light weight approach to analyze memory problems.

When customers ask for suggestions on GC tuning, my recommendation is to keep it simple: fixed size 1 GB heaps (on the Sun VM), and don’t use more than 75% of the heap. I usually don’t recommend any specific tuning parameters, as the GC algorithms are constantly improving and any exotic flags that may have worked in older JVMs (assuming they helped in the first place) may not work so well in newer JVMs. The best advice I can give is to not fill up the heap as this will cause more frequent full collections.

Inside the Modern JVM

NFJS tends to cover languages in the JVM other than Java (such as Groovy, Scala, JRuby, etc.) This is a testament to the strength and viability of the modern JVM. Brian covered some of the advancements and (quite frankly) rocket science that goes into the JVM, HotSpot in particular. No matter what happens to Java (which isn’t going away any time soon), the JVM will be around for a very long time.

In a nutshell: why is Java, a supposedly interpreted language, faster than C++ in many cases? The answer is that the JVM determines which optimizations to make at run time instead of compile time, which is the opposite approach of C++ and other native languages. Optimization at runtime is far more effective, since the JVM has hard statistics of real world usage to draw on as opposed to the speculation and guessing that happens when everything has to be compiled to machine code before execution.

The overall theme of this talk (and the previous ones) is to write simple clean code – the runtime recognizes common usage patterns in Java and is built to optimize these patterns.

Java Collections

Ted Neward gave an engaging and entertaining talk on the Java Collections API. To be honest I was familiar with most of the material, but he is a fun speaker to watch, in spite of the fact that he gave me a good ribbing for showing up to his talk after it had already started! He is quite biased against arrays and towards collections, which made me think back to a web/remote services API I designed a few years ago. I exclusively used arrays as the collection type for this API for two reasons:

1. To make SOAP/cross language interoperability much simpler (least common denominator – every language does arrays)
2. There were no generics at the time; using arrays instead of generics in the interfaces meant that I could explicitly define the type of the array

The second item is not as important anymore now that we have generics, so I’m inclined to agree that arrays should be used sparingly nowadays.

Another interesting point is that iteration over collections should be done using a functor instead of a plain iterator. For example:

List<Name> names = ...; 
MyListOps.apply(new MyApplyFn<Name>() { 
  public void apply(Name n) { 
    // use n 
  } 
}, names);

This allows the possibility of processing the collection in multiple threads.

What’s coming in Java 7

This was Ted Neward’s next talk which was just as interesting and opinionated. Here are the highlights:

• The release is targeted for early 2010
• There is no official JSR for Java 7
• Most of the information on what is going into Java 7 can be found on the blog of Joe Darcy of Sun.
• Alex Miller has a huge page on his blog detailing what is in and what is out. This is information that is is aggregating off the web.

One of the most compelling additions to Java 7 is JSR 292, which introduces the bytecode invokedynamic. The implications of this addition are narrated by Charles Nutter of JRuby. There are other syntactic conveniences making it in; however it will not include closures (a fairly controversial topic.)