Inspired by Actual Events: Observations From A History of Java Backwards Incompatibility

For the most part, Java is a very backwards compatible programming language. The advantage of this is that large systems can generally be upgraded to use newer versions of Java in a relatively easier fashion than would be possible if compatibility was broken on a larger scale. A primary disadvantage of this is that Java is stuck with some design decisions that have since been realized to be less optimal than desired, but must be left in place to maintain general backwards compatibility. Even with Java's relatively strong tie to backwards compatibility, there are differences in each major release of Java that can break Java-based applications when they are upgraded. These potential breaks that can occur, most commonly in "corner cases", are the subject of this post.

Sun Microsystems and Oracle have provided fairly detailed outlines of compatibility issues associated with Java upgrades. My point is not to cover everyone of these issues in everyone of the versions, but to instead highlight some key incompatibility issues introduced with each major release of Java that either personally impacted me or had more significant effect on others. Links at the bottom of this post are provided to the Sun/Oracle Java versions' compatibility documents for those seeking greater coverage.

Upgrading to JDK 1.2

With hindsight, it's not surprising that this early release in Java fixed several incompatibilities of the implementation with the specification. For example, the JDK 1.2 compatibility reference states, The String hash function implemented in 1.1 releases did not match the function specified in the first edition of the Java Language Specification, and was, in fact, unimplementable." It adds, "the implemented function performed very poorly on certain classes of strings" and explains that "the new String hash function in version 1.2" was implemented to "to bring the implementation into accord with the specification and to fix the performance problems." Although it was anticipated that this change to String.hashCode() would not impact most applications, it was acknowledged that "an application [that] has persistent data that depends on actual String hash values ... could theoretically be affected." This is a reminder that it's not typically a good idea to depend on an object's hashCode() method to return specific codes.

Upgrading to JDK 1.3

The JDK 1.3 compatibility reference mentions several changes that brought more implementation conformance with the JDK specification. One example of this was the change that introduced "name conflicts between types and subpackages":

According to ... the Java Language Specification, ... it is illegal for a package to contain a class or interface type and a subpackage with the same name. This rule was almost never enforced prior to version 1.3. The new compiler now enforces this rule consistently. A package, class, or interface is presumed to exist if there is a corresponding directory, source file, or class file accessible on the classpath or the sourcepath, regardless of its content.

JDK 1.3 also introduced a change to the "implementation of method java.lang.Double.hashcode."

Upgrading to JDK 1.4

The upgrade effort I was leading on a project to move to JDK 1.4 ended up taking more time than estimated due to JDK 1.4's change so that "the compiler now rejects import statements that import a type from the unnamed namespace." In other words, JDK 1.4 took away the ability to import a class defined without an explicit package. We did not realize this would be an issue for us because the code that it impacted was code generated by a third-party tool. We had not control over the generation of the code to force the generated classes to be in named packages and so they were automatically part of the "unnamed namespace." This meant that, with JDK 1.4, we could no longer compile these generated classes along with our own source code. Discovering this and working around this change took more time than we had anticipated or what we thought was going to be a relatively straightforward JDK version upgrade. The same JDK 1.4 compatibility reference also states the most appropriate solution when one controls the code: "move all of the classes from the unnamed namespace into a named namespace."

Upgrading to Java SE 5 (1.5)

I wrote about Java SE 5's change to BigDecimal.toString() in my previous post On the Virtues of Avoiding Parsing or Basing Logic on toString() Result. The Java SE 5 compatibility reference simply states, "The J2SE 5.0 BigDecimal's toString() method behaves differently than in earlier versions."

Upgrading to Java SE 6 (1.6)

The issue that harassed me most when upgrading to Java SE 6 was the inclusion of JAXB with JDK 6. This issue is not listed in the Java SE 6 compatibility reference because the nature of this issue does not technically meet the definition of a compatibility issue as documented here. However, anyone using a separately downloaded JAXB JAR before moving to Java SE 6 likely ran into the classloader issues I ran into. The solution most of us used to get past this was to place our preferred JAXB JAR in the directory specified as part of the Java Endorsed Standards Override Mechanism (deprecated as of Java 8 and removed in Java 9).

Upgrading to Java 7 (1.7)

Any uses of the com.sun.image.codec.jpeg package were broken when upgrading to Java 7. The Java 7 compatibility reference states, "The com.sun.image.codec.jpeg package was added in JDK 1.2 (Dec 1998) as a non-standard way of controlling the loading and saving of JPEG format image files. This package was never part of the platform specification and it has been removed from the Java SE 7 release. The Java Image I/O API was added to the JDK 1.4 release as a standard API and eliminated the need for the com.sun.image.codec.jpeg package."

Another incompatibility reintroduced in Java 7 is actually another example of making an implementation better conform to the specification. In this case, in Java SE 6, methods that had essentially the same erased signature but with different return types were seen as two different methods. This does not conform with the specification and Java 7 fixed this. More details on this issue can be found in my blog post NetBeans 7.1's Internal Compiler and JDK 6 Respecting Return Type for Method Overloading and in the Java 7 compatibility reference under "Synopsis" headings " A Class Cannot Define Two Methods with the Same Erased Signature but Two Different Return Types" and "Compiler Disallows Non-Overriding Methods with the Same Erased Signatures".

The Java 7 upgrade presented some difficulties for users of Substance as well. The Insubstantial 6.2 Release post states, "Java 7 fixes - there is a bug fix in Java's Color Choosers that broke substance 6.1. This is fixed in Substance 6.2, so it should run on Java 7 now!" The JDK 7 changes that broke Substance are documented in various places including JColorChooser with Substance look and feel, Java 7, ColorChooser causes NullPointerException in JSlider with JDK7, and Color chooser setColor not working in Java 7.

Upgrading to Java 8 (1.8)

Just as Java 7 changes impacted Substantial, Java 8 brought a change that directly impacted several popularly and widely used Java libraries. Although this change likely directly affected relatively few Java applications, it indirectly had the potential to affect many Java applications. Fortunately, the maintainers of these Java libraries tended to fix the issue quickly. This was another example of enforcement of the specification being tightened (corrected) and breaking things that used to work based on an implementation not implementing the specification correctly. In this case, the change/correction was in the byte code verifier. The JDK 8 Compatibility Guide states, "Verification of the invokespecial instruction has been tightened when the instruction refers to an instance initialization method ("<init>")." A nice overview of this issue is provided in Niv Steingarten's blog post Oracle's Latest Java 8 Update Broke Your Tools — How Did it Happen?

Upgrading to Java 9 (1.9)

It seems likely Java 9 will introduce some significant backwards compatibility issues, especially given its introduction of modularity. While it remains to be seen what these breakages are, there has already been significant uproar over the initial proposal to remove access to sun.misc.Unsafe. This is another example of where an officially unsupported API may not be used directly by most applications, but is probably used indirectly by numerous applications because libraries and products they depend upon use it. It's interesting that this has led to the Mark Reinhold proposal that internal APIs be encapsulated in JDK 9. Given the numerous compatibility issues associated with dropped and changed internal APIs between major Java revisions, this seems like a good idea.

Lessons Learned from JDK Version Compatibility Issues

Avoid taking advantage of improper implementations that violate the specification as those exploits of holes in the implementation may not work at all when the implementation is changed to enforce the specification.
Beware of and use only with caution any APIs, classes, and tools advertised as experimental or subject to removal in future releases of Java. This includes the sun.* packages and deprecated tools and APIs.
- I like the proposed JDK 9 approach of "encapsulating internal APIs in JDK 9" to deal with these frequent issues during major revision upgrades.
Don't depend on the String returned by toString() implementations for program logic.

Conclusion

Significant effort has been applied over the years to keep Java, for the most part, largely backwards compatible. However, there are cases where this backwards compatibility is not maintained. I have looked at some examples of this in this post and extracted some observations and lessons learned from those examples. Migrations to newer versions of Java tend to be easier when developers avoid using deprecated features, avoid using experimental features, and avoid using non-standard features. Also, certain coding practices such as avoiding basing logic on toString() results, can help.

Resources and References