Thursday, July 2, 2020

jcmd Increasingly the Single JDK Command-line Tool to Rule Them All

What would you miss from the JDK command line tools jinfo, jmap, and jstack if those tools were no longer available?

I've been a fan of the command-line tool jcmd for a long time. I like that it combines features and functions of many other JDK command-line tools in a single tool. I also like that I like its interactive nature that supports my asking the tool what it's capable of for a given JVM process, meaning I don't have to remember exact options when applying its different functions.

Since its inception, jcmd has been taking on functionality from various other command-line JDK tools. In a recent post on the OpenJDK jdk-dev mailing list, Stephen Fitch announced a survey intended to "gather more information and help us evaluate and understand how others are using [three 'j* tools' jinfo, jmap, jstack] in the JDK." The surveyors are "specifically interested in your use-cases and how these tools are effective for you in resolving JVM issues."

Fitch explains the need for this survey:

We are considering deprecation and (eventual) removal of the jinfo, jmap, jstack - (aka “j* tools”) and building out a future foundation for some aspect of serviceability on jcmd, however we don’t have a lot of data about how how these tools are used in practice, especially outside of Oracle.

I'm encouraged that it sounds likely that jcmd will continue to add functionality under its single approach that was formerly available with different tools.

The survey will be open through 15 July 2020 and is an opportunity for Java developers to share their experiences and use cases with the JDK command line tools jinfo, jmap, and jstack so that it's more likely those use cases will be supported in jcmd (if not already supported) in the future. Fitch's post presents the URL for the survey: https://www.questionpro.com/t/AQk5jZhiww

Select jcmd Posts

Here are some of my previous posts regarding jcmd (I have used jcmd in numerous other posts to demonstrate various output as well):

Friday, June 26, 2020

Better NullPointerException Messages Automatic in JDK 15

I discussed long-awaited and highly appreciated improvements to NullPointerException (NPE) messages in the posts "Better Default NullPointerException Messages Coming to Java?" and "Better NPE Messages in JDK 14". When this JEP 358-driven feature was added to JDK 14, a developer who wanted to benefit from these more insightful NPE messages needed to explicitly state that desire by passing the argument -XX:+ShowCodeDetailsInExceptionMessages to the Java launcher (java).

The JDK 15 Early Access Build #29 was released this week and makes the use of better NPE messages automatic. The release notes associated with this early access build state: "The default of the flag ShowCodeDetailsInExceptionMessages was changed to 'true'. The helpful NullPointerException messages of JEP 358 are now printed by default. The messages contain snippets of the code where the NullPointerException was raised."

The next screen snapshot demonstrates that the helpful NullPointerException details are provided automatically with JDK 15 Early Access Build #29.

The release notes also point out that one potential risk of having the "helpful" NullPointerException messages written by default is the accidental exposure of sensitive details. The release notes warn: "App deployers should double check the output of their web applications and similar usage scenarios. The NullPointerException message could be included in application error messages or be displayed by other means in the app. This could give remote attackers valuable hints about a potential vulnerable state of the software components being used."

The next screen snapshot demonstrates that automatic presentation of helpful NullPointerException details can be disabled via the use of java launcher option -XX:-ShowCodeDetailsInExceptionMessages (and the old -XX:+ShowCodeDetailsInExceptionMessages is still supported even though this is now the default):

One on the interesting consequences of the JDK-8233014 change to make helpful NullPointerException messages enabled by default is that there will undoubtedly be some Java developers pleasantly surprised when they upgrade to JDK 15 and start seeing suddenly far more useful messages when encountering the ubiquitous NullPointerException.

Thursday, June 4, 2020

Tycoon: Ransomware Targeting Java's JIMAGE on Multiple Platforms

The Blackberry Research and Intelligence Team and KPMG's UK Cyber Response Services Team have reported "Threat Spotlight: Tycoon Ransomware Targets Education and Software Sectors." This report outlines the "multi-platform Java ransomware targeting Windows and Linux that has been observed in-the-wild since at least December 2019" and which they've called "Tycoon."

The report provides a high level description of how the Tycoon ransomware is executed: "Tycoon ransomware comes in form of a ZIP archive containing a Trojanized Java Runtime Environment (JRE) build. The malware is compiled into a Java image file (JIMAGE) located at lib\modules within the build directory." The report describes the "sparsely documented" JIMAGE "file format file format that stores custom JRE images which is designed to be used by the Java Virtual Machine (JVM) at runtime." Additional high-level overviews of the JIMAGE file format can be found in JIMAGE - Java Image File Format, How Modules Are Packaged in Java 9, What Is a Custom Runtime Image in Java 9?, So what is a .jimage?, Alan Bateman's description, and slide 49 of JDK 9 Java Platform Module System. The JIMAGE format was introduced with JDK 9 modularity (Project Jigsaw).

Alan Bateman (owner of JEP 220 ["Modular Run-Time Images"])has explained why it's difficult to find documentation on the JIMAGE format: "The format is deliberately not documented and best to assume it will change and evolve over time as it inhales new optimizations." The jdk.jlink module documentation provides very brief mention of the jimage command-line tool and mentions that it is used "for inspecting the JDK implementation-specific container file for classes and resources." It also points out that there is no API for accessing jimage, which is different than for the module's two other tools (jlink and jmod). Similarly, there is no jimage tool reference on the "Java Development Kit Version 14 Tool Specifications" page even though jlink, jmod, and many other tools are featured there.

The previously mentioned report states that "the ransomware is triggered by executing a shell script that runs the Main function of the malicious Java module using the java -m command." The report also talks about Tycoon using Image File Execution Options Injection with Windows registry and then appearing to target Windows and Linux: "The malicious JRE build contains both Windows and Linux versions of this script." This reports adds that Tycoon's "Trojanized Java Runtime Environment (JRE) build" exists as a ZIP file placed in lib/module.

The "Conclusions" section of "Tycoon Ransomware Targets Education and Software Sectors" makes some interesting conclusions based on research observations. In particular, I find it interesting to read why they believe Tycoon in a targeted attack. The authors also point out, "This is the first sample we've encountered that specifically abuses the Java JIMAGE format to create a custom malicious JRE build."

I recommend reading the report "Threat Spotlight: Tycoon Ransomware Targets Education and Software Sectors" directly, but for those interested in summaries and others' observations related to this report, the following may be of interest.

The discovery of Tycoon is likely to bring significantly more attention to JIMAGE than ever before.

Saturday, May 30, 2020

Octopus Scanner: Java Build Tools and Malware

Alvaro Muñoz recently posted "The Octopus Scanner Malware: Attacking the open source supply chain" on the GitHub Security Lab site. I found this post to be interesting for a number of reasons, including its detailed coverage of how the Octopus Scanner malware works and how it was discovered, how the GitHub Security Incident Report Team (SIRT) went about addressing it, how it affected a popular Java IDE, and how GitHub works to detect and address risks to open source software deployed on its site.

Muñoz calls Octopus Scanner "an OSS supply chain malware" and writes that 26 GitHub-hosted open source projects "were backdoored by this malware." In the post, Muñoz describes in detail how Octopus Scanner works. The post in its entirety is worth reading, but here are some highlights describing Octopus Scanner:

  • "Octopus Scanner malware is only interested in the pre-jar and post-jar tasks" of "a NetBeans project build."
  • "The malware disguises itself as an ocs.txt file, but we can easily determine is is actually a Java Archive (JAR) file."
  • "The malware also infected any JAR files that were available in the project, such as dependencies - not necessarily just build artifacts."
  • "The octopus.dat payload is the binary that actually performs the NetBeans build infections."
  • "cache.dat is responsible for backdooring the built classes so that when these classes get executed, they will infect the underlying system."
  • Octopus Scanner is designed to target "UNIX-like systems", MacOS, and Windows.

I found the highly detailed approach to diagnosing Octopus Scanner's behavior in the GitHub post to be fascinating and insightful reading. It was especially insightful to see the tools and methods used to understand better how Octopus Scanner behaves. For example, they used ClassFileTransformer and "a Bytecode manipulation library such as Javassist or ByteBuddy to inject our analysis code" into the "class responsible for decrypting the blob ... right before it actually gets loaded into the JVM."

Besides the interesting details of how Octopus Scanner works and how it was discovered and researched, other interesting insights in this GitHub post are related to the risks open source builds face. Muñoz writes, "Infecting build artifacts is a means to infect more hosts since the infected project will most likely get built by other systems and the build artifacts will probably be loaded and executed on other systems as well." Muñoz adds, "In an OSS context, it gives the malware an effective means of transmission since the affected projects will presumably get cloned, forked, and used on potentially many different systems. The actual artifacts of these builds may spread even further in a way that is disconnected from the original build process and harder to track down after the fact."

Muñoz opens the post and concludes the post with some discussions about this and other attempts at sabotaging open source products and their builds. One chilling thought is included in the Conclusion: "Since the primary-infected users are developers, the access that is gained is of high interest to attackers since developers generally have access to additional projects, production environments, database passwords, and other critical assets. There is a huge potential for escalation of access, which is a core attacker objective in most cases."

Thursday, April 30, 2020

Exact Absolute Integral Numbers in JDK 15

JDK 15 Early Access Build b18 introduced new methods to the Math and StrictMath classes that will throw ArithmeticException on provided values outside the range supported by the methods without overflow. These methods bring to the concept of "absolute value" in Java what the methods such as Math.addExact, Math.subtractExact, and Math.multiplyExact have brought to basic arithmetic functions.

Prior to JDK 15, Integer.MIN_VALUE and Long.MIN_VALUE caused the respective methods Math.abs and StrictMath.abs to return the same negative number as represented by the MIN_VALUE largest negative possible value. This behavior is described in the Javadoc documentation for the affected methods and is demonstrated by the code shown below (and available on GitHub):

Demonstrating Pre-JDK 15 Absolute Value Methods

/**
 * Demonstrates "absExact" methods added to {@link Math}
 * and {@link StrictMath} with JDK 15 Early Access Build b18
 * (JDK-8241374: https://bugs.openjdk.java.net/browse/JDK-8241374).
 */
public class AbsoluteExactness
{
   public void demonstrateMathAbsInteger(final int integer)
   {
      out.println("Math.abs(" + integer + "): " + Math.abs(integer));
   }

   public void demonstrateMathAbsLong(final long longNumber)
   {
      out.println("Math.abs(" + longNumber + "L): " + Math.abs(longNumber));
   }

   public void demonstrateStrictMathAbsInteger(final int integer)
   {
      out.println("StrictMath.abs(" + integer + "): " + StrictMath.abs(integer));
   }

   public void demonstrateStrictMathAbsLong(final long longNumber)
   {
      out.println("StrictMath.abs(" + longNumber + "L): " + StrictMath.abs(longNumber));
   }

   public static void main(final String[] arguments)
   {
      final AbsoluteExactness instance = new AbsoluteExactness();

      // Demonstrate pre-JDK 15 Math/StrictMath "abs" functions on minimum values.
      instance.demonstrateMathAbsInteger(Integer.MIN_VALUE+1);
      instance.demonstrateMathAbsInteger(Integer.MIN_VALUE);
      instance.demonstrateMathAbsLong(Long.MIN_VALUE+1);
      instance.demonstrateMathAbsLong(Long.MIN_VALUE);
      instance.demonstrateStrictMathAbsInteger(Integer.MIN_VALUE+1);
      instance.demonstrateStrictMathAbsInteger(Integer.MIN_VALUE);
      instance.demonstrateStrictMathAbsLong(Long.MIN_VALUE+1);
      instance.demonstrateStrictMathAbsLong(Long.MIN_VALUE);
   }
}

When the above code is executed, the following output is written:

Math.abs(-2147483647): 2147483647
Math.abs(-2147483648): -2147483648
Math.abs(-9223372036854775807L): 9223372036854775807
Math.abs(-9223372036854775808L): -9223372036854775808
StrictMath.abs(-2147483647): 2147483647
StrictMath.abs(-2147483648): -2147483648
StrictMath.abs(-9223372036854775807L): 9223372036854775807
StrictMath.abs(-9223372036854775808L): -9223372036854775808

This output demonstrates that the maximum negative allowable value in the int and long ranges leads to that same value being returned from the appropriate abs method on Math and StrictMath.

JDK 15 Early Access Build b18 introduces absExact methods that throw ArithmeticException in this case instead of returning the negative value. This is demonstrated with the following code (also available on GitHub):

Demonstrating JDK 15-Introduced absExact Methods

public class AbsoluteExactness
{
   public void demonstrateMathAbsExactInteger(final int integer)
   {
      try
      {
         out.println("Math.absExact(" + integer + "): " + Math.absExact(integer));
      }
      catch (ArithmeticException exception)
      {
         err.println("Math.absExact(" + integer + "): " + exception);
      }
   }

   public void demonstrateMathAbsExactLong(final long longNumber)
   {
      try
      {
         out.println("Math.absExact(" + longNumber + "L): " + Math.absExact(longNumber));
      }
      catch (ArithmeticException exception)
      {
         err.println("Math.absExact(" + longNumber + "L): " + exception);
      }
   }

   public void demonstrateStrictMathAbsExactInteger(final int integer)
   {
      try
      {
         out.println("StrictMath.absExact(" + integer + "): " + StrictMath.absExact(integer));
      }
      catch (ArithmeticException exception)
      {
         err.println("StrictMath.absExact(" + integer + "):" + exception);
      }
   }

   public void demonstrateStrictMathAbsExactLong(final long longNumber)
   {
      try
      {
         out.println("StrictMath.absExact(" + longNumber + "L): " + StrictMath.absExact(longNumber));
      }
      catch (ArithmeticException exception)
      {
         err.println("StrictMath.absExact(" + longNumber + "L): " + exception);
      }
   }

   public static void main(final String[] arguments)
   {
      final AbsoluteExactness instance = new AbsoluteExactness();

      // Demonstrate JDK 15-introduced Math/StrictMath "absExact" functions
      // on minimum values.
      instance.demonstrateMathAbsExactInteger(Integer.MIN_VALUE+1);
      instance.demonstrateMathAbsExactInteger(Integer.MIN_VALUE);
      instance.demonstrateMathAbsExactLong(Long.MIN_VALUE+1);
      instance.demonstrateMathAbsExactLong(Long.MIN_VALUE);
      instance.demonstrateStrictMathAbsExactInteger(Integer.MIN_VALUE+1);
      instance.demonstrateStrictMathAbsExactInteger(Integer.MIN_VALUE);
      instance.demonstrateStrictMathAbsExactLong(Long.MIN_VALUE+1);
      instance.demonstrateStrictMathAbsExactLong(Long.MIN_VALUE);
   }

The output from this code is shown next and demonstrates the clear exception message that is thrown when the MIN_VALUE is passed to the absExact methods.

Math.absExact(-2147483647): 2147483647
Math.absExact(-2147483648): java.lang.ArithmeticException: Overflow to represent absolute value of Integer.MIN_VALUE
Math.absExact(-9223372036854775807L): 9223372036854775807
Math.absExact(-9223372036854775808L): java.lang.ArithmeticException: Overflow to represent absolute value of Long.MIN_VALUE
StrictMath.absExact(-2147483647): 2147483647
StrictMath.absExact(-2147483648):java.lang.ArithmeticException: Overflow to represent absolute value of Integer.MIN_VALUE
StrictMath.absExact(-9223372036854775807L): 9223372036854775807
StrictMath.absExact(-9223372036854775808L): java.lang.ArithmeticException: Overflow to represent absolute value of Long.MIN_VALUE

I find it generally better to have an exception thrown for a surprising edge case than to have "something" returned that requires me to read the Javadoc to find out what that case is and what is returned in that case. The exception makes it obvious that the edge case was encountered rather than the discovery of a negative number being returned from an absolute value function call only being realized sometime later and "downstream" in the code. If nothing else, the mere presence of the Math.absExact and StrictMath.absExact methods should imply to Java developers that there is some "non-exact" possibilities to consider when using Java's math libraries to compute an absolute value and that realization might lead to reading the Javadoc to find out what those non-exact cases are.

Tuesday, February 18, 2020

Updates on Valhalla and Amber Records (Mid-February 2020)

Two posts on OpenJDK mailing lists today summarize the status of Project Valhalla and Project Amber's Records.

Valhalla

Brian Goetz begins his post "Valhalla -- finding the primitives" with this sentence, "I think its worth reflecting on how far we've come in Valhalla, both for the specific designs in the VM and language, and the clarity of the basic concepts." This post contains a brief history of the major design models that have been employed with Valhalla to this point and talks about the advantages and disadvantages of each of these models.

About "Q World," Goetz writes (I've added the emphasis), "The idea was that we would declare a class as either a value class or a 'regular' class, and we would derive various properties based on that." Goetz outlines these properties:

Q-World: Differentiating Properties of Regular Versus Value Classes
PropertyRegular ClassesValue Classes
Identity?YesNo
Nullable?YesNo
Reference types?YesNo

Goetz explains that "there were many aspects which were either confusing or unsatisfying" (he provides several examples of these) about the "Q World" approach and that the approach lacked a "clean story for migration."

Goetz describes "conflating" certain "distinctions" when working with "L World":

  • "nullable vs non-nullable"
  • "pass-by-reference vs pass-by-value / flattened"
  • "reference type vs value type"
  • "identity-ful vs identity-free"

Goetz discusses how the current state of Valhalla builds on lessons learned from the "Q World" and "L World" stages. He writes:

The primitive that Valhalla introduces into class declaration is whether the instances of the class have identity or not. Traditional classes are now revealed to be "identity classes"; the new kind (identity-free) are called "inline classes". (This might not be the final word on the subject.)

Another lesson learned in Valhalla efforts is summarized:

A big AHA of the recent iterations is that it makes sense to talk about both values of inline classes and references to those values. Reference type has (almost) nothing to do with inline vs identity -- it has to do with whether the value set of the type contains values, or references.

Based on these summarized observations, Goetz writes about the "ref/val notation for the types":

"Inline" is a way of saying "identity free" when declaring classes, but it doesn't say anything (yet) about the semantics of how we represent variables on the heap or pass them on the stack. For this, we need an additional property of the type, and ref vs val seems to ideally describe what we mean."

Amber Records

In the post "[records] Summary so far," Gavin Bierman writes about "some remaining design decisions concerning records." This post outlines the following questions/design decisions and then provides partial or complete answers to the design decisions (see the post for the additional details related to each design decision).

  1. Q1. Should the distinguished supertype of records java.lang.Record be renamed?
  2. Q2. Accessibility of mandated members.
  3. Q3. Nesting.
  4. Q4. Abstract records.
  5. Q5. Deconstruction patterns.
  6. Q6. @deprecated
  7. Q7. .equals and .toString are slow.
  8. Q8. Translation strategy.
  9. Q9. Hashing.
  10. Q10. Special annotation for explicit declaration of accessors.
  11. Q11. Changing the spec of .toString method
  12. Q12. Transactional methods
  13. Q13. Factory methods instead of constructors.

Although Records are available as a preview feature in JDK 14, it's obvious much work is still being done on them. This preview status provides an opportunity for Records to be used and tested to gain feedback that can change how they are implemented. The thirteen design decisions listed above (some of which are decidedly no change and some which are possible changes) demonstrate that Records are still going to be fine-tuned.

Saturday, February 1, 2020

JDK 14/JEP 305 instanceof Pattern Matching "Smart Casts"

I generally view the presence of of the instanceof operator in Java code as a "red flag," meaning that it's not necessarily wrong to use instanceof in certain situations, but its use sometimes indicates a design issue that could be resolved in a cleaner way as described in some resources referenced at the end of this post (including resources about similar type checking functionality in languages other than Java).

Although I've seen instanceof used several times when it does not need to be, I've run into even more situations where it was not easy to avoid instanceof. This is particularly true when working with legacy code bases and certain libraries and frameworks in which I have no ability to refactor relationships between classes to support interfaces, method overriding, and other tactics that can be used to remove the need for instanceof.

A very common technique employed with instanceof is to immediately cast to the type checked in the conditional using instanceof. JEP 305 ["Pattern Matching for instanceof (Preview)"] provides an example of this common pattern and I've slightly adapted that example here:

if (object instanceof String)
{
    final String string = (String) object;
    // Do something with the 'string' variable typed as String
}

Benji Weber has posted on using reflection and on using lambda expressions to achieve Kotlin-like "instanceof smart casts." Fortunately, JDK 14 and JEP 305 bring built-in language support (albeit preview status) for this approach.

JDK 14 introduces a preview feature that allows the instanceof conditional and associated cast to be implemented completely within the conditional. The effect on the above code example is shown next:

if (object instanceof String string)
{
    // Do something with the 'string' variable typed as String
}

This preview feature is available in the JDK 14 Early Access Builds and I'm using JDK 14 Early Access Build 34 for my examples in this post.

The JEP 305 preview feature in JDK 14 is a small nicety whose advantage is more obvious in lengthy if-then-else conditional statements. The next two code listings provide a comparison of the "old way" of calling instanceof and explicitly casting to the "new preview way" of using instanceof pattern matching.

Traditional instanceof Coupled with Explicit Cast
static void makeAnimalNoises(final Object animal)
{
   if (animal instanceof Dog)
   {
      final Dog dog = (Dog) animal;
      out.println(dog.bark());
   }
   else if (animal instanceof Cat)
   {
      final Cat cat = (Cat) animal;
      out.println(cat.meow());
   }
   else if (animal instanceof Duck)
   {
      final Duck duck = (Duck) animal;
      out.println(duck.quack());
   }
   else if (animal instanceof Horse)
   {
      final Horse horse = (Horse) animal;
      out.println(horse.neigh());
   }
   else if (animal instanceof Cow)
   {
      final Cow cow = (Cow) animal;
      out.println(cow.moo());
   }
   else if (animal instanceof Lion)
   {
      final Lion lion = (Lion) animal;
      out.println(lion.roar());
   }
   else
   {
      out.println("ERROR: Unexpected animal: " + animal);
   }
}

JDK 14/JEP 305 Preview Feature

static void makeAnimalNoises(final Object animal)
{
   if (animal instanceof Dog dog)
   {
      out.println(dog.bark());
   }
   else if (animal instanceof Cat cat)
   {
      out.println(cat.meow());
   }
   else if (animal instanceof Duck duck)
   {
      out.println(duck.quack());
   }
   else if (animal instanceof Horse horse)
   {
      out.println(horse.neigh());
   }
   else if (animal instanceof Cow cow)
   {
      out.println(cow.moo());
   }
   else if (animal instanceof Lion lion)
   {
      out.println(lion.roar());
   }
   else
   {
      out.println("ERROR: Unexpected animal: " + animal);
   }
}

The full code is on GitHub and the difference between the old approach and new preview approach is available.

Because instanceof pattern matching is a preview feature, the code using this feature must be compiled with the javac flags --enable-preview and -source 14. It must be executed with java flag --enable-preview.

Conclusion

For more details on how this feature is implemented, see the post "RFR: JDK-8237528: Inefficient compilation of Pattern Matching for instanceof." Pattern matching support for instanceof is another Amber-provided step toward reduced boilerplate code in Java.

 

Resources on Issues Using instanceof