Fix: Spring-based JNDI lookup can't see .properties file

After refactoring one of our internal applications at work to use Spring XML configuration file based JNDI lookup today, several of the application's unit tests which involve testing of JNDI lookups started failing. The following error appeared for each failed test in the jUnit XML log file:

<error message="Error creating bean with name '[dsName]':
Invocation of init method failed; nested exception is
  javax.naming.NoInitialContextException: Need to specify class name in
  environment or system property, or as an applet parameter, or in an
  application resource file:  java.naming.factory.initial"
type="org.springframework.beans.factory.BeanCreationException">
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory
  .initializeBean(AbstractAutowireCapableBeanFactory.java:1338)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory
  .doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
... (snipped for brevity) ...
at org.springframework.context.support.ClassPathXmlApplicationContext
  .<init>(ClassPathXmlApplicationContext.java:83)
at [internal namespace].util.TestUtil.getSpringContext(TestUtil.java:167)

Cause

We discovered that the cause of the problem in our case was that, as the error message indicated, Spring wasn't seeing a .properties file present in the project which set a value for the java.naming.factory.initial property.

The line in our Spring context .xml file looked like:

<jee:jndi-lookup id="[dsName]" jndi-name="jdbc/[dsPath]/[dsName]"
  expected-type="javax.sql.DataSource" />

The line in our .properties file (which Spring wasn't finding properly), which set the naming factory class to a class provided by our J2EE application server (WebLogic), was:

java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory

The .properties file was located in the same folder as the Spring context .xml file.

Solution

In the Spring context .xml file, just prior to the jee:jndi-lookup tag, we added a bean of type PropertiesFactoryBean to act as a pointer to the .properties file:

<bean id="jdbcConfiguration"
  class="org.springframework.beans.factory.config.PropertiesFactoryBean">
  <property name="location" value="classpath:productInquiry.properties"/>
</bean>

Then, we added a reference to that new bean to the existing jee:jndi-lookup tag (the added part is bolded):

<jee:jndi-lookup id="ot1ds" jndi-name="jdbc/[dspath]/[dsname]"
  expected-type="javax.sql.DataSource" environment-ref="jdbcConfiguration"/>

This change fixed the problem, and got our tests to run successfully.

I hope this is helpful to someone!

Getting JAX-WS to work with WebLogic 9.2

I spent several hours at work today troubleshooting problems related to getting JAX-WS (as included with Apache CXF) working in a Java web application running under BEA (Oracle) WebLogic 9.2.  This post contains a summary of the errors I encountered, and their resolutions -- hopefully this will save someone some of the troubleshooting time that I spent today.

Background

Some brief background: I was building some new SOAP web service integrations into one of our internal Java applications, which runs on WebLogic server.  This application was the client side of the integration.  I used Spring, Apache CXF, and JAX-WS to build the integrations.

I first coded the SOAP integrations into a simple stand-alone Java application, and that worked fine.  However, when I plugged my code and the CXF .jar files into the "real" application running on my local WebLogic server, I ran into a couple of errors.

Error #1: "java.lang.NoSuchMethodError: javax.jws.WebMethod.exclude"

Upon running my application under WebLogic 9.2, I got this error in the WebLogic server log file:

org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'internalBeanName': Instantiation of bean failed;
nested exception is org.springframework.beans.factory.BeanDefinitionStoreException:
Factory method [public java.lang.Object org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create()]
threw exception; nested exception is java.lang.NoSuchMethodError: javax.jws.WebMethod.exclude()Z

This error turned out to be caused by the fact that in addition to the javax.jws.WebMethod class included in the geronimo-ws-metadata_2.0_spec-1.1.2.jar file included in the distribution of CXF that I downloaded, another (apparently older) implementation of that class is also included in the weblogic.jar file included with WebLogic 9.2.  WebLogic by default assigned priority to classes from its own weblogic.jar file over classes included with my application; as a result, WebLogic tried to use the older implementation of the javax.jws.WebMethod class, which does not include the exclude() method, and the NoSuchMethodError occurred when the other JAX-WS code tried to access that method.

To fix this, I modified my weblogic.xml and weblogic-application.xml files, to instruct WebLogic to give priority to the JAX-WS implementation in my own supplied .jar files rather than its own JAX-WS implementation. 

In my weblogic.xml, I added:

<container-descriptor>
  <prefer-web-inf-classes>true</prefer-web-inf-classes>
</container-descriptor>

For more on this weblogic.xml change, see documentation on a similar issue affecting Apache Axis.

In my weblogic-application.xml, I added:

<prefer-application-packages>
    <package-name>org.apache.xerces.*</package-name>
    <package-name>javax.jws.*</package-name>
</prefer-application-packages>

For more on this weblogic-application.xml change, see the WebLogic configuration documentation in the CXF documentation.

Error #2: "Your JAXP provider [...] does not support XML Schema"

Having resolved the NoSuchMethodError, I re-deployed my application, and got this error upon running the app:

org.springframework.beans.factory.BeanDefinitionStoreException: Parser configuration exception parsing XML from class path resource [mySpringContext.xml]; nested exception is javax.xml.parsers.ParserConfigurationException: Unable to validate using XSD: Your JAXP provider [org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@nnnnnnn] does not support XML Schema. Are you running on Java 1.4 with Apache Crimson? Upgrade to Apache Xerces (or Java 1.5) for full XSD support.

The problem here was that there was an an old (2003) xerces.jar file present in my Java project, which contained an old JAXP provider class which did not support XSD. 

To fix this, I deleted the old xerces.jar file, and redeployed the project.  (I also needed to delete WebLogic's cache, located on my machine at \bea\user_projects\domains\mydomain\servers\myserver\tmp, to completely get rid of the old cached xerces.jar file.)

Java: Getting a String with the current call stack

Occasionally, in debugging, it can be useful to get the current call stack trace of a program as a String. Here's a simple Java method that will do this:

private static String getCallStackString()
{
 java.io.ByteArrayOutputStream byteArrayOutputStream = new java.io.ByteArrayOutputStream();
 java.io.PrintStream printStream = new java.io.PrintStream(byteArrayOutputStream);
 Throwable throwable = new Throwable("Current call stack:");
 throwable.printStackTrace(printStream);
 return byteArrayOutputStream.toString();
}

Note that this works even in pre-1.4 Java versions, since it doesn't use the getStackTrace method (introduced in Java 1.4). The output of this method is a string along the lines of the following:

java.lang.Throwable: Current call stack:
 at JavaTest.getCallStack(JavaTest.java:36)
 at JavaTest.main(JavaTest.java:24)

A minor variant of this method can be used to get the call stack of an existing Exception:

private static String getCallStackString(Throwable throwable)
{
 java.io.ByteArrayOutputStream byteArrayOutputStream = new java.io.ByteArrayOutputStream();
 java.io.PrintStream printStream = new java.io.PrintStream(byteArrayOutputStream);
 throwable.printStackTrace(printStream);
 return byteArrayOutputStream.toString();
}

C#/Java/C++: Combining a variable assignment and evaluation

Pop Quiz! C#/Java/C++/Javascript/(probably others, too!) programmers, off the top of your head, what's the result of evaluating a variable assignment? In other words, to take a specific example, what is the output of this Java code snippet:

int n;
System.out.println(n = 50);

(Feel free to substitute in Console.Out.WriteLine (C#) or good old printf (C++) for the System.out.println in that snippet, depending on your language of choice.)

The answer is: 50. In Java and the other languages mentioned, the result of the evaluation of a variable assignment is the value being assigned.

I just came across this construct myself for the first time while I recently was doing a code review of a colleague's Java code. Somehow, prior to that code review, I had managed to go for over a decade of developing in these various languages without running across this!

The reason I hasn't run across this before may have to do with code readability. Doing two different things at once (in this case, a combined variable assignment and evaluation) often isn't very good for code readability (and therefore for ease of maintainability); in the general case, then, it probably makes the most sense for the variable assignment and evaluation to just be separated into two separate lines of code.

However, as my colleague's code demonstrated, combining an assignment with an evaluation can be useful when setting up a loop where the same statement is executed to assign a value to the loop variable both before the loop starts, and on each subsequent iteration of the loop. For example, here's a Java example of reading data from an input file a line at a time, using a java.io.BufferedReader:

String inputLine;
while ((inputLine = bufferedReader.readLine()) != null)
{
    // Do something with inputLine...
}

The while statement in this case combines the assignment of the variable inputLine to the line of text read from the BufferedReader, with the check to stop looping when inputLine is null.

In the past, I've written the same logic in this manner:

String inputLine = bufferedReader.readLine();
while (inputLine != null)
{
    // Do something with inputLine...

    inputLine = bufferedReader.readLine();
}

I had always been kind of annoyed over the need to repeat the assignment (inputLine = bufferedReader.readLine()) in two different places.

For writing loops like this in the future, I'll have to think more about whether the gain in code brevity (and debatably, in elegance) from using the former approach (the combined assignment/evaluation in the while statement) is worth the potential cost for future maintainers in the readability of the code.

Java: Displaying negative percentage values in red and in parentheses

Recently, I was looking to write some code in Java for a financial application that would output percentage-format numbers, rounded to two decimal places, with negative numbers being displayed in parentheses and in red color. This was an internal application which only needed to work in the US/English locale.

For example, given the values 0.12345 and -0.12345, the formatted output needed to be, respectively:

12.35%
(12.35%)

Since for this application the output only needs to be formatted in the default locale, it can be done with one of the constructors of the Java DecimalFormat class. If the output will be displayed in a web browser (and only in a web browser), the HTML to display the red color for negative values can be embedded in the string passed to the NumberFormat constructors as well:

NumberFormat redNegativePercentTwoDecimalsFormat = new java.text.DecimalFormat(
  "0.00%;'<span style=\"color:#FF0000\">'(0.00%)'</span>'");

The created NumberFormat instance can then be used to output values in our desired format:

System.out.println(redNegativePercentTwoDecimalsFormat.format(0.12345f));
System.out.println(redNegativePercentTwoDecimalsFormat.format(-0.12345f));

Which produces the desired output (when viewed in a web browser):

12.35%
(12.35%)

Fix: ClassCastException in Struts getRequestProcessor under WebLogic 7

At work yesterday, we were having a problem with one of our internal web applications running on BEA WebLogic Server 7 SP 7, Struts 1.1, and Java 1.4, where following an application redeployment, users of the application would get HTTP 500 errors. The WebLogic server output log file showed multiple ClassCastException errors with the following call stack:

java.lang.ClassCastException
at org.apache.struts.action.ActionServlet.getRequestProcessor(ActionServlet.java:871)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1508)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
. . .
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2642)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:262)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:224)

After some troubleshooting, we determined that this error would only occur when the application was under some load (from being hit by multiple users) while it was starting up; the error wouldn't occur if only a single user accessed the application while it was starting.

So, restarting the application while it was not under load (e.g. overnight) turned out to be a viable workaround in our case. Although we were back up and running at this point, I continued to research the issue to get a better understanding of the underlying cause, and to come up with a better solution should we ever need to restart the server during the work day in the future.

Researching the Issue

Some research via Google didn't turn up any obvious solutions on the issue. The exception was being thrown from the Struts class ActionServlet.getRequestProcessor; since Struts is open source, I decided to download the Struts source, and add some additional instrumentation to the getRequestProcessor method to see if that would shed any additional light on the issue.

ActionServlet.java line 871 (from the ClassCastException stack trace) turned out to be the second of these two lines, at the top of the ActionServlet.getRequestProcessor method:

String key = Globals.REQUEST_PROCESSOR_KEY + config.getPrefix();
RequestProcessor processor = (RequestProcessor)getServletContext().getAttribute(key);

So the exception was happening on the attempt to cast the result of getServletContext().getAttribute(key) to type RequestProcessor.

I added some logging to the getRequestProcessor method to log the class name of the result of the getServletContext().getAttribute(key) call to stderr (so that it would be visible in the WebLogic server log), along with the name of the current thread (so I would hopefully be able to get visibility into whether different application threads were getting different results).

Object o = getServletContext().getAttribute(key);
if (o == null)
{ 
  System.err.println("$$$ " + java.lang.Thread.currentThread().getName() + ": " 
    + "getServletContext().getAttribute(key) is null"); 
}
else
{
  System.err.println("$$$ " + java.lang.Thread.currentThread().getName() + ": " 
    + "type=" + o.getClass().toString());
}

(The "$$$" here is just a poor man's bookmark to let me easily search for my debug logging in the server log file.)

In the WebLogic log file, the type of the object causing the ClassCastException upon the attempt to cast it to type RequestProcessor was of type PIRequestProcessor, a custom type used by my company's application. I was initially confused by this, because PIRequestProcessor is declared to extend RequestProcessor in its class declaration, so casting a PIRequestProcessor to a RequestProcessor should be a valid operation. Yet this operation was definitely causing the ClassCastException.

At this point, I remembered a post on Java class loaders that I had just recently read on Kevin Bourrillion's blog (some slightly NSFW content). From Kevin's post:

But now we're finally getting to the interesting part: every class in memory in your runtime environment can be uniquely identified by the pair of (a) its full name (b) the class loader that loaded it

And also:

You may have heard someone explain, or you may have explained yourself, "see, you can't cast a foo.Bar to a foo.Bar here even though it's the same class, because they came from different class loaders, so there's funny class loader hoodoo going on there."

Aha! I improved my instrumentation on the getRequestProcessor method to include the classloader of the PIRequestProcessor instance, and of the current thread:

System.err.println("$$$ " + java.lang.Thread.currentThread().getName() + ": " 
  + type=" + o.getClass().toString() 
  + " | classloader of o: " + o.getClass().getClassLoader().toString() 
  + " | current thread classloader: " + java.lang.Thread.currentThread().getContextClassLoader());

Sure enough, I got two sets of output in the log. One set like the following, for the first thread that got into the getRequestProcessor method, which looked like this:

$$$ ExecuteThread: '5' for queue: 'default': type=class com.gfs.mps.application.productInquiry.struts.PIRequestProcessor instanceof:true classloader of o: weblogic.utils.classloaders.ChangeAwareClassLoader@4b4b50 finder: weblogic.utils.classloaders.MultiClassFinder@6edffb current thread classloader: weblogic.utils.classloaders.ChangeAwareClassLoader@4b4b50 finder: weblogic.utils.classloaders.MultiClassFinder@6edffb

And another set of logged output, for the subsequent threads that get into getRequestProcessor, like this:

$$$ ExecuteThread: '7' for queue: 'default': type=class com.gfs.mps.application.productInquiry.struts.PIRequestProcessor instanceof:false classloader of o: weblogic.utils.classloaders.ChangeAwareClassLoader@4b4b50 finder: weblogic.utils.classloaders.MultiClassFinder@6edffb current thread classloader: weblogic.utils.classloaders.ChangeAwareClassLoader@dfb821 finder: weblogic.utils.classloaders.MultiClassFinder@1a1569b

(Bolded emphasis added by me; note the different class loader ID in the second log output.) The problem was that WebLogic was using two different class loaders in the different threads. The PIRequestProcessor[4b4b50] instance (i.e. a PIRequestProcessor instance created via the class loader with ID 4b4b50) was created and cached in the first thread in the call to getRequestProcessor. When the cached PIRequestProcessor was retrieved by the second thread in its call to getRequestProcessor, the PIRequestProcessor[4b4b50] instance was returned, and when the attempt was made to cast that to a RequestProcessor[dfb821] (created by the dfb821 class loader rather than the 4b4b50 class loader), the ClassCastException resulted.

Most likely this issue between Struts 1.1 and WebLogic Server 7 SP 7 has been addressed in the current versions of one or both products; both are fairly old versions of their respective products. (A poster in that BEA forum thread claims to have been sent a patch by BEA, CR189815, that resolves this issue in WebLogic 8.1.) But in the meantime, I still wanted to come up with a solution to the issue for our application running in that environment.

The Solution

In my initial research on Google, one of the items I had come across was an article on avoiding unnecessary JSP recompilations under WebLogic 8.1 on the BEA site by Nagesh Susarla. Among other things, the article mentions a WebLogic Server parameter, "servlet-reload-check-secs", which can be set to have WebLogic not monitor during runtime whether any new Java class files have been put into place. I also came across a thread in the BEA forums which mentioned someone else running WebLogic Server having a similar problem with a ClassCastException (not really in similar circumstances my own issue, but with a similar exception call stack) and with one of the replies suggesting that the WebLogic setting to disable to disable servlet reloading at runtime be set.

I decided to give that solution a try. Initially it wasn't obvious to me where the WebLogic 7 equivalent of the servlet-reload-check-secs parameter could be set, or even whether that parameter was supported at all under WebLogic 7; my web searches turned out some documentation on the setting for WebLogic 8.1, but not for earlier versions.

After some poking around in the WebLogic 7 console, I did find the place where the equivalent of "servlet-reload-check-secs" can be set: The setting is called "Reload Period", located in the console at [domain] | Deployments | Web Applications | [app name] | Configuration tab | Files sub-tab. The setting's default is 1; setting the value to -1 and restarting the WebLogic server disables WebLogic's runtime servlet reload checking for the web application where the setting was made. (The underlying WebLogic domain config.xml setting is called "ServletReloadCheckSecs".)

After making that setting change, things started working correctly! In my log, I could see that WebLogic was no longer using different class loaders in calls to the Struts ActionServlet.getRequestProcessor method; therefore, no ClassCastException was occurring, and the application was loading properly.