We have a bunch of “identical” CentOS 6.2 servers (in terms of Java version, OS version, JBoss AS7 7.1, version of our app’s war file) where most of them are now failing when creating a javax.xml.ws.Service. One of them is still working and the same JBoss AS7 and war files work locally on our developer machines (Macs). The service exists, its wsdl location URL and service name are the same on the working box and the non-working one.
I’ve diff’d the entire JBoss AS7 directory which includes all of the JBoss libraries, scripts, configs and our war between the working machine and a non-working one. The only diffs I see are related to the IP addresses of the machine itself.
Here’s the cause of the stack trace we’re seeing with a few lines omitted:
...a lot of stuff omitted...
Caused by: java.lang.NoClassDefFoundError: org/jboss/xb/binding/ObjectModelFactory
at org.jboss.ws.core.jaxws.spi.ServiceDelegateImpl.<init>(ServiceDelegateImpl.java:137) [jbossws-native-core-3.1.1.GA.jar:3.1.1.GA]
at org.jboss.ws.core.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:63) [jbossws-native-core-3.1.1.GA.jar:3.1.1.GA]
at javax.xml.ws.Service.<init>(Service.java:57) [jboss-jaxws-api_2.2_spec-2.0.0.Final.jar:2.0.0.Final]
at schemas.dynamics.microsoft.nav.system.SystemService.<init>(SystemService.java:41) [navisionws-1001.0-SNAPSHOT.jar:]
...our code omitted...
... 79 more
Caused by: java.lang.ClassNotFoundException: org.jboss.xb.binding.ObjectModelFactory from [Module "deployment.arinweb_public-1001.0-SNAPSHOT.war:main" from Service Module Loader]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:423)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
... 86 more
We’ve tried a bunch of things including:
- Moving the entire jboss directory from the non-working machine to the working machine where it starts to work.
- Doing the opposite of #1 where the jboss that worked on one machine no longer works on the non-working machine.
- Compared all environment variables on the two machines and they match.
I’d like suggestions on what to try next to figure out what the problem is. Either that or the solution would be great.
EDIT – classloading problems while JBoss is starting deployment:
We’re getting some classloading issues while JBoss starts deployment of each app. I’ve compared these warnings between the working machine and the non-working one and they are the same. Here’s an example (there are 37 of the first warning type and 55 of the second on both servers):
18:20:04,352 WARN [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015893: Encountered invalid class name 'implementingClassName=org.jboss.shrinkwrap.impl.base.ConfigurableArchiveImpl' for service type 'org.jboss.shrinkwrap.spi.Configurable'
16:34:16,370 WARN [org.jboss.as.server.deployment] (MSC service thread 1-40) Class Path entry shrinkwrap-resolver-impl-maven-1.0.0-beta-6.jar in "/content/snipped_war_name.war/WEB-INF/lib/snipped_jar_name.jar" does not point to a valid jar for a Class-Path reference.
Thank you all for the suggestions. After a lot of debugging, we found the issue and I’ll post the details here for the one-in-a-million chance someone has the same issue:
These jars we package in our wars were the culprit:
JBoss AS 7 packages its own version which is newer than the one we were packaging. Once we removed these jars from our wars and bounced the server, the problem seems to go away.
Our long term solution to remove these dependencies was to upgrade JBoss web services from 3.1.1.GA to 4.0.1.GA which is CXF based. This involved a bunch of code changes but allowed us to remove the bad dependencies.