client warning IOP00410219

[jarodzhong]

afer 30 minutes from test of specvrit2013, one or two clients report warnings like this:

-> Apr 20, 2015 11:32:52 AM com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl waitForResponse
-> WARNING: "IOP00410219: (COMM_FAILURE) Communications timeout waiting for response.  Exceeded 1,800,000 milliseconds"
-> org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 219 completed: Maybe
->      at logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3921)
->      at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3936)
->      at com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:187)
->      at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1149)
->      at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:282)
->      at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:394)
->      at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:364)
->      at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:235)
->      at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:187)
->      at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
->      at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
->      at org.spec.jappserver.mfg.workorderses.ejb._WorkOrderSes_DynamicStub.scheduleWorkOrder(org/spec/jappserver/mfg/workorderses/ejb/_WorkOrderSes_DynamicStub.java)
->      at org.spec.jappserver.driver.PlannedLine.createVehicle(PlannedLine.java:416)
->      at org.spec.jappserver.driver.PlannedLine.createVehicle(PlannedLine.java:375)
->      at org.spec.jappserver.driver.PlannedLine.run(PlannedLine.java:193)

When the test finish, it reprots
 "■Application Server Errors:
■PlannedLines Vehicle Rate FAILED
"

servier : glassfish 4.1
java:
[root@client1 SPECvirt]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

[AnoopGupta]

Thanks for using SPECvirt_sc2013 benchmark.
How many tiles are you running?

Could you please tar up the following and attach:
 - Client log files showing the warning
 - The html report, or the repot showing Application Server Errors
 - Complete GlassFish server log

On the client, you could also look at the latest result dir in SPECjAppServer2004/output/ to see *.err files.
If only some of the Mfg planned line transaction are failing, then it could just be network tuning related.

Please do ensure that firewall is turned off. You can look at the full disclosure tgz of SPECvirt_sc2013 results for tuning.

Do let us know if you are able to resolve this issue.

[jarodzhong]

test result 1-1.0

[AnoopGupta]

I did not find any client log files in your tars.
Also, did not find the GlassFish server log from the AppVM for the 4th tile, where validation error appears.

From your tars, it looks like MfgDriver could not run any transaction for the 4th tile.

Please provide the SPECjAppServer2004/output/<runID>/* files from the tile that has this issue.

Is it always the same one or two tiles that have this validation error?

[jarodzhong]

1 Is it always the same one or two tiles that have this validation error?
LOAD_SCALE_FACTORS value:1.0  sometimes tile3 ,somtetime tie0,1,3
LOAD_SCALE_FACTORS value:0.1  there is no  this validation errors.

2 /opt/SPECjAppServer2004/output/XX
Atomicity.html  dealer.err      emulator.err  Mfg.summary       result.props
Audit.report    Dealer.summary  loline.err    PerfData.txt      SPECjAppServer.summary
Dealer.detail   delivery.err    Mfg.detail    plannedlines.err  ValidationErr.txt

plannedlines.err   like this:
      1 2015-04-10 14:19:28:539: M1:120: Error occured in scheduleWorkOrderjava.rmi.MarshalException: CORBA COMM_FAILURE 1398079707         Maybe; nested exception is:
      2     org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 219 completed: Maybe
      3 2015-04-10 14:49:28:544: M1:255: Error occured in scheduleWorkOrderjava.rmi.MarshalException: CORBA COMM_FAILURE 1398079707         Maybe; nested exception is:
      4     org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 219 completed: Maybe
      5 2015-04-10 15:19:33:556: M1:393: Error occured in scheduleWorkOrderjava.rmi.MarshalException: CORBA COMM_FAILURE 1398079707         Maybe; nested exception is:
      6     org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 219 completed: Maybe
      7 2015-04-10 15:50:13:604: M1:531: Error occured in scheduleWorkOrderjava.rmi.MarshalException: CORBA COMM_FAILURE 1398079707         Maybe; nested exception is:
      8     org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 219 completed: Maybe
      9 2015-04-10 14:19:26:142: M1:193: Exception in update WorkOrder(3430472, 1) java.rmi.MarshalException: CORBA COMM_FAILURE 139        8079707 Maybe; nested exception is:
     10     org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 219 completed: Maybe
  ..........

[AnoopGupta]

Thanks for the update.

If LOAD_SCALE_FACTORS value:0.1 is running for 4 tiles without any issues, then configuration is fine.
Communication failures at LOAD_SCALE_FACTORS value:1.0 implies that you may be missing some tuning. Basically, some RMI driver threads are unable to connect to your GlassFish instances. 

The GlassFish server log may have some relevant information. Please see <your GlassFish installation dir>/glassfish4/glassfish/domains/spec*/logs/server.log

You can look at the full disclosure tgz of SPECvirt_sc2013 results for network tuning and any GlassFish tuning in domain.xml.

You may also try a LOAD_SCALE_FACTORS of less than 1.0, like 0.4, 0.8, etc, to see when it starts failing. Your setup may not have enough resources to support LOAD_SCALE_FACTORS=1.0.