AppServer write exceptions during test

[bcarson]

I ran a test for only the AppServer (NUM_WORKLOADS=1)
Should I be concerned about the validity of the test results with the plethora of "Connection:sendRequest: write exception, retry ..." in the attached log?
The test seems to have succeeded however I also see these errors in the log:
"AbstractSJASLoad> Application error has already been cancelled"
"O1:2041: error adding cars to cart for customer 2590686 - Part 2 java.lang.NullPointerException"
"O1:1697: error adding cars to cart for customer 1105229 - Part 2 java.lang.NullPointerException"
"O1:956: error adding cars to cart for customer 1440489 - Part 2 java.lang.NullPointerException"

Should we expect a few random errors since it's under heavy load or does this point to the need for a settings adjustment?

Thanks,
-Bruce

[lroderic]

Bruce, a handful of the "error adding cars" is OK, but the "write exception, retry" isn't. If you're running one workload, the load isn't enough to be causing these errors. Please zip up the /opt/SPECvirt/*.out files, Control.config, /opt/glassfish3/glassfish/domains/spec2004-1/config/domain.xml, and /opt/glassfish3/glassfish/domains/spec2004-1/logs/server.log. Thanks.

[bcarson]

See attachment. There are a lot of errors in the server.log that I can't explain.

[lroderic]

Also please post /opt/SPECvirt/Clientmgr*.out.

fwiw, You had set NUM_WORKLOADS=4, so you were running them all. But you should be able to run them all. Are you using 10GbE between the client and tile? Or is the client on the same host as the tile?

Lisa

[bcarson]

Yeah, I had started another test with NUM_WORKLOADS=4 prior to receiving your log requests but anything prior to 11:30 in the logs was definitely with NUM_WORKLOADS=1.
The test client server and workload VMs are communication on 1GigE NICs.
As for /opt/SPECvirt/Clientmgr*.out I stranglely don't see it on this test client - does it only appear once the test is done (there is a test running right now).

[lroderic]

If you're running appserver, 1GbE between the client and appserver VM will be OK. A full tile requires about 1.5 Gb, so running a full tile will saturate a 1 GbE NIC.

The /opt/SPECvirt/Clientmgr*.out files get created immediately when the harness starts the Clientmgr.sh processes for each workload. The format is Clientmgr1_1088.out for prime client output, Clientmgr1_1098.out for appserver, Clientmgr1_1096.out for webserver, and so on. Do you have enough disk space on the client?

[bcarson]

This is space on the client and the only out file is still primectrl.out
[root@g1llloadgen002 20151209-113710]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root
                           916G  358G  511G  42%  /
/dev/sda1           99M      27M   68M  29%  /boot
tmpfs                  24G           0   24G     0% /dev/shm

The NUM_WORKLOADS=4 test completed (successful enough to have a results.htm) and the only errors I see are in the logs for AppServer (attached) and WebServer:
client-172.19.106.79_1091.log shows a lot of "Connection:sendRequest: write exception, retry"
client-172.19.106.79_1010.log shows a lot of "SPECweb_Support: [ERROR] SocketTimeoutException encountered during run"

[lroderic]

While I look into the errors, please copy/paste the contents of /opt/SPECvirt/Clientmgr.sh here.

[bcarson]

At /opt/SPECvirt/ the client only has clientmgr.jar

[lroderic]

Did you run /opt/clientvm_scripts/makeme_client.sh? Did you download the helper scripts from https://www.spec.org/forums/index.php?action=dlattach;topic=15.0;attach=54?

runspecvirt.sh calls Clientmgr.sh which contains:

java -jar clientmgr.jar -p 1098 -log > Clientmgr$1_1098.out 2>&1 &
java -jar clientmgr.jar -p 1096 -log > Clientmgr$1_1096.out 2>&1 &
java -jar clientmgr.jar -p 1094 -log > Clientmgr$1_1094.out 2>&1 &
java -jar clientmgr.jar -p 1092 -log > Clientmgr$1_1092.out 2>&1 &
java -jar clientmgr.jar -p 1088 -log > Clientmgr$1_1088.out 2>&1 &

[bcarson]

I've been starting a test this way:
pkill -9 java; cd /opt/SPECvirt; ulimit -n 100000; screen -d -S app -m java -jar clientmgr.jar -p 1098 -log; screen -d -S web -m java -jar clientmgr.jar -p 1096 -log; screen -d -S mail -m java -jar clientmgr.jar -p 1094 -log; screen -d -S batch -m java -jar clientmgr.jar -p 1092 -log; screen -d -S ctrl -m java -jar clientmgr.jar -p 1088 -log; screen -d -S specvirt -m java -jar specvirt.jar -l

I think I like your way better.   I'll try it...

Thanks!
-Bruce

[bcarson]

I'm still not seeing any other .out files but here's the Clientmgr.sh info

[root@g1llloadgen002 SPECvirt]# cat Clientmgr.sh
# Clientmgr [tile_index]
# Script called from runspecvirt.sh
#
java -jar clientmgr.jar -p 1098 -log > Clientmgr$1_1098.out 2>&1 &
java -jar clientmgr.jar -p 1096 -log > Clientmgr$1_1096.out 2>&1 &
java -jar clientmgr.jar -p 1094 -log > Clientmgr$1_1094.out 2>&1 &
java -jar clientmgr.jar -p 1092 -log > Clientmgr$1_1092.out 2>&1 &
java -jar clientmgr.jar -p 1088 -log > Clientmgr$1_1088.out 2>&1 &

[lroderic]

I highly recommend you download the example VM guide and both sets of scripts from https://www.spec.org/forums/index.php?topic=15.0.  It makes it much easier to support you.

Meanwhile, please confirm you're using the domain.xml I posted at https://www.spec.org/forums/index.php?topic=44.0. Let's get appserver working before you add the other workloads, so keep NUM_WORKLOADS=1.

[bcarson]

As you suggested I'll be focusing only on NUM_WORKLOADS=1

I'm using all the helper scripts (that I know about) and I'm starting the test with ./runspecvirt.sh but I'm still not seeing any of the logs specified in Clientmgr.sh
java -jar clientmgr.jar -p 1098 -log > Clientmgr$1_1098.out 2>&1 &
java -jar clientmgr.jar -p 1096 -log > Clientmgr$1_1096.out 2>&1 &
java -jar clientmgr.jar -p 1094 -log > Clientmgr$1_1094.out 2>&1 &
java -jar clientmgr.jar -p 1092 -log > Clientmgr$1_1092.out 2>&1 &
java -jar clientmgr.jar -p 1088 -log > Clientmgr$1_1088.out 2>&1 &

df -h shows plenty of free space so not sure why I don't see the different .out files. But if I go into the /logs/ directory will those logs contain the same info? Such as:
client-172.19.106.79_1091.log
prime-g1llloadgen002.loadtest.glbt1.gdg_1098.log

[lroderic]

It looks like it. Please post their contents or attach them. Thanks.