Errors in client while running runspecvirt.sh

[Miles]

Sorry, I have so many questions and errors as follows:
primectrl.out
2017-06-15 15:06:33:662 Thu Jun 15 15:06:33 CST 2017
2017-06-15 15:06:33:721 RMI server started: client1:9990
2017-06-15 15:06:33:724 [INFO] This is a perf-only benchmark run. Skipping active idle polling interval.
2017-06-15 15:06:33:724 PrimeControl: preparing client drivers.
2017-06-15 15:06:33:760 PrimeControl: starting clients...
2017-06-15 15:06:33:803 PrimeControl: PTDs not used for this benchmark run!
2017-06-15 15:06:33:803 PrimeControl: starting 4 masters.
2017-06-15 15:07:51:003 PrimeControl: waiting on 4 prime client(s).
2017-06-15 15:07:51:335 [ERROR] Received abort signal from client1:1092. Terminating.
2017-06-15 15:07:51:336 PrimeControl: sending abortTest() to prime clients.
2017-06-15 15:07:51:336 PrimeControl: id=3, abortID=3
2017-06-15 15:07:51:336 PrimeControl: id=2, abortID=3
2017-06-15 15:07:51:336 PrimeControl: id=0, abortID=3
2017-06-15 15:07:51:336 PrimeControl: id=1, abortID=3
2017-06-15 15:07:51:336 PrimeControl: stopping remote client processes
.
2017-06-15 15:07:53:340 PrimeControl: stopping local client threads
2017-06-15 15:07:53:341 Terminating client processes
2017-06-15 15:07:53:341 Sending termination signal to clientmgr processes
2017-06-15 15:07:53:341 Terminating prime client processes
2017-06-15 15:07:53:341 Sending termination signal to prime clientmgr processes
2017-06-15 15:07:53:341 Exiting...
2017-06-15 15:07:53:341 PrimeControl: terminating run. Please wait...
2017-06-15 15:07:53:343 specvirt: benchmark run failed!
2017-06-15 15:07:53:343 specvirt: Done!

Clientmgr1_1092.out
2017-06-15 15:06:13:673 Creating clientmgr using RMI Registry port 1092
2017-06-15 15:06:13:763 client1:1092 ready...
2017-06-15 15:07:50:990 Starting client1:1092
2017-06-15 15:07:50:991 Command line: [java, -jar, specbatch.jar, -sv, client1, -svp, 9990, -id, 3, -tile, 0, -wkld, 3]
2017-06-15 15:07:51:002 Started client1:1092

Messages from: client1:1092
-> 2017-06-15 15:07:51:090 Looking up SPECvirt controller: client1
-> 2017-06-15 15:07:51:224 masterID: 3, tile: 0, workload: 3
-> 2017-06-15 15:07:51:225 hostname: client1
-> 2017-06-15 15:07:51:248 loadscale:1.0 copycount:10
-> 2017-06-15 15:07:51:267 RMI server started: client1:9903
-> 2017-06-15 15:07:51:334 [ERROR] VM or client clocks are out of time sync with prime client. Aborting run.
-> Prime: 1497510471310/1497510471334
-> batchserver: 1497510467372

Clientmgr1_1094.out
2017-06-15 15:06:13:722 Creating clientmgr using RMI Registry port 1094
2017-06-15 15:06:13:809 client1:1094 ready...
2017-06-15 15:07:50:986 Starting client1:1094
2017-06-15 15:07:50:987 Command line: [java, -jar, specimap.jar, -calibrate, -sv, client1, -svp, 9990, -id, 2, -tile, 0, -wkld, 2]
2017-06-15 15:07:50:989 Started client1:1094

Messages from: client1:1094
-> 2017-06-15 15:07:51:087 Looking up SPECvirt controller: client1
-> 2017-06-15 15:07:51:215 masterID: 2, tile: 0, workload: 2
-> 2017-06-15 15:07:51:215 RMI hostname: client1
-> fixedFile: IMAP_fixed.rc; configFile: IMAP_config.rc; workloadFile: IMAP_sysinfo.rc
-> 2017-06-15 15:07:51:313 RMI server started: client1:9902
-> 2017-06-15 15:07:51:316 SpecimapControl: Message initialization=false
-> 2017-06-15 15:07:51:316 SpecimapControl: Folder initialization=false
-> 2017-06-15 15:07:51:316 SpecimapControl: Message verification=false
-> 2017-06-15 15:07:51:316 SpecimapControl: Folder verification=false
-> 2017-06-15 15:07:51:316 SpecimapControl: Message clean=false
-> 2017-06-15 15:07:51:316 SpecimapControl: Folder clean=false
-> 2017-06-15 15:07:51:316 SpecimapControl: rampUp=true
-> 2017-06-15 15:07:51:316 SpecimapControl: warmUp=true
-> 2017-06-15 15:07:51:316 SpecimapControl: benchmark=true
-> 2017-06-15 15:07:51:316 SpecimapControl: overwritten=false
-> 2017-06-15 15:07:51:316 SpecimapControl: calibrate=true
-> 2017-06-15 15:07:51:317 ******************** Specmail benchmark started
-> 2017-06-15 15:07:51:338 Adding host client1:1200
-> 2017-06-15 15:07:51:343 Checking for all required parameters in config.rc
-> 2017-06-15 15:07:51:343 All required parameters in config.rc file are present
-> 2017-06-15 15:07:51:343 TEST LOAD = 100%
-> 2017-06-15 15:07:51:343 Setting workload...
-> 2017-06-15 15:07:51:344 Current load factor is: 100%
-> 2017-06-15 15:07:51:344 Provisioned users involved in this test:500
-> 2017-06-15 15:07:51:344 Workload: client1:1200,CT,test1-test500,mailserver,6
-> 2017-06-15 15:07:51:348 Adding clients...
-> 2017-06-15 15:07:51:348 Adding host[0]: client1:1200
-> 2017-06-15 15:07:51:351 Initializing clients (this may take a few minutes)...
-> 2017-06-15 15:07:51:417 [ERROR] VM or client clocks are out of time sync with prime client. Aborting run.
-> Prime: 1497510471393/1497510471417
-> mailserver: 1497510468413
-> Error: Problem encoding the raw results file.
2017-06-15 15:07:55:342 Terminating processes. Please wait...
2017-06-15 15:07:55:343 Killing master procs ...
2017-06-15 15:07:55:343 Done killing procs ...

Clientmgr1_1098.out
2017-06-15 15:06:13:662 Creating clientmgr using RMI Registry port 1098
2017-06-15 15:06:13:796 client1:1098 ready...
Stopping Appserver (will fail if already stopped)

Stopping Emulator (will fail if already stopped)
Clearing Appserver message queue and log files
Remove postgres transaction logs
Restore the Database
Truncate largeorder table to clear pending requests from previous runs
Starting Appserver
Starting Emulator
2017-06-15 15:07:50:986 Starting client1:1098
2017-06-15 15:07:50:987 Command line: [java, -jar, specjapp.jar, -sv, client1, -svp, 9990, -id, 0, -tile, 0, -wkld, 0]
2017-06-15 15:07:50:990 Started client1:1098

Messages from: client1:1098
-> 2017-06-15 15:07:51:074 Looking up SPECvirt controller: client1
-> 2017-06-15 15:07:51:205 masterID: 0, tile: 0, workload: 0
-> 2017-06-15 15:07:51:205 hostname: client1
-> Hostname of prime client: client1
-> 2017-06-15 15:07:51:223 Thu Jun 15 15:06:33 CST 2017
-> 2017-06-15 15:07:51:262 RMI server started: client1:9900
-> 2017-06-15 15:07:51:267 Total clients: 1
-> 2017-06-15 15:07:51:267 Adding host client1:1091
-> 2017-06-15 15:07:51:286 Setting up clients...
-> 2017-06-15 15:07:52:010 calling getHostVM() on jappclient...
-> 2017-06-15 15:07:53:663 Remote exception in setup() from client1:1091
-> java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
->    java.io.EOFException
-> 2017-06-15 15:07:53:663 Aborting...
-> 2017-06-15 15:07:53:667 Exception occurred calling specvirt controller's abortTest() method. Exception was java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
->    java.net.ConnectException: Connection refused
-> 2017-06-15 15:07:53:668 clientsRunning = true
-> 2017-06-15 15:07:53:668 calling client.exit()...

[Miles]

I have problems in my infraserver, so I didn't include it in this run.
The date and time in all VMs and client are correct.
I have no idea why "[ERROR] VM or client clocks are out of time sync with prime client. Aborting run" exists.

[lroderic]

Hey Miles,

The harness requires that all VMs are time sync'ed with the clients and prime client. To solve this, you can sync the time manually between runs using /opt/SPECvirt/timesynctiles.sh. It's a helper script already in runspecvirt.sh. Try it manually, then rerun a test and let us know.

fyi, set the following in Control.config if you want to run a short test and see that everything is connecting:

RAMP_SECONDS = 180
WARMUP_SECONDS= 300
POLL_INTERVAL_SEC = 1800

Lisa

[Miles]

Hi Lisa
I have executed timesynctiles.sh but still got the same error.
It didn't resolve my problem.
Thanks or your help.

[lroderic]

What's the screen output when you run this command manually? What hypervisor are you using? If vSphere, are the VMware Tools in the VM up to date? Are you using VMware Tools to sync VM time to the hypervisor? Are you using NTP on the hypervisor? Physical or virtual clients?

Lisa

[abond]

Hey Miles,

Is the time being out of sync still an issue in your environment?

Thanks,
Andy

[Miles]

Hi Lisa and Andy
My host OS is Redhat 7.0.
VM tool is the attached, Virtual Machine Manager 1.4.0.
I use physical machine as my client.
The same screen output when I run this command manually.

Now I modified the CLOCK_SKEW_ALLOWED value
in Control.config to get rid of the issue.

Thanks.

[abond]

Yes, the CLOCK_SKEW_ALLOWED option will allow you to progress and get things running.  So that is good.  Just note that if the clocks are skewed too much you may end up with a lot of post run validation check errors because of the time skew.  Post run checks on expected transaction counts and mixes are done based on the master client's clock, which may cause the checks to look at the incorrect sets of data on the VMs if the clocks are skewed too much.  So you may have to come back a revisit this issue once things are running to completion to correct those errors. 

Andy

[lroderic]

This really should be working. Would you run the attached script and post the output? Thanks, Lisa

   sh -x ./timesynctiles.sh client1 hostname