fail to setup appserver with v1.04

[johnnyli]

pollme.jar is killed after launch ./runspecvirt.sh.

[root@client1 SPECvirt]# java -jar /opt/SPECpoll/pollme.jar -n client1 -p 1902
Creating RMI listener using RMI Registry port 1902
client1/192.168.1.107:1902 ready...
Killed

set DEBUG_LEVEL to 5 in Control.config get some more logs
Do I need to install JAVA 1.7.0 for all VMs? Currently only appserver1 install JAVA 1.7.0.

[root@client1 SPECvirt]# ./runspecvirt.sh
runspecvirt.sh started at: Sun Jul 1 22:59:56 EDT 2018
Tile count = 1
SPECvirt Host = client1
Tile count = 1
client1:
Sun Jul  1 23:00:11 EDT 2018
dbserver1
Sun Jul  1 23:00:11 EDT 2018
appserver1
Sun Jul  1 23:00:11 EDT 2018
batchserver1
Sun Jul  1 23:00:11 EDT 2018
mailserver1
Sun Jul  1 23:00:11 EDT 2018
infraserver1
Sun Jul  1 23:00:12 EDT 2018
webserver1
Sun Jul  1 23:00:12 EDT 2018
Preparing to start clientmgr processes for 1 tile(s) -
you have 20 seconds to Control-C to cancel
Starting 1 tile test
2018-07-01 23:00:57:521 Redirecting STDERR to primectrl.out
2018-07-01 23:00:57:521 Redirecting STDOUT to primectrl.out
To monitor test, type command: tail -f primectrl.out
[root@client1 SPECvirt]# tail -f primectrl.out
2018-07-01 23:00:57:731 PrimeControl: switch: jobNumber=10; id=2
2018-07-01 23:00:57:731 PrimeControl: got job: 10; id=3
2018-07-01 23:00:57:731 PrimeControl: switch: jobNumber=10; id=3
2018-07-01 23:00:57:731 PrimeControl: getting job... id=3
2018-07-01 23:00:57:731 PrimeControl: got job: 10; id=1
2018-07-01 23:00:57:731 PrimeControl: switch: jobNumber=10; id=1
2018-07-01 23:00:57:731 PrimeControl: got job: 10; id=0
2018-07-01 23:00:57:731 PrimeControl: switch: jobNumber=10; id=0
2018-07-01 23:01:11:488 PrimeControl: getting job... id=1
2018-07-01 23:01:26:924 PrimeControl: getting job... id=2
2018-07-01 23:04:36:608 PrimeControl: getting job... id=0
2018-07-01 23:04:36:608 PrimeControl: calling primeJob.setJob(0).
2018-07-01 23:04:36:608 PrimeControl: setting job: 0
2018-07-01 23:04:36:609 PrimeControl: got job: 0; id=0
2018-07-01 23:04:36:609 PrimeControl: switch: jobNumber=0; id=0
2018-07-01 23:04:36:610 PrimeControl: master[0][0] sleeping 20 sec.
2018-07-01 23:04:36:610 PrimeControl: got job: 0; id=2
2018-07-01 23:04:36:610 PrimeControl: switch: jobNumber=0; id=2
2018-07-01 23:04:36:610 PrimeControl: got job: 0; id=1
2018-07-01 23:04:36:610 PrimeControl: master[0][2] sleeping 20 sec.
2018-07-01 23:04:36:610 PrimeControl: switch: jobNumber=0; id=1
2018-07-01 23:04:36:610 PrimeControl: master[0][1] sleeping 20 sec.
2018-07-01 23:04:36:611 PrimeControl: got job: 0; id=3
2018-07-01 23:04:36:611 PrimeControl: switch: jobNumber=0; id=3
2018-07-01 23:04:36:611 PrimeControl: master[0][3] sleeping 20 sec.
2018-07-01 23:04:56:610 PrimeControl: calling startMaster: client1:1098 for tile  0, benchmark 0
2018-07-01 23:04:56:610 PrimeControl: calling startMaster: client1:1094 for tile  0, benchmark 2
2018-07-01 23:04:56:611 PrimeControl: calling startMaster: client1:1096 for tile  0, benchmark 1
2018-07-01 23:04:56:611 PrimeControl: calling startMaster: client1:1092 for tile  0, benchmark 3
2018-07-01 23:04:56:628 PrimeControl: getting job... id=0
2018-07-01 23:04:56:628 PrimeControl: getting job... id=1
2018-07-01 23:04:56:633 PrimeControl: getting job... id=2
2018-07-01 23:04:56:636 PrimeControl: getting job... id=3
2018-07-01 23:04:56:641 PrimeControl: waiting on 4 prime client(s).
2018-07-01 23:04:56:851 Sending config to client1:1098
2018-07-01 23:04:56:860 Sending config to client1:1096
2018-07-01 23:04:56:863 Sending config to client1:1092
2018-07-01 23:04:56:874 Sending config to client1:1094
2018-07-01 23:04:56:899 specvirt: primeRmiStarted = 1; id=0
2018-07-01 23:04:56:903 specvirt: primeRmiStarted = 2; id=3
2018-07-01 23:04:56:944 [ERROR] Received abort signal from client1:1092. Terminating.
2018-07-01 23:04:56:944 PrimeControl: sending abortTest() to prime clients.
2018-07-01 23:04:56:944 PrimeControl: setting job: 6
2018-07-01 23:04:56:944 PrimeControl: got job: 6; id=3
2018-07-01 23:04:56:944 PrimeControl: switch: jobNumber=6; id=3
2018-07-01 23:04:56:944 PrimeControl: got job: 6; id=2
2018-07-01 23:04:56:944 specvirt: primeRmiStarted = 3; id=2
2018-07-01 23:04:56:944 PrimeControl: id=3, abortID=3
2018-07-01 23:04:56:944 PrimeControl: getting job... id=3
2018-07-01 23:04:56:944 PrimeControl: got job: 6; id=1
2018-07-01 23:04:56:946 PrimeControl: switch: jobNumber=6; id=1
2018-07-01 23:04:56:946 PrimeControl: id=1, abortID=3
2018-07-01 23:04:56:946 PrimeControl: getting job... id=1
2018-07-01 23:04:56:944 PrimeControl: switch: jobNumber=6; id=2
2018-07-01 23:04:56:946 PrimeControl: id=2, abortID=3
2018-07-01 23:04:56:946 PrimeControl: getting job... id=2
2018-07-01 23:04:56:946 PrimeControl: got job: 6; id=0
2018-07-01 23:04:56:947 PrimeControl: switch: jobNumber=6; id=0
2018-07-01 23:04:56:947 PrimeControl: id=0, abortID=3
2018-07-01 23:04:56:947 PrimeControl: getting job... id=0
2018-07-01 23:04:56:951 PrimeControl: stopping remote client processes
2018-07-01 23:04:56:951 PrimeControl: setting job: 1
2018-07-01 23:04:56:951 PrimeControl: client: waiting for all done...done=0
2018-07-01 23:04:56:951 PrimeControl: got job: 1
2018-07-01 23:04:56:951 PrimeControl: calling client.stopClients() ...
2018-07-01 23:04:56:976 specvirt: primeRmiStarted = 4; id=1
2018-07-01 23:04:57:041 PrimeControl: client1:1094 (PRIME_HOST[0][2]) run complete; numStarted = -1
.
2018-07-01 23:04:57:642 PrimeControl: calling primeJob.setJob(1)
2018-07-01 23:04:57:642 PrimeControl: setting job: 1
2018-07-01 23:04:57:642 PrimeControl: got job: 1; id=0
2018-07-01 23:04:57:642 PrimeControl: switch: jobNumber=1; id=0
2018-07-01 23:04:57:642 PrimeControl: got job: 1; id=3
2018-07-01 23:04:57:643 PrimeControl: Naming lookup for 192.168.1.107:9900
2018-07-01 23:04:57:643 PrimeControl: switch: jobNumber=1; id=3
2018-07-01 23:04:57:643 PrimeControl: got job: 1; id=1
2018-07-01 23:04:57:643 PrimeControl: Naming lookup for 192.168.1.107:9903
2018-07-01 23:04:57:643 PrimeControl: switch: jobNumber=1; id=1
2018-07-01 23:04:57:643 PrimeControl: got job: 1; id=2
2018-07-01 23:04:57:643 PrimeControl: switch: jobNumber=1; id=2
2018-07-01 23:04:57:643 PrimeControl: Naming lookup for 192.168.1.107:9902
2018-07-01 23:04:57:643 PrimeControl: Naming lookup for 192.168.1.107:9901
2018-07-01 23:04:57:653 PrimeControl: done in Naming lookup for 192.168.1.107:9902
2018-07-01 23:04:57:653 PrimeControl: calling getHostVMs()...
2018-07-01 23:04:57:653 PrimeControl: done in Naming lookup for 192.168.1.107:9900
2018-07-01 23:04:57:654 PrimeControl: calling getHostVMs()...
2018-07-01 23:04:57:654 PrimeControl: hostVMs[2] = mailserver:8001
2018-07-01 23:04:57:654 PrimeControl: calling getBuildNumber()...
2018-07-01 23:04:57:655 PrimeControl: buildNum[2] = 80
2018-07-01 23:04:57:655 PrimeControl: getting job... id=2
2018-07-01 23:04:57:658 PrimeControl: hostVMs[0] = specdelivery:8001,specdb:8001
2018-07-01 23:04:57:658 PrimeControl: calling getBuildNumber()...
2018-07-01 23:04:58:955 PrimeControl: client: getting job...done=1
2018-07-01 23:04:58:956 PrimeControl: stopping local client threads
2018-07-01 23:04:58:956 PrimeControl: setting job: 2
2018-07-01 23:04:58:956 PrimeControl: client: waiting for all done...done=0
2018-07-01 23:04:58:956 PrimeControl: got job: 2
2018-07-01 23:04:58:956 PrimeControl: exiting clientThread.
2018-07-01 23:04:58:959 Terminating client processes
2018-07-01 23:04:58:959 PrimeControl: setting job: 1
2018-07-01 23:04:58:959 Sending termination signal to clientmgr processes
2018-07-01 23:04:58:959 PrimeControl: setting job: 2
2018-07-01 23:04:58:959 Terminating prime client processes
2018-07-01 23:04:58:959 PrimeControl: setting job: 4
2018-07-01 23:04:58:959 Sending termination signal to prime clientmgr processes
2018-07-01 23:04:58:959 PrimeControl: setting job: 5
2018-07-01 23:04:58:959 Exiting...
2018-07-01 23:04:58:959 calling control.stop()...
2018-07-01 23:04:58:959 PrimeControl: terminating run. Please wait...
2018-07-01 23:04:58:959 PrimeControl: got job: 4; id=2
2018-07-01 23:04:58:960 PrimeControl: switch: jobNumber=4; id=2
2018-07-01 23:04:58:960 PrimeControl: stopping prime client client1:1094
2018-07-01 23:04:58:963 specvirt: benchmark run failed!
2018-07-01 23:04:58:963 specvirt: Done!
^C
[root@client1 SPECvirt]# cat Clientmgr1_1092.out
2018-07-01 23:00:37:649 Creating clientmgr using RMI Registry port 1092
2018-07-01 23:00:37:692 client1:1092 ready...
2018-07-01 23:04:56:617 Starting client1:1092
2018-07-01 23:04:56:619 Command line: [java, -jar, specbatch.jar, -sv, client1, -svp, 9990, -id, 3, -tile, 0, -wkld, 3]
2018-07-01 23:04:56:634 Started client1:1092

Messages from: client1:1092
-> 2018-07-01 23:04:56:736 Looking up SPECvirt controller: client1
-> 2018-07-01 23:04:56:863 masterID: 3, tile: 0, workload: 3
-> 2018-07-01 23:04:56:863 hostname: client1
-> 2018-07-01 23:04:56:888 SPECbatch prime result directory:20180701-230057
-> 2018-07-01 23:04:56:888 loadscale:1.0 copycount:10
-> 2018-07-01 23:04:56:903 RMI server started: client1:9903
-> 2018-07-01 23:04:56:904 Getting host VM name for masterID=3
-> 2018-07-01 23:04:56:905 Host VM names for masterID=3: batchserver:1901
-> 2018-07-01 23:04:56:943 Remote Exception calling setConfig() on client1:1902. Aborting...
-> RemoteException was: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
->    java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
->    java.net.ConnectException: Connection refused (Connection refused)
[root@client1 SPECvirt]# cat Clientmgr1_1094.out
2018-07-01 23:00:37:666 Creating clientmgr using RMI Registry port 1094
2018-07-01 23:00:37:704 client1:1094 ready...
Sun Jul  1 23:00:56 EDT 2018: Starting mailInit.sh
Stopping Dovecot Imap: [  OK  ]
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864  5443812  11577140  32% /
tmpfs                  1962248       80   1962168   1% /dev/shm
/dev/vda1               487652    40116    421936   9% /boot
/dev/vdb              30832636 12009036  17250736  42% /mailstore
Sun Jul  1 23:01:15 EDT 2018: Starting dovecot
Starting Dovecot Imap: [  OK  ]
     11      93     759
Sun Jul  1 23:01:26 EDT 2018: Ending mailInit.sh
2018-07-01 23:04:56:618 Starting client1:1094
2018-07-01 23:04:56:619 Command line: [java, -jar, specimap.jar, -calibrate, -sv, client1, -svp, 9990, -id, 2, -tile, 0, -wkld, 2]
2018-07-01 23:04:56:631 Started client1:1094

Messages from: client1:1094
-> 2018-07-01 23:04:56:740 Looking up SPECvirt controller: client1
-> 2018-07-01 23:04:56:873 masterID: 2, tile: 0, workload: 2
-> 2018-07-01 23:04:56:873 RMI hostname: client1
-> fixedFile: IMAP_fixed.rc; configFile: IMAP_config.rc; workloadFile: IMAP_sysinfo.rc
-> 2018-07-01 23:04:56:943 RMI server started: client1:9902
-> 2018-07-01 23:04:56:948 SpecimapControl: Message initialization=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Folder initialization=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Message verification=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Folder verification=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Message clean=false
-> 2018-07-01 23:04:56:948 SpecimapControl: Folder clean=false
-> 2018-07-01 23:04:56:948 SpecimapControl: rampUp=true
-> 2018-07-01 23:04:56:948 SpecimapControl: warmUp=true
-> 2018-07-01 23:04:56:948 SpecimapControl: benchmark=true
-> 2018-07-01 23:04:56:948 SpecimapControl: overwritten=false
-> 2018-07-01 23:04:56:948 SpecimapControl: calibrate=true
-> 2018-07-01 23:04:56:953 ******************** Specmail benchmark started
-> 2018-07-01 23:04:56:981 Adding host client1:1200
-> 2018-07-01 23:04:56:987 Checking for all required parameters in config.rc
-> 2018-07-01 23:04:56:987 All required parameters in config.rc file are present
-> 2018-07-01 23:04:56:987 TEST LOAD = 100%
-> 2018-07-01 23:04:56:987 Setting workload...
-> 2018-07-01 23:04:56:988 Current load factor is: 100%
-> 2018-07-01 23:04:56:988 Provisioned users involved in this test:500
-> 2018-07-01 23:04:56:988 Workload: client1:1200,CT,test1-test500,mailserver,6
-> 2018-07-01 23:04:56:991 Adding clients...
-> 2018-07-01 23:04:56:991 Adding host[0]: client1:1200
-> 2018-07-01 23:04:56:992 Initializing clients (this may take a few minutes)...
-> [ERROR] specimapclient specimapclient_Stub[UnicastRef [liveRef: [endpoint:[192.168.1.107:10421](remote),objID:[2ede5b1:16458efa554:-7fff, -6394004371953872297]]]] returned a faulty response to getVmSysTimes() call. Check this client's console output for more information about the cause.
-> 2018-07-01 23:04:57:041 Abortiing.
-> Error: Problem encoding the raw results file.
2018-07-01 23:05:00:961 Terminating processes. Please wait...
2018-07-01 23:05:00:962 Killing master procs ...
2018-07-01 23:05:00:962 Done killing procs ...
[root@client1 SPECvirt]# cat Clientmgr1_1096.out
2018-07-01 23:00:37:636 Creating clientmgr using RMI Registry port 1096
2018-07-01 23:00:37:682 client1:1096 ready...
Sun Jul  1 23:00:57 EDT 2018: Starting webInit.sh
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864  5472620  11548332  33% /
Stopping httpd: [  OK  ]
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864  5472632  11548320  33% /
tmpfs                   510048      228    509820   1% /dev/shm
/dev/vda1               487652    40116    421936   9% /boot
/dev/vdb              61796348 33286072  25364548  57% /home/webfiles/downloads
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864  5426744  11594208  32% /
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864  5426768  11594184  32% /
tmpfs                  5064832       80   5064752   1% /dev/shm
/dev/vda1               487652    40116    421936   9% /boot
infraserver1-int:/home/webfiles/downloads
                      61796352 33286016  25364608  57% /var/www/html/support/downloads
Sun Jul  1 23:01:11 EDT 2018: Ending webInit.sh
2018-07-01 23:04:56:615 Starting client1:1096
2018-07-01 23:04:56:617 Command line: [java, -jar, specweb.jar, -sv, client1, -svp, 9990, -id, 1, -tile, 0, -wkld, 1]
2018-07-01 23:04:56:626 Started client1:1096

Messages from: client1:1096
-> 2018-07-01 23:04:56:733 Looking up SPECvirt controller: client1
-> 2018-07-01 23:04:56:859 masterID: 1, tile: 0, workload: 1
-> 2018-07-01 23:04:56:859 hostname: client1
-> 2018-07-01 23:04:56:929 Sun Jul 01 23:04:56 EDT 2018
-> 2018-07-01 23:04:56:976 RMI server started: client1:9901
-> 2018-07-01 23:04:56:979 SpecwebControl: **** SPECweb2005 benchmark started
-> 2018-07-01 23:04:56:981 SpecwebControl: * Running SPECweb_Support workload
-> 2018-07-01 23:04:56:981 Configuration: Clearing workload.
-> 2018-07-01 23:04:56:984 RemoteLoadGen: Total clients: 1
-> 2018-07-01 23:04:57:006 Connection: [ERROR] IOException creating socket. Exception is:
-> java.net.ConnectException: Connection refused (Connection refused)
-> server name: infraserver; port: 81
-> 2018-07-01 23:04:57:007 Connection: [ERROR] IOException creating socket. Exception is:
-> java.net.ConnectException: Connection refused (Connection refused)
-> server name: infraserver; port: 81
-> 2018-07-01 23:04:57:007 RemoteLoadGen: [ERROR] Unable to successfully initialize workload variables. Terminating.
-> 2018-07-01 23:04:57:007 SpecwebControl: [ERROR] Could not create all client threads.
-> 2018-07-01 23:04:57:007 SpecwebControl: [ERROR] setupWorkload() failed!
-> 2018-07-01 23:04:57:007 SpecwebControl: [ERROR] runTests() failed!
-> 2018-07-01 23:04:57:007 SpecwebControl: [ERROR] Benchmark run failed!
-> 2018-07-01 23:04:57:007 SpecwebControl: Terminating run. Please wait...
[root@client1 SPECvirt]# cat Clientmgr1_1098.out
2018-07-01 23:00:37:698 Creating clientmgr using RMI Registry port 1098
2018-07-01 23:00:37:723 client1:1098 ready...
Sun Jul  1 23:00:57 EDT 2018: Starting jAppInitRstr.sh (restore database from backup)
Stopping Appserver (will fail if already stopped)
Waiting for the domain to stop ....
Command stop-domain executed successfully.

Stopping Emulator (will fail if already stopped)
Clearing Appserver message queue and log files
Remove postgres transaction logs
Stopping postgresql service: [  OK  ]

/var/lib/pgsql/data is missing. Use "service postgresql initdb" to initialize the cluster first.
[FAILED]
Restore the Database
Restore Started...
Sun Jul  1 23:01:27 EDT 2018
Sun Jul  1 23:04:12 EDT 2018
Restore Completed
Filesystem           1K-blocks    Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864 5614536  11406416  33% /
tmpfs                  4030348     228   4030120   1% /dev/shm
/dev/vda1               487652   40116    421936   9% /boot
/dev/vdb              30832636 6813064  22446708  24% /dbstore
Truncate largeorder table to clear pending requests from previous runs
TRUNCATE TABLE
DROP TABLE
CREATE TABLE
Starting Appserver
Waiting for spec2004-1 to start .............
Successfully started the domain : spec2004-1
domain  Location: /opt/glassfish3/glassfish/domains/spec2004-1
Log File: /opt/glassfish3/glassfish/domains/spec2004-1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.
Starting Emulator
Tomcat started.
Sun Jul  1 23:04:36 EDT 2018: Ending jAppInitRstr.sh
2018-07-01 23:04:56:615 Starting client1:1098
2018-07-01 23:04:56:617 Command line: [java, -jar, specjapp.jar, -sv, client1, -svp, 9990, -id, 0, -tile, 0, -wkld, 0]
2018-07-01 23:04:56:624 Started client1:1098

Messages from: client1:1098
-> 2018-07-01 23:04:56:725 Looking up SPECvirt controller: client1
-> 2018-07-01 23:04:56:850 masterID: 0, tile: 0, workload: 0
-> 2018-07-01 23:04:56:851 hostname: client1
-> Hostname of prime client: client1
-> 2018-07-01 23:04:56:875 Sun Jul 01 23:00:57 EDT 2018
-> 2018-07-01 23:04:56:899 RMI server started: client1:9900
-> 2018-07-01 23:04:56:899 Total clients: 1
-> 2018-07-01 23:04:56:899 Adding host client1:1091
-> 2018-07-01 23:04:56:913 Setting up clients...
-> 2018-07-01 23:04:57:654 calling getHostVM() on jappclient...
-> 2018-07-01 23:04:59:275 Remote exception in setup() from client1:1091
-> java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
->    java.io.EOFException
-> 2018-07-01 23:04:59:276 Aborting...
-> 2018-07-01 23:04:59:283 Exception occurred calling specvirt controller's abortTest() method. Exception was java.rmi.ConnectException: Connection refused to host: 192.168.1.107; nested exception is:
->    java.net.ConnectException: Connection refused (Connection refused)
-> 2018-07-01 23:04:59:284 clientsRunning = true
-> 2018-07-01 23:04:59:284 calling client.exit()...

[johnnyli]

I see there are "Unknown host:" error in infraserver, webserver and dbserver.
but /etc/hosts seems ok in each VM

[root@infraserver downloads]# cat /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: infraserver1-int-int; nested exception is:
   java.net.UnknownHostException: infraserver1-int-int
java.rmi.UnknownHostException: Unknown host: infraserver1-int-int; nested exception is:
   java.net.UnknownHostException: infraserver1-int-int
   at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
   at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
   at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
   at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
   at sun.rmi.registry.RegistryImpl_Stub.rebind(RegistryImpl_Stub.java:147)
   at java.rmi.Naming.rebind(Naming.java:177)
   at org.spec.specpoll.pollme.main(pollme.java:195)
Caused by: java.net.UnknownHostException: infraserver1-int-int
   at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   at java.net.Socket.connect(Socket.java:589)
   at java.net.Socket.connect(Socket.java:538)
   at java.net.Socket.<init>(Socket.java:434)
   at java.net.Socket.<init>(Socket.java:211)
   at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
   at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
   at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
   ... 6 more

[root@webserver Desktop]# cat /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: webserver1-int-int; nested exception is:
   java.net.UnknownHostException: webserver1-int-int
java.rmi.UnknownHostException: Unknown host: webserver1-int-int; nested exception is:
   java.net.UnknownHostException: webserver1-int-int
   at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
   at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
   at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
   at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
   at sun.rmi.registry.RegistryImpl_Stub.rebind(RegistryImpl_Stub.java:147)
   at java.rmi.Naming.rebind(Naming.java:177)
   at org.spec.specpoll.pollme.main(pollme.java:195)
Caused by: java.net.UnknownHostException: webserver1-int-int
   at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   at java.net.Socket.connect(Socket.java:589)
   at java.net.Socket.connect(Socket.java:538)
   at java.net.Socket.<init>(Socket.java:434)
   at java.net.Socket.<init>(Socket.java:211)
   at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
   at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
   at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
   ... 6 more

[root@dbserver Desktop]# cat /tmp/pollme.out
[ERROR] RMI server failure:
Exception: java.rmi.ser[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: dbserver-int; nested exception is:
   java.net.UnknownHostException: dbserver-int
java.rmi.UnknownHostException: Unknown host: dbserver-int; nested exception is:
   java.net.UnknownHostException: dbserver-int
   at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616)
   at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
   at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
   at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
   at sun.rmi.registry.RegistryImpl_Stub.rebind(RegistryImpl_Stub.java:147)
   at java.rmi.Naming.rebind(Naming.java:177)
   at org.spec.specpoll.pollme.main(pollme.java:195)
Caused by: java.net.UnknownHostException: dbserver-int
   at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   at java.net.Socket.connect(Socket.java:589)
   at java.net.Socket.connect(Socket.java:538)
   at java.net.Socket.<init>(Socket.java:434)
   at java.net.Socket.<init>(Socket.java:211)
   at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
   at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
   at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
   ... 6 more

[root@infraserver downloads]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
192.168.1.146     infraserver infraserver1
192.168.1.130     webserver webserver1
192.168.1.195     mailserver mailserver1
192.168.1.141     appserver appserver1 specdelivery specemulator
192.168.1.196     dbserver dbserver1
192.168.1.125     batchserver batchserver1

192.168.1.107   client1 specdriver

# Internal VM-to-VM only communications
192.168.1.177       infraserver1-int
192.168.1.131       webserver1-int
192.168.1.150       appserver1-int
192.168.1.164       dbserver1-int specdb

[RamyaMeruva]

Hi Johnnyli,

1. Can u please install JAVA 7 on all VMs.

2. I dont understand why the hostname is " infraserver1-int-int" when its supposed to be "infraserver1-int"  and same is the case on webserver. Please ensure that u have correct entries of all VMs (IPs & hostnames) in /etc/hosts on all VMs.

[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: webserver1-int-int;

[ERROR] RMI server bind fails: java.rmi.UnknownHostException: Unknown host: infraserver1-int-int; nested exception is:

3. Reboot all the vms & use the helper script timesynctiles.sh to make sure that all Vms are time syced.

4. Also you might want to try just running the following command by itself especially on infraserver1-int & webserver1-int and see if it works fine
java -jar /opt/SPECpoll/pollme.jar -n `hostname`-int -p 8001

5. let DEBUG_LEVEL be 5 and Re-run

Thanks,
Ramya

[johnnyli]

I'm rebuilding all VMs, will back to you once done. Thanks

Some quick questions
1, In infraserver, as i test, I must run ./setup_files.sh before ./make_infraserver.sh vdb, other wise there are many errors. It's different order in the guide. Am i right?

2, How to do mailserver warmup, should i do the warmup after all VMs setup done?
Mailserver warmup
 Use the harness to run a test using mailserver and populate the indexes, creating a "warmed up" mailstore. See Section 5 for running a test.
Dovecot requires an initial "mailserver warmup run" with a duration of one to two hours depending on your storage subsystem latencies, CPU performance, and so on. You may see significant IMAP timeout errors in the Clientmgr_1088.out log during the first 20 or 30 minutes of the initial warmup run, and you can ignore these. Wait at least 30 minutes after all IMAP timeouts disappear in the IMAP server log, then stop the test, back up the existing mailstore, and use this for all future runs.

3, Do I need to do mailserver backup for 1 tile test?
Mailserver backup
After warming up the mailstore, issue the following to back it up:
 cd /opt/other_scripts
 ./backupmailstore.sh
You can copy and use this "warmed up" mailstore backup on the other tiles; that is, you do not need to create a backup mailstore on the other mailservers.

[johnnyli]

rollback to java1.7 and rebuild all VMs, still failed to run runspecvirt.sh
No transaction is detected by pollme in infraserver1-int and webserver1-int
All log and config files are attached.

2018-07-04 07:01:35:282 PrimeControl: stopping prime client client1:1092
2018-07-04 07:01:35:282 calling control.stop()...
2018-07-04 07:01:35:282 PrimeControl: terminating run. Please wait...
2018-07-04 07:01:35:282 PrimeControl: got job: 4; id=2
2018-07-04 07:01:35:283 PrimeControl: switch: jobNumber=4; id=2
2018-07-04 07:01:35:283 PrimeControl: stopping prime client client1:1094
2018-07-04 07:01:35:286 specvirt: benchmark run failed!
2018-07-04 07:01:35:286 specvirt: Done!


[root@infraserver infravm_scripts]# hostname infraserver1
[root@infraserver infravm_scripts]# java -jar /opt/SPECpoll/pollme.jar -n `hostname`-int -p 8001
Creating RMI listener using RMI Registry port 8001
infraserver1-int/192.168.1.148:8001 ready...

[root@webserver webvm_scripts]# hostname webserver1
[root@webserver webvm_scripts]# java -jar /opt/SPECpoll/pollme.jar -n `hostname`-int -web -p 8001
Creating RMI listener using RMI Registry port 8001
webserver1-int/192.168.1.149:8001 ready...

[johnnyli]

Reboot all VM, set NUM_WORKLOADS = 1 in Control.config, everthing seems good. Please have a look.

[root@client1 SPECvirt]# tail -50 primectrl.out
0,0,2018-07-04 09:58:53:329,98754,7.1,98694,3.4,197226,8.6,293946,1.75

0,0,2018-07-04 09:59:03:330,98869,7.1,98807,3.4,197431,8.6,294264,1.75

0,0,2018-07-04 09:59:13:328,98976,7.1,98926,3.4,197645,8.6,294558,1.75

0,0,2018-07-04 09:59:23:329,99086,7.1,99043,3.4,197857,8.6,294868,1.75

0,0,2018-07-04 09:59:33:328,99203,7.1,99157,3.4,198083,8.6,295189,1.75

0,0,2018-07-04 09:59:43:329,99319,7.1,99289,3.4,198332,8.6,295509,1.75

0,0,2018-07-04 09:59:53:328,99449,7.1,99412,3.4,198557,8.6,295843,1.75

0,0,2018-07-04 10:00:03:328,99566,7.1,99527,3.4,198836,8.6,296299,1.75

0,0,2018-07-04 10:00:13:327,99717,7.1,99683,3.4,199150,8.6,296746,1.75

0,0,2018-07-04 10:00:23:327,99869,7.1,99849,3.4,199498,8.6,297209,1.75

0,0,2018-07-04 10:00:33:329,100037,7.1,100026,3.4,199800,8.6,297660,1.75

0,0,2018-07-04 10:00:43:328,100288,7.1,100255,3.4,200283,8.6,298545,1.75

0,0,2018-07-04 10:00:53:328,100555,7.1,100563,3.4,200918,8.6,299471,1.75

0,0,2018-07-04 10:01:03:328,100864,7.0,100916,3.4,201557,8.6,300378,1.75

0,0,2018-07-04 10:01:13:328,101210,7.0,101267,3.4,202254,8.6,301319,1.75

0,0,2018-07-04 10:01:23:328,101566,7.0,101645,3.4,202920,8.6,302171,1.75

0,0,2018-07-04 10:01:33:328,101863,7.0,101956,3.4,203552,8.6,303007,1.75

0,0,2018-07-04 10:01:43:328,102196,7.0,102294,3.4,204166,8.6,303828,1.75

0,0,2018-07-04 10:01:53:328,102533,7.0,102610,3.4,204813,8.6,304635,1.75

0,0,2018-07-04 10:02:03:328,102855,7.0,102910,3.4,205447,8.5,305431,1.75

0,0,2018-07-04 10:02:13:328,103136,7.0,103207,3.4,206023,8.5,306180,1.75

0,0,2018-07-04 10:02:23:329,103425,6.9,103483,3.3,206595,8.5,306944,1.75

0,0,2018-07-04 10:02:33:329,103727,6.9,103758,3.3,207213,8.5,307745,1.75

0,0,2018-07-04 10:02:43:328,103973,6.9,104017,3.3,207731,8.5,308340,1.75

0,0,2018-07-04 10:02:53:328,104217,6.9,104262,3.3,208206,8.5,308972,1.75

[root@dbserver ~]# cat /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
dbserver/192.168.1.178:8001 ready...
dbserver: 1530707072081

[root@appserver ~]# cat /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
appserver/192.168.1.151:8001 ready...
2018-07-04 08:24:31:825 contacting host 'specdb'...
appserver: 1530707071873
specdb: 1530707072081

[johnnyli]

set NUM_WORKLOADS = 1, all the test passed. Detail results and logs are attached

[root@client1 SPECvirt]# ./pollmecheck.sh
Tile 1 VMs:
dbserver1:
root      2317     1  0 00:08 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n dbserver1-int -p 8001
appserver1:
root      2069     1  0 00:08 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n appserver -app -p 8001
batchserver1:
root      1963     1  0 00:08 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n batchserver -p 8001
mailserver1:
root      1970     1  0 00:08 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n mailserver -p 8001
infraserver1:
root      2049     1  0 00:08 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n infraserver1-int -p 8001
webserver1:
root      2097     1  0 00:10 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n webserver1-int -web -p 8001

[root@client1 SPECvirt]# tail -f primectrl.out
2018-07-05 02:39:10:532 PrimeControl: clients in runtime decremented: 0
2018-07-05 02:39:10:536 PrimeControl: stopping result polling.
2018-07-05 02:39:10:536 PrimeControl: setting job: 2
2018-07-05 02:39:10:536 PrimeControl: waiting for all workloads to stop...
2018-07-05 02:39:10:539 PrimeControl: finished pollMaster().
2018-07-05 02:39:10:539 PrimeControl: getting job... id=0
2018-07-05 02:39:10:539 PrimeControl: got job: 2; id=0
2018-07-05 02:39:10:539 PrimeControl: switch: jobNumber=2; id=0
2018-07-05 02:39:10:539 Sending resetCounters(false) command.
2018-07-05 02:39:10:541 PrimeControl: getting job... id=0
2018-07-05 02:39:13:616 PrimeControl: client1:1098 (PRIME_HOST[0][0]) run complete; numStarted = 0
2018-07-05 02:39:13:617 PrimeControl: calling doneMonitor.notifyAll().
2018-07-05 02:39:20:537 PrimeControl: all workloads stopped
2018-07-05 02:39:20:537 PrimeControl: stopping data reporting.
2018-07-05 02:39:20:537 PrimeControl: setting job: 8
2018-07-05 02:39:20:537 PrimeControl: exiting dataReporter thread.
2018-07-05 02:39:20:537 PrimeControl: got job: 8; id=0
2018-07-05 02:39:20:537 PrimeControl: switch: jobNumber=8; id=0
2018-07-05 02:39:20:538 PrimeControl: calling getValidationRept() ...
2018-07-05 02:39:20:540 No validation errors reported by workloads.
2018-07-05 02:39:20:540 PrimeControl: getting job... id=0
2018-07-05 02:39:20:540 PrimeControl: setting job: 9
2018-07-05 02:39:20:540 PrimeControl: got job: 9; id=0
2018-07-05 02:39:20:540 PrimeControl: switch: jobNumber=9; id=0
2018-07-05 02:39:20:540 PrimeControl: calling getResFiles() ...
2018-07-05 02:39:20:551 PrimeControl: aggregate audit...
2018-07-05 02:39:20:555 PrimeControl: getting job... id=0
2018-07-05 02:39:20:555 PrimeControl: validating aggregate audit...
2018-07-05 02:39:20:556 PrimeControl: setting job: 12
2018-07-05 02:39:20:556 PrimeControl: got job: 12; id=0
2018-07-05 02:39:20:557 PrimeControl: switch: jobNumber=12; id=0
2018-07-05 02:39:20:566 PrimeControl: getting job... id=0
2018-07-05 02:39:20:566 PrimeControl: setting job: 13
2018-07-05 02:39:20:566 PrimeControl: got job: 13; id=0
2018-07-05 02:39:20:566 PrimeControl: switch: jobNumber=13; id=0
2018-07-05 02:39:20:573 PrimeControl: getting job... id=0
2018-07-05 02:39:20:573 PrimeControl: calling primeJob.setJob(4)
2018-07-05 02:39:20:573 PrimeControl: setting job: 4
2018-07-05 02:39:20:573 PrimeControl: got job: 4; id=0
2018-07-05 02:39:20:573 PrimeControl: switch: jobNumber=4; id=0
2018-07-05 02:39:20:573 PrimeControl: stopping prime client client1:1098
2018-07-05 02:39:22:576 PrimeControl: getting job... id=0
2018-07-05 02:39:22:610 PrimeControl: calling primeJob.setJob(11).
2018-07-05 02:39:22:610 PrimeControl: setting job: 11
2018-07-05 02:39:22:610 PrimeControl: got job: 11; id=0
2018-07-05 02:39:22:610 PrimeControl: switch: jobNumber=11; id=0
2018-07-05 02:39:22:610 PrimeControl: getting job... id=0
2018-07-05 02:39:22:610 PrimeControl: calling primeJob.setJob(5)
2018-07-05 02:39:22:610 PrimeControl: setting job: 5
2018-07-05 02:39:22:611 PrimeControl: got job: 5; id=0
2018-07-05 02:39:22:611 PrimeControl: switch: jobNumber=5; id=0
2018-07-05 02:39:22:611 PrimeControl: exiting primeClientThread.
2018-07-05 02:39:22:611 PrimeControl: stopping clients.
2018-07-05 02:39:22:611 PrimeControl: stopping remote client processes
2018-07-05 02:39:22:611 PrimeControl: setting job: 1
2018-07-05 02:39:22:611 PrimeControl: client: waiting for all done...done=0
2018-07-05 02:39:22:611 PrimeControl: got job: 1
2018-07-05 02:39:22:611 PrimeControl: calling client.stopClients() ...
2018-07-05 02:39:24:614 PrimeControl: client: getting job...done=1
2018-07-05 02:39:24:614 PrimeControl: stopping local client threads
2018-07-05 02:39:24:614 PrimeControl: setting job: 2
2018-07-05 02:39:24:614 PrimeControl: client: waiting for all done...done=0
2018-07-05 02:39:24:614 PrimeControl: got job: 2
2018-07-05 02:39:24:614 PrimeControl: exiting clientThread.
2018-07-05 02:39:24:614 PrimeControl: done stopping clients.
  > Loading Raw Result File..

2018-07-05 02:39:24:654 calling control.stop()...
2018-07-05 02:39:24:654 PrimeControl: terminating run. Please wait...
2018-07-05 02:39:25:657 specvirt: Done!

[root@appserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
appserver/192.168.1.151:8001 ready...
2018-07-05 00:19:09:083 contacting host 'specdb'...
appserver: 1530764349092
specdb: 1530764349295

[root@dbserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
dbserver1-int/192.168.1.152:8001 ready...
dbserver1-int: 1530764349295

[johnnyli]

set NUM_WORKLOADS = 2, the test is failed. Detail results and logs are attached, any suggestion?

[root@client1 SPECvirt]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
192.168.1.180     infraserver infraserver1
192.168.1.137     webserver webserver1
192.168.1.161     mailserver mailserver1
192.168.1.151     appserver appserver1 specdelivery specemulator
192.168.1.178     dbserver dbserver1
192.168.1.170     batchserver batchserver1

192.168.1.132   client1 specdriver

# Internal VM-to-VM only communications
192.168.1.145       infraserver1-int
192.168.1.149       webserver1-int
192.168.1.173      appserver1-int
192.168.1.152       dbserver1-int specdb

[root@client1 other_scripts]# ./test_hostentries.sh
Testing the external IP addresses of the Virtual Machines and client

Pinging Infraserver1 VM (infraserver1)...Passed!
Pinging Webserver1 VM (webserver1)...Passed!
Pinging Mailserver1 VM (mailserver1)...Passed!
Pinging Appserver1 VM (specdelivery)...Passed!
Pinging DBserver1 VM (dbserver1)...Passed!
Pinging Batchserver1 VM (batchserver1)...Passed!
Pinging Client1 VM (client1)...Passed!

Testing the internal IP addresses of the Virtual Machines
This does not work on the client since it has no internal
network adapter. Run this test on web/infraserver and app/dbserver.

Pinging Infraserver1 VM (infraserver1-int)...Passed!
Pinging Webserver1 VM (webserver1-int)...Passed!
Pinging Appserver1 VM (appserver1-int)...Passed!
Pinging DBserver VM (specdb)...Passed!

[root@client1 SPECvirt]# ./pollmecheck.sh
Tile 1 VMs:
dbserver1:
root      2392     1  0 03:19 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n dbserver1-int -p 8001
appserver1:
root      2117     1  0 03:19 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n appserver -app -p 8001
batchserver1:
root      1963     1  0 03:19 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n batchserver -p 8001
mailserver1:
root      1968     1  0 03:21 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n mailserver -p 8001
infraserver1:
root      2050     1  0 03:19 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n infraserver1-int -p 8001
webserver1:
root      2175     1  0 03:20 ?        00:00:00 java -jar /opt/SPECpoll/pollme.jar -n webserver1-int -web -p 8001

[root@client1 SPECvirt]# tail -f primectrl.out
2018-07-05 03:25:20:681 PrimeControl: entered startMasters(); remoteMasters.length = 2
2018-07-05 03:25:20:682 PrimeControl: getting job... id=0
2018-07-05 03:25:20:684 PrimeControl: calling primeJob.setJob(10).
2018-07-05 03:25:20:684 PrimeControl: setting job: 10
2018-07-05 03:25:20:684 PrimeControl: getting job... id=1
2018-07-05 03:25:20:684 PrimeControl: got job: 10; id=1
2018-07-05 03:25:20:684 PrimeControl: switch: jobNumber=10; id=1
2018-07-05 03:25:20:685 PrimeControl: got job: 10; id=0
2018-07-05 03:25:20:685 PrimeControl: switch: jobNumber=10; id=0
2018-07-05 03:25:34:379 PrimeControl: getting job... id=1
2018-07-05 03:28:17:260 PrimeControl: getting job... id=0
2018-07-05 03:28:17:260 PrimeControl: calling primeJob.setJob(0).
2018-07-05 03:28:17:260 PrimeControl: setting job: 0
2018-07-05 03:28:17:261 PrimeControl: got job: 0; id=0
2018-07-05 03:28:17:261 PrimeControl: switch: jobNumber=0; id=0
2018-07-05 03:28:17:261 PrimeControl: got job: 0; id=1
2018-07-05 03:28:17:261 PrimeControl: master[0][0] sleeping 20 sec.
2018-07-05 03:28:17:261 PrimeControl: switch: jobNumber=0; id=1
2018-07-05 03:28:17:261 PrimeControl: master[0][1] sleeping 20 sec.
2018-07-05 03:28:37:261 PrimeControl: calling startMaster: client1:1098 for tile  0, benchmark 0
2018-07-05 03:28:37:262 PrimeControl: calling startMaster: client1:1096 for tile  0, benchmark 1
2018-07-05 03:28:37:270 PrimeControl: getting job... id=0
2018-07-05 03:28:37:271 PrimeControl: getting job... id=1
2018-07-05 03:28:37:271 PrimeControl: waiting on 2 prime client(s).
2018-07-05 03:28:37:420 Sending config to client1:1098
2018-07-05 03:28:37:452 Sending config to client1:1096
2018-07-05 03:28:37:454 specvirt: primeRmiStarted = 1; id=0
2018-07-05 03:28:37:543 specvirt: primeRmiStarted = 2; id=1
.
2018-07-05 03:28:38:272 PrimeControl: calling primeJob.setJob(1)
2018-07-05 03:28:38:272 PrimeControl: setting job: 1
2018-07-05 03:28:38:272 PrimeControl: got job: 1; id=1
2018-07-05 03:28:38:272 PrimeControl: switch: jobNumber=1; id=1
2018-07-05 03:28:38:272 PrimeControl: got job: 1; id=0
2018-07-05 03:28:38:273 PrimeControl: switch: jobNumber=1; id=0
2018-07-05 03:28:38:273 PrimeControl: Naming lookup for 192.168.1.132:9901
2018-07-05 03:28:38:273 PrimeControl: Naming lookup for 192.168.1.132:9900
2018-07-05 03:28:38:285 PrimeControl: done in Naming lookup for 192.168.1.132:9900
2018-07-05 03:28:38:285 PrimeControl: calling getHostVMs()...
2018-07-05 03:28:38:288 PrimeControl: hostVMs[0] = specdelivery:8001,specdb:8001
2018-07-05 03:28:38:289 PrimeControl: calling getBuildNumber()...
2018-07-05 03:28:40:275 RMI name lookup error for  prime client 192.168.1.132 on port 9901: exception java.rmi.ConnectException: Connection refused to host: 192.168.1.132; nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)
2018-07-05 03:28:40:275 PrimeControl: done in Naming lookup for 192.168.1.132:9901
2018-07-05 03:28:40:275 PrimeControl: calling getHostVMs()...
2018-07-05 03:28:40:275 PrimeControl: [ERROR] masters[0] build numbers (null) do not match the specvirt prime controller's (80). Please update complete harness and retry.
2018-07-05 03:28:40:276 PrimeControl: [ERROR] masters[1] build numbers (null) do not match the specvirt prime controller's (80). Please update complete harness and retry.
2018-07-05 03:28:40:276 PrimeControl: [ERROR] startMasters() failed!
2018-07-05 03:28:40:276 PrimeControl: sending abortTest() to prime clients.
2018-07-05 03:28:40:276 PrimeControl: setting job: 6
Exception in thread "Thread-6" java.lang.NullPointerException
        at org.spec.virt.PrimeControl$PrimeClientThread.run(PrimeControl.java:1650)
2018-07-05 03:30:52:981 PrimeControl: buildNum[0] = 80
2018-07-05 03:30:52:981 PrimeControl: getting job... id=0
2018-07-05 03:30:52:981 PrimeControl: got job: 6; id=0
2018-07-05 03:30:52:981 PrimeControl: switch: jobNumber=6; id=0
2018-07-05 03:30:52:981 PrimeControl: id=0, abortID=-1
2018-07-05 03:30:52:981 PrimeControl: masters[0]=client1:1098
2018-07-05 03:31:01:651 specvirt: waiting on 1 prime clients.
2018-07-05 03:31:03:299 PrimeControl: getting job... id=0

[root@appserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
appserver/192.168.1.151:8001 ready...
2018-07-05 03:31:01:196 contacting host 'specdb'...
appserver: 1530775861209
specdb: 1530775861418

[root@dbserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
dbserver1-int/192.168.1.152:8001 ready...
dbserver1-int: 1530775861418

[root@webserver ~]# tail -f /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
webserver1-int/192.168.1.149:8001 ready...

[RamyaMeruva]

Hi Johnnyli,

The Connection refused errors are usually due to firewall, SELinux or SPECpoll not running. I see that u have all of these in place.Can u confirm if webserver & infraserver can ping each other via internal network? are you using Java JDK 7 on all VMs?

From the prime client, can u please check the specvirt build version of all workload VMs, clients, and prime client as follows:

java -jar /opt/SPECvirt/specvirt.jar -v

U should see the following :

SPECvirt_sc2013 v1.1, build: 80

if u don't, you need to make sure you've installed SPECvirt correctly on all VMs.

Also, are you using 10GbE between the clients and SUT? One tile uses about 1.4GbE per tile. If you don't have 10GbE, you can split off webserver onto its own client.

Can u also attach httpd error & access logs from webserver & infraserver?

Thanks,
Ramya

[johnnyli]

Hi Ramya,
  I confirmed again in all VMs, firewall and SELinux are disabled, JAVA is 1,7.0_181, specvirt version is SPECvirt_sc2013 v1.1, build: 80.
  There are httpd error & access logs in webserver, but not in infraserver
  Do I need to set $JAVA_HOME in all VMs? I see $JAVA_HOME is /usr/lib/jvm/java-openjdk in appserver and client1, is /usr/lib/jvm/jre in dbserver, and is empty in the rest.
  I don't use 10GbE LAN card, I'll find one.

[root@webserver logs]# service iptables status
iptables: Firewall is not running.
[root@webserver logs]# service ip6tables status
ip6tables: Firewall is not running.
[root@webserver logs]# cat /proc/cmdline
ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=129M@0M rd_LVM_LV=VolGroup/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet selinux=0 audit=0
[root@webserver logs]# java -version
java version "1.7.0_181"
OpenJDK Runtime Environment (rhel-2.6.14.8.el6_9-x86_64 u181-b00)
OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
[root@webserver logs]# java -jar /opt/SPECvirt/specvirt.jar -v
SPECvirt_sc2013 v1.1, build: 80
[root@webserver logs]# ping infraserver
PING infraserver (192.168.1.180) 56(84) bytes of data.
64 bytes from infraserver (192.168.1.180): icmp_seq=1 ttl=64 time=1.42 ms
^C
--- infraserver ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 959ms
rtt min/avg/max/mdev = 1.423/1.423/1.423/0.000 ms
[root@webserver logs]# ping infraserver1-int
PING infraserver1-int (192.168.1.145) 56(84) bytes of data.
64 bytes from infraserver1-int (192.168.1.145): icmp_seq=1 ttl=64 time=0.998 ms
64 bytes from infraserver1-int (192.168.1.145): icmp_seq=2 ttl=64 time=0.372 ms
64 bytes from infraserver1-int (192.168.1.145): icmp_seq=3 ttl=64 time=0.360 ms
^C
--- infraserver1-int ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2192ms
rtt min/avg/max/mdev = 0.360/0.576/0.998/0.299 ms

[root@webserver ~]# java -jar /opt/SPECvirt/specvirt.jar -v
SPECvirt_sc2013 v1.1, build: 80

[root@infraserver logs]# cd /etc/httpd/logs/
[root@infraserver logs]# ls
[root@webserver ~]# cd /etc/httpd/logs/
[root@webserver logs]# ls
access_log  error_log  ssl_access_log  ssl_error_log  ssl_request_log
[root@webserver logs]# cat access_log
192.168.1.132 - - [05/Jul/2018:03:28:37 -0400] "GET /support/init.php?BESIM_HOST=infraserver&BESIM_PORT=81&BESIM_URI=/fcgi-bin/besim_fcgi.fcgi&BESIM_PERSISTENT=false&PADDING_DIR=/var/www/html/support/dynamic_padding/&SMARTY_DIR=/var/www/html/Smarty-2.6.26/libs/&SMARTY_SUPPORT_DIR=/var/www/html/support/&SEND_CONTENT_LENGTH=true HTTP/1.1" 200 537 "-" "-"
[root@webserver logs]# cat error_log
[Thu Jul 05 03:25:33 2018] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Jul 05 03:25:34 2018] [notice] Digest: generating secret for digest authentication ...
[Thu Jul 05 03:25:34 2018] [notice] Digest: done
[Thu Jul 05 03:25:34 2018] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[root@webserver logs]# cat ssl_access_log

[root@webserver logs]# cat ssl_error_log
[Thu Jul 05 03:25:33 2018] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Jul 05 03:25:34 2018] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[root@webserver logs]# cat ssl_request_log

[johnnyli]

Use a 10Gb card for internal network, issue is the same.

[root@client1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
192.168.1.180     infraserver infraserver1
192.168.1.137     webserver webserver1
192.168.1.161     mailserver mailserver1
192.168.1.151     appserver appserver1 specdelivery specemulator
192.168.1.178     dbserver dbserver1
192.168.1.170     batchserver batchserver1

192.168.1.132   client1 specdriver

# Internal VM-to-VM only communications
10.10.10.36       infraserver1-int
10.10.10.37       webserver1-int
10.10.10.38       appserver1-int
10.10.10.39       dbserver1-int specdb

[root@client1 other_scripts]# ./test_hostentries.sh
Testing the external IP addresses of the Virtual Machines and client

Pinging Infraserver1 VM (infraserver1)...Passed!
Pinging Webserver1 VM (webserver1)...Passed!
Pinging Mailserver1 VM (mailserver1)...Passed!
Pinging Appserver1 VM (specdelivery)...Passed!
Pinging DBserver1 VM (dbserver1)...Passed!
Pinging Batchserver1 VM (batchserver1)...Passed!
Pinging Client1 VM (client1)...Passed!

Testing the internal IP addresses of the Virtual Machines
This does not work on the client since it has no internal
network adapter. Run this test on web/infraserver and app/dbserver.

Pinging Infraserver1 VM (infraserver1-int)...FAILED
Pinging Webserver1 VM (webserver1-int)...FAILED
Pinging Appserver1 VM (appserver1-int)...FAILED
Pinging DBserver VM (specdb)...FAILED

[root@webserver other_scripts]# ./test_hostentries.sh
Testing the external IP addresses of the Virtual Machines and client

Pinging Infraserver1 VM (infraserver1)...Passed!
Pinging Webserver1 VM (webserver1)...Passed!
Pinging Mailserver1 VM (mailserver1)...Passed!
Pinging Appserver1 VM (specdelivery)...Passed!
Pinging DBserver1 VM (dbserver1)...Passed!
Pinging Batchserver1 VM (batchserver1)...Passed!
Pinging Client1 VM (client1)...Passed!

Testing the internal IP addresses of the Virtual Machines
This does not work on the client since it has no internal
network adapter. Run this test on web/infraserver and app/dbserver.

Pinging Infraserver1 VM (infraserver1-int)...Passed!
Pinging Webserver1 VM (webserver1-int)...Passed!
Pinging Appserver1 VM (appserver1-int)...Passed!
Pinging DBserver VM (specdb)...Passed!

[root@client1 SPECvirt]# tail -f primectrl.out
2018-07-05 23:48:07:925 PrimeControl: entered startMasters(); remoteMasters.length = 2
2018-07-05 23:48:07:927 PrimeControl: getting job... id=0
2018-07-05 23:48:07:927 PrimeControl: getting job... id=1
2018-07-05 23:48:07:929 PrimeControl: calling primeJob.setJob(10).
2018-07-05 23:48:07:929 PrimeControl: setting job: 10
2018-07-05 23:48:07:929 PrimeControl: got job: 10; id=1
2018-07-05 23:48:07:929 PrimeControl: got job: 10; id=0
2018-07-05 23:48:07:930 PrimeControl: switch: jobNumber=10; id=1
2018-07-05 23:48:07:930 PrimeControl: switch: jobNumber=10; id=0
2018-07-05 23:48:21:689 PrimeControl: getting job... id=1
2018-07-05 23:51:18:642 PrimeControl: getting job... id=0
2018-07-05 23:51:18:643 PrimeControl: calling primeJob.setJob(0).
2018-07-05 23:51:18:643 PrimeControl: setting job: 0
2018-07-05 23:51:18:643 PrimeControl: got job: 0; id=0
2018-07-05 23:51:18:643 PrimeControl: switch: jobNumber=0; id=0
2018-07-05 23:51:18:643 PrimeControl: got job: 0; id=1
2018-07-05 23:51:18:643 PrimeControl: master[0][0] sleeping 20 sec.
2018-07-05 23:51:18:643 PrimeControl: switch: jobNumber=0; id=1
2018-07-05 23:51:18:644 PrimeControl: master[0][1] sleeping 20 sec.
2018-07-05 23:51:38:644 PrimeControl: calling startMaster: client1:1098 for tile  0, benchmark 0
2018-07-05 23:51:38:644 PrimeControl: calling startMaster: client1:1096 for tile  0, benchmark 1
2018-07-05 23:51:38:652 PrimeControl: getting job... id=0
2018-07-05 23:51:38:655 PrimeControl: getting job... id=1
2018-07-05 23:51:38:655 PrimeControl: waiting on 2 prime client(s).
2018-07-05 23:51:38:830 Sending config to client1:1098
2018-07-05 23:51:38:885 specvirt: primeRmiStarted = 1; id=0
2018-07-05 23:51:38:908 Sending config to client1:1096
2018-07-05 23:51:38:996 specvirt: primeRmiStarted = 2; id=1
.
2018-07-05 23:51:39:656 PrimeControl: calling primeJob.setJob(1)
2018-07-05 23:51:39:656 PrimeControl: setting job: 1
2018-07-05 23:51:39:656 PrimeControl: got job: 1; id=1
2018-07-05 23:51:39:656 PrimeControl: switch: jobNumber=1; id=1
2018-07-05 23:51:39:656 PrimeControl: got job: 1; id=0
2018-07-05 23:51:39:656 PrimeControl: Naming lookup for 192.168.1.132:9901
2018-07-05 23:51:39:657 PrimeControl: switch: jobNumber=1; id=0
2018-07-05 23:51:39:657 PrimeControl: Naming lookup for 192.168.1.132:9900
2018-07-05 23:51:39:669 PrimeControl: done in Naming lookup for 192.168.1.132:9900
2018-07-05 23:51:39:669 PrimeControl: calling getHostVMs()...
2018-07-05 23:51:39:672 PrimeControl: hostVMs[0] = specdelivery:8001,specdb:8001
2018-07-05 23:51:39:672 PrimeControl: calling getBuildNumber()...
2018-07-05 23:51:41:684 RMI name lookup error for  prime client 192.168.1.132 on port 9901: exception java.rmi.ConnectException: Connection refused to host: 192.168.1.132; nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)
2018-07-05 23:51:41:684 PrimeControl: done in Naming lookup for 192.168.1.132:9901
2018-07-05 23:51:41:684 PrimeControl: calling getHostVMs()...
2018-07-05 23:51:41:684 PrimeControl: [ERROR] masters[0] build numbers (null) do not match the specvirt prime controller's (80). Please update complete harness and retry.
2018-07-05 23:51:41:685 PrimeControl: [ERROR] masters[1] build numbers (null) do not match the specvirt prime controller's (80). Please update complete harness and retry.
2018-07-05 23:51:41:685 PrimeControl: [ERROR] startMasters() failed!
2018-07-05 23:51:41:685 PrimeControl: sending abortTest() to prime clients.
2018-07-05 23:51:41:685 PrimeControl: setting job: 6
Exception in thread "Thread-6" java.lang.NullPointerException
        at org.spec.virt.PrimeControl$PrimeClientThread.run(PrimeControl.java:1650)
2018-07-05 23:53:54:928 PrimeControl: buildNum[0] = 80
2018-07-05 23:53:54:928 PrimeControl: getting job... id=0
2018-07-05 23:53:54:928 PrimeControl: got job: 6; id=0
2018-07-05 23:53:54:929 PrimeControl: switch: jobNumber=6; id=0
2018-07-05 23:53:54:929 PrimeControl: id=0, abortID=-1
2018-07-05 23:53:54:929 PrimeControl: masters[0]=client1:1098
2018-07-05 23:54:05:683 specvirt: waiting on 1 prime clients.
2018-07-05 23:54:07:329 PrimeControl: getting job... id=0

[RamyaMeruva]

Hi Johnnyli,

Can you check if your infraserver nfs share is mounted on webserver before you ran Wafgen?
On my specvirt_sc setup, I have JAVA_HOME set to /usr/lib/jvm/jre. Can u change this and retry?
The connection refused errors you're seeing are probably because of some kind of network misconfiguration. From the details u shared so far, everything looks good to me. Can u also post ifconfig output from all your VMs here?

Thanks,
Ramya

[johnnyli]

Rebuild infraserver and webserver, confirmed infraserver1-int:/home/webfiles/downloads is mounted before Wafgen.
All ifconfig commands are dumped in attached.
This time the error is different. Don't it's better or worse than before.
Which JAVA_HOME need to set to /usr/lib/jvm/jre, All? I don't do this now.

[root@client1 SPECvirt]# cat primectrl.out
2018-07-09 04:35:46:588 Mon Jul 09 04:35:46 EDT 2018
2018-07-09 04:35:46:588 specvirt: maxPreRunTime = 481
2018-07-09 04:35:46:589 specvirt: runTime = 7200
2018-07-09 04:35:46:589 specvirt: runTime = 7200
2018-07-09 04:35:46:589 specvirt: runTime = 600
2018-07-09 04:35:46:589 specvirt: runTime = 600
2018-07-09 04:35:46:590 Validator: [WARNING] NUM_WORKLOADS value is: 2; should be 4
2018-07-09 04:35:46:591 Validator: [WARNING] Non-compliant configuration.
2018-07-09 04:35:46:591 [WARNING] This will be a non-compliant benchmark result!
2018-07-09 04:35:46:634 RMI server started: client1:9990
2018-07-09 04:35:46:641 modVals: [Ljava.lang.String;@41cb0ed6
2018-07-09 04:35:46:642 [INFO] This is a perf-only benchmark run. Skipping active idle polling interval.
2018-07-09 04:35:46:643 PrimeControl: preparing client drivers.
2018-07-09 04:35:46:643 PrimeControl: PRIME_HOST 0 = client1:1098
2018-07-09 04:35:46:643 PrimeControl: PRIME_HOST 1 = client1:1096
2018-07-09 04:35:46:644 PrimeControl: Master 1: client1:1098
2018-07-09 04:35:46:644 PrimeControl: Master 2: client1:1096
2018-07-09 04:35:46:645 PrimeControl: adding host client1:1098
2018-07-09 04:35:46:654 PrimeControl: adding host client1:1096
2018-07-09 04:35:46:665 First client for 0: 192.168.1.132:1091
client[0][0]: 192.168.1.132:1091
remote client "192.168.1.132" added
2018-07-09 04:35:46:670 PrimeControl: done in client for loop
2018-07-09 04:35:46:677 First client for 1: 192.168.1.132:1010
client[1][0]: 192.168.1.132:1010
2018-07-09 04:35:46:679 PrimeControl: done in client for loop
creating ClientThread-0
2018-07-09 04:35:46:681 PrimeControl: setting job: 0
2018-07-09 04:35:46:681 PrimeControl: client: waiting for all done...done=0
2018-07-09 04:35:46:681 PrimeControl: client: getting job...done=0
2018-07-09 04:35:46:681 PrimeControl: got job: 0
2018-07-09 04:35:46:681 PrimeControl: starting clients...
2018-07-09 04:35:46:681 PrimeControl: clients.length = 2
2018-07-09 04:35:46:681 PrimeControl: clients[0].length = 1
2018-07-09 04:35:46:681 PrimeControl: starting clients[0][0]: 192.168.1.132:1091
2018-07-09 04:35:46:691 PrimeControl: started client: 192.168.1.132:1091
2018-07-09 04:35:46:691 PrimeControl: clients[1].length = 1
2018-07-09 04:35:46:691 PrimeControl: starting clients[1][0]: 192.168.1.132:1010
2018-07-09 04:35:46:696 PrimeControl: started client: 192.168.1.132:1010
2018-07-09 04:35:46:696 PrimeControl: client: getting job...done=1
2018-07-09 04:35:46:696 PrimeControl: done starting clients
2018-07-09 04:35:46:696 PrimeControl: PTDs not used for this benchmark run!
2018-07-09 04:35:46:696 PrimeControl: starting 2 masters.
2018-07-09 04:35:46:697 PrimeControl: entered startMasters(); remoteMasters.length = 2
2018-07-09 04:35:46:698 PrimeControl: getting job... id=0
2018-07-09 04:35:46:699 PrimeControl: calling primeJob.setJob(10).
2018-07-09 04:35:46:699 PrimeControl: setting job: 10
2018-07-09 04:35:46:700 PrimeControl: got job: 10; id=0
2018-07-09 04:35:46:700 PrimeControl: switch: jobNumber=10; id=0
2018-07-09 04:35:46:700 PrimeControl: getting job... id=1
2018-07-09 04:35:46:700 PrimeControl: got job: 10; id=1
2018-07-09 04:35:46:700 PrimeControl: switch: jobNumber=10; id=1
2018-07-09 04:36:00:360 PrimeControl: getting job... id=1
2018-07-09 04:38:45:206 PrimeControl: getting job... id=0
2018-07-09 04:38:45:206 PrimeControl: calling primeJob.setJob(0).
2018-07-09 04:38:45:206 PrimeControl: setting job: 0
2018-07-09 04:38:45:206 PrimeControl: got job: 0; id=0
2018-07-09 04:38:45:207 PrimeControl: switch: jobNumber=0; id=0
2018-07-09 04:38:45:207 PrimeControl: got job: 0; id=1
2018-07-09 04:38:45:207 PrimeControl: master[0][0] sleeping 20 sec.
2018-07-09 04:38:45:207 PrimeControl: switch: jobNumber=0; id=1
2018-07-09 04:38:45:207 PrimeControl: master[0][1] sleeping 20 sec.
2018-07-09 04:39:05:207 PrimeControl: calling startMaster: client1:1098 for tile  0, benchmark 0
2018-07-09 04:39:05:207 PrimeControl: calling startMaster: client1:1096 for tile  0, benchmark 1
2018-07-09 04:39:05:215 PrimeControl: getting job... id=0
2018-07-09 04:39:05:215 PrimeControl: getting job... id=1
2018-07-09 04:39:05:215 PrimeControl: waiting on 2 prime client(s).
2018-07-09 04:39:05:395 Sending config to client1:1096
2018-07-09 04:39:05:400 Sending config to client1:1098
2018-07-09 04:39:05:427 specvirt: primeRmiStarted = 1; id=0
2018-07-09 04:39:05:504 specvirt: primeRmiStarted = 2; id=1
2018-07-09 04:39:05:859 [ERROR] Received abort signal from client1:1096. Terminating.
2018-07-09 04:39:05:859 PrimeControl: sending abortTest() to prime clients.
2018-07-09 04:39:05:859 PrimeControl: setting job: 6
2018-07-09 04:39:05:859 PrimeControl: got job: 6; id=1
2018-07-09 04:39:05:859 PrimeControl: switch: jobNumber=6; id=1
2018-07-09 04:39:05:859 PrimeControl: got job: 6; id=0
2018-07-09 04:39:05:859 PrimeControl: id=1, abortID=1
2018-07-09 04:39:05:860 PrimeControl: switch: jobNumber=6; id=0
2018-07-09 04:39:05:860 PrimeControl: getting job... id=1
2018-07-09 04:39:05:860 PrimeControl: id=0, abortID=1
2018-07-09 04:39:05:860 PrimeControl: getting job... id=0
2018-07-09 04:39:05:860 PrimeControl: stopping remote client processes
2018-07-09 04:39:05:860 PrimeControl: setting job: 1
2018-07-09 04:39:05:860 PrimeControl: client: waiting for all done...done=0
2018-07-09 04:39:05:860 PrimeControl: got job: 1
2018-07-09 04:39:05:860 PrimeControl: calling client.stopClients() ...
.
2018-07-09 04:39:06:215 PrimeControl: calling primeJob.setJob(1)
2018-07-09 04:39:06:216 PrimeControl: setting job: 1
2018-07-09 04:39:06:216 PrimeControl: got job: 1; id=0
2018-07-09 04:39:06:216 PrimeControl: switch: jobNumber=1; id=0
2018-07-09 04:39:06:216 PrimeControl: got job: 1; id=1
2018-07-09 04:39:06:216 PrimeControl: Naming lookup for 192.168.1.132:9900
2018-07-09 04:39:06:216 PrimeControl: switch: jobNumber=1; id=1
2018-07-09 04:39:06:216 PrimeControl: Naming lookup for 192.168.1.132:9901
2018-07-09 04:39:06:228 PrimeControl: done in Naming lookup for 192.168.1.132:9900
2018-07-09 04:39:06:228 PrimeControl: calling getHostVMs()...
2018-07-09 04:39:06:231 PrimeControl: hostVMs[0] = specdelivery:8001,specdb:8001
2018-07-09 04:39:06:231 PrimeControl: calling getBuildNumber()...
2018-07-09 04:39:07:865 PrimeControl: client: getting job...done=1
2018-07-09 04:39:07:865 PrimeControl: stopping local client threads
2018-07-09 04:39:07:865 PrimeControl: setting job: 2
2018-07-09 04:39:07:865 PrimeControl: client: waiting for all done...done=0
2018-07-09 04:39:07:865 PrimeControl: got job: 2
2018-07-09 04:39:07:866 PrimeControl: exiting clientThread.
2018-07-09 04:39:07:866 Terminating client processes
2018-07-09 04:39:07:866 PrimeControl: setting job: 1
2018-07-09 04:39:07:866 Sending termination signal to clientmgr processes
2018-07-09 04:39:07:866 PrimeControl: setting job: 2
2018-07-09 04:39:07:866 Terminating prime client processes
2018-07-09 04:39:07:866 PrimeControl: setting job: 4
2018-07-09 04:39:07:866 Sending termination signal to prime clientmgr processes
2018-07-09 04:39:07:866 PrimeControl: setting job: 5
2018-07-09 04:39:07:866 Exiting...
2018-07-09 04:39:07:866 calling control.stop()...
2018-07-09 04:39:07:867 PrimeControl: terminating run. Please wait...
2018-07-09 04:39:07:869 specvirt: benchmark run failed!
2018-07-09 04:39:07:869 specvirt: Done!

[root@client1 SPECvirt]# cat Clientmgr1_1096.out
2018-07-09 04:35:26:623 Creating clientmgr using RMI Registry port 1096
2018-07-09 04:35:26:658 client1:1096 ready...
Mon Jul  9 04:35:45 EDT 2018: Starting webInit.sh
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864  5009552  12011400  30% /
Stopping httpd: [  OK  ]
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864  5009556  12011396  30% /
tmpfs                   510048       72    509976   1% /dev/shm
/dev/vda1               487652    40116    421936   9% /boot
/dev/vdb              51475068 33286072  15567556  69% /home/webfiles/downloads
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864  5044960  11975992  30% /
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
                      17938864  5044996  11975956  30% /
tmpfs                  5064832       72   5064760   1% /dev/shm
/dev/vda1               487652    40116    421936   9% /boot
infraserver1-int:/home/webfiles/downloads
                      51475072 33286016  15567616  69% /var/www/html/support/downloads
Mon Jul  9 04:35:59 EDT 2018: Ending webInit.sh
2018-07-09 04:39:05:211 Starting client1:1096
2018-07-09 04:39:05:212 Command line: [java, -jar, specweb.jar, -sv, client1, -svp, 9990, -id, 1, -tile, 0, -wkld, 1]
2018-07-09 04:39:05:214 Started client1:1096

Messages from: client1:1096
-> 2018-07-09 04:39:05:314 Looking up SPECvirt controller: client1
-> 2018-07-09 04:39:05:394 masterID: 1, tile: 0, workload: 1
-> 2018-07-09 04:39:05:394 hostname: client1
-> 2018-07-09 04:39:05:490 Mon Jul 09 04:39:05 EDT 2018
-> 2018-07-09 04:39:05:503 RMI server started: client1:9901
-> 2018-07-09 04:39:05:509 SpecwebControl: **** SPECweb2005 benchmark started
-> 2018-07-09 04:39:05:510 SpecwebControl: * Running SPECweb_Support workload
-> 2018-07-09 04:39:05:510 Configuration: Clearing workload.
-> 2018-07-09 04:39:05:512 RemoteLoadGen: Total clients: 1
-> 2018-07-09 04:39:05:535 HttpRequestSched: [ERROR] Valid SERVER_TIME value not provided in header.
-> 2018-07-09 04:39:05:535 HttpRequestSched: [ERROR] Response was:
-> HTTP/1.1 200 OK
-> Date: Mon, 09 Jul 2018 08:39:05 GMT
-> Server: Apache/2.2.15 (CentOS)
-> X-Powered-By: PHP/5.3.3
-> Content-Length: 139
-> Content-Type: text/html; charset=UTF-8
->
-> <html>
->    <head>
->       <title>SPECweb2005 Support Workload Init</title>
->    </head>
->    <body>
-> Error: Could not write to init_vars.php
->    </body>
-> </html>
-> 2018-07-09 04:39:05:535 SPECweb_Support: [ERROR] Error! setServerDate() failed.
-> 2018-07-09 04:39:05:536 RemoteLoadGen: [ERROR] Unable to successfully initialize workload variables. Terminating.
-> 2018-07-09 04:39:05:536 SpecwebControl: [ERROR] Could not create all client threads.
-> 2018-07-09 04:39:05:536 SpecwebControl: [ERROR] setupWorkload() failed!
-> 2018-07-09 04:39:05:536 SpecwebControl: [ERROR] runTests() failed!
-> 2018-07-09 04:39:05:536 SpecwebControl: [ERROR] Benchmark run failed!
-> 2018-07-09 04:39:05:538 SpecwebControl: Terminating run. Please wait...

[RamyaMeruva]

JAVA_HOME on client1, set it to set to /usr/lib/jvm/jre.

-Ramya