Author Topic: runspecvirt fail (Read 46562 times)

lroderic · « **Reply #30 on:** September 19, 2017, 01:07:33 PM »

On both client1 and wclient1, please issue the command ifconfig and post the results here. Also, please post the contents of each client's /etc/hosts.

Lisa

zgy · « **Reply #31 on:** September 19, 2017, 08:55:48 PM »

[root@client1 SPECvirt]# ifconfig
eth0 Link encap:Ethernet HWaddr 52:54:00:80:3E:47
inet addr:172.21.128.242 Bcast:172.21.128.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe80:3e47/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13767853 errors:0 dropped:0 overruns:0 frame:0
TX packets:11040281 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:21092275871 (19.6 GiB) TX bytes:1277646784 (1.1 GiB)

eth1 Link encap:Ethernet HWaddr 52:54:00:80:3E:48
inet6 addr: fe80::5054:ff:fe80:3e48/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:541182 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:58465062 (55.7 MiB) TX bytes:508 (508.0 b)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:15244 errors:0 dropped:0 overruns:0 frame:0
TX packets:15244 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1700356 (1.6 MiB) TX bytes:1700356 (1.6 MiB)

virbr0 Link encap:Ethernet HWaddr 52:54:00:79:40:07
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

[root@wclient1 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 52:54:00:A0:4E:9B
inet addr:172.21.128.241 Bcast:172.21.128.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fea0:4e9b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10895663 errors:0 dropped:0 overruns:0 frame:0
TX packets:8802015 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:16612923171 (15.4 GiB) TX bytes:915310848 (872.9 MiB)

eth1 Link encap:Ethernet HWaddr 52:54:00:B0:80:D8
inet6 addr: fe80::5054:ff:feb0:80d8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:664266 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:71517156 (68.2 MiB) TX bytes:488 (488.0 b)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:5570 errors:0 dropped:0 overruns:0 frame:0
TX packets:5570 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:544845 (532.0 KiB) TX bytes:544845 (532.0 KiB)

virbr0 Link encap:Ethernet HWaddr 52:54:00:79:40:07
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

[root@client1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

# Internal VM-to-VM only communications
10.10.1.220 infraserver1-int
10.10.1.221 webserver1-int
10.10.1.224 appserver1-int
10.10.1.225 dbserver1-int specdb

[root@wclient1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

# Internal VM-to-VM only communications
10.10.1.220 infraserver1-int
10.10.1.221 webserver1-int
10.10.1.224 appserver1-int
10.10.1.225 dbserver1-int specdb

As https://www.spec.org/virt_sc2013/docs/SPECvirt_ClientHarnessUserGuide.html#mozTocId969790 said, /etc/hosts will be updateed like this:

192.168.122.10 client1
192.168.122.11 wclient1

192.168.122.20 client2
192.168.122.21 wclient2

But VM_EXAMPLE shows us /etc/hosts is :
192.168.1.1 infraserver infraserver1
192.168.1.2 webserver webserver1
192.168.1.3 mailserver mailserver1
192.168.1.4 appserver appserver1 specdelivery specemulator
192.168.1.5 dbserver dbserver1
192.168.1.6 batchserver batchserver1

192.168.1.8 client1 specdriver

So In Multy client "specdriver" is not needed? Or it is a typo?

Thansk.

lroderic · « **Reply #32 on:** September 20, 2017, 11:28:20 AM »

Here's your trouble:

eth1 Link encap:Ethernet HWaddr 52:54:00:80:3E:48
inet6 addr: fe80::5054:ff:fe80:3e48/64 Scope:Link

and

# Internal VM-to-VM only communications
10.10.1.220 infraserver1-int
...

The clients need to use only one vNIC, and it uses the external network to communicate with the workload VMs. You've got a second vNIC for the internal network on the client, and you need to disable/remove that on the client. In Section 2.4 of the Example VM guide:

The script also sets up hostnames for the internal network interface (eth1 between the VMs only) to
use the following internal IP addresses. Only network-intensive workload VMs have a second interface defined.

I'll update the guide to clarify that the clients only use the external network. Workload VMs use both the external and internal network.

On the client, disable and remove the internal vNIC:
ifdown eth1
rm /etc/sysconfig/network-scripts/ifcfg-eth1
On the client, remove all the entries for the internal addresses (10.10.1.x) from /etc/hosts. The client only talks over the external network, so you need the external addresses only.
On the client, ping webserver and start SPECpoll to see that the client can contact the workloads VMs.

I have a picture but have to figure out how to post it here.

Regarding specdriver, that's not a problem. That needs to be there.

Lisa

lroderic · « **Reply #33 on:** September 20, 2017, 12:22:16 PM »

Here's the picture. External network (blue) for all workload VMs and clients. Internal network (green) only between infra/web and app/db.

With the internal network, you're isolating the traffic between the VMs to offload it from the external network.

Lisa

zgy · « **Reply #34 on:** September 21, 2017, 09:08:42 AM »

Thanks Lisa

Now I used 10Ge card and remove the eth1 and result is: (Also fail if not use wclient.)
primectrl.out:
2017-09-21 20:45:38:090 [ERROR] wclient1:1096 (PRIME_HOST[0][1]) failed to enter run phase before start of polling interval!
2017-09-21 20:45:38:090 PrimeControl: dumping polling start response times...
2017-09-21 20:45:38:090 client1:1098 (PRIME_HOST[0][0]) msec after pollStart: 8
2017-09-21 20:45:38:091 [ERROR] wclient1:1096 (PRIME_HOST[0][1]) msec after pollStart: 9223372036854775807
2017-09-21 20:45:38:091 client1:1094 (PRIME_HOST[0][2]) msec after pollStart: 16
2017-09-21 20:45:38:091 client1:1092 (PRIME_HOST[0][3]) msec after pollStart: 7
2017-09-21 20:45:38:091 PrimeControl: [ERROR] one or more workloads failed to start runtime before start of polling interval. Aborting.
2017-09-21 20:45:38:091 PrimeControl: sending abortTest() to prime clients.
2017-09-21 20:45:38:091 PrimeControl: id=0, abortID=-1
2017-09-21 20:45:38:091 PrimeControl: id=1, abortID=-1
2017-09-21 20:45:38:092 PrimeControl: masters[0]=client1:1098
2017-09-21 20:45:38:092 PrimeControl: masters[1]=wclient1:1096
2017-09-21 20:45:38:092 PrimeControl: id=3, abortID=-1
2017-09-21 20:45:38:092 PrimeControl: masters[3]=client1:1092
2017-09-21 20:45:38:092 PrimeControl: id=2, abortID=-1
2017-09-21 20:45:38:092 PrimeControl: masters[2]=client1:1094
2017-09-21 20:45:54:960 PrimeControl: stopping remote client processes
2017-09-21 20:45:56:970 PrimeControl: stopping local client threads
2017-09-21 20:45:56:971 PrimeControl: [ERROR] startMasters() failed!
2017-09-21 20:45:56:971 PrimeControl: terminating run. Please wait...
2017-09-21 20:45:57:977 specvirt: benchmark run failed!
2017-09-21 20:45:57:977 specvirt: Done!

ERROR info from wclient1:
[root@wclient1 logs]# grep -rn ERROR .
./Clientmgr1_1096.out:138:-> Checking keyname:IGNORE_WPD_ERRORS with value: 0for compliance.
./20170921-201929/prime-wclient1_1096.log:131:Checking keyname:IGNORE_WPD_ERRORS with value: 0for compliance.
./20170921-201929/client-172.21.128.241_1010.log:575939:2017-09-21 20:45:46:925 MTFF[178]: [ERROR] Incorrect file size returned. Expected: 240; received: 0
./20170921-201929/client-172.21.128.241_1010.log:575941:2017-09-21 20:45:47:053 MTFF[269]: [ERROR] Incorrect file size returned. Expected: 240; received: 0
./20170921-201929/client-172.21.128.241_1010.log:575942:2017-09-21 20:45:47:052 MTFF[135]: [ERROR] Incorrect file size returned. Expected: 80; received: 0
./20170921-201929/client-172.21.128.241_1010.log:575943:2017-09-21 20:45:47:006 MTFF[249]: [ERROR] Incorrect file size returned. Expected: 260; received: -252
./Clientmgr1_1088w.out:575945:-> 2017-09-21 20:45:46:925 MTFF[178]: [ERROR] Incorrect file size returned. Expected: 240; received: 0
./Clientmgr1_1088w.out:575947:-> 2017-09-21 20:45:47:053 MTFF[269]: [ERROR] Incorrect file size returned. Expected: 240; received: 0
./Clientmgr1_1088w.out:575948:-> 2017-09-21 20:45:47:052 MTFF[135]: [ERROR] Incorrect file size returned. Expected: 80; received: 0
./Clientmgr1_1088w.out:575949:-> 2017-09-21 20:45:47:006 MTFF[249]: [ERROR] Incorrect file size returned. Expected: 260; received: -252
[root@wclient1 logs]# grep -rn Error .
[root@wclient1 logs]#

ERROR info from client1:
[root@client1 20170921-201929]# grep -rn ERROR .
./prime-client1_1094.log:51:RESULT.IMAP_APPEND.100.ERROR"0"
./prime-client1_1094.log:60:RESULT.IMAP_FETCH_NUM_RFC822HEADER.100.ERROR"0"
./prime-client1_1094.log:69:RESULT.IMAP_FETCH_RFC822.100.ERROR"0"
./prime-client1_1094.log:78:RESULT.IMAP_LOGIN.100.ERROR"0"
./prime-client1_1094.log:87:RESULT.IMAP_LOGOUT.100.ERROR"0"
./prime-client1_1094.log:96:RESULT.IMAP_SEARCH_ALL_SUBJECT.100.ERROR"0"
./prime-client1_1094.log:105:RESULT.IMAP_SEARCH_UNSEEN.100.ERROR"0"
./prime-client1_1094.log:114:RESULT.IMAP_SELECT_INBOX.100.ERROR"0"
./prime-client1_1094.log:123:RESULT.IMAP_SELECT_INBOX.SPEC.100.ERROR"0"
./prime-client1_1094.log:132:RESULT.IMAP_UID_STORE_NUM_UNSET_FLAGS_SEEN.100.ERROR"0"
./primectrl.out:73:2017-09-21 20:45:38:090 [ERROR] wclient1:1096 (PRIME_HOST[0][1]) failed to enter run phase before start of polling interval!
./primectrl.out:76:2017-09-21 20:45:38:091 [ERROR] wclient1:1096 (PRIME_HOST[0][1]) msec after pollStart: 9223372036854775807
./primectrl.out:79:2017-09-21 20:45:38:091 PrimeControl: [ERROR] one or more workloads failed to start runtime before start of polling interval. Aborting.
./primectrl.out:91:2017-09-21 20:45:56:971 PrimeControl: [ERROR] startMasters() failed!
./Clientmgr1_1094.out:70:-> RESULT.IMAP_APPEND.100.ERROR"0"
./Clientmgr1_1094.out:79:-> RESULT.IMAP_FETCH_NUM_RFC822HEADER.100.ERROR"0"
./Clientmgr1_1094.out:88:-> RESULT.IMAP_FETCH_RFC822.100.ERROR"0"
./Clientmgr1_1094.out:97:-> RESULT.IMAP_LOGIN.100.ERROR"0"
./Clientmgr1_1094.out:106:-> RESULT.IMAP_LOGOUT.100.ERROR"0"
./Clientmgr1_1094.out:115:-> RESULT.IMAP_SEARCH_ALL_SUBJECT.100.ERROR"0"
./Clientmgr1_1094.out:124:-> RESULT.IMAP_SEARCH_UNSEEN.100.ERROR"0"
./Clientmgr1_1094.out:133:-> RESULT.IMAP_SELECT_INBOX.100.ERROR"0"
./Clientmgr1_1094.out:142:-> RESULT.IMAP_SELECT_INBOX.SPEC.100.ERROR"0"
./Clientmgr1_1094.out:151:-> RESULT.IMAP_UID_STORE_NUM_UNSET_FLAGS_SEEN.100.ERROR"0"
[root@client1 20170921-201929]# grep -rn Error .
./prime-client1_1094.log:136:Error unmarshaling return header; nested exception is:
./Clientmgr1_1094.out:155:-> Error unmarshaling return header; nested exception is:
[root@client1 20170921-201929]#

I donot think the ./prime-client1_1094.log and ./Clientmgr1_1094.out is the reseaon that due to benchmark failed, because befor the Error(Error unmarshaling return header; nested exception is), it had receive a abort signal which i think caused by wclient1:1096.

paste more info about Clientmgr1_1088w.out:
[root@wclient1 logs]# tail -100 Clientmgr1_1088w.out
-> 2017-09-21 20:46:00:031 WorkloadScheduler[2285]: Scheduler stopped.
-> 2017-09-21 20:46:00:031 WorkloadScheduler[2285]: Stopping workload thread
-> 2017-09-21 20:45:48:536 WorkloadScheduler[1291]: Scheduler stopped.
-> 2017-09-21 20:46:00:031 WorkloadScheduler[1291]: Stopping workload thread
-> 2017-09-21 20:45:48:536 WorkloadScheduler[1289]: Scheduler stopped.
-> 2017-09-21 20:46:00:031 WorkloadScheduler[1289]: Stopping workload thread
-> 2017-09-21 20:45:48:535 WorkloadScheduler[1284]: Scheduler stopped.
-> 2017-09-21 20:46:00:031 WorkloadScheduler[1284]: Stopping workload thread
-> 2017-09-21 20:45:48:535 WorkloadScheduler[1281]: Scheduler stopped.
-> 2017-09-21 20:46:00:032 WorkloadScheduler[1281]: Stopping workload thread
-> 2017-09-21 20:45:48:535 WorkloadScheduler[1280]: Scheduler stopped.
-> 2017-09-21 20:46:00:032 WorkloadScheduler[1280]: Stopping workload thread
-> 2017-09-21 20:45:48:534 SPECweb_Support[1337]: State Switch [5,4]
-> 2017-09-21 20:45:48:533 SPECweb_Support[2020]: State Switch [2,1]
-> 2017-09-21 20:46:00:032 WorkloadScheduler[1337]: Scheduler stopped.
-> 2017-09-21 20:46:00:032 WorkloadScheduler[1337]: Stopping workload thread
-> 2017-09-21 20:46:00:032 WorkloadScheduler[2020]: Scheduler stopped.
-> 2017-09-21 20:46:00:032 WorkloadScheduler[2020]: Stopping workload thread
-> 2017-09-21 20:45:48:533 SPECweb_Support[1303]: STATE 2; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:032 SPECweb_Support[1303]: STATE 2; FILE BYTES READ = 29545
-> 2017-09-21 20:45:48:507 SPECweb_Support[2264]: State Switch [2,3]
-> 2017-09-21 20:46:00:032 WorkloadScheduler[2264]: Scheduler stopped.
-> 2017-09-21 20:46:00:032 WorkloadScheduler[2264]: Stopping workload thread
-> 2017-09-21 20:45:48:507 SPECweb_Support[1442]: STATE 2; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:032 SPECweb_Support[1442]: STATE 2; FILE BYTES READ = 29543
-> 2017-09-21 20:45:48:507 SPECweb_Support[2489]: STATE 3; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:032 SPECweb_Support[2489]: STATE 3; FILE BYTES READ = 57407
-> 2017-09-21 20:45:48:507 SPECweb_Support[2447]: State Switch [1,3]
-> 2017-09-21 20:45:48:507 SPECweb_Support[1487]: State Switch [1,1]
-> 2017-09-21 20:46:00:033 WorkloadScheduler[2447]: Scheduler stopped.
-> 2017-09-21 20:46:00:033 WorkloadScheduler[2447]: Stopping workload thread
-> 2017-09-21 20:46:00:033 WorkloadScheduler[1487]: Scheduler stopped.
-> 2017-09-21 20:46:00:033 WorkloadScheduler[1487]: Stopping workload thread
-> 2017-09-21 20:45:48:506 SPECweb_Support[2121]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:033 SPECweb_Support[2121]: STATE 1; FILE BYTES READ = 19934
-> 2017-09-21 20:45:48:483 SPECweb_Support[1735]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:033 SPECweb_Support[1735]: STATE 1; FILE BYTES READ = 19749
-> 2017-09-21 20:46:00:034 SPECweb_Support[2121]: State Switch [1,3]
-> 2017-09-21 20:46:00:034 WorkloadScheduler[2121]: Scheduler stopped.
-> 2017-09-21 20:46:00:034 WorkloadScheduler[2121]: Stopping workload thread
-> 2017-09-21 20:45:48:483 SPECweb_Support[1779]: State Switch [0,2]
-> 2017-09-21 20:45:48:483 SPECweb_Support[2062]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:034 SPECweb_Support[2062]: STATE 1; FILE BYTES READ = 19753
-> 2017-09-21 20:46:00:034 WorkloadScheduler[1779]: Scheduler stopped.
-> 2017-09-21 20:46:00:034 WorkloadScheduler[1779]: Stopping workload thread
-> 2017-09-21 20:45:48:483 SPECweb_Support[1882]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:034 SPECweb_Support[1882]: STATE 1; FILE BYTES READ = 20324
-> 2017-09-21 20:45:48:483 SPECweb_Support[2387]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:034 SPECweb_Support[2387]: STATE 1; FILE BYTES READ = 19928
-> 2017-09-21 20:45:48:483 SPECweb_Support[1402]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:034 SPECweb_Support[1402]: STATE 1; FILE BYTES READ = 19944
-> 2017-09-21 20:45:48:483 SPECweb_Support[1445]: State Switch [2,3]
-> 2017-09-21 20:45:48:483 SPECweb_Support[1810]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:034 SPECweb_Support[1810]: STATE 1; FILE BYTES READ = 19932
-> 2017-09-21 20:46:00:034 WorkloadScheduler[1445]: Scheduler stopped.
-> 2017-09-21 20:46:00:034 WorkloadScheduler[1445]: Stopping workload thread
-> 2017-09-21 20:45:48:483 SPECweb_Support[2429]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:035 SPECweb_Support[2429]: STATE 1; FILE BYTES READ = 20379
-> 2017-09-21 20:45:48:483 SPECweb_Support[1997]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:035 SPECweb_Support[1997]: STATE 1; FILE BYTES READ = 20318
-> 2017-09-21 20:45:48:483 SPECweb_Support[2112]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:035 SPECweb_Support[2112]: STATE 1; FILE BYTES READ = 20377
-> 2017-09-21 20:45:48:483 SPECweb_Support[2498]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:035 SPECweb_Support[2498]: STATE 1; FILE BYTES READ = 20383
-> 2017-09-21 20:45:48:483 SPECweb_Support[1339]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:035 SPECweb_Support[1339]: STATE 1; FILE BYTES READ = 20322
-> 2017-09-21 20:46:00:036 SPECweb_Support[1339]: State Switch [1,3]
-> 2017-09-21 20:46:00:036 WorkloadScheduler[1339]: Scheduler stopped.
-> 2017-09-21 20:46:00:036 WorkloadScheduler[1339]: Stopping workload thread
-> 2017-09-21 20:45:48:482 SPECweb_Support[1538]: State Switch [1,1]
-> 2017-09-21 20:45:48:458 SPECweb_Support[1830]: State Switch [2,3]
-> 2017-09-21 20:46:00:036 WorkloadScheduler[1538]: Scheduler stopped.
-> 2017-09-21 20:46:00:036 WorkloadScheduler[1538]: Stopping workload thread
-> 2017-09-21 20:46:00:036 WorkloadScheduler[1830]: Scheduler stopped.
-> 2017-09-21 20:46:00:036 WorkloadScheduler[1830]: Stopping workload thread
-> 2017-09-21 20:45:48:458 SPECweb_Support[1384]: STATE 5; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:036 SPECweb_Support[1442]: State Switch [2,3]
-> 2017-09-21 20:46:00:036 SPECweb_Support[1384]: STATE 5; FILE BYTES READ = 115794
-> 2017-09-21 20:46:00:036 WorkloadScheduler[1442]: Scheduler stopped.
-> 2017-09-21 20:46:00:036 WorkloadScheduler[1442]: Stopping workload thread
-> 2017-09-21 20:45:48:456 SPECweb_Support[2480]: State Switch [4,3]
-> 2017-09-21 20:45:48:456 SPECweb_Support[1611]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:037 SPECweb_Support[1611]: STATE 1; FILE BYTES READ = 19924
-> 2017-09-21 20:46:00:037 WorkloadScheduler[2480]: Scheduler stopped.
-> 2017-09-21 20:46:00:037 WorkloadScheduler[2480]: Stopping workload thread
-> 2017-09-21 20:45:48:456 SPECweb_Support[1823]: State Switch [1,3]
-> 2017-09-21 20:46:00:037 WorkloadScheduler[1823]: Scheduler stopped.
-> 2017-09-21 20:46:00:037 WorkloadScheduler[1823]: Stopping workload thread
-> 2017-09-21 20:45:48:456 SPECweb_Support[1541]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:037 SPECweb_Support[1541]: STATE 1; FILE BYTES READ = 19940
-> 2017-09-21 20:45:48:456 SPECweb_Support[1750]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:037 SPECweb_Support[1750]: STATE 1; FILE BYTES READ = 20326
-> 2017-09-21 20:45:48:455 SPECweb_Support[1654]: STATE 1; RESPONSE LENGTH = 20480
-> 2017-09-21 20:46:00:037 SPECweb_Support[1654]: STATE 1; FILE BYTES READ = 20322
2017-09-21 20:46:00:346 Terminating processes. Please wait...
2017-09-21 20:46:00:347 Killing client procs ...
2017-09-21 20:46:00:350 Done killing procs ...

zgy · « **Reply #35 on:** September 21, 2017, 09:08:52 AM »

ifconfig info :

[root@client1 SPECvirt]# ./ifconfig.sh
Tile 1 VMs:
dbserver1:
eth0 Link encap:Ethernet HWaddr 52:54:00:67:75:DB
inet addr:172.24.11.224 Bcast:172.24.11.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe67:75db/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11419 errors:0 dropped:0 overruns:0 frame:0
TX packets:731 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:794184 (775.5 KiB) TX bytes:104432 (101.9 KiB)

eth1 Link encap:Ethernet HWaddr 52:54:00:19:ED:DC
inet addr:10.10.1.225 Bcast:10.10.1.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe19:eddc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4080641 errors:0 dropped:0 overruns:0 frame:0
TX packets:3982337 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:912088505 (869.8 MiB) TX bytes:1007856027 (961.1 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:12246 errors:0 dropped:0 overruns:0 frame:0
TX packets:12246 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1633832 (1.5 MiB) TX bytes:1633832 (1.5 MiB)

appserver1:
eth0 Link encap:Ethernet HWaddr 52:54:00:8A:70:70
inet addr:172.24.11.223 Bcast:172.24.11.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe8a:7070/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1386278 errors:0 dropped:0 overruns:0 frame:0
TX packets:6411295 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:253066345 (241.3 MiB) TX bytes:8402372287 (7.8 GiB)

eth1 Link encap:Ethernet HWaddr 52:54:00:A6:BE:29
inet addr:10.10.1.224 Bcast:10.10.1.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fea6:be29/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4027232 errors:0 dropped:0 overruns:0 frame:0
TX packets:4039657 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1013511798 (966.5 MiB) TX bytes:906414850 (864.4 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:151645 errors:0 dropped:0 overruns:0 frame:0
TX packets:151645 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:25168594 (24.0 MiB) TX bytes:25168594 (24.0 MiB)

batchserver1:
eth0 Link encap:Ethernet HWaddr 52:54:00:62:D2:6D
inet addr:172.24.11.225 Bcast:172.24.11.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe62:d26d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11910 errors:0 dropped:0 overruns:0 frame:0
TX packets:373 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:777703 (759.4 KiB) TX bytes:64593 (63.0 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:664 errors:0 dropped:0 overruns:0 frame:0
TX packets:664 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:65257 (63.7 KiB) TX bytes:65257 (63.7 KiB)

mailserver1:
eth0 Link encap:Ethernet HWaddr 52:54:00:5F:8F:10
inet addr:172.24.11.222 Bcast:172.24.11.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe5f:8f10/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:191942 errors:0 dropped:0 overruns:0 frame:0
TX packets:289904 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:39773093 (37.9 MiB) TX bytes:322604850 (307.6 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:339 errors:0 dropped:0 overruns:0 frame:0
TX packets:339 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:29220 (28.5 KiB) TX bytes:29220 (28.5 KiB)

infraserver1:
eth0 Link encap:Ethernet HWaddr 52:54:00:DD:AF:B8
inet addr:172.24.11.220 Bcast:172.24.11.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fedd:afb8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1374108 errors:0 dropped:0 overruns:0 frame:0
TX packets:1379470 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:97859836 (93.3 MiB) TX bytes:274606426 (261.8 MiB)

eth1 Link encap:Ethernet HWaddr 52:54:00:D6:C5:5A
inet addr:10.10.1.220 Bcast:10.10.1.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fed6:c55a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2310511 errors:0 dropped:0 overruns:0 frame:0
TX packets:35061596 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:178925296 (170.6 MiB) TX bytes:52865583612 (49.2 GiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:439 errors:0 dropped:0 overruns:0 frame:0
TX packets:439 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:38667 (37.7 KiB) TX bytes:38667 (37.7 KiB)

webserver1:
eth0 Link encap:Ethernet HWaddr 52:54:00:0B:6B:A2
inet addr:172.24.11.221 Bcast:172.24.11.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe0b:6ba2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:9155011 errors:0 dropped:0 overruns:0 frame:0
TX packets:56127619 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1229199744 (1.1 GiB) TX bytes:75202972358 (70.0 GiB)

eth1 Link encap:Ethernet HWaddr 52:54:00:2F:3B:74
inet addr:10.10.1.221 Bcast:10.10.1.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe2f:3b74/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:35106428 errors:0 dropped:0 overruns:0 frame:0
TX packets:2265609 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:52871140880 (49.2 GiB) TX bytes:161157414 (153.6 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:14024 errors:0 dropped:0 overruns:0 frame:0
TX packets:14024 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1323402 (1.2 MiB) TX bytes:1323402 (1.2 MiB)

client1:
eth0 Link encap:Ethernet HWaddr 52:54:00:4F:9B:F3
inet addr:172.21.128.242 Bcast:172.21.128.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe4f:9bf3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6859780 errors:0 dropped:0 overruns:0 frame:0
TX packets:1654916 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:8832075123 (8.2 GiB) TX bytes:388737854 (370.7 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:4966 errors:0 dropped:0 overruns:0 frame:0
TX packets:4966 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:643363 (628.2 KiB) TX bytes:643363 (628.2 KiB)

wclient1:
eth0 Link encap:Ethernet HWaddr 52:54:00:8A:CB:C8
inet addr:172.21.128.241 Bcast:172.21.128.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe8a:cbc8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:54832580 errors:0 dropped:0 overruns:0 frame:0
TX packets:7834945 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:75118203239 (69.9 GiB) TX bytes:1021146124 (973.8 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1192 errors:0 dropped:0 overruns:0 frame:0
TX packets:1192 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:110359 (107.7 KiB) TX bytes:110359 (107.7 KiB)

[root@client1 SPECvirt]# ./hostname.sh
infraserver1
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

# Internal VM-to-VM only communications
10.10.1.220 infraserver1-int
10.10.1.221 webserver1-int
10.10.1.224 appserver1-int
10.10.1.225 dbserver1-int specdb

websserver1
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

# Internal VM-to-VM only communications
10.10.1.220 infraserver1-int
10.10.1.221 webserver1-int
10.10.1.224 appserver1-int
10.10.1.225 dbserver1-int specdb

appserver1:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

# Internal VM-to-VM only communications
10.10.1.220 infraserver1-int
10.10.1.221 webserver1-int
10.10.1.224 appserver1-int
10.10.1.225 dbserver1-int specdb

dbserver1:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

# Internal VM-to-VM only communications
10.10.1.220 infraserver1-int
10.10.1.221 webserver1-int
10.10.1.224 appserver1-int
10.10.1.225 dbserver1-int specdb

batchserver1:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

# Internal VM-to-VM only communications
10.10.1.220 infraserver1-int
10.10.1.221 webserver1-int
10.10.1.224 appserver1-int
10.10.1.225 dbserver1-int specdb

mailserver1:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

# Internal VM-to-VM only communications
10.10.1.220 infraserver1-int
10.10.1.221 webserver1-int
10.10.1.224 appserver1-int
10.10.1.225 dbserver1-int specdb

wclient1:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

client1:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
##

##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.

# External VM-to-client communications
172.24.11.220 infraserver infraserver1
172.24.11.221 webserver webserver1
172.24.11.222 mailserver mailserver1
172.24.11.223 appserver appserver1 specdelivery specemulator
172.24.11.224 dbserver dbserver1
172.24.11.225 batchserver batchserver1

172.21.128.242 client1 specdriver
172.21.128.241 wclient1

lroderic · « **Reply #36 on:** September 21, 2017, 10:29:17 AM »

Thanks for the details. You've got /etc/hosts and the vNICs correct on both the clients and the workload VMs. You're getting further!

Code: [Select]

msec after pollStart: ...
means that somewhere a VM is out of time sync. It happens on occasion if you don't regularly sync everything. Before you start a test, run the helper script timesynctiles.sh to sync the time and date on the clients and workload VMs. I always invoke this in my runspecvirt script so that I never encounter this.

I also think you're at a point where you can stop the verbose output of SPECweb in prime-client1_1094.log. In Control.config, DEBUG_LEVEL = 3 is plenty. If you edited webserver's Test.config to use verbose, you can reset that back to the default.

Lisa

zgy · « **Reply #37 on:** September 21, 2017, 11:07:23 AM »

Thanks , and what do you mean "If you edited webserver's Test.config to use verbose";
How to config it ?

lroderic · « **Reply #38 on:** September 21, 2017, 11:42:43 AM »

Set each webserver's /opt/SPECweb2005/Test.config so that DEBUG_LEVEL = 0.

Lisa

zgy · « **Reply #39 on:** September 21, 2017, 11:49:41 AM »

I have run the timesynctiles.sh, set DEBUG_LEVEL=1, and rerun it , I notice that timechecktiles.sh will not sync the wclient.
And May be it is the reason of the pre ERROR. ( [ERROR] wclient1:1096 (PRIME_HOST[0][1]) msec after pollStart: 9223372036854775807)

for i in `seq 1 $tiles`;
do
echo " "
echo $CLIENT$i:
ssh $CLIENT$i date
echo dbserver$i:
ssh dbserver$i date
echo appserver$i:
ssh appserver$i date
echo batchserver$i:
ssh batchserver$i date
echo mailserver$i:
ssh mailserver$i date
echo infraserver$i:
ssh infraserver$i date
echo webserver$i:
ssh webserver$i date
done

zgy · « **Reply #40 on:** September 21, 2017, 12:03:13 PM »

HI, I rerun , but it fail again

Because This time I set DEBUG_LEVEL=1, So no show ( [ERROR] wclient1:1096 (PRIME_HOST[0][1]) msec after pollStart: 9223372036854775807) this time

[root@client1 SPECvirt]# cat primectrl.out
2017-09-21 23:31:37:417 Thu Sep 21 23:31:37 CST 2017
2017-09-21 23:31:37:471 RMI server started: client1:9990
2017-09-21 23:31:37:478 [INFO] This is a perf-only benchmark run. Skipping active idle polling interval.
2017-09-21 23:31:37:478 PrimeControl: preparing client drivers.
2017-09-21 23:31:37:555 PrimeControl: starting clients...
2017-09-21 23:31:37:555 PrimeControl: starting clients...
2017-09-21 23:31:37:581 PrimeControl: PTDs not used for this benchmark run!
2017-09-21 23:31:37:582 PrimeControl: starting 4 masters.
2017-09-21 23:35:23:263 PrimeControl: waiting on 4 prime client(s).
.
2017-09-21 23:35:24:287 setting hostsReady = true
2017-09-21 23:37:39:554 PrimeControl: Workload and prime controller builds: 80
2017-09-21 23:37:51:347 specvirt: clock sync check completed successfully
2017-09-21 23:37:51:348 specvirt: initiating workload ramp-up.
2017-09-21 23:37:51:348 Polling start time = Thu Sep 21 23:57:52 CST 2017
2017-09-21 23:37:51:348 Polling end time = Fri Sep 22 01:57:52 CST 2017
2017-09-21 23:42:53:126 PrimeControl: all workloads started.
2017-09-21 23:57:54:348 [ERROR] wclient1:1096 (PRIME_HOST[0][1]) failed to enter run phase before start of polling interval!
2017-09-21 23:57:54:349 PrimeControl: [ERROR] one or more workloads failed to start runtime before start of polling interval. Aborting.
2017-09-21 23:57:54:349 PrimeControl: sending abortTest() to prime clients.
2017-09-21 23:58:15:726 PrimeControl: stopping remote client processes
2017-09-21 23:58:17:732 PrimeControl: stopping local client threads
2017-09-21 23:58:17:733 PrimeControl: [ERROR] startMasters() failed!
2017-09-21 23:58:17:733 PrimeControl: terminating run. Please wait...
2017-09-21 23:58:18:738 specvirt: benchmark run failed!
2017-09-21 23:58:18:738 specvirt: Done!

[root@wclient1 SPECvirt]# tail -100 Clientmgr1_1096.out
-> Checking keyname:FILE_4_DATA with value: bar,60,0.8for compliance.
-> Checking keyname:CLASS_6_DIST with value: 0.100for compliance.
-> Checking keyname:FILE_2_DATA with value: ccc,34,0.66for compliance.
-> Checking keyname:FILE_31_DATA with value: H_Service_Tag_Unkown,2170,0.0for compliance.
-> Checking keyname:ITERATIONS with value: 1for compliance.
-> Checking keyname:DYN_CMD_0_EXPECTED_PCT with value: 8.1072for compliance.
-> Checking keyname:IGNORE_WPD_ERRORS with value: 0for compliance.
-> Checking keyname:TIME_TOLERABLE with value: 5000for compliance.
-> Checking keyname:THINK_INTERVAL with value: 1for compliance.
-> Checking keyname:MARKER_FREQ with value: 4096for compliance.
-> 2017-09-21 23:35:25:868 SpecwebControl: Setting client overthink time allowed to 72 sec.
-> 2017-09-21 23:35:25:870 SpecwebControl: **** SPECweb2005 benchmark started
-> 2017-09-21 23:35:25:870 SpecwebControl: Not a multi-load or multi-step run..
-> 2017-09-21 23:35:25:873 SpecwebControl: * Running SPECweb_Support workload
-> 2017-09-21 23:35:25:873 SpecwebControl: wclient1:1010 has 2500 connections.(3)
-> 2017-09-21 23:35:25:873 Configuration: Clearing workload.
-> 2017-09-21 23:35:25:874 SpecwebControl: Adding workload(s)...
-> 2017-09-21 23:35:25:874 Configuration: Adding workload 0
-> 2017-09-21 23:35:25:877 RemoteLoadGen: Total clients: 1
-> 2017-09-21 23:35:25:878 RemoteLoadGen: Adding host wclient1:1010
-> 2017-09-21 23:35:25:895 SPECweb_Support: Entered primeClientInit()
-> 2017-09-21 23:35:25:902 HttpRequestSched: server: webserver; port: 80; Web server init() command:
-> GET /support/init.php?BESIM_HOST=infraserver&BESIM_PORT=81&BESIM_URI=/fcgi-bin/besim_fcgi.fcgi&BESIM_PERSISTENT=false&PADDING_DIR=/var/www/html/support/dynamic_padding/&SMARTY_DIR=/var/www/html/Smarty-2.6.26/libs/&SMARTY_SUPPORT_DIR=/var/www/html/support/&SEND_CONTENT_LENGTH=true HTTP/1.1
-> Accept: */*
-> Host: webserver
->
->
->
-> 2017-09-21 23:35:25:909 HttpRequestSched: Web server init command response:
-> HTTP/1.1 200 OK
-> Date: Thu, 21 Sep 2017 15:35:22 GMT
-> Server: Apache/2.2.15 (CentOS)
-> X-Powered-By: PHP/5.3.3
-> Content-Length: 538
-> Content-Type: text/html; charset=UTF-8
->
-> <html>
-> <head>
-> <title>SPECweb2005 Support Workload Init</title>
-> </head>
-> <body>
-> SERVER_SOFTWARE = Apache/2.2.15 (CentOS)
-> REMOTE_ADDR = 172.21.128.241
-> SCRIPT_NAME = /support/init.php
-> QUERY_STRING = BESIM_HOST=infraserver&BESIM_PORT=81&BESIM_URI=/fcgi-bin/besim_fcgi.fcgi&BESIM_PERSISTENT=false&PADDING_DIR=/var/www/html/support/dynamic_padding/&SMARTY_DIR=/var/www/html/Smarty-2.6.26/libs/&SMARTY_SUPPORT_DIR=/var/www/html/support/&SEND_CONTENT_LENGTH=true
-> SERVER_TIME = 1506008122000
-> </body>
-> </html>
->
-> 2017-09-21 23:35:25:910 HttpRequestSched: Server time is: Thu, 21 Sep 2017 15:35:22 GMT
-> 2017-09-21 23:35:25:910 HttpRequestSched: Server date is: 2017-09-21
-> 2017-09-21 23:35:25:910 HttpRequestSched: condGetReq: If-Modified-Since: Thu, 21 Sep 2017 15:35:22 GMT
->
->
-> 2017-09-21 23:35:25:910 SPECweb_Support: LOAD = 2500
-> 2017-09-21 23:35:25:910 HttpRequestSched: BeSim init() command:
-> GET /fcgi-bin/besim_fcgi.fcgi?3&0&1506009035&2500 HTTP/1.1
-> Accept: */*
-> Host: infraserver
->
->
->
-> 2017-09-21 23:35:27:420 HttpRequestSched: Besim init command response:
-> HTTP/1.1 200 OK
-> Date: Thu, 21 Sep 2017 15:35:23 GMT
-> Server: Apache/2.2.15 (CentOS)
-> Cache-control: no-store
-> Content-Length: 306
-> Connection: close
-> Content-Type: text/html; charset=UTF-8
->
-> <html>
-> <head><title>SPECweb2005 BESIM</title></head>
-> <body>
-> SERVER_SOFTWARE = Apache/2.2.15 (CentOS)
-> REMOTE_ADDR = 172.21.128.241
-> SCRIPT_NAME = /fcgi-bin/besim_fcgi.fcgi
-> QUERY_STRING = 3&0&1506009035&2500
-> <pre>
-> 0
-> DONE ResetDate = 20170921, Time=1506009035,Load=2500,SL=25
-> </pre>
-> </body></html>
->
->
-> 2017-09-21 23:35:27:423 RemoteLoadGen: Initializing clients...
-> 2017-09-21 23:35:34:458 RemoteLoadGen: setting client tile ID to 0
-> 2017-09-21 23:35:34:577 SpecwebControl: Waiting for SPECvirt start signal.
-> 2017-09-21 23:37:53:457 sleeping for 1 seconds.
-> 2017-09-21 23:37:54:458 SpecwebControl: *** Starting Iteration 1 of 1
-> 2017-09-21 23:37:54:458 RemoteLoadGen: 300-second ramp-up starting.
-> 2017-09-21 23:37:54:458 RemoteLoadGen: Setting load to 2500
-> 2017-09-21 23:42:55:231 RemoteLoadGen: Finished starting clients.
-> 2017-09-21 23:42:55:235 SpecwebControl: Warming up for 900 seconds.
-> 2017-09-21 23:57:55:238 SpecwebControl: Clearing results.
-> 2017-09-21 23:57:56:460 Received abort signal from prime controller. Terminating.
-> 2017-09-21 23:57:56:462 SpecwebControl: Terminating run. Please wait...
-> 2017-09-21 23:58:00:053 SpecwebControl: Starting 7200-second runtime.
-> 2017-09-21 23:58:00:065 RemoteLoadGen: stopping client wclient1:1010; abort=true
-> 2017-09-21 23:58:17:512 SpecwebControl: Stopping remote clients.

lroderic · « **Reply #41 on:** September 21, 2017, 12:15:38 PM »

Quote from: zgy on September 21, 2017, 11:49:41 AM

I have run the timesynctiles.sh, set DEBUG_LEVEL=1, and rerun it , I notice that timechecktiles.sh will not sync the wclient.
And May be it is the reason of the pre ERROR. ( [ERROR] wclient1:1096 (PRIME_HOST[0][1]) msec after pollStart: 9223372036854775807)

for i in `seq 1 $tiles`;
do
echo " "
echo $CLIENT$i:
ssh $CLIENT$i date
echo dbserver$i:
ssh dbserver$i date
...

You forgot to set the time/date on the wclient, so add it to timesynctiles.sh.

In Control.config, keep

Code: [Select]

DEBUG_LEVEL = 3

zgy · « **Reply #42 on:** September 21, 2017, 12:28:20 PM »

ok, I have add wclient to timesynctiles.sh last time.

I will reopen the DEBUG_LEVEL=3 this time to see what happen.

zgy · « **Reply #43 on:** September 21, 2017, 12:58:25 PM »

HI Lisa

The error is same:

[root@client1 SPECvirt]# cat primectrl.out
2017-09-22 00:26:46:464 Fri Sep 22 00:26:46 CST 2017
2017-09-22 00:26:46:464 specvirt: maxPreRunTime = 1201
2017-09-22 00:26:46:464 specvirt: runTime = 7200
2017-09-22 00:26:46:465 specvirt: runTime = 7200
2017-09-22 00:26:46:465 specvirt: runTime = 7200
2017-09-22 00:26:46:465 specvirt: runTime = 7200
2017-09-22 00:26:46:465 specvirt: runTime = 600
2017-09-22 00:26:46:465 specvirt: runTime = 600
2017-09-22 00:26:46:465 specvirt: runTime = 600
2017-09-22 00:26:46:465 specvirt: runTime = 600
2017-09-22 00:26:46:521 RMI server started: client1:9990
2017-09-22 00:26:46:529 [INFO] This is a perf-only benchmark run. Skipping active idle polling interval.
2017-09-22 00:26:46:530 PrimeControl: preparing client drivers.
2017-09-22 00:26:46:530 PrimeControl: PRIME_HOST 0 = client1:1098
2017-09-22 00:26:46:530 PrimeControl: PRIME_HOST 1 = wclient1:1096
2017-09-22 00:26:46:530 PrimeControl: PRIME_HOST 2 = client1:1094
2017-09-22 00:26:46:531 PrimeControl: PRIME_HOST 3 = client1:1092
2017-09-22 00:26:46:532 PrimeControl: Master 1: client1:1098
2017-09-22 00:26:46:532 PrimeControl: Master 2: wclient1:1096
2017-09-22 00:26:46:532 PrimeControl: Master 3: client1:1094
2017-09-22 00:26:46:532 PrimeControl: Master 4: client1:1092
2017-09-22 00:26:46:535 PrimeControl: adding host client1:1098
2017-09-22 00:26:46:542 PrimeControl: adding host wclient1:1096
2017-09-22 00:26:46:550 PrimeControl: adding host client1:1094
2017-09-22 00:26:46:556 PrimeControl: adding host client1:1092
2017-09-22 00:26:46:570 First client for 0: 172.21.128.242:1091
2017-09-22 00:26:46:588 First client for 1: 172.21.128.241:1010
2017-09-22 00:26:46:661 First client for 2: 172.21.128.242:1200
2017-09-22 00:26:46:668 First client for 3: 172.21.128.242:1902
2017-09-22 00:26:46:673 PrimeControl: starting clients...
2017-09-22 00:26:46:673 PrimeControl: starting clients...
2017-09-22 00:26:46:674 PrimeControl: clients.length = 4
2017-09-22 00:26:46:674 PrimeControl: clients.length = 4
2017-09-22 00:26:46:674 PrimeControl: clients[0].length = 1
2017-09-22 00:26:46:674 PrimeControl: clients[0].length = 1
2017-09-22 00:26:46:674 PrimeControl: clients[1].length = 1
2017-09-22 00:26:46:674 PrimeControl: starting clients[0][0]: 172.21.128.242:1091
2017-09-22 00:26:46:674 PrimeControl: starting clients[1][0]: 172.21.128.241:1010
2017-09-22 00:26:46:694 PrimeControl: started client: 172.21.128.242:1091
2017-09-22 00:26:46:694 PrimeControl: clients[1].length = 1
2017-09-22 00:26:46:694 PrimeControl: clients[2].length = 1
2017-09-22 00:26:46:694 PrimeControl: starting clients[2][0]: 172.21.128.242:1200
2017-09-22 00:26:46:697 PrimeControl: started client: 172.21.128.242:1200
2017-09-22 00:26:46:697 PrimeControl: clients[3].length = 1
2017-09-22 00:26:46:697 PrimeControl: starting clients[3][0]: 172.21.128.242:1902
2017-09-22 00:26:46:699 PrimeControl: started client: 172.21.128.242:1902
2017-09-22 00:26:46:699 PrimeControl: started client: 172.21.128.241:1010
2017-09-22 00:26:46:699 PrimeControl: clients[2].length = 1
2017-09-22 00:26:46:699 PrimeControl: clients[3].length = 1
2017-09-22 00:26:46:700 PrimeControl: PTDs not used for this benchmark run!
2017-09-22 00:26:46:700 PrimeControl: starting 4 masters.
2017-09-22 00:30:16:456 PrimeControl: master[0][0] sleeping 20 sec.
2017-09-22 00:30:16:456 PrimeControl: master[0][2] sleeping 20 sec.
2017-09-22 00:30:16:456 PrimeControl: master[0][3] sleeping 20 sec.
2017-09-22 00:30:16:456 PrimeControl: master[0][1] sleeping 20 sec.
2017-09-22 00:30:36:486 PrimeControl: waiting on 4 prime client(s).
2017-09-22 00:30:36:817 Sending config to client1:1098
2017-09-22 00:30:36:818 Sending config to client1:1094
2017-09-22 00:30:36:823 Sending config to client1:1092
2017-09-22 00:30:36:860 Sending config to wclient1:1096
2017-09-22 00:30:36:970 specvirt: waiting on 3 prime clients.
2017-09-22 00:30:37:104 specvirt: waiting on 2 prime clients.
.
2017-09-22 00:30:37:507 setting hostsReady = true
2017-09-22 00:30:45:689 specvirt: waiting on 1 prime clients.
2017-09-22 00:32:52:742 PrimeControl: Workload and prime controller builds: 80
2017-09-22 00:32:52:742 PrimeControl: awaiting runtime started signal from prime clients
2017-09-22 00:33:04:418 specvirt: clock sync check completed successfully
2017-09-22 00:33:04:418 specvirt: initiating workload ramp-up.
2017-09-22 00:33:04:418 Polling start time = Fri Sep 22 00:53:05 CST 2017
2017-09-22 00:33:04:419 Polling end time = Fri Sep 22 02:53:05 CST 2017
2017-09-22 00:38:06:180 PrimeControl: all workloads started.
2017-09-22 00:53:07:418 [ERROR] wclient1:1096 (PRIME_HOST[0][1]) failed to enter run phase before start of polling interval!
2017-09-22 00:53:07:419 PrimeControl: dumping polling start response times...
2017-09-22 00:53:07:419 client1:1098 (PRIME_HOST[0][0]) msec after pollStart: 7
2017-09-22 00:53:07:419 [ERROR] wclient1:1096 (PRIME_HOST[0][1]) msec after pollStart: 9223372036854775807
2017-09-22 00:53:07:419 client1:1094 (PRIME_HOST[0][2]) msec after pollStart: 20
2017-09-22 00:53:07:419 client1:1092 (PRIME_HOST[0][3]) msec after pollStart: 8
2017-09-22 00:53:07:419 PrimeControl: [ERROR] one or more workloads failed to start runtime before start of polling interval. Aborting.
2017-09-22 00:53:07:419 PrimeControl: sending abortTest() to prime clients.
2017-09-22 00:53:07:419 PrimeControl: id=0, abortID=-1
2017-09-22 00:53:07:419 PrimeControl: id=1, abortID=-1
2017-09-22 00:53:07:419 PrimeControl: masters[0]=client1:1098
2017-09-22 00:53:07:420 PrimeControl: masters[1]=wclient1:1096
2017-09-22 00:53:07:420 PrimeControl: id=3, abortID=-1
2017-09-22 00:53:07:420 PrimeControl: id=2, abortID=-1
2017-09-22 00:53:07:420 PrimeControl: masters[3]=client1:1092
2017-09-22 00:53:07:420 PrimeControl: masters[2]=client1:1094
2017-09-22 00:53:28:167 PrimeControl: stopping remote client processes
2017-09-22 00:53:30:176 PrimeControl: stopping local client threads
2017-09-22 00:53:30:177 PrimeControl: [ERROR] startMasters() failed!
2017-09-22 00:53:30:177 PrimeControl: terminating run. Please wait...
2017-09-22 00:53:31:183 specvirt: benchmark run failed!
2017-09-22 00:53:31:183 specvirt: Done!

But Now I have sync on all vms,
[root@client1 SPECvirt]# ./synccheck.sh
infraserver1:
Fri Sep 22 00:56:19 CST 2017
websserver1:
Fri Sep 22 00:56:20 CST 2017
appserver1:
Fri Sep 22 00:56:20 CST 2017
dbserver1:
Fri Sep 22 00:56:20 CST 2017
batchserver1:
Fri Sep 22 00:56:20 CST 2017
mailserver1:
Fri Sep 22 00:56:20 CST 2017
wclient1:
Fri Sep 22 00:56:21 CST 2017
client1:
Fri Sep 22 00:56:21 CST 2017

lroderic · « **Reply #44 on:** September 21, 2017, 01:05:30 PM »

Reboot all VMs, rerun the time sync script, and retry. What platform are you on? Are you using the hypervisor tools to sync the VM clock to the hypervisor? Try using NTP on the server, have both client and client sync to that, then sync the VMs to the clients.

Lisa

News:

Author Topic: runspecvirt fail (Read 46562 times)

lroderic

Re: runspecvirt fail

zgy

Re: runspecvirt fail

lroderic

Re: runspecvirt fail

lroderic

Re: runspecvirt fail

zgy

Re: runspecvirt fail

zgy

Re: runspecvirt fail

lroderic

Re: runspecvirt fail

zgy

Re: runspecvirt fail

lroderic

Re: runspecvirt fail

zgy

Re: runspecvirt fail

zgy

Re: runspecvirt fail

lroderic

Re: runspecvirt fail

zgy

Re: runspecvirt fail

zgy

Re: runspecvirt fail

lroderic

Re: runspecvirt fail