SPEC Community
Product Support => SPECvirt_sc2013 => Topic started by: onlyloveu2 on October 27, 2015, 09:37:51 PM
-
run runspecvirt.sh ....
0,0,2015-10-28 09:17:13:086,0,0.0,2,10.0,0,0.0,3853,25.00
0,1,2015-10-28 09:17:13:078,1289441,1289441,0,0,589479289457,5934284695,1289330,89,22,263,391940,0
0,2,2015-10-28 09:17:13:081,912576,912576,0,0,26350344,0,2938
0,3,2015-10-28 09:17:13:073,1,719,1,0,24,2120,719,0
2015-10-28 09:17:23:071 PrimeControl: stopping result polling.
2015-10-28 09:17:23:071 PrimeControl: waiting for all workloads to stop...
2015-10-28 09:17:33:071 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:17:48:072 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:18:03:073 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:18:18:073 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:18:33:073 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:18:48:073 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:19:03:074 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:19:18:074 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:19:33:074 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:19:48:075 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:20:03:075 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:20:18:075 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:20:33:075 PrimeControl: waiting for 2 masters to finish
2015-10-28 09:20:48:076 PrimeControl: waiting for 1 masters to finish
2015-10-28 09:21:03:076 PrimeControl: waiting for 1 masters to finish
2015-10-28 09:21:18:077 PrimeControl: waiting for 1 masters to finish
2015-10-28 09:21:33:077 PrimeControl: waiting for 1 masters to finish
2015-10-28 09:21:48:078 PrimeControl: waiting for 1 masters to finish
2015-10-28 09:22:03:078 PrimeControl: waiting for 1 masters to finish
2015-10-28 09:22:11:233 PrimeControl: all workloads stopped
2015-10-28 09:22:11:304 Workload validation errors reported!:
0-0-0 = Error reading file '/opt/SPECjAppServer2004/output/4/Atomicity.html'
0-0-1 = Dealer Purchase Transaction Mix: FAILED
0-0-2 = Dealer Manage Transaction Mix: FAILED
0-0-3 = Dealer Browse Transaction Mix: FAILED
0-0-4 = Dealer 90% Response Time FAILED
0-0-5 = Dealer Avg. Response Time FAILED
0-0-6 = Manage Cycle Time Avg. FAILED
0-0-7 = LargeOrderLine Vehicle Rate FAILED
0-0-8 = PlannedLines Vehicle Rate FAILED
0-0-9 = Manufacturing 90% Response Time FAILED
0-0-10 = Manufacturing Avg. Response Time FAILED
2015-10-28 09:22:11:310 PrimeControl: aggregate audit...
2015-10-28 09:22:11:311 PrimeControl: aggregate audit...
2015-10-28 09:22:11:316 PrimeControl: aggregate audit...
2015-10-28 09:22:11:377 PrimeControl: aggregate audit...
2015-10-28 09:22:11:377 PrimeControl: validating aggregate audit...
2015-10-28 09:22:13:529 PrimeControl: stopping clients.
2015-10-28 09:22:13:529 PrimeControl: stopping remote client processes
2015-10-28 09:22:15:586 PrimeControl: stopping local client threads
> Loading Raw Result File..
2015-10-28 09:22:15:722 PrimeControl: terminating run. Please wait...
2015-10-28 09:22:16:727 specvirt: Done!
-
result
-
Hello. You can see that ap/dbserver isn't working with the 25.00 second response time:
0,0,2015-10-28 09:17:13:086,0,0.0,2,10.0,0,0.0,3853,25.00
Please set NUM_WORKLOADS=1 so that you're running only the app/dbserver to help troubleshoot. Also if you haven't done this already, please set DEBUG_LEVEL=2. Then post Clientmgr1_1088.out, Clientmgr1_1098.out, and primectrl.out. The perf.html doesn't help us at this point.
-
[root@client1 SPECvirt]# cat Control.config|grep -A10 ^NUM_WORKLO
NUM_WORKLOADS = 1
WORKLOAD_LABEL[0] = "Application Server"
WORKLOAD_LABEL[1] = "Web Server"
WORKLOAD_LABEL[2] = "Mail Server"
WORKLOAD_LABEL[3] = "Batch Server"
BACKEND_VM_LABEL[0] = "Database Server"
BACKEND_VM_LABEL[1] = "Infrastructure Server"
# POLL_MASTERS controls whether or not to request polling data from the prime
[root@client1 SPECvirt]# cat Control.config|grep ^DEBUG
DEBUG_LEVEL = 2
[root@client1 SPECvirt]#
[root@client1 SPECvirt]# tail -20f primectrl.out
2015-10-28 22:40:42:476 Wed Oct 28 22:40:42 CST 2015
2015-10-28 22:40:42:480 Validator: [WARNING] NUM_WORKLOADS value is: 1; should be 4
2015-10-28 22:40:42:481 Validator: [WARNING] Non-compliant configuration.
2015-10-28 22:40:42:481 [WARNING] This will be a non-compliant benchmark result!
2015-10-28 22:40:42:543 RMI server started: client1:9990
2015-10-28 22:40:42:558 [INFO] This is a perf-only benchmark run. Skipping active idle polling interval.
2015-10-28 22:40:42:559 PrimeControl: preparing client drivers.
2015-10-28 22:40:42:619 PrimeControl: starting clients...
2015-10-28 22:40:42:632 PrimeControl: PTDs not used for this benchmark run!
2015-10-28 22:40:42:633 PrimeControl: starting 1 masters.
2015-10-28 22:42:23:883 PrimeControl: waiting on 1 prime client(s).
.
2015-10-28 22:42:24:903 setting hostsReady = true
2015-10-28 22:44:43:244 PrimeControl: Workload and prime controller builds: 79
2015-10-28 22:44:43:244 PrimeControl: awaiting runtime started signal from prime clients
2015-10-28 22:44:43:246 PrimeControl: all workloads started.
2015-10-28 22:45:12:683 specvirt: clock sync check completed successfully
2015-10-28 22:45:12:683 specvirt: initiating workload ramp-up.
2015-10-28 22:45:12:684 Polling start time = Wed Oct 28 23:05:13 CST 2015
2015-10-28 22:45:12:684 Polling end time = Thu Oct 29 01:05:13 CST 2015
-
Is that all?
-
this is all
-
I can't read that 7z file. Would you please use another tool or just post the contents of the log files?
-
zip
-
As we continue to debug, in Control.config please set POLL_INTERVAL_SEC = 900 so that you don't have to wait an hour as well as WARMUP_SECONDS= 300.
Check the open file limit on the client:
# ulimit -n
If you get a number such as 1024, then the ulimit hasn't been set. Check the /etc/sysctl.conf file to make sure that you have a line like:
fs.file-max = 500000
Then check /etc/bashrc to see if it has near the top:
# overide open files limit of 1024
ulimit -n 100000 > /dev/null 2>&1
Then reboot the client and rerun.
-
[root@client1 SPECvirt]# cat Control.config|grep ^WARMUP_SEC
WARMUP_SECONDS= 900
[root@client1 SPECvirt]# cat Control.config|grep ^POLL_INT
POLL_INTERVAL_SEC = 900
[root@client1 SPECvirt]# uptime
06:35:41 up 1 min, 1 user, load average: 0.04, 0.02, 0.00
[root@client1 SPECvirt]# ulimit -n
500000
[root@client1 SPECvirt]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.file-max = 1000000
vm.swappiness = 0
vm.overcommit_memory = 0
net.core.optmem_max = 20000000
net.core.rmem_default = 20000000
net.core.rmem_max = 20000000
net.core.wmem_default = 20000000
net.core.wmem_max = 20000000
net.core.somaxconn = 8192
net.ipv4.tcp_max_tw_buckets = 500000
net.ipv4.tcp_mem = 20000000 20000000 200000000
net.ipv4.tcp_rmem = 20000000 20000000 200000000
net.ipv4.tcp_wmem = 20000000 20000000 200000000
net.ipv4.ip_local_port_range = 4096 65535
net.ipv4.tcp_tw_reuse = 1
[root@client1 SPECvirt]# tail -6 /etc/security/limits.conf
* hard nofile 500000
* soft nofile 500000
* hard nproc 20000
* soft nproc 20000
[root@client1 SPECvirt]#
-
without html output.
-
From the Technical Support doc, log into the appserver and verify that emulator is accessible. Otherwise check the server log where emulator application is deployed:
# curl http://specemulator:8080/Emulator/EmulatorServlet?cmd=switchlog
200 OK
Verify that SPECjAppServer application is available on AppServer:
# curl http://specdelivery:8000/Supplier/DeliveryServlet?cmd=switchlog
200 OK
Verify that database is properly configured and accessible by the SPECjAppServer application on AppServer:
# curl http://specdelivery:8000/SPECjAppServer/app?action=atomicityTests
The resulting text should show PASSED for the three Atomicity tests.
In /etc/hosts, does appserver have the host aliases specemulator and specdelivery? That is, on the client and appserver, do you have something like:
192.168.1.4 appserver appserver1 specdelivery specemulator
Also, check ulimit on all VMs, not just the client.
-
all VMs ulimit -n set 500000 ?
-
Yes. It won't hurt.
-
After resize dbserver memory to 32G, the problem (can not start glassfish service) solved.
rerun runspecvirt.sh meet a new problem again.
-
The dbserver doesn't need that much memory. It could be that the JVM for GlassFish on the appserver or client couldn't allocate enough memory.
As we continue to debug, please set NUM_WORKLOADS=2 to run only appserver and webserver. In Clientmgr_1088.out you have:
Messages from: 10.4.3.71:1010
-> 2015-10-30 13:40:12:791 Connection: [ERROR] Bad status: 404
-> 2015-10-30 13:40:12:791 Connection: bad status ======= buf_len=478 headerEnd=161 contentLen=317 chunkEnd-1========
-> 2015-10-30 13:40:12:792 Connection: HTTP/1.1 404 Not Found
-> Date: Fri, 30 Oct 2015 05:40:12 GMT
-> Server: Apache/2.2.15 (CentOS)
-> Content-Length: 317
-> Content-Type: text/html; charset=iso-8859-1
->
-> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
-> <html><head>
-> <title>404 Not Found</title>
-> </head><body>
-> <h1>Not Found</h1>
-> <p>The requested URL /support/downloads/dir0000000102/download0_0 was not found on this server.</p>
-> <hr>
-> <address>Apache/2.2.15 (CentOS) Server at webserver Port 443</address>
-> </body></html>
->
-> 2015-10-30 13:40:12:792 HttpRequestSched: [ERROR] Bad response (-1). Request was GET /support/downloads/dir0000000102/download0_0 HTTP/1.1
This indicates that the webserver can't find the network file system where the webstore files reside. On the webserver you should have the following in /etc/fstab:
infraserver1-int:/home/webfiles/downloads /var/www/html/support/downloads nfs defaults 0 0
Would you log into webserver and do an:
# ls -l /var/www/html/support/downloads
When you ran setup_files.sh for webserver, did you encounter any errors?
-
I increase appserver memory to 32g,input error at last reply,
ok, i will check nfs on webserver. run webserver's deploy script not found error,
perhaps after reboot webserver,can not mount nfs.
-
[root@client1 SPECvirt]# curl http://specemulator:8080/Emulator/EmulatorServlet?cmd=switchlog
200 OK
[root@client1 SPECvirt]# curl http://specemulator:8080/Emulator/EmulatorServlet?cmd=switchlog
200 OK
[root@client1 SPECvirt]# curl http://specdelivery:8000/SPECjAppServer/app?action=atomicityTests
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
<META http-equiv="Content-Style-Type" content="text/css">
<TITLE>Atomicity Tests</TITLE>
</HEAD>
<BODY bgcolor="#ffffff" link="#000099" vlink="#000099">
<TABLE width="645" align="left">
<UL>
<FONT size="5" color="#000000"><b>SPECjAppServer2004 Transaction Atomicity Test</b></FONT>
<BR></BR>
</UL>
<P><BR>
</P>
<UL>
<b>Definition:</b> The System Under Test must guarantee that all Transactions are atomic; the system will either perform all individual operations on the data, or will assure that no partially-completed operations leave any effects on the data. The tests that were executed with their results below were used to determine if the System Under Test you are operating on meets all the transactional atomicity requirements. If any of the tests have a result of "FAILED" your system is not set up to ensure transaction atomicity
</UL>
<UL>
<b>Test 1: </b>This test checks to see if the proper transaction atomicity levels are being upheld in transactions associated with the benchmark. This test case drives placing an order for immediate insertion into the dealerships inventory. An exception is raised after placing the order and while adding the inventory to the dealer�s inventory table. This should cause the transaction changes to be removed from the database and all other items returned to how they existed before the transaction took place. This test case has three steps which are as follows to verify atomicity
<p>
1.) Query database to check how many inventory items the dealership has, the dealerships account balance, and the number of orders which have been placed for the dealer inside of the dealer domain. These numbers are the initial metrics that the final test cases should line up with after rolling back the transaction.
</p>
2.) Drives the above transaction which causes a transaction rollback exception to occur.
<p>
3.) Query database to check how many inventory items the dealership has, the dealerships account balance, and the number of orders which have been placed for the dealer inside of the dealer domain. These numbers should equal those in step 1) for the test case to be successful and the transaction to have been atomic.
</p>
<LI>Atomicity Test One: <b><FONT color="#ff0000">PASSED</b></FONT></LI><BR></BR>
<p></p>
<b>Test 2: </b>This test transaction simply tests that the application server is working properly and that it is able to insert an order as in Atomicity test 1 but without causing the exception and have it show up in the database.
<LI>Atomicity Test Two: <b><FONT color="#ff0000">PASSED</b></FONT></LI><BR></BR>
<p></p>
<b>Test 3: </b>This test checks to see if the proper transaction atomicity levels are being upheld in transaction associated with the benchmark and specifically the messaging subsystem in this test case. This test case drives placing a order which contains a large order and an item to be insert immediately into the dealerships inventory. An exception is raised after placing the order and while adding the inventory to the dealer�s inventory table. This should cause the transaction changes to be removed from the database, messages removed from queue and all other items returned to how they existed before the transaction took place. This test case has three steps which are as follows to verify atomicity.
<p>
1.) Query database to check how many inventory items the dealership has, the dealerships account balance, and the number of orders which have been placed for the dealer inside of the dealer domain. Also the large order table is queried to check how many large orders exist in the database before we drive the transaction. These numbers are the initial metrics that the final test cases should line up with after rolling back the transaction.
</p>
2.) Drives the above listed transaction which causes a transaction rollback exception to occur.
<p>
3.) Query database to check how many inventory items the dealership has, the dealerships account balance, and the number of orders which have been placed for the dealer inside of the dealer domain. Also query the large order table to check how many large orders there are in the table. These numbers should equal those in step 1) for the test case to be successful and the transaction to have been atomic.
</p>
<LI>Atomicity Test Three: <b><FONT color="#ff0000">PASSED</b></FONT></LI><BR></BR>
</UL>
</TABLE>
</BODY>
</HTML>
[root@client1 SPECvirt]# ssh root@appserver "cat /etc/hosts"
Warning: Permanently added 'appserver' (RSA) to the list of known hosts.
127.0.0.1 localhost
##
# Defaults used for the SPECvirt_sc2013 Example VM Setup Guide.
# External VM-to-client communications
10.4.3.61 webserver webserver1
10.4.3.62 infraserver infraserver1
10.4.3.63 mailserver mailserver1
10.4.3.65 appserver appserver1 specdelivery specemulator
10.4.3.64 dbserver dbserver1
10.4.3.66 batchserver batchserver1
10.4.3.71 client1 specdriver
# Internal VM-to-VM only communications
10.10.1.62 infraserver1-int
10.10.1.61 webserver1-int
10.10.1.65 appserver1-int
10.10.1.64 dbserver1-int specdb
10.10.1.71 client1-int
[root@client1 SPECvirt]# ssh root@appserver "cat /etc/hosts"|grep "appserver appserver1 specdelivery specemulator"
10.4.3.65 appserver appserver1 specdelivery specemulator
[root@client1 SPECvirt]# ssh root@dbserver "cat /etc/hosts"|grep "appserver appserver1 specdelivery specemulator"
10.4.3.65 appserver appserver1 specdelivery specemulator
[root@client1 SPECvirt]#
[root@client1 SPECvirt]#
[root@client1 SPECvirt]#
-
last debug log
-
OK, appserver is working even though you're getting those Http errors. Please check recent submissions for how other have tuned their networks in /etc/sysctl.conf. I'll also do some research. Because it's a short test, it will be non-compliant and QoS will FAIL. That's expected.
So on to webserver. Have you figured out what's up with the webstore that NFS serves from the infraserver to the webserver?
-
yes, infraserver has nfs service, webserver mounted it shared directory.
I collect all server configure info. please help me to check.
i want to get a non-error, complete(db,app,batch,mail,web) specvirt2013 performance result. thank you.
-
On infraserver:
/etc/init.d/httpd stop
rm -f /tmp/besim_support.globals
/etc/init.d/httpd start
On webserver, paste here the first two lines of the output of:
more /var/www/html/support/downloads/dir0000000102/download0_0
-
On Infraserver:
[root@infraserver ~]# /etc/init.d/httpd stop
Stopping httpd: [ OK ]
[root@infraserver ~]# ls -l /tmp/besim_support.globals
-rw-r--r-- 1 apache apache 336 Oct 31 12:14 /tmp/besim_support.globals
[root@infraserver ~]# rm -f /tmp/besim_support.globals
[root@infraserver ~]# /etc/init.d/httpd start
Starting httpd: [ OK ]
[root@infraserver ~]#
On webserver:
[root@webserver ~]# head -2 /var/www/html/support/downloads/dir0000000102/download0_0
104858 0_support/downloads/dir0000000102/download0_0
0*a(-a6qoy*/w%b+L7V61h|os@w_zA45pf^Anoj;y"bRidO7+]"AXKo1UWVGo_t
[root@webserver ~]# ls -l /var/www/html/support/downloads/dir0000000102/download0_0
-rw-r--r-- 1 apache apache 104858 Oct 26 23:19 /var/www/html/support/downloads/dir0000000102/download0_0
[root@webserver ~]#
-
Let's see if the client can see the web page. Restart httpd on both the infraserver and webserver. From the client, issue:
curl http://webserver/support/downloads/dir0000000102/download0_0|more
curl https://webserver/support/downloads/dir0000000102/download0_0|more
-
[root@client1 ~]# curl http://webserver/support/downloads/dir0000000102/download0_0|head -20
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 104858 0_support/downloads/dir0000000102/download0_0
0*a(-a6qoy*/w%b+L7V61h|os@w_zA45pf^Anoj;y"bRidO7+]"AXKo1UWVGo_t
HeZ{)O](ip!K.Q;20f+JG93p8ar9y:.D\O0X-AObpvK)n6pdEKXHj8E$$&@kg:*
RaIA?5,<---G$B<Z53)Q);oLsES{V#LEA#(wtOxg\Uc.6-u`\\gPX>BTWZp_Z-t
N%V@U`p=9Z8G:|%DCs"UJifi[Ow!/H6O(J{!y36:][q@d5B--wegp076oFw>-AJ
kc.Z/pWA_Iann./4J^EwJt.8Twp>WRx_wwwWjwOV/[q=.9SKbEq:gsQ"ALOY?i`
yswocp?m2ODU9dDY*;BzQW=Lr7sSgi{yk+z0^wi3zR3-WRe+UVS|F{OG+,2o'>O
l6CQzm{PRK%k4gJ|[wE<#kYnYiQGAPUi-y"q95Cc_DCJ7$b1!Mcjh8Ee'`Rb'8.
8(`]5;[l{^@U.k#avqb"t4EeA-a?bA@"8kMn'_U@mH16#J8\2oI@e=w*tiM/&JD
)QL.#Q_Z-C-S>TYKe&IWa]9+WPkwwM_PdVnv|9KFw{AMiE2j@GJY-.a@**NAAI+
[,z"GhG9<[)PnUwGZjGLxW9a.b/rgrRPM-Q"LP!_S=tp*T<t/>hZXas3iIyB*=f
{U*zS?W^DRgYA#9YP,G]ofzVQ7-AV4mN#"N*H|0P(a![+TR@|81nvoi)BCfNvIE
hzd)bCsIZ$*>(uO4Ki|uMq^LBfQc,g,^nE=%6dfh:W']gYXry(<RTW`@oov3F^=
9]a(`h|IE|D:ID9DsuHi>)Gd$_S%UB3OB0C4|E-q-ePgZ"wbP7MJ@K:xMvs3,#4
@F?CfQQW|NH&a4@gDo.)>m#uOd`[UB?.<MbFQs[T:YGA$\1ST-4!(hezUl{^Z"C
Lo|aF?AO1p>1=hj#b\JH%/X|#W+<`Z.RRL,XEqBT37XB!Kt"8au|.MlRvH4Ag8C
)1vcbFFp[`ZPC-=X68L_+l!x=hkq{/VlbJD^0t9^Nu<#UvNFR(Dk8Gc#dCtHo|[
hs?xdU3'3ja{Iz||vZE]Q32kWpJV0-&xcs.+BT8NVxmdt69x*)&Y!^V^g9rWFDo
S+*fRE@7SSQ9b5O9,>WcIj[ng7wR=g*yMZr`tP`GB/]x3n|`vyY{!IaW;]va($C
H27bW]V%0kd#V?.[;^`eEuU<?96zMbNXd<(!`@Cr|h*x4{'uW$;xyUi=4SxR&e'
29 102k 29 30727 0 0 3095k 0 --:--:-- --:--:-- --:--:-- 9.7M
curl: (23) Failed writing body (2041 != 16384)
[root@client1 ~]# curl https://webserver/support/downloads/dir0000000102/download0_0|head -20
curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
[root@client1 ~]# curl -k https://webserver/support/downloads/dir0000000102/download0_0|head -20
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 104858 0_support/downloads/dir0000000102/download0_0
0*a(-a6qoy*/w%b+L7V61h|os@w_zA45pf^Anoj;y"bRidO7+]"AXKo1UWVGo_t
HeZ{)O](ip!K.Q;20f+JG93p8ar9y:.D\O0X-AObpvK)n6pdEKXHj8E$$&@kg:*
RaIA?5,<---G$B<Z53)Q);oLsES{V#LEA#(wtOxg\Uc.6-u`\\gPX>BTWZp_Z-t
N%V@U`p=9Z8G:|%DCs"UJifi[Ow!/H6O(J{!y36:][q@d5B--wegp076oFw>-AJ
kc.Z/pWA_Iann./4J^EwJt.8Twp>WRx_wwwWjwOV/[q=.9SKbEq:gsQ"ALOY?i`
yswocp?m2ODU9dDY*;BzQW=Lr7sSgi{yk+z0^wi3zR3-WRe+UVS|F{OG+,2o'>O
l6CQzm{PRK%k4gJ|[wE<#kYnYiQGAPUi-y"q95Cc_DCJ7$b1!Mcjh8Ee'`Rb'8.
8(`]5;[l{^@U.k#avqb"t4EeA-a?bA@"8kMn'_U@mH16#J8\2oI@e=w*tiM/&JD
)QL.#Q_Z-C-S>TYKe&IWa]9+WPkwwM_PdVnv|9KFw{AMiE2j@GJY-.a@**NAAI+
[,z"GhG9<[)PnUwGZjGLxW9a.b/rgrRPM-Q"LP!_S=tp*T<t/>hZXas3iIyB*=f
{U*zS?W^DRgYA#9YP,G]ofzVQ7-AV4mN#"N*H|0P(a![+TR@|81nvoi)BCfNvIE
hzd)bCsIZ$*>(uO4Ki|uMq^LBfQc,g,^nE=%6dfh:W']gYXry(<RTW`@oov3F^=
9]a(`h|IE|D:ID9DsuHi>)Gd$_S%UB3OB0C4|E-q-ePgZ"wbP7MJ@K:xMvs3,#4
@F?CfQQW|NH&a4@gDo.)>m#uOd`[UB?.<MbFQs[T:YGA$\1ST-4!(hezUl{^Z"C
Lo|aF?AO1p>1=hj#b\JH%/X|#W+<`Z.RRL,XEqBT37XB!Kt"8au|.MlRvH4Ag8C
)1vcbFFp[`ZPC-=X68L_+l!x=hkq{/VlbJD^0t9^Nu<#UvNFR(Dk8Gc#dCtHo|[
hs?xdU3'3ja{Iz||vZE]Q32kWpJV0-&xcs.+BT8NVxmdt69x*)&Y!^V^g9rWFDo
S+*fRE@7SSQ9b5O9,>WcIj[ng7wR=g*yMZr`tP`GB/]x3n|`vyY{!IaW;]va($C
H27bW]V%0kd#V?.[;^`eEuU<?96zMbNXd<(!`@Cr|h*x4{'uW$;xyUi=4SxR&e'
31 102k 31 32768 0 0 308k 0 --:--:-- --:--:-- --:--:-- 10.4M
curl: (23) Failed writing body (4096 != 16384)
[root@client1 ~]#
-
SSL is running if this works:
curl -k http://webserver/support/downloads/dir0000000102/download0_0|head -20
Check to see if something is listening at port 443:
netstat -ant|grep 443
Should show
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
Also, do you have the following at the end of /etc/httpd/conf.d/ssl.conf:
ServerName webserver:443
-
[root@webserver ~]# cat /etc/httpd/conf.d/ssl.conf|grep 443
Listen 443
<VirtualHost _default_:443>
#ServerName www.example.com:443
ServerName webserver:443
[root@webserver ~]# netstat -tunlp | grep 443
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1904/httpd
[root@webserver ~]#
[root@client1 ~]# curl -k http://webserver/support/downloads/dir0000000102/download0_0|head -20
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 104858 0_support/downloads/dir0000000102/download0_0
0*a(-a6qoy*/w%b+L7V61h|os@w_zA45pf^Anoj;y"bRidO7+]"AXKo1UWVGo_t
HeZ{)O](ip!K.Q;20f+JG93p8ar9y:.D\O0X-AObpvK)n6pdEKXHj8E$$&@kg:*
RaIA?5,<---G$B<Z53)Q);oLsES{V#LEA#(wtOxg\Uc.6-u`\\gPX>BTWZp_Z-t
N%V@U`p=9Z8G:|%DCs"UJifi[Ow!/H6O(J{!y36:][q@d5B--wegp076oFw>-AJ
kc.Z/pWA_Iann./4J^EwJt.8Twp>WRx_wwwWjwOV/[q=.9SKbEq:gsQ"ALOY?i`
yswocp?m2ODU9dDY*;BzQW=Lr7sSgi{yk+z0^wi3zR3-WRe+UVS|F{OG+,2o'>O
l6CQzm{PRK%k4gJ|[wE<#kYnYiQGAPUi-y"q95Cc_DCJ7$b1!Mcjh8Ee'`Rb'8.
8(`]5;[l{^@U.k#avqb"t4EeA-a?bA@"8kMn'_U@mH16#J8\2oI@e=w*tiM/&JD
)QL.#Q_Z-C-S>TYKe&IWa]9+WPkwwM_PdVnv|9KFw{AMiE2j@GJY-.a@**NAAI+
[,z"GhG9<[)PnUwGZjGLxW9a.b/rgrRPM-Q"LP!_S=tp*T<t/>hZXas3iIyB*=f
{U*zS?W^DRgYA#9YP,G]ofzVQ7-AV4mN#"N*H|0P(a![+TR@|81nvoi)BCfNvIE
hzd)bCsIZ$*>(uO4Ki|uMq^LBfQc,g,^nE=%6dfh:W']gYXry(<RTW`@oov3F^=
9]a(`h|IE|D:ID9DsuHi>)Gd$_S%UB3OB0C4|E-q-ePgZ"wbP7MJ@K:xMvs3,#4
@F?CfQQW|NH&a4@gDo.)>m#uOd`[UB?.<MbFQs[T:YGA$\1ST-4!(hezUl{^Z"C
Lo|aF?AO1p>1=hj#b\JH%/X|#W+<`Z.RRL,XEqBT37XB!Kt"8au|.MlRvH4Ag8C
)1vcbFFp[`ZPC-=X68L_+l!x=hkq{/VlbJD^0t9^Nu<#UvNFR(Dk8Gc#dCtHo|[
hs?xdU3'3ja{Iz||vZE]Q32kWpJV0-&xcs.+BT8NVxmdt69x*)&Y!^V^g9rWFDo
S+*fRE@7SSQ9b5O9,>WcIj[ng7wR=g*yMZr`tP`GB/]x3n|`vyY{!IaW;]va($C
H27bW]V%0kd#V?.[;^`eEuU<?96zMbNXd<(!`@Cr|h*x4{'uW$;xyUi=4SxR&e'
33 102k 33 34783 0 0 296k 0 --:--:-- --:--:-- --:--:-- 314k
curl: (23) Failed writing body (297 != 14600)
[root@client1 ~]# curl -k https://webserver/support/downloads/dir0000000102/download0_0|head -20
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 104858 0_support/downloads/dir0000000102/download0_0
0*a(-a6qoy*/w%b+L7V61h|os@w_zA45pf^Anoj;y"bRidO7+]"AXKo1UWVGo_t
HeZ{)O](ip!K.Q;20f+JG93p8ar9y:.D\O0X-AObpvK)n6pdEKXHj8E$$&@kg:*
RaIA?5,<---G$B<Z53)Q);oLsES{V#LEA#(wtOxg\Uc.6-u`\\gPX>BTWZp_Z-t
N%V@U`p=9Z8G:|%DCs"UJifi[Ow!/H6O(J{!y36:][q@d5B--wegp076oFw>-AJ
kc.Z/pWA_Iann./4J^EwJt.8Twp>WRx_wwwWjwOV/[q=.9SKbEq:gsQ"ALOY?i`
yswocp?m2ODU9dDY*;BzQW=Lr7sSgi{yk+z0^wi3zR3-WRe+UVS|F{OG+,2o'>O
l6CQzm{PRK%k4gJ|[wE<#kYnYiQGAPUi-y"q95Cc_DCJ7$b1!Mcjh8Ee'`Rb'8.
8(`]5;[l{^@U.k#avqb"t4EeA-a?bA@"8kMn'_U@mH16#J8\2oI@e=w*tiM/&JD
)QL.#Q_Z-C-S>TYKe&IWa]9+WPkwwM_PdVnv|9KFw{AMiE2j@GJY-.a@**NAAI+
[,z"GhG9<[)PnUwGZjGLxW9a.b/rgrRPM-Q"LP!_S=tp*T<t/>hZXas3iIyB*=f
{U*zS?W^DRgYA#9YP,G]ofzVQ7-AV4mN#"N*H|0P(a![+TR@|81nvoi)BCfNvIE
hzd)bCsIZ$*>(uO4Ki|uMq^LBfQc,g,^nE=%6dfh:W']gYXry(<RTW`@oov3F^=
9]a(`h|IE|D:ID9DsuHi>)Gd$_S%UB3OB0C4|E-q-ePgZ"wbP7MJ@K:xMvs3,#4
@F?CfQQW|NH&a4@gDo.)>m#uOd`[UB?.<MbFQs[T:YGA$\1ST-4!(hezUl{^Z"C
Lo|aF?AO1p>1=hj#b\JH%/X|#W+<`Z.RRL,XEqBT37XB!Kt"8au|.MlRvH4Ag8C
)1vcbFFp[`ZPC-=X68L_+l!x=hkq{/VlbJD^0t9^Nu<#UvNFR(Dk8Gc#dCtHo|[
hs?xdU3'3ja{Iz||vZE]Q32kWpJV0-&xcs.+BT8NVxmdt69x*)&Y!^V^g9rWFDo
S+*fRE@7SSQ9b5O9,>WcIj[ng7wR=g*yMZr`tP`GB/]x3n|`vyY{!IaW;]va($C
H27bW]V%0kd#V?.[;^`eEuU<?96zMbNXd<(!`@Cr|h*x4{'uW$;xyUi=4SxR&e'
31 102k 31 32768 0 0 303k 0 --:--:-- --:--:-- --:--:-- 10.4M
curl: (23) Failed writing body (4096 != 16384)
[root@client1 ~]#
so, webserver https serivce port should be have no problem ?
-
Please connect to webserver and infraserver and post what's in /etc/sysctl.conf. Also on webserver and infraserver, make sure you have the following in /etc/security/limits.conf:
* hard nofile 1000000
* soft nofile 1000000
* hard nproc 100000
* soft nproc 100000
Then retry your test.
-
Last login: Wed Nov 4 05:57:08 2015 from 10.4.3.159
[root@infraserver ~]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536
# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
#
# SPECvirt_sc2013 tunings for infraserver
#
fs.file-max = 500000
net.core.somaxconn = 81920
net.ipv4.tcp_max_tw_buckets = 1500000
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_filter = 1
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_syn_retries = 4
net.ipv4.tcp_retries2 = 7
vm.swappiness=0
[root@infraserver ~]#
Last login: Wed Nov 4 13:47:40 2015 from 10.4.3.159
[root@webserver ~]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536
# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
#
# SPECvirt_sc2013 tunings for webserver
#
fs.file-max = 500000
net.core.somaxconn = 81920
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_filter = 1
net.ipv4.tcp_max_tw_buckets = 1500000
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_timestamps = 0
vm.swappiness=0
[root@webserver ~]#
-
Run test, the appserver and client1 need postgresql.jar?
I just run a test, discovered in the Clientmgr1_1088.out
-> Configuring 2 LargeOLAgent(s)...
-> woRatePerAgent for LO: 59.99988000024
-> Nov 05, 2015 3:50:05 AM com.sun.enterprise.v3.server.CommonClassLoaderServiceImpl findDerbyClient
-> INFO: Cannot find javadb client jar file, derby jdbc driver will not be available by default.
-> Nov 05, 2015 3:50:05 AM org.glassfish.enterprise.iiop.impl.GlassFishORBManager getCorbalocURL
-> INFO: list ==> specdelivery:3700
-> Nov 05, 2015 3:50:05 AM org.glassfish.enterprise.iiop.impl.GlassFishORBManager getCorbalocURL
-> INFO: corbaloc url ==> iiop:1.2@specdelivery:3700
-> Nov 05, 2015 3:50:09 AM com.sun.enterprise.v3.server.CommonClassLoaderServiceImpl findDerbyClient
-> INFO: Cannot find javadb client jar file, derby jdbc driver will not be available by default.
-> Nov 05, 2015 3:50:09 AM org.glassfish.enterprise.iiop.impl.GlassFishORBManager getCorbalocURL
-> INFO: list ==> specdelivery:3700
-> Nov 05, 2015 3:50:09 AM org.glassfish.enterprise.iiop.impl.GlassFishORBManager getCorbalocURL
-> INFO: corbaloc url ==> iiop:1.2@specdelivery:3700
->
-> Warmup = Thu Nov 05 03:54:13 CST 2015
-> SteadyState = Thu Nov 05 03:59:13 CST 2015
-> Rampdown = Thu Nov 05 04:14:13 CST 2015
-> Finish = Thu Nov 05 04:14:13 CST 2015
if it need, where path it find?
Can immediately reply me, I am on-line to wait for ..
This problem kept on for a long time.
-
last test debug info...
-
Congratulations, your short test ran. Ignore the findDerbyClient errors.
Is your appserver still set to 32 GB or memory? If so, set appserver to have 2 VCPUs and 8 GB memory, set NUM_WORKOADS=1, and rerun to make sure appserver is still good. If that's good, then set NUM_WORKLOADS=4 and run a short test.
-
appserver is 2 VCPUs and 8G memory now.
[root@appserver bin]# cat /proc/cpuinfo |grep ^processor
processor : 0
processor : 1
[root@appserver bin]# free -m
total used free shared buffers cached
Mem: 7873 7137 736 0 29 262
-/+ buffers/cache: 6845 1028
Swap: 4999 0 4999
[root@appserver bin]#
last test result
-
Go ahead and set NUM_WORKLOADS=4 and run a short test. If that looks good, run a long one.
-
fail NUM_WORKLOAD=4 short test
-
Find out a problem? Could you solve?
-
I think it's a tuning problem. I'm asking the experts - hold please.
-
appserver set 4cpu 16G memory , run NUM_WORKLOADS=4 done.
why can not set appserver 4cpu 16G ?
-
Please post the GlassFish server.log file.
-
glassfish log
-
We need to see the server.log after you see any Null Pointer exceptions or your run fails.
The server.log you attached is right after restarting GlassFish, so I don't see any errors.
-
ok, I run test again.
-
When you run, please check for your max network usage during the run, especially when you get errors in the client logs.
If there is network tuning or bandwidth issue, the 1098 client may show Null pointer exceptions.
Also, try after reload / restore your database.
-
After reload dbserver database, run test result.
-
Glassfish server.log shows:
java.sql.SQLException: Error in allocating a connection. Cause: In-use connections equal max-pool-size and expired max-wait-time. Cannot allocate more connections.
Check the max-pool-size and max-wait-time tunings in GlassFish domain.xml. You may have to increase and try. Please compare with domain.xml from a recent SPECvirt_sc2013 result that used GlassFish to determine which tunings you may need.
-
[root@appserver opt]# find . -name "domain.xml"
./glassfish3/glassfish/lib/templates/domain.xml
./glassfish3/glassfish/domains/spec2004-1/config/domain.xml
./glassfish3/glassfish/domains/domain1/config/domain.xml
./SPECjAppServer2004/bin/domain.xml
./appvm_scripts/files/domain.xml
./appvm_scripts/files/Glassfish312_SPECjApp_BuildScripts/domain.xml
[root@appserver opt]#
is ./glassfish3/glassfish/domains/spec2004-1/config/domain.xml ?
</domain>[root@appserver opt]# cat ./glassfish3/glassfish/domains/spec2004-1/config/domain.xml | grep max
<connector-connection-pool max-pool-size="250" pool-resize-quantity="16" steady-pool-size="16" name="jms/QueueConnectionFactory" resource-adapter-name="jmsra" connection-definition-name="javax.jms.QueueConnectionFactory">
<property name="max-poolsize" value="1000"></property>
<property name="max-pool-size" value="512"></property>
<jvm-options>-Dosgi.shell.telnet.maxconn=1</jvm-options>
<http default-virtual-server="server" max-connections="250">
<http default-virtual-server="server" max-connections="250">
<http default-virtual-server="__asadmin" max-connections="250" encoded-slash-enabled="true">
<thread-pool name="admin-thread-pool" max-thread-pool-size="50" max-queue-size="256"></thread-pool>
<thread-pool name="thread-pool-1" max-thread-pool-size="200"></thread-pool>
<jvm-options>-Dosgi.shell.telnet.maxconn=1</jvm-options>
<http default-virtual-server="__asadmin" max-connections="250">
<thread-pool max-thread-pool-size="200" name="thread-pool-1"></thread-pool>
[root@appserver opt]#
-
1. How to set appserver's domain.xml max-pool-size, not found max-wait-time.
2. How to set dbserver's shared_buffers and max_connections.
3. all servers and client configure files in Attachments, please check.
-
client configure
-
I'm sorry this is taking so long. Please set NUM_WORKLOADS=1 and POLL_INTERVAL_SEC=2400 to run only appserver for a longer time. Then zip and post the .out files. I want to see if it's a problem with appserver over time or appserver once you add the other workloads.
-
On the client, edit /opt/SPECjAppServer2004/config/run.properties and change:
dlrAgentMinHeapMB = 512
dlrAgentMaxHeapMB = 512
Then rerun a test.
Lisa
-
return error...