Author Topic: Webserver workload some request time outs but still 100% QOS  (Read 79626 times)

andchri

  • Newbie
  • *
  • Posts: 9
  • Karma: +1/-0
Webserver workload some request time outs but still 100% QOS
« on: December 14, 2016, 07:15:28 AM »
I have been trying to tune the webserver workload for some time and a typical log I am getting is as follows:
-------------------------------------------------------------------------------------------------------------------------------------------------
client-192.168.1.8_1010.log  prime-client1_1096.log       
oaf783-kz-1# more 20161213-175618/client-192.168.1.8_1010.log
2016-12-13 17:56:18:822 Creating specwebclient using RMI Registry port 1010
2016-12-13 17:56:18:849 oaf783-kz-1:1010 ready...
2016-12-13 17:57:17:894 LoadGenerator: Setting sleep delay.
2016-12-13 17:57:24:276 specwebclient: setting tile ID to 0
2016-12-13 17:57:24:281 Looking up polling host: webserver:8001
2016-12-13 17:57:25:385 LoadGenerator: Ramping up for 150 seconds.
2016-12-13 17:57:25:385 LoadGenerator: Starting 2500 threads.
2016-12-13 17:59:55:385 LoadGenerator: All threads running, test starting
2016-12-13 18:01:51:016 SPECweb_Support: [ERROR] Expected 10 product categories, 0 found.
2016-12-13 18:01:51:016 SPECweb_Support: [ERROR] STATE 0; Expected 51700 bytes appended to response but found 3524287
2016-12-13 18:01:51:016 SPECweb_Support: [ERROR] STATE 0; RESPONSE INVALID on GET /support/index.php HTTP/1.1
2016-12-13 18:03:27:452 SPECweb_Support: [ERROR] Expected 10 product categories, 0 found.
2016-12-13 18:03:27:452 SPECweb_Support: [ERROR] STATE 0; Expected 51700 bytes appended to response but found 314574
2016-12-13 18:03:27:452 SPECweb_Support: [ERROR] STATE 0; RESPONSE INVALID on GET /support/index.php HTTP/1.1
2016-12-13 18:04:15:606 SPECweb_Support: [ERROR] Expected 60 optons, 0 found.
2016-12-13 18:04:15:606 SPECweb_Support: [ERROR] STATE 3; Expected 52700 bytes appended to response but found 18000
2016-12-13 18:04:15:606 SPECweb_Support: [ERROR] STATE 3; RESPONSE INVALID on GET /support/product.php?id=3054 HTTP/1.1
2016-12-13 18:04:52:545 SPECweb_Support: [ERROR] Expected 60 optons, 0 found.
2016-12-13 18:04:52:545 SPECweb_Support: [ERROR] STATE 3; Expected 52700 bytes appended to response but found 1510
2016-12-13 18:04:52:545 SPECweb_Support: [ERROR] STATE 3; RESPONSE INVALID on GET /support/product.php?id=2102 HTTP/1.1
2016-12-13 18:04:55:397 WorkloadScheduler: Clearing statistics...
2016-12-13 18:04:55:402 WorkloadScheduler: Clearing statistics...
2016-12-13 18:07:09:630 SPECweb_Support: [ERROR] Expected 60 optons, 0 found.
2016-12-13 18:07:09:630 SPECweb_Support: [ERROR] STATE 3; Expected 52700 bytes appended to response but found 80
2016-12-13 18:07:09:630 SPECweb_Support: [ERROR] STATE 3; RESPONSE INVALID on GET /support/product.php?id=1667 HTTP/1.1
2016-12-13 18:11:03:846 SPECweb_Support: [ERROR] STATE 5; RESPONSE DYNAMIC CONTENT NOT VALIDATED! EXPECTED 5188365, GOT -1
2016-12-13 18:11:03:846 SPECweb_Support: [ERROR] STATE 5; Expected 113300 bytes appended to response but found 250
2016-12-13 18:11:03:846 SPECweb_Support: [ERROR] STATE 5; RESPONSE INVALID on GET /support/file.php?id=8145 HTTP/1.1
2016-12-13 18:11:36:540 SPECweb_Support: [ERROR] Expected 60 optons, 0 found.
2016-12-13 18:11:36:541 SPECweb_Support: [ERROR] STATE 3; Expected 52700 bytes appended to response but found 110
2016-12-13 18:11:36:541 SPECweb_Support: [ERROR] STATE 3; RESPONSE INVALID on GET /support/product.php?id=1855 HTTP/1.1
2016-12-13 18:21:04:623 SPECweb_Support: [ERROR] Expected 60 optons, 0 found.
2016-12-13 18:21:04:623 SPECweb_Support: [ERROR] STATE 3; Expected 52700 bytes appended to response but found 27500
2016-12-13 18:21:04:623 SPECweb_Support: [ERROR] STATE 3; RESPONSE INVALID on GET /support/product.php?id=2282 HTTP/1.1
2016-12-13 18:24:55:414 LoadGenerator: Threads stopping.
2016-12-13 18:24:55:415 LoadGenerator: Ramping down for 180 seconds.
2016-12-13 18:24:58:882 MTFF[45]: [ERROR] Incorrect file size returned. Expected: 110; received: 0
2016-12-13 18:27:56:368 LoadGenerator: Finished ramping down.
2016-12-13 18:27:56:380 LoadGenerator: Threads finished.
2016-12-13 18:27:56:380 LoadGenerator: Collecting run statistics.
2016-12-13 18:27:57:381 catalog requests: 24963
2016-12-13 18:27:57:381 download requests: 14574
2016-12-13 18:27:57:381 file requests: 28923
2016-12-13 18:27:57:381 fileCatalog requests: 48109
2016-12-13 18:27:57:381 home requests: 17390
2016-12-13 18:27:57:381 product requests: 52794
2016-12-13 18:27:57:381 search requests: 26958
2016-12-13 18:27:57:381 Total requests: 213711, successful requests: 213707
2016-12-13 18:27:57:981 Threads finished...
2016-12-13 18:27:57:982 LoadGenerator: Returning statistics.

That is a shortened poll time of only 20 minutes.
The summary  results for that particular run are as follows:

Performance
Tile #   Pct Load    Web Server    Per-Tile
Score    Overall
Score
1    100%    178.09    98.95    98.95

Quality Of Service (QOS)
Tile #   Pct Load    Web Server    Per-Tile
Score    Overall
Score
1    100%    1.00    1.00    100.00%

Web Server
Web Wkload    Req/sec    Good    Tolerable    Fail    Valid. Errors
Support             178.09          213704           3              0                   4

I have managed to get up to 179 Req/sec but have not eliminated the time out errors.

The webserver is showing the errors in it's log also:
------------------------------------------------------------------------
[Tue Dec 13 18:11:37.397220 2016] [:error] [pid 4053] [client 192.168.1.8:47534] PHP Warning:  fopen(http://infraserver:81/fcgi-bin/besim_fcgi.fcgi?3&5): failed to open stream: Connection refused in /var/www/html/support/inc.common.php on line 56
[Tue Dec 13 18:11:37.397276 2016] [:error] [pid 4053] [client 192.168.1.8:47534] PHP Fatal error:  Call to undefined function report() in /var/www/html/support/inc.common.php on line 57
[Tue Dec 13 18:21:05.479760 2016] [:error] [pid 3784] [client 192.168.1.8:45195] PHP Warning:  fopen(http://infraserver:81/fcgi-bin/besim_fcgi.fcgi?3&4&2282): failed to open stream: Connection refused in /var/www/html/support/inc.common.php on line 56
[Tue Dec 13 18:21:05.479841 2016] [:error] [pid 3784] [client 192.168.1.8:45195] PHP Fatal error:  Call to undefined function report() in /var/www/html/support/inc.common.php on line 57
webserver1#

My ulimits seem fine, have already upped them from the standard.

I tried upping the vcpus slightly for both the prime client and webserver guest, which didn't have a significant effect.
I seem to have 50 - 60% idle time for cpu throughout the run while I have 24G of virtual memory for both guests and that does not appear to be a bottle neck either.

My webserver is apache2.4 with php 5.6. and I am using the prefork mpm on apache with the following tunings:

webserver
----------------
Timeout 20
KeepAlive on
MaxKeepAliveRequests 0
KeepAliveTimeout 60

<IfModule prefork.c>
StartServers      8
MinSpareServers   5
MaxSpareServers   20
ServerLimit     10240
MaxRequestWorkers       10240
MaxConnectionsPerChild  4000 
</IfModule>

infraserver
----------------
Timeout 20
KeepAlive on
MaxKeepAliveRequests 100
KeepAliveTimeout 60

<IfModule prefork.c>
StartServers     200
MinSpareServers  20
MaxSpareServers  50
ServerLimit     8000
MaxRequestWorkers       8000
MaxConnectionsPerChild  32000
</IfModule>

my php settings are:
---------------------------------
webserver1# more /etc/php/5.6/conf.d/extensions.ini
;
; Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved.
;

; To enable PHP extensions, uncomment the desired line(s) below, or add similar
; lines to another active PHP configuration (.ini) file.
;extension=apcu.so
;extension=bcmath.so
extension=bz2.so
;extension=calendar.so
extension=curl.so
;extension=dba.so
;extension=enchant.so
;extension=exif.so
extension=ftp.so
extension=gd.so
extension=gettext.so
;extension=gmp.so
;extension=intl.so
;extension=ldap.so
;extension=mbstring.so
;extension=mysql.so
extension=mysqli.so
;extension=opcache.so
extension=openssl.so
;extension=pcntl.so
extension=pdo_mysql.so
;extension=readline.so
;extension=shmop.so
extension=snmp.so
;extension=soap.so
;extension=sockets.so
;extension=suhosin.so
;extension=sysvmsg.so
;extension=sysvsem.so
;extension=sysvshm.so
extension=tidy.so
;extension=wddx.so
;extension=xmlrpc.so
;extension=xsl.so
;extension=zip.so

I recently moved from apache22 with php-5.3 (imposed my newer OS build not by choice) and some of the extensions are a little diffrent, namely apcu in 5.6 instead of 5.3 and also suhosin.
I have tried a number of runs with different tunings for those extensions with little or no effect on the end result.

I suppose my main question is wether or not these time outs are a deal breaker since the QOS is still being returned as 100% ?

I would prefer to get rid of them if at all possible but my tuning attempts are increasingly seeming like shots in the dark rather than informed experiments so if anyone has input or advice on that side I would greatly appreciate it.

many thanks,

Andrew

ChrisFloyd

  • Moderator
  • Jr. Member
  • *****
  • Posts: 52
  • Karma: +2/-0
Re: Webserver workload some request time outs but still 100% QOS
« Reply #1 on: December 14, 2016, 01:36:45 PM »
...
I have tried a number of runs with different tunings for those extensions with little or no effect on the end result.

I suppose my main question is wether or not these time outs are a deal breaker since the QOS is still being returned as 100% ?

I would prefer to get rid of them if at all possible but my tuning attempts are increasingly seeming like shots in the dark rather than informed experiments so if anyone has input or advice on that side I would greatly appreciate it.

many thanks,

Andrew

Given that your QoS is 100%, and the web transaction rate (178/sec) is on par with expected throughputs, the workload appears to be running correctly.  I suspect that those errors are caused by the very short warmup time of 150 secs.  Could you please try increasing the warmup time to at least 300 seconds and see if that eliminates those problems?  *Note, when many tiles are running and with the host server under high load, it is not uncommon to see a handful of IMAP errors (for mail), HTTP request/timeout errors (for web), and few DealerAgent errors for SPECjAppserver2004. Assuming each workload QoS is still within tolerance and the throughput rates are decent, a small number of these types of errors can be ignored.

Thanks,

Chris
« Last Edit: December 14, 2016, 01:43:27 PM by ChrisFloyd »

andchri

  • Newbie
  • *
  • Posts: 9
  • Karma: +1/-0
Re: Webserver workload some request time outs but still 100% QOS
« Reply #2 on: December 15, 2016, 12:05:37 PM »
Great thanks Chris,

Andrew