Web Server QoS Errors

[zoubotao]

These are all my vms：
appserver1、mailserver1、webserver1、infraserver1、batchserver1  in numa node 1；

appserver2、mailserver2、webserver2、infraserver2、batchserver2  in numa node 1；

appserver3、mailserver3、webserver3、infraserver3、batchserver3  in numa node 1；

appserver4、mailserver4、webserver4、infraserver4、batchserver4  in numa node 2；

appserver5、mailserver5、webserver5、infraserver5、batchserver5  in numa node 2；

appserver6、mailserver6、webserver6、infraserver6、batchserver6  in numa node 2；

dbserver1、dbserver5 in numa node 0；

appserver is configurated：4 core、8GB ram；

dbserver is configurated：12 core、28GB ram；

infraserver is configurated：1 core、2GB ram；

webserver is configurated：2 core、12GB ram；

mailserver is configurated：1 core、1GB ram；

batchserver is configurated：2 core、1GB ram；

All vm use Mellanox 25Gb vf；

When I ran 4 tiles, I can have a good result; but when I run 6 tiles, the QoS of webserver4，webserver5，webserver6 cannot pass：
Run 1
Tile 4
Web Server Errors:
Iteration 1 failed 95% TIME GOOD QoS. Achieved: 90.1%
Iteration 1 failed 99% TIME TOLERABLE QoS. Achieved: 96.9%
Tile 5
Web Server Errors:
Iteration 1 failed 95% TIME GOOD QoS. Achieved: 90.4%
Iteration 1 failed 99% TIME TOLERABLE QoS. Achieved: 97.1%
Tile 6
Web Server Errors:
Iteration 1 failed 95% TIME GOOD QoS. Achieved: 89.8%
Iteration 1 failed 99% TIME TOLERABLE QoS. Achieved: 97.0%

I am confused about it.

[RamyaMeruva]

Hi Zoubotao,

This seems more like resourcing/tuning issue. Since you've already got 4T passing, I would recommend u to first try 5T and monitor resource utilization.

Also, you can refer to latest published results on how to tune your testbed to support more tiles.

Thanks,
Ramya

[zoubotao]

Thank you for your reply. I will do it as you said.

[sachinkhanna48]

Hi Zoubotao,

This seems more like resourcing/tuning issue. Since you've already got 4T passing, I would recommend u to first try 5T and monitor resource utilization.

Also, you can refer to the latest published results on how to tune your testbed to support more tiles.

Thanks,
Ramya

Hi, I am facing the same problem please help me!

Thanks

[ChrisFloyd]

sachinkhanna48,

How many Tiles have you been able to run successfully?  More than 1 Tile?   If so, can you verify that you have sufficient network and storage resources to handle multiple tiles?  If you are running multiple tiles, are you using at least 10Gb NICs on the clients (or dedicated 1Gb NICs pre VM), and/or have you verified you do not have a client network bottleneck? Note, each Tile will require close to 1Gbit traffic on the SUT (as well as on the client/driver VM).  Disk IO requirements per tile are ~1000 IOPs (or more, depending on the VM memory configuration of your VMs), 

While recent published results may configure more resources than are technically necessary for SPECvirt_sc2013 results, they provide a reasonable idea of network and disk requirements for optimal performance.  Perhaps look at some published results with lower tile counts (older results), as well as recent results using the latest storage and network components: 

https://www.spec.org/virt_sc2013/results/specvirt_sc2013_perf.html

[ChrisFloyd]

Are you using 10GbE (or greater) network infrastructure on your client VMs and SUT VMs?  (i.e.,  10GbE NICs on client hosts, 10GbE network on SUT VMs, 10GbE network switch)

If you have 1 tile working successfully and passing QoS, but QoS fails at higher tile counts, this indicates a tuning problem - not a benchmark harness or workload functional problem.  All tiles drive the same load for the same workload type - therefore if higher tile counts are not passing webserver QoS, the culprit is likely a bottleneck on the SUT network or client network.  Each tile will require ~800Gbit on both the SUT and Client infrastructure.

[Bellantoni_874]

Is there any solution for such type of errors because I m still getting the same errors.
Please provide us proper guidance for such errors.

[Shuster512]

sachinkhanna48,

How many Tiles have you been able to run successfully?  More than 1 Tile?   If so, can you verify that you have sufficient network and storage resources to handle multiple tiles?  If you are running multiple tiles, are you using at least 10Gb NICs on the clients (or dedicated 1Gb NICs pre VM), and/or have you verified you do not have a client network bottleneck? Note, each Tile will require close to 1Gbit traffic on the SUT (as well as on the client/driver VM).  Disk IO requirements per tile are ~1000 IOPs (or more, depending on the VM memory configuration of your VMs), 

While recent published results may configure more resources than are technically necessary for SPECvirt_sc2013 results, Facetime for PC they provide a reasonable idea of network and disk requirements for optimal performance.  Perhaps look at some published results with lower tile counts (older results), as well as recent results using the latest storage and network components: 

https://www.spec.org/virt_sc2013/results/specvirt_sc2013_perf.html

Thanks for the taking interest and help.

[ChrisFloyd]

No problem, Shuster512.  I'm glad we were able to help.

[Charles9388]

Hi Zoubotao,

This seems more like resourcing/tuning issue. Since you've already got 4T passing, I would recommend u to first try 5T and monitor resource utilization. upsers

Also, you can refer to latest published results on how to tune your testbed to support more tiles.

Thanks,
Ramya

Did you have a fix on this issue? Facing same issue but no response from anyone

[AsheetHakoo]

Hi Charles9388
can you give some details of your setup and at which scale are you seeing the issues.
Are you also monitoring the Network utilization of the SUT ? Generally this seems to be Network resource issue, but do send across the scale and the SUT information and we can take it from there.
thanks

[Vargas69]

When I ran 4 tiles, I can have a good result; but when I run 6 tiles, the QoS of webserver4，webserver5，webserver6 cannot pass：
Run 1
Tile 4
Web Server Errors:
Iteration 1 failed 95% TIME GOOD QoS. Achieved: 90.1%
Iteration 1 failed 99% TIME TOLERABLE QoS. Achieved: 96.9%
Tile 5
Web Server Errors:
Iteration 1 failed 95% TIME GOOD QoS. Achieved: 90.4%
Iteration 1 failed 99% TIME TOLERABLE QoS. Achieved: 97.1%
Tile 6
Web Server Errors:
Iteration 1 failed 95% TIME GOOD QoS. Achieved: 89.8%
Iteration 1 failed 99% TIME TOLERABLE QoS. Achieved: 97.0%

I am confused about
web servers require threads and processes to serve HTTP requests. Each TCP connection to the web server occupies one of these threads respectively processes.

[Hegmann]

Why do you need quality of service for a web application? Well, web servers require threads and processes to serve HTTP requests. Each TCP connection to the web server occupies one of these threads respectively processes. Sometimes a server gets too busy to serve every request due to the lack of free processes or threads. Another parameter requiring control by mod_qos is the available bandwidth: all clients communicate to the server over a network link with limited bandwidth. Overfilling the link results in network congestion and poor performance.