Validation Errors after SPECvirt run completed

[Miles]

Hi
A 8T4W run completed but many validation errors appeared on the report as follows:

Code: [Select]

Application Server Errors: ◾Dealer 90% Response Time FAILED
◾Dealer Avg. Response Time FAILED
◾Purchase Cycle Time Avg. FAILED
◾Manage Cycle Time Avg. FAILED
◾Browse Cycle Time Avg. FAILED
◾Vehicle Purchasing Rate FAILED
◾Largeorder Vehicle Purchase Rate FAILED
◾Regular Vehicle Purchase Rate FAILED
◾LargeOrderLine Vehicle Rate FAILED
◾PlannedLines Vehicle Rate FAILED
◾Manufacturing 90% Response Time FAILED

◾ Web Server Errors: ◾Iteration 1 failed 95% TIME GOOD QoS. Achieved: 74.8%
◾Iteration 1 failed 99% TIME TOLERABLE QoS. Achieved: 84.3%
◾Weighted percentage difference (1.65%) for product in Iteration 1 is too high. Expected: 274495 requests, Actual: 256246
◾Weighted percentage difference (1.44%) for fileCatalog in Iteration 1 is too high. Expected: 249555 requests, Actual: 233596
◾Sum of weighted percentage difference (6.42%) exceeds 1.5% for Iteration 1
◾Page home errors (1537) >= 0.01 * Requests ( 84269) = 842.69
◾Page search errors (4003) >= 0.01 * Requests ( 130547) = 1305.47
◾Page catalog errors (4406) >= 0.01 * Requests ( 121184) = 1211.84
◾Page product errors (11323) >= 0.01 * Requests ( 256246) = 2562.46
◾Page fileCatalog errors (10097) >= 0.01 * Requests ( 233596) = 2335.96
◾Page file errors (4147) >= 0.01 * Requests ( 140695) = 1406.95
◾Page download errors (1082) >= 0.01 * Requests ( 70335) = 703.35
◾Total validation errors (36595) >= 0.005 * TotalRequests ( 1036872) =5184.36

Code: [Select]

To enable SSLv3, log into the client and:

    Remove SSLv3 and MD5withRSA from the jdk.tls.disabledAlgorithms property in the $JDK_HOME/jre/lib/security/java.security file (SPEC virt_sc assumes an MD5-based SSLv3 cipher).
    Add SSL_RSA_WITH_RC4_128_MD5 to the jdk.tls.legacyAlgorithms list in the $JDK_HOME/jre/lib/security/java.security file.
    Remove !MD5 from the SSLCipherSuite list in /etc/httpd/conf.d/ssl.conf file.
My webserver is RHEL6.5, JAVA version is 1.7.0.45.
I can not find the "jdk.tls.legacyAlgorithms" security/java.security file.
Does other option have the same effect?

Thanks.

[DavidSchmidt]

Hi Miles.

It looks like you have multiple tiles going. The errors you are seeing appear to be due to resource or tuning constraints. The user forum is intended to be for help to ensure a user's harness is functioning as intended from a benchmark perspective. Optimization and tuning is unfortunately beyond the scope of the forum. I would recommend referring to published results for suggestions on how to optimize your tunings, particularly around network tuning.

Regarding your question about the SSLv3 functionality, you should check in the java.security file for the java you are using and look for a line like:

   jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, \
    EC keySize < 224

You would need to remove "SSLv3" and "MD5withRSA" from the line. That should allow you to use the default encryption and cipher key for SPECvirt_sc2013.

[Miles]

Hi
The validation errors will not appear if I add 1 dbserver into my 4T4W
(E.g dbserver1 for appserver1 and appserver2; dbserver2 for appserver3 and
appserver4)
It is the only way to make my application workload good.
I am trying and looking for the solution to meet the method in User's Guide.

[DavidSchmidt]

Hi Miles,

When you try to use a single dbserver, what are the entries in your hosts file for the appserver, dbserver and specdb on your clients and appserver/dbserver? Ideally, you should be referring to "appserver" and "dbserver" for each tile, with these names pointing to the appropriate IP addresses for the app/db server on that tile. For example, if you are using a single NIC per VM, your client and VMs' hosts file should look like:

Tile 1
   192.168.1.1   infraserver infraserver1
   192.168.1.2   webserver webserver1
   192.168.1.3   mailserver webserver1
   192.168.1.4   appserver appserver1 specdelivery specemulator
   192.168.1.5   dbserver dbserver1 specdb
   192.168.1.6   batchserver batchserver1
   
   192.168.1.8   specdriver client client1
   192.168.1.8   wclient wclient1

Tile 2
   192.168.2.1   infraserver infraserver2
   192.168.2.2   webserver webserver2
   192.168.2.3   mailserver webserver2
   192.168.2.4   appserver appserver2 specdelivery specemulator
   192.168.1.5   dbserver dbserver2 specdb
   192.168.2.6   batchserver batchserver2
   
   192.168.2.8   specdriver client client2
   192.168.2.8   wclient wclient2

All of your control scripts should utilize the non-indexed names so that they are portable across tiles. Also note that for tile 2, the dbserver is referencing the IP for dbserver1, even though we call it dbserver2. The fact that your setup works if you have individual dbserver VMs for each tile implies that there is some mismatch between your hosts files and your control scripts.

[Miles]

Hi
According to User's Guide, "the database server VM requires at least two vCPUs per tile"
I increased vCPUs of dbserver to "8" and completed a 4T4W run without any validation errors.
Then I extended to 8T4W, validation errors appear again in application and web workloads.
Do each workload VMs need more vCPUs/vRAM when the amount of tiles are extended?

Thanks.

[DavidSchmidt]

Hi Miles. No, the dbserver is the only one that has different requirements for more tiles since each dbserver handles up to 4 appservers.

[Miles]

Hi
There are 33 errors in the 9T4W report and low QOS (i.e Application Server of Tile 5)

But Pct Load in each Tile is 100%, it means my SUT can run with more tiles, right?
Is the bad QOS from the wrong setting?

I just increase vCPU/vRAM to each VM so far, but can not get a result which has a Pct Load < 100% in last tile yet.

Thanks.

[ChrisFloyd]

Miles,

The QoS values for your app server are very low (e.g., under 0.95) which is indicative of a system that is either in need of further tuning or possibly at the maximum load it can support.  For example, a SPECjAppServer QoS score of .96 means that 4% of the transactions are NOT meeting the required response time criteria (i.e., the transactions are taking too long - for whatever reason).   A QoS that is failing (typically under ~95% for the three QoS constrained workloads) is an indication that you should be either *decreasing* (not increasing) the Tile count, or tuning the OS/Hypervisor and/or guest software stack to address the excessive transaction response times for the affected workload(s).

*Note, the guidance that we can provide on this forum is limited to assistance related to initial harness, guest VM, and the example guest software implementation. From your posts, it seems you have successfully run multiple Tiles on your testbed.  Scale-up performance tuning assistance is beyond the scope of our support capacity. Tuning such environments depend heavily on your particular hardware/software infrastructure and associated configuration.

I would recommend closely reviewing the details of the hardware and software tuning for published SPECvirt_sc2013 results and compare with your existing hardware and software setup.  Assuming you are using a similar hardware and software configuration, there should be enough information in the FDR and data collection .tgz to provide further tuning information.

[Miles]

Hi
Thanks for reply.

Now I have a rookie question.
"How do I know how many tiles I can build on my SUT?"
Is there a method to predict at the first?

I just predict through
  1.how much memory allocated on the SUT and 
  2.how much vRAM each tile needs
to determine the number of tiles to build.

Any method more precise?

Thanks.

[ChrisFloyd]

Determining how many Tiles your environment can support is actually somewhat of an 'advanced' topic.   

However, once you are running a few tiles, you may want to check the CPU utilization of the host, and get a rough idea as to how many more Tiles you could support.  Note, many environments don't scale-up/scale-out linearly - so your maximum Tile limit may also be bounded by the performance capabilities of your disk-subsystem, SUT network, and host memory (among other things, including even the client/driver CPU or network capacity).

[Miles]

Hi
Now I know I need more runs to get the most tiles

Thanks for your reply.

[Miles]

Hi
What is the calculation formula for overall score?

For example, below is the result of tile-1

Code: [Select]

Tile #, Pct Load, Application,  Web,  Mail,  Batch,  Per-Tile Score
   1        100%     166.42    179.43 126.95 PASSED   94.53
How to calculate to get 94.53 through the score of each workload?

Thanks.

[ChrisFloyd]

Miles,

See section 1.3 of the SPECvirt_sc2013 Design Guide:

particularly:

"We calculate the overall score by taking each component workload in each tile and normalizing it against its theoretical maximum for the pre-defined load level. The normalized throughput scores for each tile are averaged arithmetically to create a per-tile submetric, and the submetrics for all tiles are added to get the overall performance metric."

The three values for the theoretical maximum scores for the SPECweb, mail, and SPECjApp workloads can be seen in the "FIXED WORKLOAD PROPERTIES" section of Control.config.

specifically:

# WORKLOAD_SCORE_TMAX_VALUE is the theoretical maximum throughput rate for each
# workload. Comment these values out if you do not want to normalize scores to
# the theoretical max. Setting the value to 0 has the effect of not using this
# workload's score in calculating the result.

WORKLOAD_SCORE_TMAX_VALUE[0] = 174.30
WORKLOAD_SCORE_TMAX_VALUE[1] = 179.98
WORKLOAD_SCORE_TMAX_VALUE[2] = 143.60
WORKLOAD_SCORE_TMAX_VALUE[3] = 0

...So in your case,
App: 166.42 / 174.30. = 0.9547
Web: 179.43 / 179.98 = 0.9969
Mail: 126.95 / 143.60 = 0.8841
-----------------------------------
Average=                       .9453

0.9453 * 100 = 94.53  (score for that tile)

[Miles]

Hi
Thanks very much !!

[Miles]

Hi

Code: [Select]

WORKLOAD_SCORE_TMAX_VALUE[0] = 174.30
WORKLOAD_SCORE_TMAX_VALUE[1] = 179.98
WORKLOAD_SCORE_TMAX_VALUE[2] = 143.60
The values are defined in Control.config but no statement to describe the metrics.
Can I refer to any documents to know how to calculate?

Thanks.