SPEC Community

Product Support => SERT => Topic started by: DarwinSmith on July 10, 2020, 02:25:46 PM

Title: Score and Transactions per Second
Post by: DarwinSmith on July 10, 2020, 02:25:46 PM

I think I have a misunderstanding of “Score” from the detailed SERT results Performance Data table

Explanation of “Score" from SERT-resultfilefields.html
11.5.4 Score
The fields in this column show the worklet specific score for each interval, which is calculated dividing the sum of all "Transaction Count" values for this interval by the "Elapsed Measurement Time (s)". For the calibration intervals a field showing the "Calibration Result" score is added in this column.

It sounds to me like “Score” is transactions per second. However, the numbers don’t add up for me. For example:

From SERT results-details.txt...
                                                    PERFORMANCE DATA                                                       
   Phase    | Interval | Actual |      Score      |  Host  | Client |   Elapsed   |   Transaction   | Transaction  | Transaction
            |          |  Load  |                 |   CV   |   CV   | Measurement |                 |    Count     |   Time (s) 
            |          |        |                 |        |        |  Time (s)   |                 |              |             
----------- | -------- | ------ | --------------- | ------ | ------ | ----------- | --------------- | ------------ | ------------
Measurement |   100%   | 99.9%  |   872,964.835   |  0.0%  |  1.2%  |   30.067    |   TxCryptoAes   |  26,234,086  |  1,776.203 
            |   75%    | 75.0%  |   655,372.195   |  0.0%  |  1.1%  |   30.067    |   TxCryptoAes   |  19,697,941  |  1,126.161 
            |   50%    | 50.0%  |   437,016.609   |  0.0%  |  1.1%  |   30.060    |   TxCryptoAes   |  13,132,541  |   692.204   
            |   25%    | 25.0%  |   218,529.155   |  0.0%  |  1.2%  |   30.057    |   TxCryptoAes   |  6,566,870   |   329.571   

However, when I do the calculation: 26,234,086 / 30.067 = 872,502.093

Title: Re: Score and Transactions per Second
Post by: JeremyArnold on July 14, 2020, 12:27:34 PM

Thanks for your question.

Unfortunately, the documentation in SERT-resultfilefields.html isn't completely accurate; we'll work on getting that updated.

For a single client JVM, the interval score is indeed calculated as the number of transactions divided by elapsed time.  But nearly all SERT worklets use multiple client JVMs running at the same time.  When we combine the scores from multiple client JVMs:

• The total Score is the sum of the scores of each JVM
• The transaction count is the sum of the counts of each JVM
• The elapsed time is calculated from the earliest start time divided by the latest end time of any JVM

The intervals start and end at almost exactly the same time in each JVM, so the elapsed times should be very close.  But the overall elapsed time will generally be slightly longer than the elapsed time of each JVM.  So if you divide the total transaction count by the overall elapsed time, you will tend to get a slightly lower score calculation than the reported value (the sum of the scores of each JVM).  The values are normally very close (in this case, within about 0.05%), but not identical.

The per-JVM metrics are available in the results.xml file if you want to double-check the calculations yourself.  Due to the large number of JVMs in some environments it isn't practical for all of these values to be included in results-details.txt.

I hope this helps -- if you have any further questions, just let us know.