SPECweb2009 Release 1.20 Backend Simulator (BeSim) Design Document

Version 1.20, Last modified 2010-04-29


Overview

SPECweb2009 includes new workloads to represent web server performance for applications such as online banking and e-commerce.  In real life the web servers supporting these applications would need to communicate with backend application and database servers.  The current literature on these Internet applications show an architecture where the web server or web server farm interfaces with an application server or servers via web services protocols (e.g. SOAP over HTTP); the application server connects to the database server.   Since the WEB subcommittee's focus is on measuring the performance of the web server, the SPECweb2009 benchmark needs to include a lightweight and efficient Backend Simulator (BeSim) to stand in for the Application and Database server portion of the solution architecture.  The Backend Simulator must be able to behave like a perfectly sized backend system for the SPECweb2009 web server under test (SUT) and not become a bottleneck that would effect the measurement of the SUT.

The Backend Simulator runs on one or more systems that are accessible by the system under test (SUT) and by the prime client.   The workload specific web server API code interfaces with BeSim via HTTP requests to keep the simulation more true to life and keep the processing required to something familiar to people who've worked with web servers and SPECweb99.  The workload specific API code sends requests to BeSim to obtain data that is typically acquired from backend systems such as a user's bank balance or a registered user's shopping preferences.   The data in the BeSim response is then used by the API code to complete the page to be returned as the response to a client's request.
 

Basic Goals and Functions for BeSim


BeSim Implementation

The key requirements for BeSim's implementation are portability, easy of maintenance, and easy of extension to support new workloads.  To meet these requirements, BeSim makes use of existing HTTP software and is implemented as a dynamic API module.  This approach leverages already developed HTTP software that knows all about handling multiple HTTP requests and allows for rapid prototyping for new workloads.  

BeSim consists of a common set of core code that is used within one of several common web server API's.  If a faster BeSim is needed, users can either tune the web server software or select a faster web server.

 

How BeSim Works

The BeSim system is connected to the Prime Client and the SUT, when a SPECweb2009 test is started, the Prime Client sends an initialization message in the form of a dynamic HTTP request with a query string containing all the parameters the BeSim software needs for the test.

Besim initializes its data structures and writes out a small file (i.e. /tmp/besim_<workload>.globals) so if there are multiple copies of BeSim (such as multiple fast-cgi instances) they can synchronize. 

A value exchanged during initialization is used by BeSim to construct some of the data in the responses. The clients, which also have this value use it to validate the dynamic data returned by the SUT in its responses to the clients dynamic requests.

Most dynamic HTTP/HTTPS requests sent by the clients to the SUT require that the SUT send a request to BeSim to acquire one or more pieces of data.  The SUT uses this data to construct a response to the client's request.   The lightweight nature of BeSim allows it to respond quickly to these requests so that the overhead does not significantly impact the SUT. 

A standardized request and response format is used between the SUT and BeSim.  All requests sent to BeSim are in the form:

        GET /<api-dir>/<besim_api>?<Workload_Type>&<Command_Type>&<param=value>[&<param=value>]

Responses returned by BeSim are in the form:

        HTTP 200 OK
        Cache-control: no-store
        Content-type: text/html
        Content-Length: [length of all return text - excluding headers]

        <html>
        <head><title>SPECweb2009 BESIM</title></head>
        <body>
        <p>SERVER_SOFTWARE = [ServerSoftware]
        <p>REMOTE_ADDR = [RemoteAddr]
        <p>SCRIPT_NAME = [ScriptName]
        <p>QUERY_STRING = [QueryString]
        <pre>
        [BeSIM MessageText]
        </pre>
        </body></html>

The BeSim MessageText includes a status value of "0\n" for OK and "1\n" for ERROR.  If  BeSim returns an Error it also returns an Error Message to indicate why it failed.  If the request was OK, the rest of the Message Text contains the data for the response.  The workload specific BeSim request-response sequences are included in the appendixes below.

Building BeSim

To build BeSim, the tester must select and install web server software on the system to server as the BeSim Box.  They must also select a compatible BeSim API.  SPECweb2009 supplies Fast-CGI, ISAPI, and NSAPI implementations.  The directions for building each API are included with the BeSim software in the Make_Readme and in the User documentation.   The kit also includes a set of perl scripts to allow simple testing of BeSim operations.

Appendix A.  BeSim For Banking Request-Response Sequences


Command (# - Name)
Script Mapping
DB Op
Request Parameters Response
0 - Reset
Prime Client Only
NA
N_TIME&\
N_MinUserID&\
N_MaxUserID&Load&\
Check_URL_Base&\
Num_check_subdirs
S_Status (DONE)
1 - LoginPasswd
Login
Select
N_UserID


S_Status
N_Password

2 - AcctBalances
Login, Order_check, Transfer
Select
N_UserID


S_Status
N_NumberOfAccts Lines:
N_Acct_#&N_Type&D_Balance
3 -AcctSummary
Account_summary
Select
N_UserID



S_Status
N_NumberOfAcctsLines:
N_Acct_#&N_Type&D_Balance&\
D_TotalDeposit&D_AvgDeposit&\
D_TotalWitdrawal&D_AvgWithdrawal
4 - CheckDetail1
Check_detail_html and
Check_detail_image
Select


N_UserID&N_Check



S_Status
N_Acct&S_FrontURL&S_BackURL

Note: URL's for check images are created
using the base URL path info supplied with
the Reset (0) Command and appending:
"/user<10char_UID>/
<CIF|CIB><6digit_Check_No.>"
(where CIF stands for check image front
and CIB stands for check image back)
5 - Bill Payment
Bill_pay
Select
N_UserID

S_Status
N_NumberOfPayees Lines:
N_PayeeId&D_Payment&N_Date
6 - PostPayee
Post_payee
Insert
N_UserID&N_PayeeID&\
S_Name&S_Address&\
S_City&S_State&\
N_Zip&S_Phone
S_Status (DONE)
N_Confirmation

7 - QuickPay
Quick_pay
Insert
N_UserID&N_PayeeId&\
N_Date&D_Amount
S_Status (DONE)
N_Confirmation

8 - ReviewBillPay
Bill_pay_status_output
Select
N_UserID&N_StartDate&\
N_EndDate
S_Status
N_NumberOfPayees Lines:
N_PayeeId&N_Date&D_Amount
9 - Profile
Profile
Select
N_userID
S_Status
S_Address&S_Email&S_Phone
10 - ChangeProfile
Change_profile
Update
N_UserID& \
S_Address&\
S_Email&S_Phone
S_Status (DONE)
N_Confirmation
11 - PlaceChkOrder
Place_check_order
Select,Update,Insert
N_UserID&N_Acct#&\
N_Date&D_Price
S_Status (DONE)
N_Confirmation
12  - PostTransfer
Post_transfer
Select,Update,Insert,Insert,Select

N_UserID&N_Acct1&\
D_Amount&\
N_Acct2,N_Date
N_NumberOfAccts Linesi(2):
N_Acct_# &D_Balance



Data Formats

Userid - char[10]
Password - char[32] (encrypted using MD5 is returned by BeSim to webserver)
(Client passes char[10] to webserver)
Account number - char[10]
Account Type - int
Account Balance - numeric 10.2
Total Deposit - numeric 10.2
Avg Deposit - numeric 10.2
Total withdrawal - numeric 10.2
Avg withdrawal - numeric 10.2
Front image location - var char[100] (path)
Back image location - var char[100]
Payee id  - char[20]
Payment amount - numeric 10.2
Date - in format yyyymmdd
Payee name - var char[100]
Street Address - var char[100] – use %20 for blank character in request strings
City - var char[20]
Zip - char[5]
Phone number - char[20]
email address - char[30] – use %40 for @ character in request strings
S_Status – int string ["0\n"=OK, non-zero=error]

Note: "\"  line continuation, not part of actual string


Appendix B.  BeSim For Ecommerce Request-Response Sequences


Command (# - Name)
Script Mapping
DB Op
Request Parameters Response
0 - Reset
Prime Client Only
NA
N_TIME&N_MinRegion&N_MaxRegion
S_Status (DONE)
1 - Search
Search
Select

N_Region&S_Segment&\
S_Keyword1+S_Keyword2...

S_Status
N_Results Lines:                S_Segment&S_Title&\
S_URL&S_Overview
2 - ProductLines
browse
Select

N_Region&S_CustType

S_Status
N_NumberOfProductLines
(scales with N_Load):
S_ProductLine1
S_ProductLine2...
3 - ProductModels
browse_productline
Select

N_Region&S_CustType&\
S_ProductLine

S_Status
\N_ProductModelLines
(scales with N_Load):
S_ProductName&\
S_ProductHighlights&\
S_Feature1&S_FeatureDesc1&\
S_Feature2&S_FeatureDesc2&\
S_Feature3&S_FeatureDesc3...
4 - ProductDetails
productdetail
Select

N_Region&S_CustType&\
S_Product

S_Status
N_ProductDetail Lines:
S_ProductDetail1&\
S_ProductDetail2...
5 - CustomizationChoices
customize
Select

N_Region&S_CustType&\
S_ProductModel &N_Stage


S_Status
N_NumberOfComponents Lines
(varies with N_Stage):
S_ComponentType&\
S_ComponentName&\
S_ComponentID&\
D_Price
6 - GetPrice
cart & customize
Select
N_Region&S_Item&\
S_ComponentID1&\
S_ComponentID2...
S_Status
D_Price
S_Currency
7 - CheckoutLogin
login
Select
N_Region&S_UserID


S_Status
S_Password

8 - CheckoutRegister
login
Insert
 N_Region&\
S_FirstName&S_LastName&\
S_Email&S_Password
S_Status
N_Confirmation
9 - SaveCart
cart
Insert

N_Region&S_Email&\
N_Items&\
S_Item1&\
N_Quantity1&\
N_Components&\
S_ComponentID1&\
S_ComponentID2&\
S_Item2&\
N_Quantity2&\
N_Components&\
S_ComponentID1&\
S_ComponentID2...
S_Status
N_Confirmation


10 - SubmitOrder
confirm
Insert

N_Region&S_Email&S_ShipFirstName&\
S_ShipLastName&S_ShipAddress&\
S_ShipCity&S_ShipState&N_ShipZip&\
S_ShipPhone&S_ShipMethod&\
S_BillFirstName&S_BillLastName&\
S_BillAddress&S_BillCity&S_BillState&\
N_BillZip&S_BillPhone&S_CardType&\
N_CardNo&N_ExpMo&N_ExpYr&\
N_Items&\
S_Item1&N_Quantity1&\
N_Components&\
S_ComponentIDn&\
S_Item2&N_Quantity2&\
N_Components&\
S_ComponentIDn.
S_Status
S_ShipDate&N_Confirmation

11 - GetRegions
cart & customize
Select
 N_Region_flag
(where 0 or any value returns list of all regions)
S_Status
N_NumberRegions Lines:
S_Region


Data Formats

Region – char[8], i.e. “region1”-“region80”
Segment – var char[11], specifies the search segment, i.e. “techsupport”, “all”
KeywordN – var char[128] – search keywords (user input)
Title – char[16] – brief title of a document (could be random characters)
URL – char[128] – URL to the document (starts with http://, could be random after that)
Overview – char[256] – a brief description of the document (could be random)
CustType – char[4], i.e. “home”, “mlb”
ProductLine – var char[16], i.e. “desktop”, “server”, “storage”
ProductModels – int
ProductModel – var char [16], i.e. “desktop5”, “server16”
Stage – int, described below:

Stage #

Stage Type

Avg. # component types returned

Avg. # BeSim selections returned (per component)

1

DetailedComponents

20

5

2

ServiceSupport

6

4

3

Accessories

7

6

 Component Type – var char[128], i.e. “CPU”, “Memory”, “OS”
Component Name – var char[128], i.e. “Dual Processor Intel Xeon, 2.0GHz/1MB Cache”
Price – int, i.e. 199 for $199
UserID – email address - char[120] – use %40 for @ character in request strings
Password - char[32], since this is encrypted using MD5 is returned by BeSim to webserver
(Client passes char[10] to webserver)
FirstName – var char[14]
LastName – var char[15]
Address – var char[50]
City – var char[30]
State – char[2]
Zip – char[5]
Phone – char[15]
ShipMethod – char[5]
CardName – char[80]
CardType – char[16]
CardNo – long
ExpMo – int, 1-12
ExpYr – int, i.e. 2004
ShipDate - in format MM/DD/YYYY
Confirmation – long
S_Status – int strine ["0\n"=OK, non-zero=error]

Note: "\"  line continuation, not part of actual string


Appendix C.  BeSim For Support Request-Response Sequences


Command (# - Name)
Script Mapping
DB Op
Request Parameters
Response
0 - Reset
Prime Client Only
NA
N_TIME&N_Load S_Status (DONE)
1 - Category Listing
Index
Select
(none) S_Status
N_Results:
N_ProductCategoryID1&S_ProductCategory1
N_ProductCategoryID2&S_ProductCategory2...
2 - Product Listing
Catalog
Select
N_ProductCategoryID S_Status
N_Results:
N_ProductID1&S_Product1
N_ProductID2&S_Product2...
3 - Search
Search
Select

S_keyword1+\
S_keyword2+\
S_keyword3...
S_Status
N_Results:
N_ProductID1&S_Product1
N_ProductID2&S_Product2...
4 - Download Categories
Product
Select

N_ProductID S_Status
N_Results:
S_DownloadCategoryID1
S_DownloadCategoryID2...
5 - Languages
Product
Select

(none)
S_Status
N_Results:
S_Language1
S_Language2...
6 - Operating Systems
Product
Select

N_ProductID S_Status
N_Results:
S_OS1
S_OS2...

7 - Files
FileCatalog
Select

S_ProductID&\
S_DownloadCategory&\
S_Language&S_OS
S_Status
N_Results:             
N_FileID1&S_FileName1&S_Date&N_Size1
T_Description1                
N_FileID2&S_FileName2&S_Date&N_Size2
T_Description2....
8 - File
File
Select

N_ProductID&\
S_DownloadCategory&\
S_Language&S_OS
S_Status
S_FileName&S_Date&N_Size&\
S_DownloadURL
T_Description
T_AdditionalInfo



Data Formats

All N_*ID – unsigned int
S_Status – "0\n" if OK, non-zero if not
S_ProductCategorychar[25], i.e. “Servers”
S_Productchar[50], i.e. “ProEdge Ultra 1500”
S_DownloadCategorychar[25], i.e. “BIOS Updates”
S_Languagechar[25], i.e. “Brazilian Portuguese”
S_OS – char[50], i.e. “Operating System Y 2004 SP2”
S_FileName
varchar(255), i.e. “DisplayDriver1_4423.exe”, “web2004-1.00.tgz”
T_Descriptionvarchar(255), i.e. “Vendor XYZ Driver Update 44.23 (Service Release 2)”
T_AdditionalInfoN – text block; usually installation instructions, which can be in the 1KB-2KB range
S_DownloadURLvarchar(255), starts with ‘http://’
S_Datedatetime
N_Sizebigint, size in bytes

 Note: "\"  line continuation, not part of actual string



Copyright © 2005-2010 Standard Performance Evaluation Corporation.  All rights reserved.