SPEC SFS(R)2014_vdi Result DDN Storage : SFA14KX with GridScaler SPEC SFS2014_vdi = 800 Desktops (Overall Response Time = 2.58 msec) =============================================================================== Performance =========== Business Average Metric Latency Desktops Desktops (Desktops) (msec) Ops/Sec MB/Sec ------------ ------------ ------------ ------------ 80 0.6 16000 234 160 1.7 32001 469 240 2.2 48001 698 320 2.5 64002 934 400 2.6 80003 1164 480 2.7 96003 1403 560 2.8 112004 1637 640 3.1 128004 1873 720 3.4 144005 2102 800 3.7 160005 2343 =============================================================================== Product and Test Information ============================ +---------------------------------------------------------------+ | SFA14KX with GridScaler | +---------------------------------------------------------------+ Tested by DDN Storage Hardware Available 08 2018 Software Available 09 2018 Date Tested 09 2018 License Number 4722 Licensee Locations Santa Clara To address the comprehensive needs of High Performance Computing and Analytics environments, the revolutionary DDN SFA14KX Hybrid Storage Platform is the highest performance architecture in the industry that delivers up to 60GB/s of throughput, extreme IOPs at low latency with industry-leading density in a single 4U appliance. By integrating the latest high-performance technologies from silicon, to interconnect, memory and flash, along with DDN's SFAOS - a real-time storage engine designed for scalable performance, the SFA14KX outperforms everything on the market. Leveraging over a decade of leadership in the highest end of Big Data, the GRIDScaler Parallel File system solution running on the SFA14KX provides flexible choices for Enterprise-grade data protection, availability features, and the performance of a parallel file system coupled with DDN's deep expertise and history of supporting highly efficient, large-scale deployments. Solution Under Test Bill of Materials ===================================== Item No Qty Type Vendor Model/Name Description ---- ---- ---------- ---------- ---------- ----------------------------------- 1 1 Storage DDN SFA14KX Dual Intel(R) Xeon(R) CPU E5-2650 Appliance Storage (FC) v2 @ 2.60GHz 2 25 Network Mellanox ConnectX- Dual-port QSFP, EDR IB (100Gb/s) / Adapter VPI 100GigE, PCIe 3.0 x16 8GT/s (MCX4121A- XCAT) 3 6 GridScaler Supermicro SYS-1027R- Dual Intel(R) Xeon(R) CPU E5-2667 Server WC1RT v3 @ 3.20GHz, 96GB of memory per Server 4 19 GridScaler Supermicro SYS-1027R- Dual Intel Xeon(R) CPU E5-2650 v2 @ Clients WC1RT 2.60GHz, 128GB of memory per client 5 2 Switch Mellanox SB7700 36 port EDR Infiniband switch 6 72 Drives HGST SDLL1DLR40 HGST Hitachi Ultrastar SS200 400GB 0GCCA1 MLC SAS 12Gbps Mixed Use (SE) 2.5-inch Internal Solid State Drive (SSD) 7 50 Drives Toshiba AL14SEB030 AL14SEB-N Enterprise Performance N Boot HDD 8 6 FC Adapter QLogic QLogic QLogic 32Gb 2-port FC to PCIe Gen3 QLE2742 x8 Adapter Configuration Diagrams ====================== 1) sfs2014-20180924-00044.config1.png (see SPEC SFS2014 results webpage) Component Software ================== Item Name and No Component Type Version Description ---- ------------ ------------ ------------ ----------------------------------- 1 Server Nodes Distributed GRIDScaler Distributed file system software Filesystem 5.0.2 that runs on server nodes. 2 Client Nodes Distributed GRIDScaler Distributed file system software Filesystem 5.0.2 that runs on client nodes. 3 Client Nodes Operating RHEL 7.4 The Client operating system - System 64-bit Red Hat Enterprise Linux version 7.4. 4 Storage Storage SFAOS 11.1.0 SFAOS - real-time storage Operating Appliance Appliance System designed for scalable performance. Hardware Configuration and Tuning - Physical ============================================ +----------------------------------------------------------------------+ | GRIDScaler NSD Server configuration | +----------------------------------------------------------------------+ Parameter Name Value Description --------------- --------------- ---------------------------------------- nsdBufSpace 70 The parameter nsdBufSpace specifies the percent of pagepool which can be utilized for NSD IO buffers. nsdMaxWorkerThr 1536 The parameter nsdMaxWorkerThreads sets eads the maximum number of NSD threads on an NSD server that will be concurrently transferring data with NSD clients. pagepool 4g The Pagepool parameter determines the size of the file data cache. nsdSmallThreadR 3 The parameter nsdSmallThreadRatio atio determines the ratio of NSD server queues for small IO's (default less than 64KiB) to the number of NSD server queues that handle large IO's (> 64KiB). nsdThreadsPerQu 8 The parameter nsdThreadsPerQueue eue determines the number of threads assigned to process each NSD server IO queue. +----------------------------------------------------------------------+ | GRIDScaler common configuration | +----------------------------------------------------------------------+ Parameter Name Value Description --------------- --------------- ---------------------------------------- maxMBpS 30000 The maxMBpS option is an indicator of the maximum throughput in megabytes that can be submitted per second into or out of a single node. ignorePrefetchL yes Specifies that only maxMBpS and not the UNCount number of LUNs should be used to dynamically allocate prefetch threads. verbsRdma enable Enables the use of RDMA for data transfers. verbsRdmaSend yes Enables the use of verbs send/receive for data transfers. verbsPorts mlx5_0/1 8x Lists the Ports used for the communication between the nodes. workerThreads 1024 The workerThreads parameter controls an integrated group of variables that tune the file system performance in environments that are capable of high sequential and random read and write workloads and small file activity. maxReceiverThre 64 The maxReceiverThreads parameter is the ads number of threads used to handle incoming network packets. Hardware Configuration and Tuning Notes --------------------------------------- None Software Configuration and Tuning - Physical ============================================ +----------------------------------------------------------------------+ | GridScaler Clients | +----------------------------------------------------------------------+ Parameter Name Value Description --------------- --------------- ---------------------------------------- maxFilesToCache 4m The maxFilesToCache (MFTC) parameter controls how many file descriptors (inodes) each node can cache. maxStatCache 40m The maxStatCache parameter sets aside pageable memory to cache attributes of files that are not currently in the regular file cache. pagepool 16g The Pagepool parameter determines the size of the file data cache. openFileTimeout 86400 Determines the max amount of seconds we allow inode informations to stay in cache after the last open of the file before discarding them. maxActiveIalloc 8 Determines how many inode allocation Segs segments a individual client is allowed to select free inodes from in parallel. syncInterval 30 Specifies the interval (in seconds) in which data that has not been explicitly committed by the client is synced systemwide. prefetchAggress 1 Defines how aggressively to prefetch iveness data. Software Configuration and Tuning Notes --------------------------------------- Detailed description of the configuration and tuning options can be found here --> https://www.ibm.com/developerworks/community/wikis/home?lang=ja#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Tuning%20Parameters Service SLA Notes ----------------- None Storage and Filesystems ======================= Item Stable No Description Data Protection Storage Qty ---- ------------------------------------- ------------------ -------- ----- 1 72 SSDs in SFA14KX (FC) DCR 8+2p Yes 72 2 50 mirrored internal 300GB 10K SAS RAID-1 No 50 Boot drives Number of Filesystems 1 Total Capacity 20.3 TiB Filesystem Type GRIDScaler Filesystem Creation Notes ------------------------- A single filesystem with 1MB blocksize (no separate metadata disks) in scatter allocation mode was created. The filesystem inode limit was set to 1 .5 Billion. Storage and Filesystem Notes ---------------------------- The SFA14KX had 24 8+2p virtual disks with a 128KB strip size created out of a single 72 drive DCR storage pool. Transport Configuration - Physical ================================== Item Number of No Transport Type Ports Used Notes ---- --------------- ---------- ----------------------------------------------- 1 100Gb EDR 25 mlx5_0 2 16 Gb FC 24 FC0-3 Transport Configuration Notes ----------------------------- 2x 36-port switches in a single infiniband fabric, with 8 ISLs between them. Management traffic used IPoIB, data traffic used RDMA on the same physical adapter. Switches - Physical =================== Total Used Item Port Port No Switch Name Switch Type Count Count Notes ---- -------------------- --------------- ------ ----- ------------------------ 1 Client Mellanox 100Gb EDR 36 19 The default SB7700 configuration was used on the switch 2 Server Mellanox 100Gb EDR 36 6 The default SB7700 configuration was used on the switch Processing Elements - Physical ============================== Item No Qty Type Location Description Processing Function ---- ---- -------- -------------- ------------------------- ------------------- 1 2 CPU SFA14KX Dual Intel(R) Xeon(R) CPU Storage unit E5-2650 v2 @ 2.60GHz 2 19 CPU client nodes Dual Intel Xeon(R) CPU Filesystem client, E5-2650 v2 @ 2.60GHz load generator 3 6 CPU server nodes Dual Intel(R) Xeon(R) CPU GRIDScaler Server E5-2667 v3 @ 3.20GHz Processing Element Notes ------------------------ None Memory - Physical ================= Size in Number of Description GiB Instances Nonvolatile Total GiB ------------------------- ---------- ---------- ------------ ------------ Cache in SFA System 128 2 NV 256 GRIDScaler client node 128 19 V 2432 system memory GRIDScaler Server node 96 6 V 576 system memory Grand Total Memory Gibibytes 3264 Memory Notes ------------ The GRIDScaler clients use a portion of the memory (configured via pagepool and file cache parameter) to cache metadata and data. The GRIDScaler servers use a portion of the memory (configured via pagepool) for write Buffers. In the SFA14KX, some portion of memory is used for the SFAOS Operating system as well as data caching. Stable Storage ============== SFAOS with Declustered RAID performs rapid rebuilds, spreading the rebuild process across many drives. SFAOS also supports a range of features which improve uptime for large scale systems including partial rebuilds, enlosure redundancy, dual active-active controllers, online upgrades and more. The SFA14KX (FC) has built-in backup battery power support to allow destaging of cached data to persistent storage in case of a power outage. The system doesn't require further battery power after the destage process completed. All servers and the SFA14KX are redundantly configured. All 6 Servers have access to all data shared by the SFA14KX. In the event of loss of a server, that server's data will be failed over automatically to a remaining server with continued production service. Stable writes and commit operations in GRIDScaler are not acknowledged until the NSD server receives an acknowledgment of write completion from the underlying storage system (SFA14KX) Solution Under Test Configuration Notes ======================================= The solution under test used a GRIDSCaler Cluster optimized for small file, metadata intensive workloads. The Clients served as Filesystem clients as well as load generators for the benchmark. The Benchmark was executed from one of the server nodes. None of the component used to perform the test where patched with Spectre or Meltdown patches (CVE-2017-5754,CVE-2017-5753,CVE-2017-5715). Other Solution Notes ==================== None Dataflow ======== All 19 Clients where used to generate workload against a single Filesystem mountpoint (single namespace) accessible as a local mount on all clients. The GRIDScaler Server received the requests by the clients and processed the read or write operation against all connected DCR backed VD's in the SFA14KX. Other Notes =========== GRIDScaler are trademarks of DataDirect Network in the U.S. and/or other countries. Intel and Xeon are trademarks of the Intel Corporation in the U.S. and/or other countries. Mellanox is a registered trademark of Mellanox Ltd. Other Report Notes ================== None =============================================================================== Generated on Wed Mar 13 16:26:02 2019 by SpecReport Copyright (C) 2016-2019 Standard Performance Evaluation Corporation