SPEC SFS(R)2014_swbuild Result E8 Storage : E8 Storage X24 with IBM Spectrum Scale 5.0.1.1 SPEC SFS2014_swbuild = 1200 Builds (Overall Response Time = 0.57 msec) =============================================================================== Performance =========== Business Average Metric Latency Builds Builds (Builds) (msec) Ops/Sec MB/Sec ------------ ------------ ------------ ------------ 120 0.1 60002 831 240 0.1 120004 1664 360 0.1 180007 2497 480 0.1 240009 3328 600 0.2 300012 4160 720 0.4 360014 4993 840 0.6 420016 5824 960 1.0 479887 6654 1080 1.5 539767 7486 1200 2.4 599881 8318 =============================================================================== Product and Test Information ============================ +---------------------------------------------------------------+ | E8 Storage X24 with IBM Spectrum Scale 5.0.1.1 | +---------------------------------------------------------------+ Tested by E8 Storage Hardware Available April 2018 Software Available June 2018 Date Tested June 2018 License Number 4847 Licensee Locations Santa Clara, CA, USA E8 Storage is a pioneer in shared accelerated storage for data-intensive, high-performance applications that drive business revenue. E8 Storage's affordable, reliable and scalable solution is ideally suited for the most demanding low-latency workloads, including AI, genomics, real-time analytics, financial and trading applications, and large-scale file systems. Driven by the company's patented architecture, E8 Storage's high-performance shared NVMe storage solution delivers 10 times the performance at half the cost of existing storage products. With E8 Storage, enterprise data centers can enjoy unprecedented storage performance density and scale, delivering NVMe performance without compromising on reliability and availability. IBM Spectrum Scale helps solve the challenge of explosive growth of unstructured data against a flat IT budget. Spectrum Scale provides unified file and object software-defined storage for high performance, large scale workloads on-premises or in the cloud. Spectrum Scale includes the protocols, services and performance required by many industries, Technical Computing, Big Data, HDFS and business critical content repositories. IBM Spectrum Scale provides world-class storage management with extreme scalability, flash accelerated performance, and automatic policy-based storage tiering from flash through disk to tape, reducing storage costs up to 90% while improving security and management efficiency in cloud, big data & analytics environments. Solution Under Test Bill of Materials ===================================== Item No Qty Type Vendor Model/Name Description ---- ---- ---------- ---------- ---------- ----------------------------------- 1 15 Spectrum IBM X3650-M4 Spectrum Scale 5.0.1.1 client nodes Scale Client 2 15 Spectrum Gigabyte MH61-HD4 Spectrum Scale 5.0.1.1 client nodes Scale Client 3 30 Network Mellanox ConnectX-5 Dual port 100GbE Adapters used in Interface VPI each Spectrum Scale Server Card (MCX556A- ECAT) 4 1 Storage E8 Storage E8-X24 Dual controller storage appliance Appliance with 24 Intel Optane D4800X 750GB NVMe SSDs, 2 x Intel Xeon 2.0GHz 14-core CPU and 128GB RAM per controller. 2 x Mellanox ConnectX-4 EN network interface cards are installed per controller 5 1 Switch Mellanox MSN2700-CS 32-port 100GbE switch 2F 6 1 Switch Juniper EX4200-48T 48-port 1GbE switch Series 8PoE Configuration Diagrams ====================== 1) sfs2014-20180706-00041.config1.png (see SPEC SFS2014 results webpage) Component Software ================== Item Name and No Component Type Version Description ---- ------------ ------------ ------------ ----------------------------------- 1 Client Nodes Spectrum 5.0.1.1 The Spectrum Scale File System is a Scale File distributed file system that runs System on all client nodes to form a cluster. The cluster allows for the creation and management of single namespace file systems. 2 Client Nodes Operating RHEL 7.5 The operating system on the client System nodes was 64-bit Red Hat Enterprise Linux version 7.5 3 Client Nodes E8 Storage 2.1.2 The E8 Storage Agent is a client Agent driver which manages communication and data transfer between the client and storage appliance 4 Storage E8 Storage 2.1.2 E8 Storage software provides Appliance Software centralized management and high availability functionality for the E8 Storage solution 5 Storage Operating RHEL 7.4 The operating system on the storage Appliance System appliance was 64-bit Red Hat Enterprise Linux version 7.4. Hardware Configuration and Tuning - Physical ============================================ +----------------------------------------------------------------------+ | Spectrum Scale Client Nodes | +----------------------------------------------------------------------+ Parameter Name Value Description --------------- --------------- ---------------------------------------- numaMemoryInter yes Enables memory interleaving on NUMA leave based systems. verbsRdma enable Enables Ethernet RDMA transfers between Spectrum Scale client nodes and E8 Storage controllers verbsRdmaSend yes Enables the use of Ethernet RDMA for most Spectrum Scale daemon-to-daemon communication. verbsPorts mlx5_1/1/1 Ethernet device names and port numbers. txqueuelen 10000 Defines the transmission queue length for the Mellanox adapter Hardware Configuration and Tuning Notes --------------------------------------- The Spectrum Scale configuration parameter was set using the mmchconfig command on one of the nodes in the cluster. The verbs settings in the table above allow for efficient use of the RoCE infrastructure. The settings determine when data are transferred over IP and when they are transferred using the verbs protocol. Software Configuration and Tuning - Physical ============================================ +----------------------------------------------------------------------+ | Spectrum Scale Client Nodes | +----------------------------------------------------------------------+ Parameter Name Value Description --------------- --------------- ---------------------------------------- maxStatCache 0 Specifies the number of inodes to keep in the stat cache. workerThreads 256 Controls the maximum number of concurrent file operations at any one instant, as well as the degree of concurrency for flushing dirty data and metadata in the background and for prefetching data and metadata. maxMBpS 16k Specifies an estimate of how many megabytes of data can be transferred per second into or out of a single node. pagepool 6g Specifies the size of the cache on each node. maxFilesToCache 10m Specifies the number of inodes to cache for recently used files that have been closed. ignorePrefetchL yes Specifies that only maxMBpS and not the UNCount number of LUNs should be used to dynamically allocate prefetch threads. prefetchAggress 1 Defines how aggressively to prefetch iveness data. 1 means prefetch on 2nd access if sequential. prefetchPct 5 Specifies what percent of the pagepool (cache) can be used for prefetching. syncInterval 30 Specifies the interval (in seconds) in which data that has not been explicitly committed by the client is synced systemwide. maxActiveIalloc 8 Determines how many inode allocation Segs segments a client is allowed to select in parallel. openFileTimeout 86400 Determines the max amount of seconds inode information is allowed to stay in cache after the last file open. +----------------------------------------------------------------------+ | E8 Storage Agent | +----------------------------------------------------------------------+ Parameter Name Value Description --------------- --------------- ---------------------------------------- e8block threads 1 Specifies the number of threads used by the e8block process on the host servers Software Configuration and Tuning Notes --------------------------------------- The configuration parameters for the Spectrum Scale file system were set using the mmchconfig command on one of the nodes in the cluster. The nodes used mostly default tuning parameters. A discussion of Spectrum Scale tuning can be found in the official documentation for the mmchconfig command and on the IBM developerWorks wiki at https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/fa32927c-e904-49cc-a4cc-870bcc8e307c/page/Tuning%20Parameters (for additional information on tuning parameters see http://files.gpfsug.org/presentations/2014/UG10_GPFS_Performance_Session_v10.pdf). The E8 Storage controller used default parameters for all volumes. Service SLA Notes ----------------- There were no opaque services in use Storage and Filesystems ======================= Item Stable No Description Data Protection Storage Qty ---- ------------------------------------- ------------------ -------- ----- 1 24 x 750GB NVMe SSDs in the E8-X24 RAID-6 Yes 1 2 2 x 300GB 10K SAS HDD internal drives RAID-1 No 30 per Spectrum Scale client node (15 IBM servers) used to store the OS 3 1 x 240GB SATA SSD internal drive per NA No 15 Spectrum Scale client node (15 Gigabyte servers) used to store the OS Number of Filesystems 1 Total Capacity 14TB Filesystem Type Spectrum Scale File System Filesystem Creation Notes ------------------------- A single Spectrum Scale file system was created with a 1 MiB block size for data and metadata, and 4 KiB inode size. The 14TB file system has 1 mixed data and metadata volume (aka pool). Each client node mounted the file system. The nodes each had an ext4 file system that hosted the operating system. Storage and Filesystem Notes ---------------------------- The E8 Storage appliance has 24 750GB Intel Optane drives configured as a 22+2 RAID group for data protection, with a 16+2 stripe size. The mixed data / metadata volume was provisioned from this single RAID group, with the volume spanning all drives in the RAID group. All client nodes had shared read / write access to the single volume. The cluster used a single-tier architecture. The Spectrum Scale client nodes performed both file and block level operations. Each node had access to shared volumes, so any file operation on a node was translated to a block operation and serviced on the same node. Transport Configuration - Physical ================================== Item Number of No Transport Type Ports Used Notes ---- --------------- ---------- ----------------------------------------------- 1 1 GbE admin 32 Each Spectrum Scale node and E8 Storage network controller connects to a 1 GbE administration network with MTU=1500. 2 100 GbE cluster 27 15 client nodes have a single port connected to network the switch via a 50GbE split cable (8 switch ports), 15 client nodes have 1 x 100GbE port connected to the switch (15 switch ports), and each E8 Storage controller has 4 100GbE ports connected to the switch (4 ports). It is a shared 100GbE cluster network, set to MTU=4200. The ring buffers (rx and tx) were set to 8192 on the network adapters Transport Configuration Notes ----------------------------- The 1GbE network was used for administrative purposes and for Spectrum Scale inter-node communication. All benchmark traffic flowed through the Mellanox SN2700 100Gb Ethernet switch. 15 client nodes had a single active 50Gb Ethernet port which was connected to the switch with a split cable (2 50GbE clients per 100GbE switch port), with the remaining 15 clients direct connect at 100Gb Ethernet to the switch. Switches - Physical =================== Total Used Item Port Port No Switch Name Switch Type Count Count Notes ---- -------------------- --------------- ------ ----- ------------------------ 1 Mellanox SN2700 100Gb Ethernet 32 27 The default configuration was used on the switch 2 Juniper EX4200 1Gb Ethernet 48 32 Administrative network Series 8PoE only. The default configuration was used on the switch. Processing Elements - Physical ============================== Item No Qty Type Location Description Processing Function ---- ---- -------- -------------- ------------------------- ------------------- 1 30 CPU Spectrum Scale Intel(R) Xeon(R) CPU Spectrum Scale client nodes E5-2630 v2 2.60GHz 6-core client, E8 Agent, (IBM) load generator, device drivers 2 30 CPU Spectrum Scale Intel(R) Xeon(R) Gold Spectrum Scale client nodes 6142 CPU 2.60GHz 16-core client, E8 Agent, (Gigabyte) load generator, device drivers 3 4 CPU E8-X24 Intel(R) Xeon(R) CPU E8 Storage server, controller E5-2660 v4 2.00GHz E8 Storage RAID, 14-core device drivers Processing Element Notes ------------------------ Each of the Spectrum Scale client nodes had 2 physical processors. On 15 clients, each processor had 6 physical cores with one thread per core by default. On the other 15 clients, each processor had 16 physical cores with one thread per core by default. The client nodes were running the following kernel, which contains some but not all of the Spectre/Meltdown patches, and no other patches were applied: 3.10.0-862.el7.x86_64 The E8-X24 is a dual controller appliance, each controller has 2 physical processors. Each processor had 14 cores with one thread per core. All available Spectre/Meltdown patches have been applied to the controller CPUs. (more info here: https://access.redhat.com/security/vulnerabilities/speculativeexecution) Memory - Physical ================= Size in Number of Description GiB Instances Nonvolatile Total GiB ------------------------- ---------- ---------- ------------ ------------ Spectrum Scale client 128 30 V 3840 node system memory E8 Storage controller 16 16 V 256 system memory Grand Total Memory Gibibytes 4096 Memory Notes ------------ In the client nodes, Spectrum Scale reserves a portion of the physical memory (per the pagepool setting above) for file data and metadata caching. Some additional memory is dynamically allocated for buffers used for node to node communication and up to 10 million inode stat informations per node. In the E8 Storage controller, a portion of the physical memory is reserved for block write data and system metadata caching. Stable Storage ============== The E8 Storage controller uses a portion of internal memory to temporarily cache write data (as well as store modified data) before being written to the SSDs. Writes are acknowledged as successful once they are stored in the controller write cache, and a redundant copy is kept by the E8 Agent on the host. In the event of a controller failure, the hosts will replay the write cache for the surviving controller. In the event of a power failure, each controller has backup battery power which is combined with power-fail protection on the SSDs to ensure data is committed to SSDs prior to shutdown. Solution Under Test Configuration Notes ======================================= The solution under test was a Spectrum Scale cluster optimized for small file, metadata intensive environments. The Spectrum Scale nodes were also the load generators for the benchmark. The benchmark was executed from one of the nodes. Other Solution Notes ==================== None Dataflow ======== The 30 Spectrum Scale nodes were the load generators for the benchmark. Each load generator had access to the single namespace Spectrum Scale file system. The benchmark accessed a single mount point on each load generator. In turn each of mount points corresponded to a single shared base directory in the file system. The nodes process the file operations, and the data requests to and from the backend storage were serviced locally on each node by the E8 Client Agent. Other Notes =========== IBM and IBM Spectrum Scale are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Intel and Xeon are trademarks of the Intel Corporation in the U.S. and/or other countries. Mellanox is a registered trademark of Mellanox Ltd. Other Report Notes ================== None =============================================================================== Generated on Wed Mar 13 16:59:07 2019 by SpecReport Copyright (C) 2016-2019 Standard Performance Evaluation Corporation