SPEC SFS®2014_vda Result

Copyright © 2016-2019 Standard Performance Evaluation Corporation

DDN Storage SPEC SFS2014_vda = 3400 Streams
SFA14KXE with EXAScaler Overall Response Time = 50.07 msec


Performance Graph

Product and Test Information

SFA14KXE with EXAScaler
Tested byDDN Storage
Hardware Available08 2018
Software Available09 2018
Date Tested09 2018
License Number4722
Licensee LocationsSanta Clara

To address the comprehensive needs of High Performance Computing and Analytics environments, the revolutionary DDN SFA14KXE Hybrid Storage Platform is the highest performance architecture in the industry that delivers up to 60GB/s of throughput, extreme IOPs at low latency with industry-leading density in a single 4U appliance. By integrating the latest high-performance technologies from silicon, to interconnect, memory and flash, along with DDN's SFAOS - a real-time storage engine designed for scalable performance, the SFA14KXE outperforms everything on the market. Leveraging over a decade of leadership in the highest end of Big Data, the EXAScaler Lustre Parallel File system solution running on the SFA14KXE provides flexible choices for highest demanding performance of a parallel file system coupled with DDN's deep expertise and history of supporting highly efficient, large-scale deployments.

Solution Under Test Bill of Materials

Item NoQtyTypeVendorModel/NameDescription
11Storage ApplianceDDN StorageSFA14KXE2x Dual Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
217Network AdapterMellanoxConnectX- VPI (MCX4121A-XCAT)Dual-port QSFP, EDR IB (100Gb/s) / 100GigE, PCIe 3.0 x16 8GT/s
31Lustre MDS/MGS ServerSupermicroSYS-1027R-WC1RTDual Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 96GB of memory per Server
416EXAScaler ClientsSupermicroSYS-1027R-WC1RTDual Intel Xeon(R) CPU E5-2650 v2 @ 2.60GHz, 128GB of memory per client
52SwitchMellanoxSB770036 port EDR Infiniband switch
6420DrivesHGSTHUH721010AL4200HGST Ultrastar He10 HUH721010AL4200 - hard drive - 10 TB - SAS 12Gbs
71EnclosureDDNEF4024External FC connected HDD JBOD enclosure
84DrivesHGSTHUC156030CSS200HGST Ultrastar C15K600 HUC156030CSS200 - hard drive - 300 GB - SAS 12Gb/s
934DrivesToshibaAL14SEB030NAL14SEB-N Enterprise Performance Boot HDD
101FC AdapterQLogicQLogic QLE2742QLogic 32Gb 2-port FC to PCIe Gen3 x8 Adapter

Configuration Diagrams

  1. Configuration Diagram

Component Software

Item NoComponentTypeName and VersionDescription
1MDS/MGS Server NodesDistributed FilesystemES 4.0.0, lustre-2.10.4_ddn4Distributed file system software that runs on MDS/MGS node.
2OSS Server NodesDistributed FilesystemES 4.0.0, lustre-2.10.4_ddn4Distributed file system software that runs on the Virtual OSS VM's.
3Client NodesDistributed Filesystem lustre-client-2.10.4_ddn4-1.el7.centos.x86_64Distributed file system software that runs on client nodes.
4Client NodesOperating SystemRHEL 7.4The Client operating system - 64-bit Red Hat Enterprise Linux version 7.4.
5Storage ApplianceStorage ApplianceSFAOS 11.1.0SFAOS - real-time storage Operating System designed for scalable performance.

Hardware Configuration and Tuning - Physical

EXAScaler common configuration
Parameter NameValueDescription
lctl set_param osc.*.max_pages_per_rpc16Mmaximum number of pages per RPC
lctl set_param osc.*.max_dirty_mb1024maximum amount of outstanding dirty data that wasn't synced by the application
lctl set_param llite.*.max_read_ahead_mb2048maximum amount of cache reserved for readahead cache
lctl set_param osc.*.max_rpcs_in_flight16maximum number of parallel RPC's in flight
lctl set_param llite.*.max_read_ahead_per_file_mb1024max readahead buffer per file
lctl set_param osc.*.checksums0disable data checksums

Hardware Configuration and Tuning Notes

please check the following pages for detaild documentation on supported parameters : for osc.* parameters --> https://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#TuningClientIORPCStream for llite.* parameters --> https://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#TuningClientReadahead

Software Configuration and Tuning - Physical

EXAScaler Clients
Parameter NameValueDescription
processor.max_cstate0Defines maximum allowed c-state of CPU.
intel_idle.max_cstate0Defines maximum allowed c-state of CPU in idle mode.
idlepollset idle mode to polling for maximum performance.

Software Configuration and Tuning Notes

no special tuning was applied beyond what was mentioned above

Service SLA Notes


Storage and Filesystems

Item NoDescriptionData ProtectionStable StorageQty
1420 HDDs in SFA14KXEDCR 8+2pYes420
234 mirrored internal 300GB 10K SAS Boot drivesRAID-1No34
32 mirrored 300GB 15K SAS drives MGS RAID-1yes2
42 mirrored 300GB 15K SAS drives MDT RAID-1yes2
Number of Filesystems1
Total Capacity2863.4 TiB
Filesystem TypeLustre

Filesystem Creation Notes

A single filesystem with all MDT and OST disks was created, no additional settings where applied.

Storage and Filesystem Notes

The SFA14KXE has 8x virtual disks with a 128KB stripe size and 8+2P RAID protection created from 8 DCR storage pools in 51/1 configuration (51 drives per pool with 1 drive worth of spare space).

Transport Configuration - Physical

Item NoTransport TypeNumber of Ports UsedNotes
1100Gb EDR16client IB ports
2100Gb EDR4SFA IB ports
3100Gb EDR1MDS IB port
4100Gb EDR8ISL between switches
516 Gb FC2direct attached FC to JBOF

Transport Configuration Notes

2x 36-port switches in a single InfiniBand fabric, with 8 ISLs between them. Management traffic used IPoIB, data traffic used IB Verbs on the same physical adapter.

Switches - Physical

Item NoSwitch NameSwitch TypeTotal Port CountUsed Port CountNotes
1Client/SFA Mellanox SB7700100Gb EDR368The default configuration was used on the switch
2MDS Mellanox SB7700100Gb EDR3627The default configuration was used on the switch

Processing Elements - Physical

Item NoQtyTypeLocationDescriptionProcessing Function
12CPUSFA14KXEDual Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHzStorage unit
216CPUclient nodesDual Intel Xeon(R) CPU E5-2650 v2 @ 2.60GHzFilesystem client, load generator
31CPUServer nodesDual Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHzEXAScaler Server

Processing Element Notes


Processing Elements - Virtual

Item NoQtyTypeLocationDescriptionProcessing Function
14CPUServer nodes8 Virtual Cores from SFA ControllerEXAScaler Server

Processing Element Notes


Memory - Physical

DescriptionSize in GiBNumber of InstancesNonvolatileTotal GiB
Memory in SFA System divided for OS and cache762NV152
Memory in SFA System for VM Memory904V360
EXAScaler client node system memory6416V1024
EXAScaler MDS/MGS Server node system memory961V96
Grand Total Memory Gibibytes1632

Memory Notes

The EXAScaler filesystem utilizes local filesystem cache/memory for its caching mechanism for both clients and OST's. All resources in the system and local filesystem are available for use by Lustre. In the SFA14KXE, some portion of memory is used for the SFAOS Operating system as well as data caching.

Memory - Virtual

DescriptionSize in GiBNumber of InstancesNonvolatileTotal GiB
Memory assigned to ach OSS VM within the SFA Controller904V360
Grand Total Memory Gibibytes360

Memory Notes

Each of the EXAScaler OSS VM's has 90 GB of memory assigned for OS and caching.

Stable Storage

SFAOS with Declustered RAID performs rapid rebuilds, spreading the rebuild process across many drives. SFAOS also supports a range of features which improve uptime for large scale systems including partial rebuilds, enlosure redundancy, dual active-active controllers, online upgrades and more. The SFA14KXE has built-in backup battery power support to allow destaging of cached data to persistent storage in case of a power outage. The system doesn't require further battery power after the destage process completed. All OSS servers and the SFA14KXE are redundantly configured. All 4 Servers have access to all data shared by the SFA14KXE. In the event of loss of a server, that server's data will be failed over automatically to a remaining server with continued production service. Stable writes and commit operations in EXAScaler are not acknowledged until the OSS server receives an acknowledgment of write completion from the underlying storage system (SFA14KXE)

Solution Under Test Configuration Notes

The solution under test used a EXAScaler Cluster optimized for large file, sequential streaming workloads. The Clients served as Filesystem clients as well as load generators for the benchmark. The Benchmark was executed from one of the server nodes. None of the component used to perform the test where patched with Spectre or Meltdown patches (CVE-2017-5754,CVE-2017-5753,CVE-2017-5715).

Other Solution Notes



All 16 Clients where used to generate workload against a single Filesystem mountpoint (single namespace) accessible as a local mount on all clients. The EXAScaler Server received the requests by the clients and processed the read or write operation against all connected DCR backed VD's in the SFA14KXE.

Other Notes

EXAScaler is a trademark of DataDirect Network in the U.S. and/or other countries. Intel and Xeon are trademarks of the Intel Corporation in the U.S. and/or other countries. Mellanox is a registered trademark of Mellanox Ltd.

Other Report Notes


Generated on Wed Mar 13 16:26:24 2019 by SpecReport
Copyright © 2016-2019 Standard Performance Evaluation Corporation