History of InfiniBand vs Ethernet Performance
Version 32
InfiniBand vs Ethernet Performance
Blowing past 1GbE in your data center, Baremetal vs KVM virtio, NFS rdma vs default 40GbIB vs 40GbE vs 10GbE with SR-IOV and RDMA
NFS Export
[root@guest1 ~]# cat /etc/exports
/master1 192.168.72.0/22(rw,insecure,async,all_squash,no_root_squash) 172.16.0.0/22(rw,insecure,async,all_squash,no_root_squash)
[root@baremetal2 ~]# nfsstat -m
/mnt/56ib from 172.16.0.41:/master1
Flags: rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.16.0.14,local_lock=none,addr=172.16.0.41
Connection | 40Gb IB Mellanox/EMC SX6005 | 40Gb Eth Cisco NX3064 | 10Gb Eth Cisco NX3064 | |||||||
16K * 64KB | 16K * 64KB | 16K * 64KB | ||||||||
Server | Client | Write | Read | w/Cache | Write | Read | w/Cache | Write | Read | w/Cache |
---|---|---|---|---|---|---|---|---|---|---|
Baremetal | Baremetal | 3.2Gb/s 409MB/s | 9.8Gb/s 1227MB/s | 47.8Gb/s 5984MB/s | Gb/s MB/s | Gb/s MB/s | Gb/s MB/s | 5.6Gb/s 700MB/s | 8.69Gb/s 1086MB/s | 49.4Gb/s 6184MB/s |
KVM SR-IOV | Baremetal | 4.2Gb/s 529MB/s | 9.6Gb/s 1206MB/s | 47.2Gb/s 5910MB/s | Gb/s MB/s | Gb/s MB/s | Gb/s MB/s | 5.0Gb/s 631MB/s | 8.7Gb/s 1088MB/s | 46.4Gb/s 5802MB/s |
Gb/s MB/s | Gb/s MB/s | Gb/s MB/s | Gb/s MB/s | Gb/s MB/s | Gb/s MB/s | Gb/s MB/s | Gb/s MB/s | Gb/s MB/s | ||
Hardware
Server: Dell r730xd, Dual Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHzNetwork Adapter 40Gb: Mellanox ConnectX-3 CX354A MT27500 Family ConnectX-3
Network Adapter 10GbE: Mellanox ConnectX-2 =MT26448 ConnectX EN 10GigE, PCIe 2.0 5GT/s
Storage: 9x12TB Seagate Exos 12Gb/S SAS 7200 RPM, PERC H730P, mkfs.xfs -f -i size=512 -d su=64k,sw=8 /dev/sdb
[root@hyper0 ~]# omreport storage controller controller=0
Controller PERC H730P Mini(Embedded)
Controller
ID : 0
Status : Ok
Name : PERC H730P Mini
Slot ID : Embedded
State : Ready
Firmware Version : 25.5.3.0005
Minimum Required Firmware Version : Not Applicable
Driver Version : 07.701.17.00-rh1
Minimum Required Driver Version : Not Applicable
Storport Driver Version : Not Applicable
Minimum Required Storport Driver Version : Not Applicable
Number of Connectors : 1
Rebuild Rate : 30%
BGI Rate : 30%
Check Consistency Rate : 30%
Reconstruct Rate : 30%
Alarm State : Not Applicable
Cluster Mode : Not Applicable
SCSI Initiator ID : Not Applicable
Cache Memory Size : 2048 MB
Patrol Read Mode : Auto
Patrol Read State : Stopped
Patrol Read Rate : 30%
Patrol Read Iterations : 4
Abort Check Consistency on Error : Disabled
Allow Revertible Hot Spare and Replace Member : Enabled
Load Balance : Not Applicable
Auto Replace Member on Predictive Failure : Disabled
Redundant Path view : Not Applicable
CacheCade Capable : Not Applicable
Persistent Hot Spare : Disabled
Encryption Capable : Yes
Encryption Key Present : No
Encryption Mode : None
Preserved Cache : Not Applicable
Spin Down Unconfigured Drives : Disabled
Spin Down Hot Spares : Disabled
Spin Down Configured Drives : Disabled
Automatic Disk Power Saving (Idle C) : Disabled
Start Time (HH:MM) : Not Applicable
Time Interval for Spin Up (in Hours) : Not Applicable
T10 Protection Information Capable : No
Non-RAID HDD Disk Cache Policy : Unchanged
Current Controller Mode : RAID
Connectors
ID : 0
Status : Ok
Name : Connector 0
State : Ready
Connector Type : SAS Port RAID Mode
Termination : Not Applicable
SCSI Rate : Not Applicable
Virtual Disks
ID : 0
Status : Ok
Name : BOOT
State : Ready
Hot Spare Policy violated : Not Assigned
Encrypted : No
Layout : RAID-1
Size : 558.38 GB (599550590976 bytes)
T10 Protection Information Status : No
Associated Fluid Cache State : Not Applicable
Device Name : /dev/sda
Bus Protocol : SAS
Media : HDD
Read Policy : Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Unchanged
ID : 1
Status : Ok
Name : STORAGE
State : Ready
Hot Spare Policy violated : Not Assigned
Encrypted : No
Layout : RAID-5
Size : 89,404.00 GB (95996814032896 bytes)
T10 Protection Information Status : No
Associated Fluid Cache State : Not Applicable
Device Name : /dev/sdb
Bus Protocol : SAS
Media : HDD
Read Policy : Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Unchanged
Physical Disks
ID : 0:1:0
Status : Non-Critical
Name : Physical Disk 0:1:0
State : Online
Power Status : Spun Up
Bus Protocol : SAS
Media : HDD
Part of Cache Pool : Not Applicable
Remaining Rated Write Endurance : Not Applicable
Failure Predicted : No
Revision : E001
Driver Version : Not Applicable
Model Number : Not Applicable
T10 PI Capable : No
Certified : No
Encryption Capable : No
Encrypted : Not Applicable
Progress : Not Applicable
Mirror Set ID : Not Applicable
Capacity : 11,175.50 GB (11999601754112 bytes)
Used RAID Disk Space : 11,175.50 GB (11999601754112 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare : No
Vendor ID : SEAGATE
Product ID : ST12000NM0027
Serial No. : ZCH06GA7
Part Number : Not Available
Negotiated Speed : 12.00 Gbps
Capable Speed : 12.00 Gbps
PCIe Negotiated Link Width : Not Applicable
PCIe Maximum Link Width : Not Applicable
Sector Size : 512B
Device Write Cache : Not Applicable
Manufacture Day : Not Available
Manufacture Week : Not Available
Manufacture Year : Not Available
SAS Address : 5000C5009497687D
Non-RAID HDD Disk Cache Policy : Not Applicable
Disk Cache Policy : Not Applicable
Sub Vendor : Not Available
Available Spare : Not Available
ISE Capable : No