Connecting It Together in IBM Spectrum Fusion HCI

 

In my previous post, I wrote about the different types of servers that are available for use in the IBM Spectrum Fusion HCI product.  There are storage/compute servers and compute-only servers that make up the bulk of the system, with optional servers available to add GPU capability and to add storage networking capability.  In this post, I will take a look at the network switches that connect all these servers together and also provide the uplink ports for connecting the Spectrum Fusion HCI system to the data center network and beyond.

You might be wondering why network switches are even included in Spectrum Fusion HCI. Wouldn't it just be easier to connect all the servers to network switches that are already available in the data center?  Easier, yes, but that shortcut would certainly weaken the value of the offering.  Having integrated, redundant network switches makes the product easier to deploy, makes the system more resilient to failure, and provides dedicated high-speed connections between the servers to guarantee the performance and integrity of storage access.

There are two types of switches in Spectrum Fusion HCI: high-speed network switches and management network switches.  The high-speed switches provide the connections for application and storage network traffic.  The management network switches provide the connections for accessing the BMC (baseboard management controller) through dedicated connections network ports on each server.

The Need for Speed

To provide for resiliency and maintenance with minimal disruption, there are two identical NVIDIA SN3700C high-speed Ethernet switches in the system.  The switches are configured as an active-active pair, both operating at the same time and acting as each other's backup.  Each switch has 32x QSFP28 100Gb Ethernet ports.  These ports are used at their maximum speed to create an internal storage network that allows storage traffic to move very quickly between the nodes.  Each server is connected to two of these 100GbE ports, one on each switch.  This provides redundancy and up to 200Gbps of bandwidth.

In addition, several of the 100GbE ports are split to create 4x 25GbE ports.  Each server in the system is connected to two of these 25GbE ports, one on each switch.  These connections are used to create an internal network dedicated to application traffic, separate from the internal storage network.  In total, each server has four network connections to the high-speed switches providing up to 250Gbps of bandwidth.

Two 100GbE ports on each switch are reserved to provide uplink support from the system to the data center network for a maximum total of 400Gbps.  However, the configuration of these ports is flexible and, with the use of adapters, can accommodate connections to data center switch ports of type 1, 10, 25, 40, 50, and 100GbE.

Independent Oversight

Independent of the high-speed switches, there is a pair of NVIDIA AS4610 1GBASE-T Ethernet management switches integrated into Spectrum Fusion HCI. The switches are also configured as an active-active pair, just like the high-speed switches.  Each of these management switches has 48x 1GbE ports, which provides more than enough ports for each server in the system to connect both its management port and its alternate management port to the network.  These connections don't require high-speed connections so this switch is a good match for what is needed without adding unnecessary cost or complexity.

The management switches are not connected directly to the data center network.  Instead, the uplink ports of the management switches connect only to the high-speed switches.  In this way, the Spectrum Fusion HCI management software is able to control which users are able to execute commands using the BMC, and what commands are even allowed for users to execute.

In my next post I will dig into the details of the components in the storage/compute servers used in Spectrum Fusion HCI.  Meanwhile, if you would like review the different servers that can be used in the system, you check out my previous blog post, "The Servers that Power Spectrum Fusion HCI."

Next post: Inside the Storage/Compute Servers of IBM Spectrum Fusion HCI

Photo of network switches and cables by Paul Kauffmann
Photo of jet by Jonathan Ridley on Unsplash

The opinions expressed in this post are those of the author.

Comments

Post a Comment

Popular posts from this blog

Inside the Storage/Compute Servers of IBM Spectrum Fusion HCI

Image
Previously, I provided an overview of the IBM Spectrum Fusion HCI hardware followed by posts giving an overview of the different server and switch components that go into that system. Most of the servers in an IBM Spectrum Fusion HCI system will be storage/compute servers, so you can think of these servers as the workhorses of the system. In this post, I take a closer look at the CPU, memory, storage, and network elements inside these storage/compute servers. CPU and Memory A storage/compute server is built from a Lenovo SR645 server that is one rack unit (1U) high.  (A rack unit is an industry standard that translates to 1.75 inches; an industry standard data center rack used to hold servers and switches is 42U tall.)  Inside the server are two processor sockets that are each populated with an AMD EPYC 7302 processor.  The EPYC 7302 is a 16-core, 32-thread, processor that operates with a base clock speed of 3.0 GHz.  With two of these processors, the server has a total o
Read more

Fusion HCI Adds 8x the GPU Power

Image
IBM Storage Fusion HCI System has released a new GPU server today that takes a huge step forward in the support of AI applications.  The new server, called the G03, can be configured to have as many as 8 double-wide GPU adapter cards .   Fine-tuning and inferencing with foundation models used in AI applications increasingly requires faster and more powerful hardware, and the Fusion HCI G03 meets that need head on! The tremendous boost provided by the G03 for AI apps comes from having up to 8 of the  NVIDIA L40S  GPU PCIe double-wide adapter cards inside a single server.  The L40S is powered by the NVIDIA Ada Lovelace Architecture and is designed for generative AI and LLM training and inferencing.  It contains fourth generation tensor cores and the latest transformer engine that puts those tensor cores to good use. The design of the G03 provides flexibility for deploying GPU resources.  Many organizations will order the G03 with a full complement of 8 L40S GPUs.  But for organizations g
Read more

Fusion HCI Performance Boost for AI Apps

Image
Fine-tuning and inferencing with foundation models used in AI applications increasingly requires faster and more powerful hardware as AI foundation models grow ever larger.  In response to this need, IBM Storage Fusion HCI System released a new GPU server option earlier this month called the G02 that gives a performance boost for AI applications. The new G02 GPU server is based on the Lenovo SR650 V3.  The configuration used in the G02 includes a pair of Intel Xeon Gold 6418H 24-core "Sapphire Rapids" processors.  This processor model includes Intel Deep Learning Boost, designed to accelerate AI use cases by extending the Intel AVX-512 instruction extension with a new instruction that increases deep learning inference performance. But the big boost provided by the G02 for AI apps comes from the three  NVIDIA A100 80GB GPU PCIe adapter cards in each G02 server.  This newer version of the A100 has u p to 1.25x higher AI inference performance over the A100 40GB  that was used i
Read more