X86 Module Development
An Intel Skylake x86 Processor Module is available for installation into one of the internal module bays within the Cisco Nexus 3550-F Fusion (formerly ExaLINK Fusion). This processor is available for you to run your own applications on, allowing construction and deployment of any number of custom appliances. The CPU is not required by Cisco for running or management of the rest of the Nexus 3550-F, so you are not virtualized or sandboxed in any way - you have complete control over the processor.
The specifications of the processor module are as follows:
- Intel Core-i7 Skylake 6820EQ CPU, 4 cores @ 2.8GHz, 3.5GHz turbo, QM170 chipset
- 32GB DDR4-2133MT/s (PC4-17000)
- Dual mSATA SSDs, typically shipped with Samsung 850 EVO 250GB as RAID1 for operating system
- M.2 PCIe NVMe SSD, typically shipped with Samsung 950 PRO 512GB capable of sustained write at 1,500MByte/s
- Onboard low latency Cisco Nexus SmartNIC K35-Q (formerly ExaNIC X40), providing 8x10Gbe (2x40G via firmware update) interfaces which can be connected to other Nexus 3550-F objects (mux, switch, front panel ports etc)
- Onboard 1Gb LAN, typically patched through to front panel
- Trusted Platform Module (TPM)
- Intel AMT allows remote KVM access, including operating system install and BIOS config
- Multiple serial ports available
- Multiple FPGA development options available
- Access to PPS from front panel or Nexus 3550-F GPS receiver for highly accurate time synchronization
Top and bottom view of x86 Processor Module, showing drive bays
This module consists of a carrier board designed by Cisco, and a 3rd party x86 processor board in a COM Express form factor. This design allows for potential upgrading of the x86 processor board to newer microarchitectures - for example Intel's Kaby Lake or Cannonlake, without changing the underlying carrier board.
The 3rd party processor board currently shipping is the SH960 from DFI.
As can be seen below the carrier board features the SmartNIC K35-Q (X40), connectors for mounting mSATA's and the M.2 drive, along with all the required power supply, support and management hardware.
Block diagram of x86 Processor Module
x86 Processor Module consists of a COM Express module installed on Cisco's carrier board
x86 Processor Module installed into Nexus 3550-F
System Power Management
The Nexus 3550-F CLI is used to enable/disable power to the processor module. Assuming the processor is installed into module Y, the module can be powered up/down using the following commands:
admin@N3550-F> configure module Y power on Module Y powered on admin@N3550-F> configure module Y power off Module Y powered off
This is a hard power off, ie the equivalent of removing all power feeds to a server.
When operating, the power consumption of the processor module can be displayed using the
show power command as follows:
admin@N3550-F> show power Voltage Current ------- ------- Line card A 12.3 V 0.3 A Line card B 12.3 V 0.3 A Line card C 12.2 V 0.5 A Module X 12.3 V 0.9 A Module Y 12.2 V 2.7 A
The temperature of the processor module can be read using the
show temperature command, as follows:
admin@N3550-F> sh temp Temperature ------------------------------ Crosspoint 36.9 C 36.9 C 34.3 C 34.3 C Line card A 34.0 C Line card B 35.4 C Line card C 35.8 C Mainboard 30.9 C 34.9 C Module X 34.0 C 33.0 C Module Y 52.0 C 51.0 C
The first temperature listed above for module Y is the carrier board temperature. The second is the temperature of the SmartNIC K35-Q (X40) FPGA. There is no way to directly read the CPU temperature from the Nexus 3550-F CLI, however the CPU and FPGA are thermally coupled by the heatsink. The CPU temperature can be read by logging into the processor module itself and running an application such as lm_senors, i7z etc.
Installed Operating System
The default configuration ships with Linux CentOS 7 installed onto a RAID1 partition created from the dual mSATA drives. The M.2 NVMe drive is mounted at /data. The login for this system as it ships is:
username: root password: exablaze
Primary Serial Port
There are 2 serial ports provided on the x86 processor module, both of which are connected through to the Nexus 3550-F management processor. One of these can be used to gain console access to the x86 processor module. This is done by enabling a
serial-server on the Nexus 3550-F and connecting to it remotely via the Nexus 3550-F management port. Using this feature allows you to access the x86 processor module for management purposes without sacrificing a high speed front panel port
As can be seen in the diagram above, the remote host connects to a serial server running on the Nexus 3550-F management processor. Traffic received/transmitted on this connection is routed to the x86 processor module's serial port, where a getty session would typically be running, enabling console access to the system. Commands to configure this would be as follows:
admin@N3550-F> configure module Y serial-server port 1444 Serial server enabled for module Y on TCP port 1444
No authentication or other security is provided on this connection (other than the Access Control List for the whole management interface)
Beginning with Nexus 3550-F Fusion Software v1.16.0, you can restrict access to the connection
to only the localhost by using the
To restrict connection access only to the localhost, use the following command:
admin@N3550-F> configure module x serial-server port 9999 localhost-only Serial server enabled for module X on TCP port 9999 (localhost only)
Once the serial server is enabled on the Nexus 3550-F, various applications can be used to pass traffic to/from the connection. For example, telnet can be used on the remote host to login to the x86 processor module as follows:
$ telnet N3550-F 1444 Trying 192.168.220.11... Connected to N3550-F. Escape character is '^]'. CentOS Linux 7 (Core) Kernel 4.6.2-1.el7.elrepo.x86_64 on an x86_64 fusion_x86 login: root Password: Last login: Mon Aug 8 13:22:02 on tty1 [root@fusion_x86 ~]#
To disable the serial-server, the standard
no form of the command can be issued:
admin@N3550-F> config module Y no serial-server Serial server disabled for module Y
Secondary Serial Port
The second serial port on the x86 processor module can be used to gain access to the CLI of the Nexus 3550-F the processor module resides in.
To enable this, the Nexus 3550-F must first be instructed to enable CLI access via this serial port:
admin@N3550-F> configure module Y serial-console Serial console access enabled for module Y
We can then use a standard terminal application on the x86 module to open the serial port and gain access to the CLI login on the Nexus 3550-F:
[root@fusion_x86 ~]# picocom /dev/ttyS0 --baud 9600 picocom v1.7 port is : /dev/ttyS0 flowcontrol : none baudrate is : 9600 parity is : none databits are : 8 escape is : C-a local echo is : no noinit is : no noreset is : no nolock is : no send_cmd is : sz -vv receive_cmd is : rz -vv imap is : omap is : emap is : crcrlf,delbs, Terminal ready N3550-F login: admin Password: admin@N3550-F> admin@N3550-F>
no form of the command can be used to disable to console server:
admin@N3550-F> config module Y no serial-console Serial console disabled for module Y
Cisco Nexus SmartNIC
There is an onboard SmartNIC K35-Q (X40) embedded into the processor module. Whilst this is not in the same physical form factor as the SmartNIC K35-Q (X40) (i.e. it's not a PCIe card), the functionality and performance is exactly the same. The SmartNIC K35-Q (X40) provides 8 ultra low latency network interfaces to the host processor module, each of which can be configured for operation at 10G, 1G or 100M (with 40G firmware in development). The standard features of all Cisco Nexus SmartNICs (formerly ExaNIC) apply, including hardware timestamping, flow steering and kernel bypass for low latency transparent acceleration of sockets applications.
When the Nexus 3550-F detects an installed processor module, nine new ports are added to the system, for example Y1-Y9 assuming the module was installed into bay Y.
These interfaces/ports can be patched directly through to the front panel, or used to monitor & capture data flowing elsewhere throughout the Nexus 3550-F. The ports can be displayed with the
show port command, for example we can see the following on the Nexus 3550-F CLI:
admin@N3550-F> sh port Y* Port Status Description ---- ------- ----------- Y1 unused Y2 unused Y3 unused Y4 unused Y5 unused Y6 unused Y7 unused Y8 unused Y9 unused
Port Y9 is the onboard LAN (described below), and Y1-Y8 are the 8 SmartNIC interfaces, from exanic0:0 through to exanic0:7. As with any other port, they can have an alias and description set, get patched to another port, be added to a mux object etc.
Note: Setting the speed of a port in this case just configures the signal recovery circuitry on the Nexus 3550-F for optimal performance, it does not change the speed the SmartNIC is configured for on the Host - the user must ensure this is done independently.
In order to patch one of these SmartNIC ports to the front panel, the
patch command should be used:
admin@N3550-F> conf patch Y1 B2 Patch created between port "Y1" and port "B2"
This would allow, for example, a remote host to connect to the processor module as follows:
The motherboard within the processor module has an Intel(R) i217 Ethernet controller capable of running at 1G and 100M. This is exposed as port Y9, and would typically be patched out to allow for remote connection and management:
The onboard Ethernet controller supports Intel Active Management Technology (AMT), which allows for a number of handy IPMI/remote management capabilities. Refer to the section on AMT for further details.
The onboard Ethernet controller supports Intel Active Management Technology (AMT), which allows for a number of handy IPMI/remote management capabilities. For example, you are able to establish a remote KVM over IP session with the processor, so you can interact with the system as a local user, for example reboot, enter the BIOS, install operating systems etc. Users might be familiar with similar technologies from Dell (iDRAC), HP (iLO) etc.
Cisco ships the processor modules with AMT enabled with a password set which is
Exablaze1!. AMT is only available via the onboard LAN (ie port Y9), not any of the SmartNIC interfaces. It can be disabled via the KVM methods described below.
Intel AMT can be connected to via a number of different methods. It is configured to be assigned an IP address from a DHCP server on the network. Once the IP has been assigned, it can be discovered by searching for open ports on port 16992, for example:
$ sudo nmap --open -sT -p 16992 192.168.220.1/24 [sudo] password for donald: Starting Nmap 6.40 ( http://nmap.org ) at 2016-08-09 18:17 AEST Nmap scan report for 192.168.220.174 Host is up (-0.088s latency). PORT STATE SERVICE 16992/tcp open amt-soap-http MAC Address: 00:01:29:5D:EA:6B (DFI) Nmap done: 256 IP addresses (122 hosts up) scanned in 8.14 seconds
As can be seen above, the local DHCP server has assigned the module an IP address of 192.168.220.174. AMT has a fairly simple web interface for interacting with the system, so browsing to http://192.168.0.174:16992 and logging in as user
admin with password
Exablaze1! yields the following:
The default admin password can be changed by selecting User Accounts, and clicking the Change Admin... button.
A number of software utilities are available the allow for greater control over the system, including event & access logs, full KVM access (including into the BIOS), storage redirection, serial-over-lan etc.
Editing the x86 Module BIOS using Mesh Commander from a remote PC
Intel AMT can be completely disabled in the BIOS, however users should be aware that once this is done, the only way to re-enable it is to remove the lid of the Nexus 3550-F chassis and connect an HDMI monitor and USB keyboard to the connectors provided on the x86 module, and entering the MEBX extension within the BIOS menus.
This page was last updated on Mar-16-2021.