X86 Module Development
An Intel Skylake x86 Processor Module is available for installation into one of the internal module bays within the ExaLINK Fusion. This processor is available for you to run your own applications on, allowing construction and deployment of any number of custom appliances. The CPU is not required by Exablaze for running or management of the rest of the Fusion, so you are not virtualized or sandboxed in any way - you have complete control over the processor.
The specifications of the processor module are as follows:
- Intel Core-i7 Skylake 6820EQ CPU, 4 cores @ 2.8GHz, 3.5GHz turbo, QM170 chipset
- 32GB DDR4-2133MT/s (PC4-17000)
- Dual mSATA SSDs, typically shipped with Samsung 850 EVO 250GB as RAID1 for operating system
- M.2 PCIe NVMe SSD, typically shipped with Samsung 950 PRO 512GB capable of sustained write at 1,500MByte/s
- Onboard low latency ExaNIC X40, providing 8x10Gbe (2x40G via firmware update) interfaces which can be connected to other Fusion objects (mux, switch, front panel ports etc)
- Onboard 1Gb LAN, typically patched through to front panel
- Trusted Platform Module (TPM)
- Intel AMT allows remote KVM access, including operating system install and BIOS config
- Multiple serial ports available
- Multiple FPGA development options available
- Access to PPS from front panel or Fusion GPS receiver for highly accurate time synchronization
Top and bottom view of x86 Processor Module, showing drive bays
This module consists of a carrier board designed by Exablaze, and a 3rd party x86 processor board in a COM Express form factor. This design allows for potential upgrading of the x86 processor board to newer microarchitectures - for example Intel's Kaby Lake or Cannonlake, without changing the underlying carrier board.
The 3rd party processor board currently shipping is the SH960 from DFI.
As can be seen below the carrier board features the ExaNIC X40, connectors for mounting mSATA's and the M.2 drive, along with all the required power supply, support and management hardware.
Block diagram of x86 Processor Module
x86 Processor Module consists of a COM Express module installed on Exablaze's carrier board
x86 Processor Module installed into Fusion
System Power Management
The Fusion CLI is used to enable/disable power to the processor module. Assuming the processor is installed into module Y, the module can be powered up/down using the following commands:
[email protected]> configure module Y power on Module Y powered on [email protected]> configure module Y power off Module Y powered off
This is a hard power off, ie the equivalent of removing all power feeds to a server.
When operating, the power consumption of the processor module can be displayed using the
show power command as follows:
[email protected]> show power Voltage Current ------- ------- Line card A 12.3 V 0.3 A Line card B 12.3 V 0.3 A Line card C 12.2 V 0.5 A Module X 12.3 V 0.9 A Module Y 12.2 V 2.7 A
The temperature of the processor module can be read using the
show temperature command, as follows:
[email protected]> sh temp Temperature ------------------------------ Crosspoint 36.9 C 36.9 C 34.3 C 34.3 C Line card A 34.0 C Line card B 35.4 C Line card C 35.8 C Mainboard 30.9 C 34.9 C Module X 34.0 C 33.0 C Module Y 52.0 C 51.0 C
The first temperature listed above for module Y is the carrier board temperature. The second is the temperature of the ExaNIC X40 FPGA. There is no way to directly read the CPU temperature from the Fusion CLI, however the CPU and FPGA are thermally coupled by the heatsink. The CPU temperature can be read by logging into the processor module itself and running an application such as lm_senors, i7z etc.
Installed Operating System
The default configuration ships with Linux CentOS 7 installed onto a RAID1 partition created from the dual mSATA drives. The M.2 NVMe drive is mounted at /data. The login for this system as it ships is:
username: root password: exablaze
Primary Serial Port
There are 2 serial ports provided on the x86 processor module, both of which are connected through to the Fusion management processor. One of these can be used to gain console access to the x86 processor module. This is done by enabling a
serial-server on the Fusion and connecting to it remotely via the Fusion management port. Using this feature allows you to access the x86 processor module for management purposes without sacrificing a high speed front panel port
As can be seen in the diagram above, the remote host connects to a serial server running on the Fusion management processor. Traffic received/transmitted on this connection is routed to the x86 processor module's serial port, where a getty session would typically be running, enabling console access to the system. Commands to configure this would be as follows:
[email protected]> configure module Y serial-server port 1444 Serial server enabled for module Y on TCP port 1444
Once the serial server is enabled on the Fusion, various applications can be used to pass traffic to/from the connection. For example, telnet can be used on the remote host to login to the x86 processor module as follows:
$ telnet EXALINK-FUSION 1444 Trying 192.168.220.11... Connected to EXALINK-FUSION. Escape character is '^]'. CentOS Linux 7 (Core) Kernel 4.6.2-1.el7.elrepo.x86_64 on an x86_64 fusion_x86 login: root Password: Last login: Mon Aug 8 13:22:02 on tty1 [[email protected]_x86 ~]#
To disable the serial-server, the standard
no form of the command can be issued:
[email protected]> config module Y no serial-server Serial server disabled for module Y
Secondary Serial Port
The second serial port on the x86 processor module can be used to gain access to the CLI of the Fusion the processor module resides in.
To enable this, the Fusion must first be instructed to enable CLI access via this serial port:
[email protected]> configure module Y serial-console Serial console access enabled for module Y
We can then use a standard terminal application on the x86 module to open the serial port and gain access to the CLI login on the Fusion:
[[email protected]_x86 ~]# picocom /dev/ttyS0 --baud 9600 picocom v1.7 port is : /dev/ttyS0 flowcontrol : none baudrate is : 9600 parity is : none databits are : 8 escape is : C-a local echo is : no noinit is : no noreset is : no nolock is : no send_cmd is : sz -vv receive_cmd is : rz -vv imap is : omap is : emap is : crcrlf,delbs, Terminal ready EXALINK-FUSION login: admin Password: [email protected]> [email protected]>
no form of the command can be used to disable to console server:
[email protected]> config module Y no serial-console Serial console disabled for module Y
There is an onboard ExaNIC X40 embedded into the processor module. Whilst this is not in the same physical form factor as the ExaNIC X40 (ie it's not a PCIe card), the functionality and performance is exactly the same. The ExaNIC X40 provides 8 ultra low latency network interfaces to the host processor module, each of which can be configured for operation at 10G, 1G or 100M (with 40G firmware in development). The standard features of all ExaNICs apply, including hardware timestamping, flow steering and kernel bypass for low latency transparent acceleration of sockets applications.
When the Fusion detects an installed processor module, nine new ports are added to the system, for example Y1-Y9 assuming the module was installed into bay Y.
These interfaces/ports can be patched directly through to the front panel, or used to monitor & capture data flowing elsewhere throughout the Fusion. The ports can be displayed with the
show port command, for example we can see the following on the Fusion CLI:
[email protected]> sh port Y* Port Status Description ---- ------- ----------- Y1 unused Y2 unused Y3 unused Y4 unused Y5 unused Y6 unused Y7 unused Y8 unused Y9 unused
Port Y9 is the onboard LAN (described below), and Y1-Y8 are the 8 ExaNIC interfaces, from exanic0:0 through to exanic0:7. As with any other port, they can have an alias and description set, get patched to another port, be added to a mux object etc.
Note: Setting the speed of a port in this case just configures the signal recovery circuitry on the Fusion for optimal performance, it does not change the speed the ExaNIC is configured for on the Host - the user must ensure this is done independently.
In order to patch one of these ExaNIC ports to the front panel, the
patch command should be used:
[email protected]> conf patch Y1 B2 Patch created between port "Y1" and port "B2"
This would allow, for example, a remote host to connect to the processor module as follows:
The motherboard within the processor module has an Intel(R) i217 Ethernet controller capable of running at 1G and 100M. This is exposed as port Y9, and would typically be patched out to allow for remote connection and management:
The onboard Ethernet controller supports Intel Active Management Technology (AMT), which allows for a number of handy IPMI/remote management capabilities. Refer to the section on AMT for further details.
The onboard Ethernet controller supports Intel Active Management Technology (AMT), which allows for a number of handy IPMI/remote management capabilities. For example, you are able to establish a remote KVM over IP session with the processor, so you can interact with the system as a local user, for example reboot, enter the BIOS, install operating systems etc. Users might be familiar with similar technologies from Dell (iDRAC), HP (iLO) etc.
Exablaze ships the processor modules with AMT enabled with a password set which is
Exablaze1!. AMT is only available via the onboard LAN (ie port Y9), not any of the exanic interfaces. It can be disabled via the KVM methods described below.
Intel AMT can be connected to via a number of different methods. It is configured to be assigned an IP address from a DHCP server on the network. Once the IP has been assigned, it can be discovered by searching for open ports on port 16992, for example:
$ sudo nmap --open -sT -p 16992 192.168.220.1/24 [sudo] password for donald: Starting Nmap 6.40 ( http://nmap.org ) at 2016-08-09 18:17 AEST Nmap scan report for 192.168.220.174 Host is up (-0.088s latency). PORT STATE SERVICE 16992/tcp open amt-soap-http MAC Address: 00:01:29:5D:EA:6B (DFI) Nmap done: 256 IP addresses (122 hosts up) scanned in 8.14 seconds
As can be seen above, the local DHCP server has assigned the module an IP address of 192.168.220.174. AMT has a fairly simple web interface for interacting with the system, so browsing to http://192.168.0.174:16992 and logging in as user
admin with password
Exablaze1! yields the following:
The default admin password can be changed by selecting User Accounts, and clicking the Change Admin... button.
A number of software utilities are available the allow for greater control over the system, including event & access logs, full KVM access (including into the BIOS), storage redirection, serial-over-lan etc.
Editing the x86 Module BIOS using Mesh Commander from a remote PC
Intel AMT can be completely disabled in the BIOS, however users should be aware that once this is done, the only way to re-enable it is to remove the lid of the Fusion chassis and connect an HDMI monitor and USB keyboard to the connectors provided on the x86 module, and entering the MEBX extension within the BIOS menus.
This page was last updated on May-14-2019.