Ishwar Ramani, David Ranch

10/16/05

The benefits of IPMI are console redirection and remote management without the need for specific OS support. Since IPMI is built into the server's Ethernet1 interface, there is no need for external terminal servers and power management units. Console redirection is useful during system restarts and when there are network failures where as remote management enables remote access to server's environmental status as well as power control (reset, reboot, shutdown, etc.. All of this functionality comes regardless of the state of the operating system. SuperMicro's documentation and online FAQ entries are incomplete. This documentation is the PVL group's experience in successfully installing a SuperMicro IPMI card on a Fedora FC3 Linux server.

IPMI : Intelligent Platform Management Interface. An open industry standard for monitoring system hardware and sensors. This involves communication, hardware interface specification and accessibility. There have been three releases so far. ver 1,1.5 and 2.0. [1][2].

IPMI 1.0	Features like autonomous access, logging and control. IPMI messaging command sets, sensor data records and event messages. Acess through system interfaces like memory mapped IO , I2C bus etc.
IPMI 1.5	Ability to send IPMI messages to BMC over LAN, LAN alerting.
IPMI 2.0	Serial Over LAN enabling console redirection, access control, enhanced authentication, packet encryption, SMbus interface

BMC : Baseboard Management Controllers. IPMI compliant micro controllers that handle system event management. These are usually available as cPCI cards.

GPCagent : A SuperMicro proprietary "Graceful Power" control agent. This agent provides graceful power control features for both Linux and windows platform. GPC means the OS will shutdown gracefully before a power shutdown. It is available from the SuperMicro website. It run as a daemon (smagent) on a Linux based managed system. It requires openIPMI kernel modules to be installed on the managed system to interact with the IPMI device.[8]

i2c : A low speed (>=400khz) system management interface supported on most embedded systems. It is similar [13] to the SMBus.

ipmicli : A SuperMicro proprietary command line interface for Linux, similar in function to IPMIView, available from the SuperMicro website.[6]

IPMItool : An opensource tool[14] for accessing the IPMI device through either local or remote access. Its a command line tool that can used to perform various commands for reading and writing to the IPMI device. It is equivalent to the ipmicli proprietary tool except for console redirection which is not available on this tool.

IPMIView : A SuperMicro proprietary java applet available from SuperMicro website. Runs on both windows (tested on windows2k) and Linux (fc3) platforms. This runs on the remote system and can be used to interact with the IPMI interface on the managed system. This software provides sensor monitoring, secure login, LAN/IP configuration, chassis power control and console redirection terminal. It also provides a graceful power shutdown/restart option that requires a daemon running on the managed system.[5]

ipnmac : A SuperMicro proprietary command line tool for Linux to set the IP and mac address for the ipmi interface. This tool can be used to set the address locally on the managed system.[9]

Managed system : system which is to be managed using IPMI. The IPMI card is installed on this machine. IPMI v2.0 supports both local and remote access to the BMC. Local access is provided through a system interface like kCS (IO port). Remote access is provided through the onboard LAN interface (on IPMI supported motherboards).

OpenIPMI : An opensource IPMI project that maintains linux drivers for the IPMI device. These drivers run on the managed system and provide a local interface to the IPMI card. They also support a primitive command line utility, equivalent to the ipmicli. The utility is meant more as a sample than a working tool.[7]

Remote system : System from which the IPMI enabled server is managed. This is usually over the network.

SMbus : System Management bus. A low speed (<100khz) system management interface supported on most PC and server motherboards. It is similar [13] to the I2C bus. The BMC uses the SMbus to communicate with motherboard sensors and Ethernet interface. [3][4]

SuperoDoctor : A SuperMicro proprietary IPMI tool. Verision II is a command line tool for local access on the managed system to IPMI interface. Version III is a GUI based tool for local access but works only on windows platform. This requires the openIPMI kernel modules to be installed on the managed machine.

The server (managed system) is a SuperMicro 5014C-MR Intel based server (using the P8SCi motherboard) with Fedora Core 3 distribution installed on it. The IPMI card is a SuperMicro card (IPMI2.0) - model no: IPMI AOC-IPMI2.0-E. The IPMI client software for accessing the IPMI device is available on the SuperMicro website.

NOTE: At this time, FreeBSD's Broadcom Ethernet drivers conflict with the IPMI function. So, FreeBSD does not work as of yet.

4.a. Since the IPMI card uses the same network interface (NIC) as LAN (LAN1 if there are multiple interfaces on the motherboard), it needs to be configured with a different IP and MAC address.

NOTE: This conflicts with SuperMicro's IPMI documentation /and/ online FAQ entries.

This enables the Ethernet card to forward all IPMI traffic to the internal IPMI daughtercard. The MAC address for the IPMI card can be found on a sticker on the IPMI slot itself on the motherboard.

NOTE: This also conflicts with SuperMicro's IPMI documentation and online FAQ entries. The correct MAC entry is on the IPMI socket itself. The IPMI MAC address should NOT be the same MAC as either of the LAN1 or LAN2 MAC addresses.

4.b. Once the IPMI daughtercard is installed, the card firmware needs to be flashed in. THIS HAS TO BE DONE FIRST.

NOTE: Though not mentioned anywhere, the SuperMicro IPMI cards do NOT come with any form of IPMI code pre-installed. They are completely blank from the factory. Until code is installed into the card, it will not respond to any requests, etc.

The SuperMicro installation CD is bootable (make sure you use the newest version available from ftp://ftp.supermicro.com) and can be used to flash in the firmware matching the motherboard used with the card.

NOTE: The 5014C-MR has a P8SCi motherboard board which does not have a corresponding firmware according to SuperMicro's documentation or FAQ entries.

The P8SC8 motherboard is the closest match and it works fine on the P8SCi's IPMI daughtercards. On booting from the CD A GUI runs automatically. Select IPMI 2.0 and the P8SC8 series. This flashes the new firmware into the card. Exit to command line and run SMCFGX(X=I for intel and B for broadcom LAN chip)[11]. It is located in <CDROM>/Firmware/IPMI20/. ipnmac utility is also located in this directory. So you can set the IP (assigned by you) and the MAC (from 4a) for the IPMI LAN interface.

5.a. Once Linux has been installed and it boots up, the IP and MAC address of the IPMI card have to be configured (if not done in 4.b). Use the "ipnmac" utility found via the Supermico ftp site to set the MAC and IP address for the card. This command has to be run via sudo and prompts for the new ip and mac address.

5.b. For the GPC agent to run, the openIPMI drivers need to be installed on the managed system. The default FC3 build has openIPMI loadable modules. The modules are ipmi_devinf and ipmi_si (located in /lib/modules/2.6.9-1.667smp/kernel/drivers/char/ipmi). Check the SuperMicro documentation[12] for installation procedure, if these modules are not in the kernel.

NOTE: The ipmi_si need special parameters for loading. This was tested on a 2.6.10 kernel. It does not work with the 2.6.9-1.667(smp) kernel in the FC3 package.
modprobe ipmi_si type=kcs ports=0xca8 si_regspacings=4.

Once these modules are up and running, start the smagent.
$ smagent &

NOTE: When we tried running this daemon, the system entered the graceful restart state whenever smagent started. Make sure it works before adding to rc.local/login script(this leads to a restart loop).

5.c. Use the IPMIview Java utility from the remote system to access the IPMI card. IPMIview installation and operation are well documented in the SuperMicro website[12]. Since IPMI is designed to be OS transparent, this utility does not need any modules to be installed in the kernel. When we tried this from outside the local LAN and it worked without even citing a default gateway. We suspect the card is "inheriting" the gateway IP from the OS but this needs to be confirmed. If this configuration does not work, run IPMIview in the same LAN and configure the gateway MAC and IP .

5.d. The IPMIview tool allows power shutdown, reset, and device monitoring. We noticed that IPMI console redirection did not work though everything seemed to be configured properly. The BIOS on the P8SCi is supposed to detect the IPMI card automatically and set the console redirection options for it. Unfortunately, we were still seeing garbage in the remote console. [10]

NOTE: The current BIOS version of 1.1 shows the redirection going to "NULL" instead of something understandable like IPMI, COM1, etc.

NOTE #2: The solution was to get the latest P8SCI BIOS firmware (rev1.1) and re-flash it on the managed system. We had to do this even though the current BIOS on the board indictated it was already version 1.1.

[3]SMbus standard page. http://www.smbus.org/.

[5]SuperMicro FTP site. ftp://ftp.supermicro.com /CDR-0010_2.02_for_IPMI_Server_Managment/IPMI_Solution/Linux/Administrator/IPMIView-Linux_2.2_050503.bin

[6]SuperMicro FTP site. ftp://ftp.supermicro.com/CDR-0010_2.02_for_IPMI_Server_Managment/IPMI_Solution/Linux/Administrator/CLI

[8]SuperMicro FTP site. ftp://ftp.supermicro.com/CDR-0010_2.02_for_IPMI_Server_Managment/IPMI_Solution/Linux/GPC_Agent

[9]SuperMicro FTP site. ftp://ftp.supermicro.com/ CDR-0010_2.02_for_IPMI_Server_Managment/IPMI_Solution/Linux/Utility/ipnmac-1.4-04090401.tar.gz

[11]IPMI firmware setup documentation from SuperMicro. ftp://ftp.supermicro.com/CDR-0010_2.02_for_IPMI_Server_Managment/Firmware/IPMI20/Readme.txt

[12]IPMIView docs. ftp://ftp.supermicro.com/CDR-0010_2.02_for_IPMI_Server_Managment/Manuals/

10/16/05 - First release; acronym alpha sort; added an I2C definition; minor HTML formatitng changes

IPMI support on SuperMicro P8SCi motherboards

Index

1. Overview

2. IPMI Tools and Definitions

3. Target Platform

4. HW Installation Procedure

5. Software Installation Procedure

6. References

7. Errata