SPARC V9 Boot Process – Part 2: sysboot

Introduction

This article is a follow-up to the earlier SPARC V9 Boot Process – Part 1: bootblk and takes a closer look at the second-stage bootloader of the SPARC V9 architecture.

The design of OpenBoot firmware limits the size of the first-stage bootloader to around 8 KiB. This is just enough to locate the boot device and load the second-stage bootloader, which can be much larger and contains all the logic required to load the operating system kernel. On some operating systems, the second-stage bootloader is called ufsboot or ofwboot; however, in this article, it is referred to as sysboot.

The rest of the article describes how to build SPARC V9 development tools for cross-compilation. It also demonstrates how the first-stage bootloader can use OpenBoot client interface services to locate the boot device and load the second-stage bootloader.

Development Tools for Cross-Compilation

Because we are performing bare-metal programming, which does not rely on operating systems or standard libraries, we can easily build a set of tools to generate SPARC V9 target binaries. My build platform is Linux on ARM aarch64; however, other platforms should work just as well.

The script below performs the following tasks:

1. Downloads source code tar files for Binutils and GCC.
2. Unpacks tar files.
3. Builds Binutils and then GCC development tools.
4. Installs development tools in user's home directory.

When configuring and building tools for cross-compilation, it is essential to explicitly specify the --build, --host, and --target options:

• --build and --host specify the platforms on which the development tools are built and run, respectively.
• --target specifies the platform for which the development tools produce runtime binaries.

🖙 It is important to specify: --target=sparc64-unknown-elf, which produces 64-bit SPARC ELF binaries for a bare-metal (no operating system) target.

# Create various build directories
BUILD_ROOT="${HOME:?}/gcc_build" &&
TAR_DIR="${BUILD_ROOT:?}/tar" &&
SRC_DIR="${BUILD_ROOT:?}/src" &&
OBJ_DIR="${BUILD_ROOT:?}/obj"
mkdir -p ${TAR_DIR:?} ${SRC_DIR:?} ${OBJ_DIR:?}

# Download the latest Binutils and GCC
BINUTILS_VER="2.46.0" &&
GCC_VER="15.2.0" &&
cd ${TAR_DIR:?} &&
wget https://ftp.gnu.org/gnu/binutils/binutils-${BINUTILS_VER:?}.tar.gz &&
wget https://ftp.gnu.org/gnu/gcc/gcc-${GCC_VER:?}/gcc-${GCC_VER:?}.tar.gz

# Unpack all downloaded files
cd ${SRC_DIR:?} &&
for i in $(ls ${TAR_DIR:?}); do tar -xf ${TAR_DIR:?}/$i & done; wait

# Before building GCC we need to download a few prerequisites
cd ${SRC_DIR:?}/gcc-${GCC_VER:?} &&
/bin/sh contrib/download_prerequisites

# Set various build options
MAKE_JOBS=4 &&
BUILD="aarch64-unknown-linux-gnu" &&
HOST="${BUILD:?}" &&
TARGET="sparc64-unknown-elf" &&
PREFIX="${HOME:?}/gcc-${GCC_VER:?}_${TARGET:?}" &&
PATH="${PREFIX:?}/bin:${PATH}" &&
CC="gcc" &&
CXX="g++" &&
unset LDFLAGS &&
export PATH CC CXX

# Build and install Binutils with static standard libs
pkg="binutils-${BINUTILS_VER:?}"; cd ${OBJ_DIR:?} &&
test ! -d ${pkg:?} && mkdir ${pkg:?}; cd ${pkg:?} &&
${SRC_DIR:?}/${pkg:?}/configure --prefix=${PREFIX:?} --libdir=${PREFIX:?}/lib \
  --build="${BUILD:?}" --host="${HOST:?}" --target="${TARGET:?}" \
  --enable-shared --enable-64-bit-bfd \
  --with-static-standard-libraries --disable-nls &&
gmake -j "${MAKE_JOBS:?}" && gmake install

# Build and install GCC
pkg="gcc-${GCC_VER:?}"; cd ${OBJ_DIR:?} &&
test ! -d ${pkg:?} && mkdir ${pkg:?}; cd ${pkg:?} &&
${SRC_DIR:?}/${pkg:?}/configure --prefix=${PREFIX:?} --libdir=${PREFIX:?}/lib \
  --build="${BUILD:?}" --host="${HOST:?}" --target="${TARGET:?}" --with-cpu=v9 \
  --enable-languages=c --enable-shared --disable-libssp --disable-nls &&
gmake --output-sync=target -j "${MAKE_JOBS:?}" && gmake install

The development tools are installed at:
${HOME}/gcc-15.2.0_sparc64-unknown-elf

When compiling code for SPARC V9, the following GCC options should be used:
-ffreestanding -nostdlib -nostartfiles

Now that the development tools for cross-compilation are in place, we can proceed to the next steps: developing SPARC V9 bootloaders.

SPARC V9 Boot Device Layout

To keep things simple, I use the following boot device layout: the first 32 KiB are reserved for the Sun VTOC label and the stage-1 bootloader, while the next 1 MiB is reserved for the stage-2 bootloader.

The boot sequence is as follows:

1. The OpenBoot firmware reads the first 512 bytes and verifies the Sun VTOC label.
2. The OpenBoot firmware then reads the next 8 KiB containing the stage-1 bootloader (boot1) and executes it.
3. The stage-1 bootloader allocates a 1 MiB memory buffer, opens a handle to the specified boot device, seeks to offset 32,768, reads 1 MiB of data into the buffer, and then executes the stage-2 bootloader (boot2) from that memory.

My current stage-1 bootloader is about 5 KiB in size, but it could be expanded to roughly 8 KiB. The stage-2 bootloader is currently around 10 KiB and could be expanded to as much as 1 MiB.

My SPARC V9 test machine is a Sun T5220 that supports booting from a USB flash drive. I use both native and SPARC V9 development tools to build the Sun VTOC utility along with the stage-1 and stage-2 bootloaders, and then write them to the USB drive.

# Include SPARC V9 cross-tools in current PATH
export PATH=${HOME:?}/gcc-15.2.0_sparc64-unknown-elf/bin:${PATH}

# Build native vtoc_label utility
cd ${SRC_DIR}/vtoc_label && gcc -O2 -std=c11 -Wall -pedantic \
  -D_FILE_OFFSET_BITS=64 -D_POSIX_C_SOURCE=200809L \
  -o ${BIN_DIR}/vtoc_label vtoc_label.c

# Build SPARC V9 stage-1 bootloader
cd ${SRC_DIR}/boot1 && sparc64-unknown-elf-gcc \
  -Os -std=c11 -Wall -pedantic -ffreestanding -nostdlib -nostartfiles \
  -o ${BIN_DIR}/boot1.bin start.s main.c ofw.c util.c && \
sparc64-unknown-elf-strip ${BIN_DIR}/boot1.bin

# Build SPARC V9 stage-2 bootloader
cd ${SRC_DIR}/boot2 && sparc64-unknown-elf-gcc \
  -O2 -std=c11 -Wall -pedantic -ffreestanding -nostdlib -nostartfiles \
  -o ${BIN_DIR}/boot2.bin start.s main.c ofw.c util.c v9reg.c v9reg_save.s && \
sparc64-unknown-elf-strip ${BIN_DIR}/boot2.bin

# Write Sun VTOC disk label and bootloaders
${BIN_DIR}/vtoc_label write <device> &&
dd if=${BIN_DIR}/boot1.bin of=<device> bs=512 seek=1 &&
dd if=${BIN_DIR}/boot2.bin of=<device> bs=1024 seek=32 &&
sync

The bootloaders are placed at fixed offsets at the start of the boot device. This simple layout avoids the complexity of file systems and is straightforward to implement.

Source Code and Boot Demo

The mechanism by which the stage-1 bootloader loads and executes the stage-2 bootloader is provided by the OpenBoot client interface services. They provide specific functions for opening a boot device and reading from or writing to data at designated offsets. My stage-1 bootloader follows the following sequence:

1. Invokes the OpenBoot finddevice() client interface to open a handle for the /chosen node.
2. Invokes the OpenBoot getprop() client interface to open handles for stdin, stdout, and bootpath, which are located under the /chosen node.
3. Invokes the OpenBoot claim() client interface to allocate a memory buffer for the stage-2 bootloader.
4. Invokes the OpenBoot seek() and read() client interfaces to load the stage-2 bootloader into the allocated memory buffer.
5. Invokes the OpenBoot execute-buffer() client interface to execute the stage-2 bootloader.

Once the stage-2 bootloader starts executing, it captures the current state of the machine registers and prints them to the console. The complete source code for the bootloaders is located here: sparcv9_bootloader.tar.gz.

{0} ok boot /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@3/disk@0
Boot device: /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@3/disk@0  File and args: 

Running SPARC V9 stage-1 bootloader

Allocating memory for stage-2 bootloader:
addr=0xFE982000, size=0x100000

Reading stage-2 bootloader:
ihandle_bootpath=0xFEB50608, offset=0x8000, size=0x100000


Running SPARC V9 stage-2 bootloader

Displaying machine registers:

i0          0x00000000FEBA9D88  4273642888
i1          0x0000000000000027  39
i2          0x0000000000000000  0
i3          0x00000000000000FF  255
i4          0x0000000000000030  48
i5          0x0000000000000036  54
i6 (fp)     0x00000000FEBA93D1  4273640401
i7          0x0000000000100138  1048888

l0          0x000000000000298E  10638
l1          0x0000000000000000  0
l2          0x000000000000257D  9597
l3          0x00000000000010AC  4268
l4          0x0000000000040008  262152
l5          0x0000000000000001  1
l6          0x0000000000004000  16384
l7          0x00000000080010D0  134222032

o0          0x0000000000000000  0
o1          0x000000000003C40A  246794
o2          0x000000000004D32C  316204
o3          0x00000000F025EFE4  4029018084
o4          0x00000000FEE82F10  4276629264
o5          0x00000000FEE81798  4276623256
o6 (sp)     0x00000000FEBA9321  4273640225
o7          0x0000000000100490  1049744

g0          0x0000000000000000  0
g1          0x0000000000000021  33
g2          0x0000000000000001  1
g3          0x0000000000000000  0
g4          0x0000000000000000  0
g5          0x0000000000000000  0
g6          0x0000000000000000  0
g7          0x0000000000000000  0

pc          0x0000000000101A54
ccr         0b00010001 (xcc=c, icc=c)
asi         0x0000000000000000
tick        0x0000000E02465966

pil         13
tba         0x00000000F0200000
tl          0
gl          0
pstate      0b0000000010110 (tct=0, cle=0, tle=0, mm=0, pef=1, am=0, priv=1, ie=1)

cwp         2
cansave     4
canrestore  2
cleanwin    7
otherwin    0
wstate      0x00

Press any key to exit: 
Program terminated
{0} ok

For further details on the SPARC V9 architecture and its registers, refer to the February 1993 Microprocessor Report article listed in the References section below.

References

Microprocessor Report SPARC V9 Adds Wealth Of New Features

SPARC V9 Boot Process – Part 1: bootblk

Introduction

SPARC V9 is a 64-bit Reduced Instruction Set Computer (RISC) architecture introduced in the mid-1990s that was widely used in servers and workstations manufactured by Sun Microsystems. As its popularity gradually declined, the hardware lost the performance advantage it once held in its early years and is now considered a legacy architecture.

Despite its age and competitive obsolescence, SPARC V9 is an interesting architecture to study. It has several unique characteristics, and some of us who lived through the dot-com boom still enjoy using Sun hardware for nostalgic reasons.

This article takes a closer look at the overlooked elements of the SPARC V9 boot process. It explores the OpenBoot firmware and demonstrates how to implement a first-stage bootloader from scratch.

OpenBoot

OpenBoot represents Sun Microsystems’ implementation of the IEEE 1275 Open Firmware standard, providing facilities for hardware configuration, system diagnostics, and operating system initialisation on SPARC V9 architectures. After completing the Power-On Self-Test (POST), OpenBoot firmware displays the ok prompt (provided the autoboot property is set to false), allowing the boot process to be started from a disk or over a network.

The OpenBoot PROM includes an FCode interpreter, which enables the execution of code written in a machine-independent interpreted language known as FCode. Firmware for pluggable storage adapters or even the first-stage bootloader can be written in FCode, allowing it to remain portable across different CPU architectures that implement a compatible FCode interpreter.

During initialisation, OpenBoot constructs the device tree — a hierarchical data structure that represents the system’s hardware and configuration parameters, and may also include firmware device drivers. The device tree consists of device nodes, where each node may have properties, methods, data, and other child nodes.

There are several standard system nodes; however, the /chosen node includes a number of useful properties that can be accessed by the first-stage bootloader. This node contains parameters chosen or specified at runtime:

Property Name	Encoding	Value
name	string	"chosen"
stdin	integer	Ihandle for the console input.
stdout	integer	Ihandle for the console output.
bootpath	string	The device path for the last boot device.
bootargs	string	The arguments to the last boot command.
memory	integer	Ihandle for the package that describes physical memory.
mmu	integer	Ihandle for the package that describes memory management unit.

Below is an example of what it looks like on a Sun T5220:

{0} ok dev /chosen
{0} ok .properties
bootargs                 
bootpath                 
mmu                      fff74080 
memory                   fff74290 
stdout                   feb51d88 
stdin                    feb51f90 
stdout-#lines            ffffffff 
name                     chosen

For example, the first-stage bootloader can use the hexadecimal value of the stdin ihandle to read text from the console, and the hexadecimal value of the stdout ihandle to write text to the console.

OpenBoot provides two primary interfaces: the user interface and the client interface. The user interface presents the ok prompt and offers a command-line environment for user interaction. The client interface supplies a set of services that can be invoked by a client program. A client program may be the first-stage bootloader, known as bootblk, and can comprise either FCode bytecodes or 64-bit SPARC V9 instructions packaged within an ELF64 binary.

In the following sections, I describe how a 64-bit ELF64 binary can call the OpenBoot client interface handler and invoke a range of services.

OpenBoot Client Interface Services

The OpenBoot firmware provides a number of services that can be invoked by the first-stage bootblk bootloader. The services are accessed via the client interface handler (a function pointer), with arguments and return values passed through an array of fixed size cells. A cell is a unit of storage in OpenBoot and on SPARC V9 its size is 8 bytes (64 bits).

The array passed to the OpenBoot client interface handler comprises the following cells:

Cell Name	Description
service	Pointer specifying the address of a '\0' terminated string of the client interface service
Num args	Integer specifying the number of input arguments to the client interface service
Num rets	Integer specifing the number of return values from the client interface service
Arg1	...
Arg2	...
ArgN	Input arguments to the client interface service
Ret1	...
Ret2	...
RetN	Return values from the client interface service

In the C programming language, a call to the write service can be implemented as shown in the following example:

/* 64-bit Open Firmware uses 64-bit storage cells */
typedef uint64_t ofw_cell_cdt;

/*
* Open Firmware ihandle is a handle/reference identifying a package instance.
* The ihandle always seems to be a 32-bit integer.
*/
typedef uint32_t ihandle_cdt;
#define OFW_IHANDLE_INVALID ((ihandle_cdt)-1)

/* Open Firmware Client Interface Function handler */
static int (*ofw_cif_handler)(void *);

/* Open Firmware ihandles for stdin and stdout */
static ihandle_cdt ihandle_stdin    = OFW_IHANDLE_INVALID;
static ihandle_cdt ihandle_stdout   = OFW_IHANDLE_INVALID;
static ihandle_cdt ihandle_bootpath = OFW_IHANDLE_INVALID;

/*
* Write to Open Firmware stdout.
*
* Arguments:
* strbuf     (in): String buffer.
* strbuf_len (in): String buffer length in chars.
*
* Pre-cond:
* 1. ofw_init() was executed earlier.
* 2. strbuf points to a valid string buffer.
* 3. strbuf_len indicates the number of string chars to write to stdout.
*
* Post-cond:
* 1. Characters written from strbuf to stdout.
*
* Return:
* On success: number of bytes copied from strbuf to stdout.
* On failure: (uint32_t)-1.
*/
uint32_t ofw_write_stdout(const char *strbuf, uint32_t strbuf_len)
{
    ofw_cell_cdt cif_arg[7];

    /* If stdout ihandle is not initialised, return -1 */
    if (ihandle_stdout == OFW_IHANDLE_INVALID) { return (uint32_t)-1; }

    cif_arg[0] = (ofw_cell_cdt)"write";        /* Service */
    cif_arg[1] = (ofw_cell_cdt)3;              /* Number of arguments */
    cif_arg[2] = (ofw_cell_cdt)1;              /* Number of return values */
    cif_arg[3] = (ofw_cell_cdt)ihandle_stdout; /* Arg1: stdout ihandle */
    cif_arg[4] = (ofw_cell_cdt)strbuf;         /* Arg2: strbuf address */
    cif_arg[5] = (ofw_cell_cdt)strbuf_len;     /* Arg3: strbuf length */
    cif_arg[6] = (ofw_cell_cdt)-1;             /* Ret1: return value */

    /* Call into cif handler */
    ASSERT(ofw_cif_handler(cif_arg) == 0);
    return (uint32_t)cif_arg[6];
}

For a specification of the client interface services, consult the IEEE 1275-1994 standard.

Sun VTOC Disk Label and Bootblk Requirements

When OpenBoot attempts to boot from a disk drive, it checks for a valid Sun VTOC disk label in the first 512-byte sector of the boot device. If a valid disk label is not found, OpenBoot displays an error message and aborts the boot process.

Once the Sun disk label has been checked, OpenBoot reads the next 16 512-byte sectors and loads bootblk first-stage bootloader. Since the number of sectors read during this stage is hard-coded into the firmware, the size of bootblk is limited to approximately 8 KiB and should fit within the 16 sectors immediately following the disk label. In my testing on a Sun T5220, I was able to load a bootblk slightly over 10 KiB in size; however, larger bootblk binaries resulted in the firmware error: "ERROR: Last Trap: Fast Data Access MMU Miss".

I developed a utility called vtoc_label, which can write a fake Sun VTOC disk label to a file or disk. The integer values within the disk label must be stored in big-endian byte order; therefore, the utility employs byte-order conversion functions to translate between host and network byte order, allowing it to generate correct disk labels even on little-endian architectures.

The bootblk is primarily written in the C programming language, with a small section containing the _start function implemented in SPARC V9 assembly. The boot stages are as follows:

1. Upon power-on, firmware begins to execute, probing devices, performing initialisation and running diagnostic routines.
2. Firmware enables the Memory Management Unit (MMU) and sets up a basic SPARC V9 trap table.
3. The boot command executes, specifying the boot device and boot arguments.
4. The first 512-byte sector is read from the boot device and the Sun VTOC disk label is checked for validity.
5. The bootblk binary is read from the next 16 512-byte sectors after the disk label.
6. The address of the OpenBoot client interface handler is copied to register %o4.
7. The bootblk binary format is checked (FCode, ELF64, etc) and then executed.
8. Control is transferred to the bootblk binary (i.e. first-stage bootloader), which can then load a larger second-stage bootloader or an operating system kernel.

Source Code and Boot Demo

The complete source code for the bootloaders is located here: sparcv9_bootloader.tar.gz. Below follows a demonstration of a simple bootblk first-stage bootloader.

Initially, I compile the vtoc_label utility and boot1.bin binary. Then, I use the vtoc_label and dd utilities to write the disk label followed by the bootloader:

# Include SPARC V9 cross-tools in current PATH
export PATH=${HOME:?}/gcc-15.2.0_sparc64-unknown-elf/bin:${PATH}

# Build native vtoc_label utility
cd ${SRC_DIR}/vtoc_label && gcc -O2 -std=c11 -Wall -pedantic \
  -D_FILE_OFFSET_BITS=64 -D_POSIX_C_SOURCE=200809L \
  -o ${BIN_DIR}/vtoc_label vtoc_label.c

# Build SPARC V9 stage-1 bootloader
cd ${SRC_DIR}/boot1 && sparc64-unknown-elf-gcc \
  -Os -std=c11 -Wall -pedantic -ffreestanding -nostdlib -nostartfiles \
  -o ${BIN_DIR}/boot1.bin start.s main.c ofw.c util.c && \
sparc64-unknown-elf-strip ${BIN_DIR}/boot1.bin

# Write Sun VTOC disk label and bootloader
${BIN_DIR}/vtoc_label write <device> &&
dd if=${BIN_DIR}/boot1.bin of=<device> bs=512 seek=1 &&
sync

Finally, I insert the USB flash drive into the Sun T5220 and boot from the drive containing my bootblk binary, which demonstrates basic input and output functionality on the system console:

{0} ok boot /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@3/disk@0
Boot device: /pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@3/disk@0  File and args: 

Running SPARC V9 stage-1 bootloader

ihandle_stdin  0x100348
ihandle_stdout 0x100358

Type your name: root
Hello root. Press any key to exit: 
Program terminated
{0} ok

In the second article: SPARC V9 Boot Process – Part 2: sysboot, I will examine how the first-stage bootloader can load and execute larger binaries and overcome the 8 KiB size limitation imposed by the firmware.

References

IEEE 1275-1994 Standard for Boot Firmware
IEEE Draft Std P1275.1/D14a Standard for Boot Firmware

SPARC T4-2: SMI link failed memory link test

During the first boot up sequence, one of my SPARC T4-2 machines started logging hardware errors. This was a brand new machine, never used before, which I unpacked myself from a sealed box.

[CPU 0:0:0] NOTICE: Initializing MCU 0 [CPU 1:0:0] NOTICE: Initializing MCU 0 [CPU 0:0:0] NOTICE: Initializing MCU 1 [CPU 1:0:0] NOTICE: Initializing MCU 1 [CPU 0:0:0] NOTICE: SMI Channel 0, SB Mapping 0 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 1:0:0] NOTICE: SMI Channel 0, SB Mapping 0 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 0:0:0] NOTICE: SMI Channel 0, SB Mapping 1 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 1:0:0] NOTICE: SMI Channel 0, SB Mapping 1 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 0:0:0] NOTICE: SMI Channel 1, SB Mapping 0 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 1:0:0] NOTICE: SMI Channel 1, SB Mapping 0 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 0:0:0] NOTICE: SMI Channel 1, SB Mapping 1 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 1:0:0] NOTICE: SMI Channel 1, SB Mapping 1 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 1:0:0] NOTICE: SMI Channel 0, SB Mapping 0 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 1:0:0] NOTICE: SMI Channel 0, SB Mapping 1 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 1:0:0] NOTICE: SMI Channel 1, SB Mapping 0 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 1:0:0] NOTICE: SMI Channel 1, SB Mapping 1 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 0:0:0] NOTICE: SMI Channel 0, SB Mapping 0 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 0:0:0] NOTICE: SMI Channel 0, SB Mapping 1 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 0:0:0] NOTICE: SMI Channel 1, SB Mapping 0 -- ERRCNT: 0x0 LNERR: 0x0 [CPU 0:0:0] NOTICE: SMI Channel 1, SB Mapping 1 -- ERRCNT: 0xf LNERR: 0x188 [CPU 0:0:0] ERROR: N0.MCU1: SMI link failed memory link test. [CPU 0:0:0] ERROR: /SYS/MB/CMP0/MCU1 failed to initialize [CPU 0:0:0] NOTICE: Updating Config Information for Guest Manager [CPU 0:0:0] NOTICE: Issuing Host warm Reset [CPU 1:0:0] NOTICE: Reconfiguring System [CPU 0:0:0] NOTICE: Reconfiguring System [CPU 1:0:0] NOTICE: MCU0: Memory Capacity is 64GB [CPU 0:0:0] NOTICE: /SYS/MB/CMP0/MCU1 is disabled [CPU 0:0:0] NOTICE: MCU0: Memory Capacity is 64GB [CPU 1:0:0] NOTICE: MCU1: Memory Capacity is 64GB [CPU 0:0:0] ERROR: /SYS/MB/CMP0/L3T1: unusable (MCU disabled). Not configured [CPU 0:0:0] ERROR: /SYS/MB/CMP0/L3T3: unusable (MCU disabled). Not configured [CPU 0:0:0] ERROR: /SYS/MB/CMP0/L3T5: unusable (MCU disabled). Not configured [CPU 0:0:0] ERROR: /SYS/MB/CMP0/L3T7: unusable (MCU disabled). Not configured [CPU 0:0:0] ERROR: /SYS/MB/CMP1/L3T0: unusable (Global symmetry rules). Not configured [CPU 0:0:0] ERROR: /SYS/MB/CMP1/L3T2: unusable (Global symmetry rules). Not configured [CPU 0:0:0] ERROR: /SYS/MB/CMP1/L3T4: unusable (Global symmetry rules). Not configured [CPU 0:0:0] ERROR: /SYS/MB/CMP1/L3T6: unusable (Global symmetry rules). Not configured [CPU 1:0:0] ERROR: /SYS/MB/CMP1/MCU0: Not configured due to partial cache mode [CPU 0:0:0] ERROR: /SYS/MB/CMP0/CORE1: Not configured (Required L3T not available) [CPU 1:0:0] ERROR: /SYS/MB/CMP1/CORE1: Not configured (Required L3T not available) [CPU 0:0:0] ERROR: /SYS/MB/CMP0/CORE3: Not configured (Required L3T not available) [CPU 1:0:0] ERROR: /SYS/MB/CMP1/CORE3: Not configured (Required L3T not available) [CPU 0:0:0] ERROR: /SYS/MB/CMP0/CORE5: Not configured (Required L3T not available) [CPU 1:0:0] ERROR: /SYS/MB/CMP1/CORE5: Not configured (Required L3T not available) [CPU 0:0:0] ERROR: /SYS/MB/CMP0/CORE7: Not configured (Required L3T not available) [CPU 1:0:0] ERROR: /SYS/MB/CMP1/CORE7: Not configured (Required L3T not available) [CPU 0:0:0] NOTICE: Usable strands: 00ff00ff00ff00ff [CPU 1:0:0] NOTICE: Usable strands: 00ff00ff00ff00ff [CPU 0:0:0] NOTICE: System memory capacity is 128GB [CPU 0:0:0] NOTICE: Enabling caches [CPU 1:0:0] NOTICE: Enabling caches [CPU 0:0:0] NOTICE: L3 Banks Enabled: 55 [CPU 1:0:0] NOTICE: L3 Banks Enabled: aa

Similar issue was previously logged with Sun/Oracle, however access to the root cause and its resolution is blocked for anyone who does not have a valid support contract. How nice of them.

I assumed the issue was related to one of the memory riser boards, so I carried out the following steps:

1. Cleared the fault in the system ILOM.
2. Powered off the system and unplugged power cables. Unplugging is required since there is normally a standby power of 3.3V.
3. Unplugged each memory riser, one by one, and then reseated individual memory modules.

Next time when I powered on the system, the fault was resolved and did not appear again.

Not sure of the actual root cause, possibly a bad connection with one of the memory modules. If anyone knows the details, let me know in the comments.