serverdude

After some time i decided to write something again. This time it’s a technical solution i’ve been working on.

Task: Shred 200+ HP Physical hosts.

Problem: Most don’t have OS running, iLo NAND is dead, SSA fails to boot.

Solution: Using debian10-live image to network boot and initiate shredding procedure.

It is not your typical scenario where you reuse a server for different applications. Since working in data center often requires to completely erase systems of any traces before giving up equipment. In this case the task scope is quite big and no physical access to servers made me think of some automated solution, since most server’s couldn’t even boot SSA to delete the disks.

After some reading and experiments with different mountable media to erase disks i just decided to go with native Linux approach.

Firstly a live system must have been made to boot via network. In this case i had access to “FOG” solution, so after few tries i was able to live boot debian10 with these parameters in the boot menu:

kernel tftp://${fog-ip}/os/debian/debian10-live/vmlinuz
initrd tftp://${fog-ip}/os/debian/debian10-live/initrd
imgargs vmlinuz boot=live components fetch=http://${fog-ip}/fog/management/debian10-live/filesystem.squashfs
boot || goto MENU

Since all servers are the same and HP P220i RAID controller doesn’t seem to support JBOD mode i decided to use “ssacli” to initiate secure erase with random random zero approach.

Therefore i made a script, to install ssacli, delete logical volumes and initiate disk erase procedure on all disks. Great! But it still required me to download the script from local ftp and execute it.

I decided to upgrade this approach and auto start the script when live system is booting and here is my approach:

kill.sh

#!/bin/bash

#install hp tool
echo "deb https://downloads.linux.hpe.com/SDR/repo/mcp stretch/current non-free" >> /etc/apt/sources.list
wget https://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub
apt-key add hpePublicKey2048_key1.pub
apt update
apt install ssacli

#disk erase commands
ssacli ctrl slot=0 ld 1 delete forced || echo "no LD 1 found"
ssacli ctrl slot=0 ld 2 delete forced || echo "no LD 2 found"
ssacli ctrl slot=1 ld 1 delete forced || echo "no LD 1 found"
ssacli ctrl slot=1 ld 2 delete forced || echo "no LD 2 found"


ssacli ctrl slot=0 pd 1I:1:1 modify erase erasepattern=random_random_zero forced || echo "no disk"
ssacli ctrl slot=0 pd 1I:1:2 modify erase erasepattern=random_random_zero forced || echo "no disk"
ssacli ctrl slot=1 pd 1I:1:1 modify erase erasepattern=random_random_zero forced || echo "no disk"
ssacli ctrl slot=1 pd 1I:1:2 modify erase erasepattern=random_random_zero forced || echo "no disk"

ssacli ctrl slot=0 pd all show
sleep 10
ifdown eno1
ifdown eno2

While the script alone is nice, next it is needed to place it inside the filesystem and execute it.

First we need to extract filesystem.squashfs which comes with live media iso file.

mount -o loop filesystem.squashfs /mnt/iso
cp -a /mnt/iso/* /tmp/temp_fs

Inside /etc/init.d i placed a script executable file which executes my kill.sh script on system startup.

/etc/init.d/kill_script

/etc/init.d/kill_script

#!/bin/sh
### BEGIN INIT INFO
# Provides:      	shredding_script
# Required-Start:	$all
# Required-Stop:
# Default-Start: 	2 3 4 5
# Default-Stop:
# Short-Description: Run kill.sh at startup
### END INIT INFO

case "$1" in
  start)
	/usr/local/bin/kill.sh
	;;
  stop)
	;;
  *)
	echo "Usage: $0 {start|stop}"
	exit 1
	;;
esac
exit 0

Place the kill script here /usr/local/bin/kill.sh (make executable of course)

Lastly it was time to pack squashfs filesystem with newly added content:

mksquashfs /tmp/temp_fs/ /tmp/filesystem.squashfs -comp xz

Place the file system inside FOG network boot directory and that’s it.

Now these were Blade systems mostly, therefore setting first boot option to Network 1 or 2 was fairly easy, then resetting the server and it automatically booted to FOG without having to access each one individually via iLo webpage.

Secure erase procedure automatically started and it saved a bunch of time and mouse click buttons. iLo does report that drive is erasing or erase complete.