2011-05-07T16:27:16+01:00

Intalling HP Proliant DL320G5p

It's already 2 years since I had a chance to burn-in and install HP Proliant DL320G5p server. Even I took notes back then, I never found time until now when suddenly a good reason came... So I'll write a bit about the hardware, installation of Debian Lenny with Xen paravirtualization and performance tests that I've ran back then. Someone might find the test numbers useful as a reference.

front-top-small.png back-small.png

Here are the HW parts:

1x SERV HP DL320G5p QC-3210,  2.13GHz/1333 2x1GB SAS/SATA, Rack
4x HP 2GB UB PC2-6400 1x2GB Kit (ML110G5, ML310G5, DL320G5p)
3x HDD HP 72GB DP 3.5" Hot Plug, SAS 15k
1x HDD HP 500GB 3,5" Hot Plug, SATA 7.2K 
2x HP SC44Ge PCI-Ex HBA
1x HP DL320G5p iLO Port Opt Kit
1x HP DL1U 4 Drive Cage
1x HP HBA SAS-SATA 4x1LN Cable Kit

or in short - quad-core 2.13GHz Xeon CPU, 8GB RAM, 3x 72GB SAS + 500GB SATA HDD, with a iLO hardware remote console.

neo:~# lspci
00:00.0 Host bridge: Intel Corporation 3200/3210 Chipset DRAM Controller (rev 01)
00:01.0 PCI bridge: Intel Corporation 3200/3210 Chipset Host-Primary PCI Express Bridge (rev 01)
00:06.0 PCI bridge: Intel Corporation 3210 Chipset Host-Secondary PCI Express Bridge (rev 01)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02)
00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 (rev 02)
00:1c.3 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4 (rev 02)
00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 02)
00:1c.5 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 6 (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
00:1d.3 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface Controller (rev 02)
00:1f.2 IDE interface: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 4 port SATA IDE Controller (rev 02)
00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA IDE Controller (rev 02)
01:02.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)
01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out  Processor (rev 03)
01:04.4 USB Controller: Hewlett-Packard Company Proliant iLO2 virtual USB controller
01:04.6 IPMI SMIC interface: Hewlett-Packard Company Proliant iLO2 virtual UART
02:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev b5)
03:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet (rev a3)
03:04.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet (rev a3)
15:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)

Putting together the hardware pieces was quite straightforward, only later I've realized I have to put the SATA disk first and only then the SAS, as I had to change the cabling when I found out that the write performance on the SATA disk was "11.3 MB/s" using SC44Ge PCI-Ex controller vs "31.6 MB/s" using the on-board Intel ICH9 controller. So I left the three SAS disks connected to SC44Ge and the single SATA disk to the main board.

To install the base Debian Lenny system I didn't had to do any special tricks, just used the virtual CDROM and went through the installer.

disk partitioning

neo:~# fdisk -l /dev/sda

Disk /dev/sda: 73.4 GB, 73407865856 bytes
255 heads, 63 sectors/track, 8924 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000c51f0

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          31      248976   fd  Linux raid autodetect
/dev/sda2              32        8924    71433022+  fd  Linux raid autodetect
neo:~# fdisk -l /dev/sdb

Disk /dev/sdb: 73.4 GB, 73407865856 bytes
255 heads, 63 sectors/track, 8924 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000da224

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          31      248976   fd  Linux raid autodetect
/dev/sdb2              32        8924    71433022+  fd  Linux raid autodetect
neo:~# fdisk -l /dev/sdc

Disk /dev/sdc: 73.4 GB, 73407865856 bytes
255 heads, 63 sectors/track, 8924 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000ee4f3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *           1          31      248976   fd  Linux raid autodetect
/dev/sdc2              32        8924    71433022+  8e  Linux LVM
neo:~# fdisk -l /dev/sdd

Disk /dev/sdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00017cb0

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1       60801   488384001   8e  Linux LVM

The first partition of the three SAS disks (sda, sdb, sdc) is in RAID1 for the /boot with the grub loader. Then the rest of the first two disks (sda, sdb) is in RAID1 as a PV for LVM. Third SAS and the SATA disks (sdc, sdd) were stand-alone and also PV for LVM. So in total 3x volume groups vg00, vg01 and vg02. vg00, vg01 with capacity 68GB and vg02 with 465GB. All of them with a different characteristic. vg00 protected from one disk failure, vg01 standalone fast SAS 15k disk and vg02 standalone SATA with a big capacity.

neo:~# pvdisplay 
  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               vg00
  PV Size               68.12 GB / not usable 2.69 MB
  Allocatable           yes 
  PE Size (KByte)       4096
  Total PE              17439
  Free PE               2819
  Allocated PE          14620
  PV UUID               FiVmLS-7f3H-0S9x-7YjQ-bKnE-7M1t-E1aYIw   

  --- Physical volume ---
  PV Name               /dev/sdc2
  VG Name               vg01
  PV Size               68.12 GB / not usable 2.81 MB
  Allocatable           yes 
  PE Size (KByte)       4096
  Total PE              17439
  Free PE               11071
  Allocated PE          6368
  PV UUID               Jxx0zz-8hBR-jfUW-ZhaQ-QXCY-J9sb-qJwqoJ
   
  --- Physical volume ---
  PV Name               /dev/sdd1
  VG Name               vg02
  PV Size               465.76 GB / not usable 1.50 MB
  Allocatable           yes 
  PE Size (KByte)       4096
  Total PE              119234
  Free PE               73858
  Allocated PE          45376
  PV UUID               wMdrbc-s7G0-rsHj-O5CS-6JH8-MJTf-lHCKcy
   

iLO

The hadware remote console is an independent piece of the hardware inside the server chassis, sharing only power supply. Using this console server can be powered on or off and it allows also to see the "screen", access "keyboard" and virtual USB CDROM of the machine remotely allowing to reinstall the server remotely from anywhere.

Xen paravirtualiztion

Why paravirtualization even when the hardware was capable of full virtualization? Even thou the guest systems has to run modified Xen-domU kernels, the paravirtualizations brings the advantage that the single partitions from host system dom0 can be directly used in the guest domU systems. So there is no need to partition and use LVM in the domU-s again. It's easy to shut down the domU and mount the partition and do maintenance on it, like for example resize.

disk speed

I proved that the system was able to use the speed of the disks in the three volume groups independently via running the badblocks read/write check on one and then on all three at the same time without noticing significant difference. So this hardware with 4 cores has a potential of running 4 really independent machines at once, with my set-up 3 as the two of the disks are joined in RAID1.

testing with hdparm -t:

/dev/sda:
 Timing buffered disk reads:  352 MB in  3.00 seconds = 117.26 MB/sec
neo:~# hdparm -t /dev/sdb

/dev/sdb:
 Timing buffered disk reads:  352 MB in  3.00 seconds = 117.33 MB/sec
neo:~# hdparm -t /dev/sdc

/dev/sdc:
 Timing buffered disk reads:  354 MB in  3.01 seconds = 117.44 MB/sec
neo:~# hdparm -t /dev/sdd

/dev/sdd:
 Timing buffered disk reads:  322 MB in  3.01 seconds = 106.94 MB/sec

---

/dev/mapper/vg00-mirror--sas:
 Timing buffered disk reads:  352 MB in  3.00 seconds = 117.25 MB/sec

/dev/mapper/vg01-single--sas:
 Timing buffered disk reads:  354 MB in  3.01 seconds = 117.59 MB/sec

/dev/mapper/vg02-single--sata:
 Timing buffered disk reads:  324 MB in  3.02 seconds = 107.42 MB/sec

SAS RAID1 disk sync:

### sas disks sync
md1 : active raid1 sda2[0] sdb2[2]
      71432896 blocks [2/1] [U_]
      [>....................]  recovery =  2.9% (2091456/71432896) finish=10.4min speed=110076K/sec

----total-cpu-usage---- --dsk/sda-----dsk/sdb-----dsk/sdc-----dsk/sdd-- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ: read  writ: read  writ: read  writ| recv  send|  in   out | int   csw 
  0   0 100   0   0   0| 110M    0 :   0   110M:   0     0 : 102M    0 |2534B 3590B|   0     0 |8306    14k
  0   0 100   0   0   0| 109M    0 :   0   109M:   0     0 : 100M    0 | 384B 3476B|   0     0 |8217    13k

bad blocks check - only one disk running

neo:~# time badblocks -s -w /dev/mapper/vg01-single--sas
Testing with pattern 0xaa: done
Reading and comparing: done
Testing with pattern 0x55: done
Reading and comparing: done
Testing with pattern 0xff: done                                
Reading and comparing: done                                
Testing with pattern 0x00: done                                
Reading and comparing: done                                

real    134m39.806s
user    3m0.267s
sys     0m16.969s

bad blocks check - all three VG at once (stripped output only real time left):

neo:/mnt# time badblocks -s -w /dev/mapper/vg00-mirror--sas
real	135m0.663s

neo:~# time badblocks -s -w /dev/mapper/vg01-single--sas 
real	135m4.589s

neo:~# time badblocks -s -w /dev/mapper/vg02-single--sata 
real	255m34.293s

bad blocks check - all three VG at once (stripped output only real time left) and inside a XEN virtual machines:

mirror:~# time badblocks -s -w /dev/sdb1
real    135m31.627s

sas:~# time badblocks -s -w /dev/sdb1
real    135m28.460s

sata:~# time badblocks -s -w /dev/sdb1
real    257m33.197s

note there is no difference in speed between a single disk and the same two disks in SW RAID1

read speeds SAS vs SATA with dd

sas:~# time dd if=/dev/sdb1 of=/dev/null bs=1M count=10000 skip=10000
10485760000 bytes (10 GB) copied, 88.9769 s, 118 MB/s

# read
sata:~# time dd if=/dev/sdb1 of=/dev/null bs=1M count=10000 skip=10000
10485760000 bytes (10 GB) copied, 93.1479 s, 113 MB/s

write speeds SAS vs SATA with dd

sas:~# time dd if=/dev/zero of=/dev/sdb1 bs=1M count=10000
10485760000 bytes (10 GB) copied, 87.8604 s, 119 MB/s

# through SC44Ge
sata:~# time dd if=/dev/zero of=/dev/sdb1 bs=1M count=10000
10485760000 bytes (10 GB) copied, 927.064 s, 11.3 MB/s

# through on-board controller later
sata:~# time dd if=/dev/zero of=/dev/sdb1 bs=1M count=10000
10485760000 bytes (10 GB) copied, 305.611 s, 34.3 MB/s

It turned out that the read speeds on the SAS disk and SATA disk were equal and the write speed was ⅓. Let's try to prove this via the backblocks run times. During the run there is the same amount data to read and write. So for the SAS disk it was 135min/2 = 67,5min for each operation. If the SATA writes would be 3x slower then the total time would be 67,5m + 3x 67,5m = 270m. Which is actually similar to the real run time of 257min. I'm pretty sure some mathematician will beat me for this prove, but ...

network

The two Ethernet ports are configured in active-backup bonding mode. So both ports can be plugged to one or two switches and only one is communicating. When one switch goes down (power down, port shut-down) the other port takes over. The bond0 virtual interface that is created from the two eth0 and eth1 interfaces is then configured in a bridge interface so that the virtual machines can get public ip from the same IP pool as the physical interface has. Here is the system configuration:

# apt-get install bridge-utils ifenslave-2.6
# echo bonding >> /etc/modules:
# echo "alias bond0 bonding" >> /etc/modprobe.d/aliases
# echo "options bonding mode=active-backup miimon=100 max_bonds=1" >> /etc/modprobe.d/aliases

# vim /etc/network/interfaces

auto br0
iface br0 inet static
    address 62.40.64.245
    netmask 255.255.255.240
    broadcast 62.40.64.247
    gateway 62.40.64.241
    bridge_ports bond0
    bridge_fd 0
    bridge_stp off
    pre-up ifconfig bond0 up
    pre-up ifconfig eth0 up
    pre-up ifconfig eth1 up
    pre-up ifenslave bond0 eth0 eth1

I've crimped a 1Gb cross-over Ethernet cable and used netperf and ifstat to measure the throughput between my laptop and the server.

ifstat (one way and then the other way):

       eth0       
 Kbps in  Kbps out
961336.7  13976.19
961350.0  14070.64
961197.4  14146.69
961434.5  14138.59

       eth0       
 Kbps in  Kbps out
20822.81  958567.5
20731.41  953624.3
20842.09  959024.5
20700.29  952988.1

netperf (both sides had the netserver and netclient ran at the same time):

sas:~# netperf -l 60 -H 62.40.64.241
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 62.40.64.241 (62.40.64.241) port 0 AF_INET : demo
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    60.02     936.60   

ant:~# netperf -l 60 -H 62.40.64.242
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 62.40.64.242 (62.40.64.242) port 0 AF_INET : demo
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    60.01     107.55   

CPU

4x:

neo:~# cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU           X3210  @ 2.13GHz
stepping	: 11
cpu MHz		: 2128.046
cache size	: 4096 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu de tsc msr pae cx8 apic mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht constant_tsc pni ssse3
bogomips	: 4262.53
clflush size	: 64
power management:

the-end

The server is running for over 2 years now hosting couple of virtual machines. Some stand-alone with own public IP, some private ones behind nginx reverse proxy. So far so good! :-)

 

PS it took me 3,5h to write this blog entry...

comments powered by Disqus