Archive for the 'Sysadmin' Category

HP Laserjet P3005: fuser disassembly and gear kit replacement

Friday, January 23rd, 2015

Here’s how to disassemble the fuser on an HP LaserJet P3005 printer, and replace the gear kit (which is a part that fails often, reference CB414-67923). First remove the sliding plastic part at the back of the printer, then remove these two screws :


Then remove the big plastic panel, bottom first so it disengages at the top.

Lift these two little tabs so you pull the part they’re attached on towards you, and remove it:


Remove the screw holding the next panel in place (then remove it by putting the cable at the left away, and be careful not to break the clip at the right). I don’t think this is necessary, in reality. I did it because I didn’t know where I was going.


Then unplug the fuser cables (the cover on the power panel comes off with no screws) :



And unscrew the four screws :


You can then pull the fuser away and reach the gears. From left to right, the first one has a plastic tab keeping it in place, which you can pull with a little screwdriver. The second one is free when you remove the first. The third one is freed by the fourth, which is held in place by a weird little plastic piece. The sizes of the pieces are rather close but not identical, watch out for that. More details on this part of the procedure.


Datacenters and power consumption

Thursday, July 30th, 2009

Now that we’re in 2009, it’s been a few years since we’re more and more aware that energy is a valuable resource that should be spared. Where I work I’m handling a medium-sized datacenter of 160 servers, and they use approximately 28 kW of power. 28.000 watts, that represents 470 light bulbs, always on, burning day and night.

To this, we can add the cooling system’s consumption, which we don’t monitor but we can safely add 3 to 8 kW for the three cooling units, depending on the outside temperature. That’s what fun about datacenters, we have to burn electricity to cool down the room heated by the servers’ electricity consumption. Consider it the equivalent of putting the oven in your fridge when you bake a cake.

We’ve been trying to minimize a bit this consumption. There’s an article there that writes about shutting down servers when they’re not in use – ie, at night, or during the week-end.

Well, we thought of it first ! :-) In my opinion, it isn’t possible to shutdown every server at night : the backups run during the night, and saving its data is much more important to a company than saving electricity – sadly in some sense… So, we can’t shutdown production servers with important data on it. Luckily, where I work, a very large part of the datacenter is used for development and testing – we’re doing cluster-oriented storage, we have clusters, and about 140 out of our 160 servers are completely unused at night: developers go home.

So, one year and half ago already, I’ve implemented a way for developers to tell whether their test servers could be shutdown at night or not (in case they have long-running tests on them). It’s not a huge success, mainly because people choose 24/24 instead of 12/24 “just in case”, I think, but with approximately 30 servers down every night and during week-ends, we still spare 8 kW more than half the time. Still better than nothing…

Besides, at home, I’m now putting my laptop to sleep when I’m not in front of it. The saving’s much less and completely nullified by the server in the cupboard, but it’s still better than the days where I had three servers in the cupboard and didn’t put my laptop to sleep !

RAID1 array enlarging

Wednesday, March 4th, 2009

Here’s a quick  recipe to easily enlarge a RAID1 array with the least possible downtime, using linux 2.6 and mdadm.

We’ll start with a two-disk setup, /dev/sda and /dev/sdb, containing two arrays, /dev/md0 and /dev/md1. /dev/md0 is mounted on / and /dev/md1 is mounted on /backup. We want to grow /dev/md1 from 230GB to 898G (switching from 250GB disks to 1TB).

/dev/md0 has /dev/sda1 and /dev/sdb1, /dev/md1 has /dev/sda3 and /dev/sdb3, while swap partitions are on /dev/sda2 and /dev/sdb2.

Obligatory warning: Use your own brain when following this procedure. Don’t follow me blindly – it’s your data at stake.

Booting on degraded array: don’t shoot yourself in the foot.

When you’ll remove one of the existing disks, your computer won’t be able to boot if grub isn’t installed on the other disk’s bootsector, so make sure that grub is installed on both disks’ MBR:
grub> find /boot/grub/menu.lst
grub> root (hd0,0)
grub> setup (hd0)
grub> root(hd1,0)
grub> setup (hd1)

Shutdown the computer, remove sdb, put in one of the new 1TB disks in place, and reboot. Booting can take some time while the initrd’s mdadm tries to find the missing disk.

You’ll boot with degraded arrays, as shown there:
#cat /proc/mdstat
md0 : active raid1 sda1[0]
19534912 blocks [1/2] [U_]

md1 : active raid1 sda3[0]
223134720 blocks [1/2] [U_]

Now, we’ll dump sda’s partition table:
#sfdisk -d /dev/sda > partitions.txt

Edit the partitions.txt file to remove the size=xxxxxxx field on the sda3 line, so that the biggest possible partition size will be used. The file will look like:

# partition table of /dev/sda
unit: sectors
/dev/sda1 : start=       63, size= 39070017, Id=fd, bootable
/dev/sda2 : start= 39070080, size=  1959930, Id=82
/dev/sda3 : start= 41030010, Id=fd
/dev/sda4 : start=        0, size=        0, Id= 0

Disk initialisation

Now partition sdb using this table:
#sfdisk /dev/sdb < partitions.txt

recreate swap if needed:
#mkswap /dev/sdb2; swapon -a

Put sdb back in the arrays:
#mdadm –manage /dev/md0 –add /dev/sdb1
#mdadm –manage /dev/md1 –add /dev/sdb3

Wait until the array is resynchronised and clean. I use:
#watch cat /proc/mdstat #(quit with Ctrl-C)

Install grub on the new disk using grub, like previously (sdb is hd1 for grub), so that you’ll be able to boot from it.

Changing the second disk

Shutdown, remove sda, put the second new disk in place of it, and reboot – make sure your BIOS is configured to try and boot on both drives.

Now you’ll have degraded arrays again:
#cat /proc/mdstat
md0 : active raid1 sdb1[0]
19534912 blocks [1/2] [_U]

md1 : active raid1 sdb3[0]
223134720 blocks [1/2] [_U]

Redo the whole disk initialisation section, this time on sda instead of sdb. Don’t forget to reinstall grub on sda.

In the end you’ll get your arrays clean as they were before, but /dev/md1 will still be 230GB instead of using the whole available room on the disks’ partitions 3.

Grow the things

Let’s ask mdadm to take the whole partitions size for md1:
#mdadm –grow /dev/md1 –size=max

You’ll have to wait for synchronisation again (watch cat /proc/mdstat).

The only remaining thing is to grow the ext3 filesystem sitting on md1, and that’s where the most downtime happen (your data won’t be available unless you do a live FS resize, which I didn’t want to test); these steps took about 30 minutes to complete for me:
#umount /dev/md1
#e2fsck -f /dev/md1 #(it’s better to force a check to avoid a resize failure)
#resize2fs /dev/md1 #(this makes the filesystem the biggest possible)
#e2fsck -f /dev/md1 #(verify that everything is OK)
#mount /dev/md1 #(and you’re done, as df -h should show you):

# df -h /dev/md1
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              898G  228G  634G  27% /backup

Rambling about half-finished RAID setups

One thing you may have noticed is that I’m installing grub on both drives. This can seem evident, but most software RAID arrays I’ve seen couldn’t boot out of the second disk for lack of an MBR. It makes the RAID setup useful when your second disk fails, but if it’s the first, you’re forced to resort to a rescue CD or PXE boot to reboot your server. This makes things much harder to fix, provokes cold sweats, downtimes, and user annoyment. Install grub on both disks. Check the system boots when removing one disk, both the first or the second, before going into production. Don’t misunderstand your RAID arrays as a backup system. RAID arrays provides redundancy and eases (a LOT) recovering from a failed disk, but it doesn’t eases recovering from two failed disks; and it doesn’t recover lost data from human mistakes either. Regarding failed disks, best results are achieved by monitoring the disks – with smartd for example – and replacing suspicious disks too soon rather than too late.

Registar switch

Tuesday, November 25th, 2008

Back in may 2001, I grabbed my first domain name, I wanted to stop switching URL each time I switched the hosting. At the time I had no debit card, and I chose the first registrar I found which accepted payment by cheque,

Since then, I grumbled and grumbled each time I had to log in to their web administration interface, for domain renewal, DNS glue records updates, etc ; but out of habit, I bought two more domain names from them : my wife‘s, and a second one I had.

Finally, last week, after more failing glue records updates, I switched my domain and my wife’s domain to I left the third one on amen, as I don’t plan on renewing it, it’s useless for me to keep two domain names.

I knew Gandi beforehand as it’s the registrar for my work’s domain, and I’m glad I did the switch. Their interface is multiple times better, it does what it has to do and doesn’t bug out when hitting Submit with not even an error message.

How to change Dell’s BIOS settings from a Linux command-line

Wednesday, May 21st, 2008

To be able to change BIOS settings from the command-line on a Dell Poweredge, you need the syscfg utility. It’s very useful when you want to change a configuration on, for example, 32 nodes at once, without having to plug screen, plug keyboard, reboot, change setting, reboot 32 times. Here is how I installed it on the CentOS 5 distribution :

# cd ; wget -q -O – | bash
# yum install srvadmin-hapi
# wget
# mkdir dtk
# mount -o loop dtk_2.5_80_Linux.iso dtk/
# cd dtk/isolinux/
# cp  SA.2 ~/SA.2.gz
# cd; gunzip SA.2.gz
# mkdir stage2
# cd stage2
# cpio -i < ../SA.2
# cd lofs
# mkdir dell
# mount -o loop dell.cramfs dell/
# mkdir -p /usr/local/sbin ; cp dell/toolkit/bin/syscfg /usr/local/sbin/
# umount dell
# cd
# umount dtk

And voilà! You can now use syscfg:

# /usr/local/sbin/syscfg –biosver
# /usr/local/sbin/syscfg –virtualization=enable

I’d have preferred an easier way, but couldn’t find syscfg’s RPM.

When deploying that to a lot of nodes, you probably don’t want to go through all the associated network downloads of the first phase (wget of the yum repository, yum, and wget of the 230MB iso), so you can take shortcuts:

# for node in $(list_of_nodes); do scp /usr/local/sbin/syscfg /var/cache/yum/dell-hardware-auto/packages/srvadmin-*.rpm $node: ; ssh $node “mkdir -p /usr/local/sbin; mv syscfg /usr/local/sbin; rpm -ivh srvadmin-*.rpm”; done;

news for few, stuff no-one cares about