The Trails and Tribulations of Tape – A Cautionary Tale

For most of us, we have some data on some computers that we’d like to keep. For some it is critical. However, it can be a minefield, and finding a suitable strategy can be difficult.

Having spent a lot of time in the past moving precious data from one hard drive to another, and using fault tolerant RAID arrays to protect important data, I decided that it would be a good idea to invest in a long term and repeatable backup, and tape seems to be the accepted way to do this.

Since I don’t want to invest the life-savings in a state of the art tape drive, an older (but still very capable) model should do for what I need.

Hence I have bought an HP LTO5 Drive – 1.5TB per tape native, and up to 3TB compressed, which depends entirely on what you are backing up. Assume 1.5TB to be on the safe side.

The LTO6 and 7 drives hold more data per tape, but the tapes are proportionately more expensive too, so LTO5 seemed like a good compromise.

However, if you’re used to using NAS or external Hard Drive as backup, tape seems to be an altogether more complicated thing and fraught with troubles.

As I have an HP server, I decided to buy an HP drive as that should ensure it was compatible. There are good second hand offers on eBay, but I found a “new unboxed” one for a good price and bought that.

Problem 1: SAS Connections

If you’re used only to SATA (as I am), SAS is quite a new and complicated thing. I checked the specs for the server I was going to put the drive in, and it says that it has SAS connections, so that should be all right? Wrong! While there are SAS connections in the server, the are all used in the 4 non-hot-swap drive bays. Ok, says I – I’ll plug in one of the HDDs into the SATA port, and free up a SAS port. Brilliant – the HDDs are fine, and the tape drive lights up.

Does Linux recognise it? Of course it doesn’t – the hard drive controller in the (albeit entry level) HP server offers SAS connectors and says SAS on the specification (I did check before buying the drive), but is only capable of driving SATA hardware. Thanks HP!

A new SAS controller card and suitable cables (an extra £200 odd) and I’m in business – or so I thought.

Problem 2: Warranty Support for Updates

So I try a backup – looks good, and then an error…

Try again… looks good… error.

Reading about the errors I was getting on various forums, the advise seems to be to update the tape drive firmware.

However, just because the product is advertised as “new” by the seller – don’t expect HPE to give you up-to-date firmware to go with it – this is “enterprise” kit you have to pay for support if the item is out of warranty – and there is no way of finding out if it is in warranty if you don’t buy from HPE directly, or from an authorised reseller. (i.e. not from eBay)

A Solution

Noting that the problems documented as fixed by the update didn’t include mine, I continued to search websites for the specific error I was getting which was only shown by dmesg, not by tar, and found that when using tar with an LTO5 drive, the block size is essential.

The advise given here was you should use at least 256kb blocksize for an LTO5 drive.

So, almost like magic, adding the parameter “-b 512” (i.e. 512 x 512b = 256kb) to the tar command fixed the problem, and I’m now backing up to tape, but only after several hours of scouring the web and pulling my hair out.

Conclusion – the Moral of this Story

Tape backup is a whole order of magnitude more complicated and expensive than an external HDD or NAS drive. Don’t start down the road of tape unless you really need it, and can invest the time and energy in getting it to work properly.

Do your research thoroughly before buying anything, especially 2nd hand, and don’t assume just because you see something on the documentation that it will work without additional hardware and/or software.

Was it all worth it? Well, I hope I’ll never need to find out…


Page Layout Settings and Headers/Footers in EA Virtual Documents

When trying to generate a document using the virtual document generation facility in Enterprise Architect, I’ve been struggling to get the formatting to match the template for my client.

It appears when generating the virtual document, the page setup/layout parameters (i.e. margins) are derived not from the first template used in the virtual document, as I would have assumed, but from the last!

Clearly, unless you apply section breaks to the templates, each template called overwrites certain parameters (including the margins) for the overall document.

While I can understand why this might be the case, it does seem much more intuitive that these details would come from the first template in any section, rather than the last, since you would usually have a fairly specific first template for any particular document type.

I also note that the same is true for headers and footers. If you have headers and footers in a cover sheet, as they are in a separate section in the created document, the headers and footers are retained, and propagate to the whole document (assuming you don’t have other headers and footers defined).

However, if the header and footers are in the in a standard document template etc., they are overwritten by the last template in the virtual document, even if there is no header/footer defined in that there not being any header/footer defined.


Migrating CentOS5 VMWare images with LVM to KVM

I’ve decided that using VMWare Server has been a useful and worthwhile experience, but I want to do stuff which would involve upgrading to one of the paid-for versions of VMWare and I don’t want to pay.

So, I’ve set up a new server to host my VMs on Fedora 14, and am migrating to using KVM instead.

I had expected that I’d have to backup the image and then recover it, but no, it seems that the open source community has done everything I need. I started here and noted the slightly scary caveat that LVM using images didn’t work.

I tried anyway, and sure enough, I got a message saying

Volume group "VolGroup00" not found
Unable to access resume device (/dev/VolGroup00/LogVol01)
mount: could not find filesystem '/dev/root'

It had to be possible. I asked at, and the reply led me to look at virt-p2v, which was a possible fallback option, but rather scary since there aren’t any hosted pre-build iso files any more.

However, reading some of the blurb on the author’s blog about manual p2v he mentioned possibly having to edit the fstab file to point the mounts in the right direction following virtualisation.

I had loaded the converted qcow2 image, using Virtual Machine Manager. When I looked at the setup I noticed that the disk setup looked like this:

Properties page for VirtIO disk.

I Googled VirtIO and found that it was a para-virtualised disk type, which meant that it had to be supported by the guest OS. This would explain why the Fedora 14 rescue disk could see the LVM but not the actual operating system.

I deleted the disk entry and replaced it with this:

Properties page for new IDE drive

Now, I started the guest, and it booted!

Text from successful boot image.

Upon booting, the X-window display didn’t work, but the automatic rescue process worked, and I got a gnome login screen. The next problem was that the mouse didn’t work properly. The Virtual Machine Manager does suggest that you add a tablet to the machine to ensure that the mouse matches the VNC mouse, but it still didn’t work after doing that. However, yet again, someone had done the work first, and I found the answer here. Slightly messy editing the xorg.conf file, but it worked.

The Disadvantages of not Keeping Current

I have been requested by a client to produce data extracts from an old bespoke system, which has been knocking about for many years and been left without upgrades for several years. A nice piece of work for me, but it’s got me thinking about the whole upgrade lifecycle issues and bespoke vs packaged solutions in general.

The reasons it’s not been upgraded are that “it ain’t broke – don’t fix it”, and that nobody wants to pay the costs to keep regression testing new upgrades.

However, there are disadvantages to letting a system continue on old techonolgy for a long period of time. Maintenance gets more difficult as fewer people are available who remember the older technology. The world moves on and the system doesn’t, so it looks antiquated, and leads users to feel irritated by the inadequate user-interface.

The net result is that a new proprietary packaged system is going to be deployed replacing the creaking bespoke system, at considerable cost. However, as with all packaged solutions, the “out of the box” fit with the business requirement is good, but not perfect. Ok, the bespoke system doesn’t fit with the business 100% either, since the business has moved on and it hasn’t, but I can’t help thinking that had the bespoke system been periodically upgraded to use the latest (or at least more recent) technologies, and the changes to the requirements, that the requirements fit could have been better and the overall cost less than the new replacement system.

A final disadvantage of remaining on older techonolgy, comes when carrying out the extract of data for migration, you realise that “that approach doesn’t work on that version, you need the next one”. Realising that several approaches aren’t possible due to the features I’m now used to being for more recent versions, it becomes frustrating to have to rethink each time.

Decimal keypad in iPhone SDK – Well Done Apple!

A few days ago I was trying to find how to get a decimal point in the number pad for an app I was writing. Having reviewed a number of discussions on the subject which suggested subclassing the keypad or adding extra buttons and all sorts of scary things, I finally resorted to the manual.

In the list of available keypad types, it turns out that there is actually a new one in SDK 4.1. Unfortunately, at present, they haven’t updated Interface Builder with the new values, so you can’t select the new values directly but you can programatically, like this:

textField.keyboardType = UIKeyboardTypeNumberPad;

so clearly they have listened to the needs expressed in these various articles, but unfortunately, this is rather well hidden in the documentation, and until it appears in the keyboard type picklist in Interface Builder, will remain so.

So well done Apple for adding UIKeyboardTypeNumberPad, but please can you add it to Interface Builder too!

Paging Enabled peculiarity in UIScrollView

Whilst developing an iPhone app, I have come some very odd behaviour in UIScrollView. Ok, I’ll admit what I was doing was somewhat unusual, but I think the issue could occur in less unusual circumstances.

The effect appeared to be that the UIScrollView was remembering the last user scroll page position, despite my changing the frame, contentSize and contentOffset of the view manually. If the user touched the scroll view after my programmatic changes, the view would immediately scroll back to the page that had last been user scrolled to.

Consider the following situation: we start with the scrollview (and content) looking like

[1 2 3 4] 5 6 7 8

where the brackets denote the visible page area. I manually scroll to show the second page:

1 2 3 4 [5 6 7 8] 

which works fine. I then change to “selected” mode programmatically, and the display now shows:

1 2 3 4 5 6 [7] 8

again, which works fine.
However, if I now tap on the UIScrollView it immediately scrolls to

1 2 3 4 [5] 6 7 8

for no apparent reason.

This was going to be a complete show stopper for me, so I spent quite a long time delving into the mysteries of UIScrollView, and to cut a long story short, I think I’ve found a bug. I’ve opened a bug report with and submitted a test case, but not heard anything yet, so I’ll update this post as and when they confirm or refute it.

However, the good news is that while investigating what was causing the problem, having guessed at the cause, which I think is a rounding error between two different member variables, responsible for handling the paging enabled mode, a workaround idea presented itself.

What I discovered was, that if the content view width was integer divisible by the number of pages (which in this case was 2 or 8 pages wide), the problem didn’t happen, whereas if the content size divided by the number of pages resulted in a fraction, the issue occurred. Thankfully, this was a fairly easy workaround to implement, and perhaps explains why it hasn’t shown up often enough for apple to have found it previously.

Deserializing serializable Java objects without a default constructor

I’ve spent the past few days implementing a serialisation process overriding the ObjectOutputStream and ObjectInputStream classes. However there seemed to be certain classes that cannot be deserialized even though they are serializable, as they have no default constructor.

The serialisation class documentation implies that the constructor for a serializable class isn’t called anyway, which makes sense as you don’t really want to set up a class from scratch when you are deserializing, just restore it.

Most replies to questions on this subject seemed to imply that it can’t be done except by the default serialization classes. This is no use if you want to override these, and since they are overridable, surely Sun woudn’t have made it impossible.

One hit during my search mentioned the ReflectionFactory class in passing, but didn’t have any detail, so I looked into it further, and sure enough, this seems to be the answer.

The Solution?

It would seem that the ReflectionFactory class can create a Constructor object on the fly, using the default constructor of the first non-serializable superclass, and the class you’re trying to instanciate.

So when overriding the deserialisation process, you can create an instance of an object that doesn’t use the default constructor, and doesn’t need one to exist by calling the constructor returned by something like the following:

ReflectionFactory rf = ReflectionFactory.getReflectionFactory();
Class type = HashMap.class; //
Class nonSerType;
for ( nonSerType = type; Serializable.class.isAssignableFrom(nonSerType); nonSerType = nonSerType.getSuperclass() ) {}
Constructor superCon, deSerCon;
superCon = nonSerType.getConstructor();
deSerCon = rf.newConstructorForSerialization(type, superCon);

where the deSerCon can be used to instanciate the deserialized object.

This process seems to work for my deserialization work, and so I’ve documented it here. I’ll admit, I’ve no idea if there are any specific limitations to this process, but for the limited range of classes I’ve tested so far, it works fine.

Server Side XSLT2 with Apache, Tomcat and Saxon

I have spent several days trying to figure out how to do server side XSLT using an Apache server using numerous suggested ways from articiles on the web about how to do it using Cocoon, and using mod-xslt2, all of which are out of date in one way or another, i.e. mod-xslt2 doesn’t work with Apache 2.2 yet, and the article describing how to do it with Cocoon used a previous version of Cocoon so didn’t help much with the current one.

However, I have now managed server side XSLT2 transformation using Apache, Tomcat, mod-jk, mod-rewrite and Saxon as the transformation engine.

The Theory

I noticed in the Saxon documentation, that one of the demos was a simple servlet. So, running this servlet on Tomcat, I should be able to do the transformations on the server.

The downside of this approach is that I was going to end up with really nasty looking URLs, such as

which is hardly convenient when writing links, and while that could be automated in the xsl transformation, it would still look ugly in the browser address or status bars.

However, Apache has mod-rewrite which does lots of exciting things with URLs. This can rewrite any incoming URL to poke the relevant parameters into the servlet.

The Practice

Install Apache, Tomcat and mod-jk. There’s loads of instructions for that on the web, and anything I say will have been better said elsewhere so I won’t.

Everything here is rather distibution dependant, so your system may be slightly different, but the ideas should be similar. I’m using Centos 5.2 here.

  1. Download and install SaxonB 9.1 for Java. Make sure you also get resources package, as that has the servlet sample.
  2. Complile the file, and create a .war file. To do this, I created a web application in Netbeans 6.5 and got it to do all the compiling and packaging etc.
  3. Using the tomcat manager app, add and release the .war file.
  4. Place or link your xml and xsl files in the servlet directory, so the servlet can access them. I have /usr/share/tomcat5/webapps/Servlets/xml and /usr/share/tomcat5/webapps/Servlets/xsl linked to my webserver directories, although you could have the xml and xsl files just in the servlet directory.
  5. I use the default virtual host, and hold the majority of the website files in the apache html directory, therefore you can’t just use the auto-generated virtual host file.Update the /usr/share/tomcat5/conf/servers.xml file: the <Host> needs some <Alias> tags to allow Tomcat to see your website when not accessed through localhost. The host section look something like this:
    <Host name="localhost" appBase="webapps"
        unpackWARs="true" autoDeploy="true"
        xmlValidation="false" xmlNamespaceAware="false">
  6. Restart your tomcat instance, and this will create the “auto” subdirectory in the /usr/share/tomcat5/conf directory.
  7. Look through the mod_jk.conf in the auto directory, and you should see a pair of lines starting JkMount. Copy these lines into the main httpd.conf file area, if like me you’re not using virtual hosts, or into the relevant <VirtualHost> element. You should have something like this:
    JkMount /Servlets ajp13
    JkMount /Servlets/* ajp13
  8. Restart httpd, then we can test that you can run the servlet from Apache. So from your browser try

    This should show an error message about missing style parameter. If you get this message then Apache and Tomcat are linked and you can proceed.

    It is probably a good idea to try and access the servelet from other hostnames/domain names that might use it, since it may work for localhost and no other hosts. Thus trying


    may be a good idea too.

  9. Now we need to rewrite the incoming URLs to use the servlet. I have all my URLs beginning xml/ rewritten to use the servlet, with a single xsl transformation. If you need different xsl files, then this will be slightly more tricky, and will need a separate rewrite rule for each xsl file, or a more complex servelet.Add the RewriteRule directives in the <Directory element appropriate for your site. I have
    <Directory "/var/www/html">
        RewriteEngine       on
        RewriteRule         (^xml/.*$)      /Servlets/SaxonServlet?source=$1&style=/xsl/main.xsl
  10. Restart httpd and assuming everything is ok, you should get your xml documents transformed using the specified xsl. That’s how this page works, as it’s entirely written in xml.

I hope this is useful to someone, and if anyone has a better way of doing it I’ll be interested to know. One of my friends insists PHP is the way to go, but I can’t see that being any easier than this.

Setting up fetchmail daemon on Centos 5.3

Unfortunately, I failed to find any suitable instructions for doing this when I was looking to set up a single fetchmail daemon for Centos. It appears this is easy on debian flavours, and perhaps there are better ways of doing it than this, but I thought I’d share the method I used…

I’ve kept the steps fairly simple, so please forgive me if some of it is completely obvious…

1 – Install fetchmail if not already installed

Check if fetchmail is already installed, and perhaps update it if it is out of date.

yum list fetchmail

will show you if you’ve already got it installed, and which versions are available for install. I used the x86_64 package version 6.3.6-1.1.el5

Use yum to install or update this as required.

2 – Configure fetchmail

Why is nothing ever simple under Centos? I tried to find a RPM fetchmailconf, the gui fetchmailrc file configurator, for Centos 5 but couldn’t find one for the current version of fetchmail.

I ended up with fetchmailconf-6.3.4-1.1_6.0.1.el5.x86_64.rpm which wouldn’t install as it depends on the fetchmail-6.3.4 rather than the 6.3.6 version I had installed, so I had to install it using

rpm -i --nodeps fetchmailconf-6.3.4-1.1_6.0.1.el5.x86_64.rpm

This at least worked, and so I used it to configure my .fetchmailrc file for my mail servers etc. I then copied the ~/.fetchmailrc file to /etc/fetchmailrc for use by the daemon.

3 – Set up a script in /etc/init.d.

I largely copied the script for the Samba daemon.

# chkconfig: - 91 35
# description: Starts and stops fetchmail in daemon mode

# Source function library.
if [ -f /etc/init.d/functions ] ; then
  . /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
  . /etc/rc.d/init.d/functions
  exit 1

# Avoid using root's TMPDIR
unset TMPDIR

# Check that fetchmailrc exists.
[ -f /etc/fetchmailrc ] || exit 6


start() {
	echo -n $"Starting $KIND services: "
	daemon --user fetchmail fetchmail -f /etc/fetchmailrc --syslog
	return $RETVAL

stop() {
	echo -n $"Shutting down $KIND services: "
	runuser fetchmail -c 'fetchmail --syslog --quit >dev/null 2>&1'
	[ "$RETVAL" -eq 0 ] && success $"$base startup" || failure $"$base startup"
	return $RETVAL

restart() {

reload() {
        echo -n $"Reloading fetchmailrc file: "
	return $RETVAL

rhstatus() {
	status fetchmail

# Allow status as non-root.
if [ "$1" = status ]; then
       exit $?

# Check that we can write to it... so non-root users stop here
[ -w /etc/fetchmailrc ] || exit 4

case "$1" in
	echo $"Usage: $0 {start|stop|restart|reload|status}"
	exit 2

exit $?

You can download this file by right clicking here.

4 – Set up a fetchmail user.

Create a user on the system called fetchmail. I used the -r option with useradd but then I had to create a home directory as the fetchmail program expects one. You can set up an alternative directory using an environment variable, but I took the easy way and just created a home directory.

You then chown the /etc/fetchmailrc file to fetchmail, and the home directory.

5 – Start it up…

Now try and start the service.

service fetchmail start

With any luck, the service starts… I suggest you then try and stop it with

service fetchmail stop

to ensure the stop code works too.

Next steps…

I then added the service using the gui services control, and this works ok, allowing me to start, stop, and configure the service for automatic startup on level 5.

However, selinux is doing something I don’t quite understand. It works using the service command from the command line, but it doesn’t work during the normal start up process. I suspect this is to do with the context task initrc_exec_t, but what I’ve tried so far doesn’t work. Someone who understands selinux better than me can probably tell me what I’m doing wrong.

I hope this is useful to someone. Let me know if you find a better way…