Migrating CentOS5 VMWare images with LVM to KVM

I’ve decided that using VMWare Server has been a useful and worthwhile experience, but I want to do stuff which would involve upgrading to one of the paid-for versions of VMWare and I don’t want to pay.

So, I’ve set up a new server to host my VMs on Fedora 14, and am migrating to using KVM instead.

I had expected that I’d have to backup the image and then recover it, but no, it seems that the open source community has done everything I need. I started here and noted the slightly scary caveat that LVM using images didn’t work.

I tried anyway, and sure enough, I got a message saying

Volume group "VolGroup00" not found
Unable to access resume device (/dev/VolGroup00/LogVol01)
mount: could not find filesystem '/dev/root'

It had to be possible. I asked at LinuxQuestions.org, and the reply led me to look at virt-p2v, which was a possible fallback option, but rather scary since there aren’t any hosted pre-build iso files any more.

However, reading some of the blurb on the author’s blog about manual p2v he mentioned possibly having to edit the fstab file to point the mounts in the right direction following virtualisation.

I had loaded the converted qcow2 image, using Virtual Machine Manager. When I looked at the setup I noticed that the disk setup looked like this:

Properties page for VirtIO disk.

I Googled VirtIO and found that it was a para-virtualised disk type, which meant that it had to be supported by the guest OS. This would explain why the Fedora 14 rescue disk could see the LVM but not the actual operating system.

I deleted the disk entry and replaced it with this:

Properties page for new IDE drive

Now, I started the guest, and it booted!

Text from successful boot image.

Upon booting, the X-window display didn’t work, but the automatic rescue process worked, and I got a gnome login screen. The next problem was that the mouse didn’t work properly. The Virtual Machine Manager does suggest that you add a tablet to the machine to ensure that the mouse matches the VNC mouse, but it still didn’t work after doing that. However, yet again, someone had done the work first, and I found the answer here. Slightly messy editing the xorg.conf file, but it worked.

The Disadvantages of not Keeping Current

I have been requested by a client to produce data extracts from an old bespoke system, which has been knocking about for many years and been left without upgrades for several years. A nice piece of work for me, but it’s got me thinking about the whole upgrade lifecycle issues and bespoke vs packaged solutions in general.

The reasons it’s not been upgraded are that “it ain’t broke – don’t fix it”, and that nobody wants to pay the costs to keep regression testing new upgrades.

However, there are disadvantages to letting a system continue on old techonolgy for a long period of time. Maintenance gets more difficult as fewer people are available who remember the older technology. The world moves on and the system doesn’t, so it looks antiquated, and leads users to feel irritated by the inadequate user-interface.

The net result is that a new proprietary packaged system is going to be deployed replacing the creaking bespoke system, at considerable cost. However, as with all packaged solutions, the “out of the box” fit with the business requirement is good, but not perfect. Ok, the bespoke system doesn’t fit with the business 100% either, since the business has moved on and it hasn’t, but I can’t help thinking that had the bespoke system been periodically upgraded to use the latest (or at least more recent) technologies, and the changes to the requirements, that the requirements fit could have been better and the overall cost less than the new replacement system.

A final disadvantage of remaining on older techonolgy, comes when carrying out the extract of data for migration, you realise that “that approach doesn’t work on that version, you need the next one”. Realising that several approaches aren’t possible due to the features I’m now used to being for more recent versions, it becomes frustrating to have to rethink each time.

Decimal keypad in iPhone SDK – Well Done Apple!

A few days ago I was trying to find how to get a decimal point in the number pad for an app I was writing. Having reviewed a number of discussions on the subject which suggested subclassing the keypad or adding extra buttons and all sorts of scary things, I finally resorted to the manual.

In the list of available keypad types, it turns out that there is actually a new one in SDK 4.1. Unfortunately, at present, they haven’t updated Interface Builder with the new values, so you can’t select the new values directly but you can programatically, like this:

textField.keyboardType = UIKeyboardTypeNumberPad;

so clearly they have listened to the needs expressed in these various articles, but unfortunately, this is rather well hidden in the documentation, and until it appears in the keyboard type picklist in Interface Builder, will remain so.

So well done Apple for adding UIKeyboardTypeNumberPad, but please can you add it to Interface Builder too!

Paging Enabled peculiarity in UIScrollView

Whilst developing an iPhone app, I have come some very odd behaviour in UIScrollView. Ok, I’ll admit what I was doing was somewhat unusual, but I think the issue could occur in less unusual circumstances.

The effect appeared to be that the UIScrollView was remembering the last user scroll page position, despite my changing the frame, contentSize and contentOffset of the view manually. If the user touched the scroll view after my programmatic changes, the view would immediately scroll back to the page that had last been user scrolled to.

Consider the following situation: we start with the scrollview (and content) looking like

[1 2 3 4] 5 6 7 8

where the brackets denote the visible page area. I manually scroll to show the second page:

1 2 3 4 [5 6 7 8] 

which works fine. I then change to “selected” mode programmatically, and the display now shows:

1 2 3 4 5 6 [7] 8

again, which works fine.
However, if I now tap on the UIScrollView it immediately scrolls to

1 2 3 4 [5] 6 7 8

for no apparent reason.

This was going to be a complete show stopper for me, so I spent quite a long time delving into the mysteries of UIScrollView, and to cut a long story short, I think I’ve found a bug. I’ve opened a bug report with and submitted a test case, but not heard anything yet, so I’ll update this post as and when they confirm or refute it.

However, the good news is that while investigating what was causing the problem, having guessed at the cause, which I think is a rounding error between two different member variables, responsible for handling the paging enabled mode, a workaround idea presented itself.

What I discovered was, that if the content view width was integer divisible by the number of pages (which in this case was 2 or 8 pages wide), the problem didn’t happen, whereas if the content size divided by the number of pages resulted in a fraction, the issue occurred. Thankfully, this was a fairly easy workaround to implement, and perhaps explains why it hasn’t shown up often enough for apple to have found it previously.

Deserializing serializable Java objects without a default constructor

I’ve spent the past few days implementing a serialisation process overriding the ObjectOutputStream and ObjectInputStream classes. However there seemed to be certain classes that cannot be deserialized even though they are serializable, as they have no default constructor.

The serialisation class documentation implies that the constructor for a serializable class isn’t called anyway, which makes sense as you don’t really want to set up a class from scratch when you are deserializing, just restore it.

Most replies to questions on this subject seemed to imply that it can’t be done except by the default serialization classes. This is no use if you want to override these, and since they are overridable, surely Sun woudn’t have made it impossible.

One hit during my search mentioned the ReflectionFactory class in passing, but didn’t have any detail, so I looked into it further, and sure enough, this seems to be the answer.

The Solution?

It would seem that the ReflectionFactory class can create a Constructor object on the fly, using the default constructor of the first non-serializable superclass, and the class you’re trying to instanciate.

So when overriding the deserialisation process, you can create an instance of an object that doesn’t use the default constructor, and doesn’t need one to exist by calling the constructor returned by something like the following:

ReflectionFactory rf = ReflectionFactory.getReflectionFactory();
Class type = HashMap.class; //
Class nonSerType;
for ( nonSerType = type; Serializable.class.isAssignableFrom(nonSerType); nonSerType = nonSerType.getSuperclass() ) {}
Constructor superCon, deSerCon;
superCon = nonSerType.getConstructor();
deSerCon = rf.newConstructorForSerialization(type, superCon);

where the deSerCon can be used to instanciate the deserialized object.

This process seems to work for my deserialization work, and so I’ve documented it here. I’ll admit, I’ve no idea if there are any specific limitations to this process, but for the limited range of classes I’ve tested so far, it works fine.

Server Side XSLT2 with Apache, Tomcat and Saxon

I have spent several days trying to figure out how to do server side XSLT using an Apache server using numerous suggested ways from articiles on the web about how to do it using Cocoon, and using mod-xslt2, all of which are out of date in one way or another, i.e. mod-xslt2 doesn’t work with Apache 2.2 yet, and the article describing how to do it with Cocoon used a previous version of Cocoon so didn’t help much with the current one.

However, I have now managed server side XSLT2 transformation using Apache, Tomcat, mod-jk, mod-rewrite and Saxon as the transformation engine.

The Theory

I noticed in the Saxon documentation, that one of the demos was a simple servlet. So, running this servlet on Tomcat, I should be able to do the transformations on the server.

The downside of this approach is that I was going to end up with really nasty looking URLs, such as

http://www.screwtape.co.uk/Servlets/SaxonServlet?source=xml/homePage.xml&style=xsl/main.xsl

which is hardly convenient when writing links, and while that could be automated in the xsl transformation, it would still look ugly in the browser address or status bars.

However, Apache has mod-rewrite which does lots of exciting things with URLs. This can rewrite any incoming URL to poke the relevant parameters into the servlet.

The Practice

Install Apache, Tomcat and mod-jk. There’s loads of instructions for that on the web, and anything I say will have been better said elsewhere so I won’t.

Everything here is rather distibution dependant, so your system may be slightly different, but the ideas should be similar. I’m using Centos 5.2 here.

  1. Download and install SaxonB 9.1 for Java. Make sure you also get resources package, as that has the servlet sample.
  2. Complile the SaxonServlet.java file, and create a .war file. To do this, I created a web application in Netbeans 6.5 and got it to do all the compiling and packaging etc.
  3. Using the tomcat manager app, add and release the .war file.
  4. Place or link your xml and xsl files in the servlet directory, so the servlet can access them. I have /usr/share/tomcat5/webapps/Servlets/xml and /usr/share/tomcat5/webapps/Servlets/xsl linked to my webserver directories, although you could have the xml and xsl files just in the servlet directory.
  5. I use the default virtual host, and hold the majority of the website files in the apache html directory, therefore you can’t just use the auto-generated virtual host file.Update the /usr/share/tomcat5/conf/servers.xml file: the <Host> needs some <Alias> tags to allow Tomcat to see your website when not accessed through localhost. The host section look something like this:
    <Host name="localhost" appBase="webapps"
        unpackWARs="true" autoDeploy="true"
        xmlValidation="false" xmlNamespaceAware="false">
        <Alias>www.screwtape.co.uk</Alias>
        <Alias>screwtape.co.uk</Alias
        ...
    </Host>
  6. Restart your tomcat instance, and this will create the “auto” subdirectory in the /usr/share/tomcat5/conf directory.
  7. Look through the mod_jk.conf in the auto directory, and you should see a pair of lines starting JkMount. Copy these lines into the main httpd.conf file area, if like me you’re not using virtual hosts, or into the relevant <VirtualHost> element. You should have something like this:
    JkMount /Servlets ajp13
    JkMount /Servlets/* ajp13
  8. Restart httpd, then we can test that you can run the servlet from Apache. So from your browser try
    http://localhost/Servlets/SaxonServelt

    This should show an error message about missing style parameter. If you get this message then Apache and Tomcat are linked and you can proceed.

    It is probably a good idea to try and access the servelet from other hostnames/domain names that might use it, since it may work for localhost and no other hosts. Thus trying

    http://<yourhostname>/Servlets/SaxonServlet

    may be a good idea too.

  9. Now we need to rewrite the incoming URLs to use the servlet. I have all my URLs beginning xml/ rewritten to use the servlet, with a single xsl transformation. If you need different xsl files, then this will be slightly more tricky, and will need a separate rewrite rule for each xsl file, or a more complex servelet.Add the RewriteRule directives in the <Directory element appropriate for your site. I have
    <Directory "/var/www/html">
        ...
        RewriteEngine       on
        RewriteRule         (^xml/.*$)      /Servlets/SaxonServlet?source=$1&style=/xsl/main.xsl
    </Directory>
  10. Restart httpd and assuming everything is ok, you should get your xml documents transformed using the specified xsl. That’s how this page works, as it’s entirely written in xml.

I hope this is useful to someone, and if anyone has a better way of doing it I’ll be interested to know. One of my friends insists PHP is the way to go, but I can’t see that being any easier than this.

Setting up fetchmail daemon on Centos 5.3

Unfortunately, I failed to find any suitable instructions for doing this when I was looking to set up a single fetchmail daemon for Centos. It appears this is easy on debian flavours, and perhaps there are better ways of doing it than this, but I thought I’d share the method I used…

I’ve kept the steps fairly simple, so please forgive me if some of it is completely obvious…

1 – Install fetchmail if not already installed

Check if fetchmail is already installed, and perhaps update it if it is out of date.

yum list fetchmail

will show you if you’ve already got it installed, and which versions are available for install. I used the x86_64 package version 6.3.6-1.1.el5

Use yum to install or update this as required.

2 – Configure fetchmail

Why is nothing ever simple under Centos? I tried to find a RPM fetchmailconf, the gui fetchmailrc file configurator, for Centos 5 but couldn’t find one for the current version of fetchmail.

I ended up with fetchmailconf-6.3.4-1.1_6.0.1.el5.x86_64.rpm which wouldn’t install as it depends on the fetchmail-6.3.4 rather than the 6.3.6 version I had installed, so I had to install it using

rpm -i --nodeps fetchmailconf-6.3.4-1.1_6.0.1.el5.x86_64.rpm

This at least worked, and so I used it to configure my .fetchmailrc file for my mail servers etc. I then copied the ~/.fetchmailrc file to /etc/fetchmailrc for use by the daemon.

3 – Set up a script in /etc/init.d.

I largely copied the script for the Samba daemon.

#!/bin/sh
#
# chkconfig: - 91 35
# description: Starts and stops fetchmail in daemon mode
#

# Source function library.
if [ -f /etc/init.d/functions ] ; then
  . /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
  . /etc/rc.d/init.d/functions
else
  exit 1
fi

# Avoid using root's TMPDIR
unset TMPDIR

# Check that fetchmailrc exists.
[ -f /etc/fetchmailrc ] || exit 6

RETVAL=0

start() {
        KIND="fetchmail"
	echo -n $"Starting $KIND services: "
	daemon --user fetchmail fetchmail -f /etc/fetchmailrc --syslog
	RETVAL=$?
	echo
	return $RETVAL
}	

stop() {
        KIND="fetchmail"
	echo -n $"Shutting down $KIND services: "
	runuser fetchmail -c 'fetchmail --syslog --quit >dev/null 2>&1'
	RETVAL=$?
	[ "$RETVAL" -eq 0 ] && success $"$base startup" || failure $"$base startup"
	echo
	return $RETVAL
}	

restart() {
	stop
	start
}	

reload() {
        echo -n $"Reloading fetchmailrc file: "
	RETVAL=$?
	echo
	return $RETVAL
}	

rhstatus() {
	status fetchmail
	RETVAL=$?
}	

# Allow status as non-root.
if [ "$1" = status ]; then
       rhstatus
       exit $?
fi

# Check that we can write to it... so non-root users stop here
[ -w /etc/fetchmailrc ] || exit 4

case "$1" in
  start)
  	start
	;;
  stop)
  	stop
	;;
  restart)
  	restart
	;;
  reload)
  	reload
	;;
  status)
  	rhstatus
	;;
  *)
	echo $"Usage: $0 {start|stop|restart|reload|status}"
	exit 2
esac

exit $?

You can download this file by right clicking here.

4 – Set up a fetchmail user.

Create a user on the system called fetchmail. I used the -r option with useradd but then I had to create a home directory as the fetchmail program expects one. You can set up an alternative directory using an environment variable, but I took the easy way and just created a home directory.

You then chown the /etc/fetchmailrc file to fetchmail, and the home directory.

5 – Start it up…

Now try and start the service.

service fetchmail start

With any luck, the service starts… I suggest you then try and stop it with

service fetchmail stop

to ensure the stop code works too.

Next steps…

I then added the service using the gui services control, and this works ok, allowing me to start, stop, and configure the service for automatic startup on level 5.

However, selinux is doing something I don’t quite understand. It works using the service command from the command line, but it doesn’t work during the normal start up process. I suspect this is to do with the context task initrc_exec_t, but what I’ve tried so far doesn’t work. Someone who understands selinux better than me can probably tell me what I’m doing wrong.

I hope this is useful to someone. Let me know if you find a better way…