December 2009 Archives

Still there after reading this and this?. What I would like to show next is Module::Build::SysPath and the usage for CPAN authors. What are the features?

• pre/post install paths (dev+test/prod)
• system paths configured once when Sys::Path is installed
• "Debian way" of handling configuration files
• automatic all files copying
• creating of empty folders (for state, log, etc. files)

The main reason for existence is to install files (configuration, templates, data files, ...) where FHS recommends and also to know where to put temporary files (like state, lock, cache, ...).

There are couple of stages that the CPAN distribution is going through. Development, smoke testing, pre install and install stage. In all of those it is good to know where our files are. :-P Before the final install the files are always inside the distribution folder, only after install the files are in the system folders. There for the Module::Build::SysPath requires a special module YouModuleName::SPc that gets overwritten with values from Sys::Path once installed.

There are, as always, couple of other ways how to store and work with non-perl modules files. Here are some that I'm aware of:

• storing data inside __DATA__ in .pm file
• storing files in the same folder as .pm files
• storing files in auto/ - File::ShareDir
• hardcoded system folders
• use Sys::Path paths directly
• ???

Everyone can choose.

Reasoning why knowing the system paths is a good idea can be found here. Now let's see how one can guess (get?) them in Perl.

There are a lot of Perl installation made to non-standard paths. Like:

• /usr/local/bin/perl
• /home/$USERNAME/local/bin/perl
• C:\Strawberry\bin\perl
• /opt/bin/perl

I was trying to find out what do this installations share and what could be used to find out if the Perl is in file hierarchy standard folder - /usr/bin/perl. The simplest way, that I found out, was to:

use Config;
if ($Config::Config{'prefix'} eq '/usr') { ... do stuff ... }

That is basically all Sys::Path does. If the installation prefix was set to "/usr" (in all Linux distributions standard Perl packages it is), than the rest of system paths is by default set to FHS. If not all of the paths are based on the prefix - $PREFIX/etc, etc.

Just check the Pod of Sys::Path for all the different paths and their usage.

too simple? but works :-)

Once upon a time I was a young system administrator. Having all the strange looking /usr, /var, /etc, ... all round me was scary and I was not sure what to "do" with all those folder trees. At some point I started to compile the extra programs that I needed. With a default prefix all ended up in /usr/local which looked safe to me. I knew that my stuff is there and the mysterious system stuff is everywhere else.

Well it worked. Having to maintain some more servers later I started to do some packaging. I was using Slackware at that time and the Slackware packaging system was really simple - just a tarball that got extracted to the root of the file system with some scripts that got executed during the package installation time. Simple and worked for me. Still I kept the stuff in /usr/local to be on the safe side.

And the time passed :-) and I'm using Debian now, but most important change is that I've lost all my respect to the file system and I've learned where and why to put files. Why to use /etc for configs, /var/log for logfiles, /var/lib for state files, /var/cache for cache files, /usr/share for templates or static data files, etc.? Simple because it is standard. Because it is standard, standard compliant distribution will stick to it. Besides being standard there are some really good reasons too. Helper tools will understand the files and then act based on the files category. Automatically rotating logs, backup-ing the important (non static distribution) files, cleaning up the temporary files etc. Well and there are also humans out there. Co-admins or newcomers, that will login to the machine and look for files or trouble shoot the programs. Knowing where to look for stuff really helps!

With today advance of virtualization techniques there is no reason to mix too many things (projects) on one server. So there is no reason to play safe with the paths and files should be put to the right place where they belong - FHS.

Have you ever had a Perl distribution extracted and wanted to install all it's dependencies afterwards? You can do it using:

cpan -i Test::Install::METArequires
perl -MTest::iMETAr -le 'Test::iMETAr->t'

The output will be TAP. One test per dependency. Is there some other (better) way to do it? Should be as always...

Reply to two commanets from "when virtualization is too expensive".

1st was from grantm. Thanks for pointing out the OpenVZ. It's much more professional tool than debootstraping and chrooting, but comes with more configuration and setting-up complexity. It's possible to share memory using OpenVZ which is the resource that we never have enough.

2nd was from Alias. Alias is pointing out the deduplication of disk data. Disk space is kind of cheep these days, comparing to memory, bandwidth and cpu processing costs when dealing with huge data. The deduplication can offer, besides saving the disk space, a better cache efficiency, that will increase disk read speed, that is never fast enough. ZFS should soon (first quarter of 2010) support deduplication. It seems that there will be no native ZFS support for Linux because of the licencing problems :-( . Easy Linux deduplication that can come in handy is cowdancer. `cp -al some-folder/ some-folder.copy && cd some-folder.copy && cow-shell` will copy "some-folder" via creating hardlinks which is really fast. Than any write operation under this shell and folder will result in removing the hardlink and creating a new file => copy-on-write. This works good with chroot-ed systems.

Yes the thing that turns your single computer to multiple computers. Why should it be expensive? Simply because the virtual machines are not sharing one disk space and one memory. To make the the virtual system run smoothly, it needs some decent amount of dedicated memory and a disk volume with some reserve so it doesn't have to be resized too often.

First find the reasons for doing virtualization, why would anyone want to run multiple machines on a single hw? Most likely to clearly separate the programs and the whole operating systems. Give the strictly defined virtual hw resources, limit the access for security reasons. And also to add one level of abstraction which then allows systems to live in a cloud. But that is a different topic.

Let's search for the solutions how not to do virtualization and fulfil some (!) of the requirements of it. Mainly the clear files and whole system separation for Perl development.

If just Perl is in the play, then compiling and installing user own Perl is an option. Simply having the Perl binary and all the installed CPAN modules in the $HOME directory of the user.

If Apache is needed or some extra binary libraries, it is still possible to compile and install to the user home, but it is quite a lot of "hand work" and not every one has time and passion to do it. Much more simple way is to use chroot. What chroot does is that it sets root of the filesystem for the child processes to a folder. And as we are in UNIX, where (nearly) anything is a file, this means a different machine. Both systems, the parent and the chroot-ed, still share the same /proc, /dev, network devices etc., but the separation is enough to be able to install programs with standard distribution commands and run them. Fair enough to have chroot-ed machine as a development machine. Benefiting of shared memory and disks pace, easy file sharing (one filesystem) and not having to maintain virtualization sw.

Here is how to create a chrooted system on Debian and switch to it:

debootstrap lenny /usr/chroot/$MACHINENAME
echo ${MACHINENAME}_chroot > /usr/chroot/$MACHINENAME/etc/debian_chroot
chroot /usr/chroot/$MACHINENAME su -