Playing around with apt-cpan and apt-pm made me wonder how to get them installed on fresh clean system without installing too many dependencies - preferably nothing at all. So my concern was - How to get a bunch of Perl scripts to the OS without affecting it? Then I've recalled hearing about pp and gave it a try. The result? => Alien::Debian::Apt::PM

Alien::Debian::Apt::PM is basically one huge (10MB) binary - alien-debian-apt-pm and a custom Build.PL that will create all the necessary symlinks once the basic binary is installed. Alien::Debian::Apt::PM has no dependencies besides Module::Build 0.36 so the system stays clean even when installed via CPAN shell. Other way how to "install" is to simply copy the binary to the system path and create those symlinks. See the distribution Pod for details how to install and use.

Subject: YAPC::Europe 2011 - Talk & Tutorial Evaluations
Date: Wed, 21 Sep 2011 15:04:10 +0100
To: Jozef Kutej <jozef@kutej.net>

Hi,

In this email, please find the feedback received from the surveys available for
YAPC::Europe 2011. 

An Explanation of the Tables:

For the table matrix below, each row represents the question asked of the respondee,
with the columns representing the rating given, graded from 1 to 10, where 1 represents
a low rating and 10 a high one. The values in each cell represent the number of
respondees who rated the question with a particular value.

Due to space restrictions, the text of each question has been truncated. The full text for
the questions are as below:

* Your prior knowledge of subject?
* Speaker's knowledge of subject?
* Speaker's presentation of subject?
* Quality of presentation materials?
* Overall presentation rating?


Demystify file system hierarchy for deployments:

-----------------------------------------------------------------------
| Questions         |  1 |  2 |  3 |  4 |  5 |  6 |  7 |  8 |  9 | 10 |
-----------------------------------------------------------------------
| Your prior knowle |  - |  - |  - |  - |  3 |  - |  - |  2 |  - |  - |
| Speaker's knowled |  - |  - |  - |  - |  1 |  - |  - |  2 |  - |  2 |
| Speaker's present |  - |  - |  - |  1 |  2 |  - |  - |  2 |  - |  - |
| Quality of presen |  - |  - |  - |  2 |  1 |  - |  - |  2 |  - |  - |
| Overall presentat |  - |  - |  - |  1 |  2 |  - |  - |  2 |  - |  - |
-----------------------------------------------------------------------

Regards,
Barbie.
--
YAPC Conference Surveys <http://yapc-surveys.org>
QA Hackathons <http://qa-hackathon.org>

My goal was to have a single command to use to install Perl modules that will prefer distribution packages over installing from CPAN.

Combining 02packages.details.txt.gz with data from MetaCPAN::API I've generated 02packages.dependencies.txt.gz. This file has similar structure, basically it is a list of module names with all the dependencies it has. Here are example lines:

DBI     1.616   Clone/0.31 DBD::AnyData/0.09 DBD::CSV/0.29 DBD::PO/2.10 DBD::RAM/0.072 DB_File ExtUtils::MakeMaker ExtUtils::MakeMaker/6.48 MLDBM Net::Daemon RPC::PlServer/0.2001 SQL::Statement/1.27 SQL::Statement/1.28 Test::Simple/0.90 perl/5.008
Moose   2.0205  Algorithm::C3 DBM::Deep Data::OptList/0.107 Data::Visitor DateTime DateTime::Calendar::Mayan DateTime::Format::MySQL Declare::Constraints::Simple Devel::GlobalDestruction Devel::PartialDump/0.14 Dist::CheckConflicts/0.02 Eval::Closure/0.04 ExtUtils::MakeMaker/6.30 File::Find::Rule HTTP::Headers IO::File IO::String List::MoreUtils/0.28 Locale::US MRO::Compat/0.05 Module::Info Module::Refresh Package::DeprecationManager/0.11 Package::Stash/0.21 Package::Stash::XS/0.18 PadWalker Params::Coerce Params::Util/1.00 Regexp::Common Scalar::Util/1.19 Sub::Exporter/0.980 Sub::Name/0.05 Task::Weaken Test::Deep Test::DependentModules/0.09 Test::Fatal/0.001 Test::Inline Test::LeakTrace Test::More/0.88 Test::Output Test::Requires/0.05 Try::Tiny/0.02 URI perl/v5.8.3
MetaCPAN::API   0.33    Any::Moose Carp ExtUtils::MakeMaker/6.30 File::Find File::Temp HTTP::Tiny JSON Module::Build/0.3601 Test::Fatal Test::More Test::TinyMocker Try::Tiny URI::Escape
Pod::POM::Web   1.17    Alien::GvaScript/1.021 AnnoCPAN::Perldoc::Filter Config Encode::Guess HTTP::Daemon List::MoreUtils List::Util MIME::Types Module::Build/0.38 Module::CoreList POSIX PPI::HTML Pod::POM/0.25 Pod::POM::View::HTML Search::Indexer/0.75 Test::More Time::HiRes URI URI::QueryParam
Pod::POM::Web::Indexer  0       Alien::GvaScript/1.021 AnnoCPAN::Perldoc::Filter Config Encode::Guess HTTP::Daemon List::MoreUtils List::Util MIME::Types Module::Build/0.38 Module::CoreList POSIX PPI::HTML Pod::POM/0.25 Pod::POM::View::HTML Search::Indexer/0.75 Test::More Time::HiRes URI URI::QueryParam
Pod::POM::Web::PSGI     0.002   CGI::Emulate::PSGI/0.12 ExtUtils::MakeMaker/6.30 Pod::POM::Web/1.17

So basically it's possible to download this file and find out offline the dependency chain of a module.

Which I did in apt-cpan. With help of Debian::Apt::PM if is possible to preferably install packages from Debian distribution repositories and only the missing modules via CPAN shell.

Here is an example:

$ apt-cpan -n install Nagios::Plugin::OverHTTP
sudo apt-get install libdata-validate-domain-perl libdata-validate-uri-perl libenv-path-perl libhtml-strip-perl libmoosex-clone-perl
sudo cpan -i Nagios::Plugin::OverHTTP Const::Fast

Just uploaded WWW-Tracking Perl distribution to the CPAN:

Goal:

Server-side web hits tracking, generic and flexible enough so that many tracking services like Google Analytics, Piwik, local file, etc. can be used depending on configuration.

Vision:

Universal enough to process many sources (headers, logs, tracking URL-s, ...) and covert or relay them to different other destinations (GA, Piwik, ...) making use of the fact that the tracking data information is the same or nearly the same for all the sources and destinations.

Implementation:

Initially tracking data needs to be gathered. Look at WWW::Tracking::Data for the complete list. Most of these data can be found in headers of the http request. Then these data can be serialized and passed on to one of the tracking services.

Bare WWW::Tracking::Data offers just as_hash and from_hash, the rest can be done by one or more plugins, like for example parsing the http headers with WWW::Tracking::Data::Plugin::Headers and passing it to WWW::Tracking::Data::Plugin::GoogleAnalytics.

Use cases:

• tracking browsers that doesn't support JavaScript (ex. mobile browsing)

• store the tracking data in local logs or files and replay it later to Piwik or Google Analytics

• track web browsing simultaneous to more tracking services (compare the results, choose the one that fits)

• aid with transition from one tracking service to another

Slides from my yesterdays talk are here, uploaded to Bratislava.pm.org page.

My thanks go to everyone who participated on this great conference and helped this event happen!

Let this blog post be also an invitation to my talk that I'll have on this years YAPC::EU::2011 in Riga and a reason "Why?" there will be more questions then answers.

There is one question that keeps me busy for quite some months now already and that is: What if the knowledge and wisdom is not in answers, but in questions?

As it kept me busy for just too long and as I'm a full-time Perl programmer, I've decided to write a code to help me answer "Why?".

package Acme::KnowledgeWisdom;

our $VERSION = '0.01';

use Moose;
use warnings FATAL => 'all';

has 'in_questions' => ( isa => 'Bool', is => 'ro', default => 1);
has 'has_already'  => ( isa => 'Bool', is => 'ro', default => 0);

sub get {
    my $kw = shift;
    
    return $kw->ask
        if $kw->in_questions;

    return 42;
}

sub ask {
    my $self = shift;
    
    return 42
        if $self->has_already;
    
    my $kw = Acme::KnowledgeWisdom->new;
    return $kw->get;
}

1;

The module is well tested ;-) and is of course on CPAN.

It's now more then one week since Perl QA Hackathon 2011 in Amsterdam that I've attended. I'll first write about the work part or the official reason why we all went there.

In the beginning every attendee wrote down a plan and then the results at the end after sitting in a meeting room with a laptop and a bunch of hacking chums™ for three days.

My goal in the beginning was to do something related to Debian-Perl. I've asked at the mailing list and got an answer from Gregor that there is an open tasks list and that providing the Debian <-> CPAN mapping would be also useful. I decided for the latter and created a web page with a look-up service:

http://pkg-perl.alioth.debian.org/cpan2deb/

How does it works? The page is a result of three scripts - dpkg-scanpmpackages, apt-pm and apt-pm-web and the Debian::Apt::PM module behind them. While only apt-pm-web was not existing before the hackathon the module also needed some love in order to be able to get the distribution name and the component information in the output too. Here is how the scripts works:

dpkg-scanpmpackages /path/to/repository-or-deb-mirror

will find all Packages.bz2 files in the given path, parse them, extract the .deb package files, look-up for all .pm files inside, analyse them for Perl package names and write all this informtion to the same folder into the PerlPackages.bz2 file. (here's example) Each .deb file has an own section in this file like:

Package: libcontextual-return-perl
Architecture: all
Filename: pool/main/libc/libcontextual-return-perl/libcontextual-return-perl_0.003001-1_all.deb
Version: 0.003001-1
Distribution: stable
Component: main
Perl-Modules: 
 Contextual::Return::Failure (0)
 Contextual::Return (0.003001)

apt-pm update

will scan all repositories (all components) listed in /etc/apt/sources.list and fetch the PerlPackages.bz2 files. There are special empty package repositories with only PerlPackages created that can be safely added to the sources.list:

deb http://pkg-perl.alioth.debian.org/~jozef-guest/pmindex/     lenny   main contrib non-free
deb http://pkg-perl.alioth.debian.org/~jozef-guest/pmindex/     squeeze main contrib non-free
deb http://pkg-perl.alioth.debian.org/~jozef-guest/pmindex/     wheezy       main contrib non-free
deb http://pkg-perl.alioth.debian.org/~jozef-guest/pmindex/     sid          main contrib non-free
deb http://pkg-perl.alioth.debian.org/~jozef-guest/pmindex/     experimental main contrib non-free

or even local file:// non-public repository will work fine. Then using `apt-pm find Module::Name` a module can be looked-up, if it is already present in any of the specified distributions/components.

apt-pm-web

will loop through all the modules listed in 02packages.details.txt.gz (on CPAN) and generate index .json files to be used on the web. These .json files with all the rest of the web files are then copied to alioth. These page is basically a page with single ajax form input field. The dynamic part is written in JavaScript in script.js. Once the form is submitted, the value is md5-ed, first two letters are used for the name of the json file which is loaded and if the module name is present the results are show on the page. This way there is no need for server-side code to run and all that alioth (webserver) has to do is to shoot static files. Also note the the url looks like this http://pkg-perl.alioth.debian.org/cpan2deb/#q=Tk::Widget where the "q" (query) parameter is delimited via "#" instead of normal "?" resulting in no page refresh when changing the url or resubmitting the form.

So what's there for you?

Now there is an easy way how to look-up Perl modules in Debian packages on the web. And there is a way to do this for a local non-public repositories too. In case you have local repository that includes packaged Perl modules and would like to have this page locally, here is a how-to guide:

1. cpan -i Debian::Apt::PM
2. editor /etc/apt/sources.list # add one/some of the above repositories and your local one
3. dpkg-scanpmpackages /path/to/local-repository
4. apt-pm update
5. apt-pm find DBI # to test if all works fine
6. git clone https://github.com/jozef/Debian-Apt-PM.git /tmp/dapm
7. cp -r /tmp/dapm/examples/web /srv/cpan2deb
8. perl -MCPAN -MCPAN::Shell -le 'CPAN::Shell->reload("index")'
9. apt-pm-web -p $HOME/.cpan/sources/modules/02packages.details.txt.gz -o /srv/cpan2deb'
10. # make the /srv/cpan2deb folder available via your favourite web server

Actually people from Ubuntu community could find use for this and create such a page for the many Ubuntu distributions that are out there. (I can not do this as I don't have a disk space and CPU to process such a huge mirror. Making it work without the need for all the .deb files that could be deleted after indexing would help, but I don't have the time to do this neither)

To be continued...

After chatting a bit at #tripwolf irc channel and having fun about the kids I've created following "Baby simulator":

perl -le 'my $FOOD_INTERVAL = 4; my $SLEEP_INTERVAL = 8; my $MAX_RESERVE = 10; while (1) { $SIG{ALRM} = sub { warn "crying\n"; system("echo ".("crying"x1000)." > /dev/dsp"); alarm 1; $reserve--; die "paternity-over => please try again :)\n" if not $reserve; }; warn "waveing and crowing\n"; $reserve = $MAX_RESERVE ; alarm $FOOD_INTERVAL; my $stomach = <STDIN>; alarm 0; print "zzzz"; sleep $SLEEP_INTERVAL; }'

The constants has to be adjusted to a current age of a kid to properly simulate real-life conditions. ;-)

Disclaimer: This software is compatible only with Linux. Upgrade your OS to a supported one® if necessary.

It's too late for Pepl to use it as a training program as his baby was born on 25th of February. I'm sending out this way congratulations to Pepl and Daniela. I wish them that their daughter Ria Johanna will bring a lot of joy and happiness to their life!

Accepted Grants for 2010Q3 doesn't make a clear border line and I got 2x congratulations that my grant was accepted. The truth is only the first three each quarter are. I just "scored" at the last 5th, non-rejected place.

When I showed Benjamin (a college of mine) my TPF 2010Q3 grant proposal, we slipped to a discussion about deploying Perl software. Nice (and recent) list of different approaches can be found @modernperlbooks.com. To sum it up => TIMTOWTDI. Which is good, but none of those is perfect. The Perl+CPAN world is way too complex.

During out discussion with Benjamin I proposed an idea of shipping the application altogether with the OS. Insane? The base Debian system is ~190MB, all the rest is needed for the application. Then deployment will be a matter of running this system on a virtual machine, somewhere in the cloud or in a simple chroot. (btw any Linux distribution can have any other Linux distribution working in a chroot) The files will never clash, all the "machines" would be dedicated. No fear of putting files where they belong to.