[draft] list of all CPAN object/package lazy loaders

Discussion:

Terrence Brannon

2005-01-19 00:02:37 UTC

I think a list of modules which dynamically load packages and/or
create objects at runtime would be useful. Please add any modules that
you know of that I overlooked... also feel free to write a
comprehensive assessment of these modules (grin).

NAME
Catalog::lazy_loaders - list of all CPAN object/package lazy loaders

DESCRIPTION
A lazy loader allows a program to dynamically (i.e., at run-time, on
demand) load a package. I have seen several such modules make their way
onto CPAN and thought it would be beneficial to have a list of all of
them somewhere.

Ideally the features of these modules would be evaluated, but cataloging
the modules is a valuable first step.

RELATED MODULES
* Object::RealizeLater by Mark Overmeer
<http://search.cpan.org/dist/Object-Realize-Later/>

* Class::Autouse by Adam Kennedy
<http://search.cpan.org/~adamk/Class-Autouse-1.14/>

* AnyLoader by Michael Schwern
<http://search.cpan.org/search?dist=anyloader>

* Class::Loader by Vipul Ved Prakash
<http://search.cpan.org/dist/Class-Loader/>

Note: this module got slagged on cpan ratings:

<http://cpanratings.perl.org/d/Class-Loader>

* load by Elizabeth Mattijsen
<http://search.cpan.org/dist/load>

* Module::Load::Conditional by Jos Boumans
<http://search.cpan.org/dist/Module-Load-Conditional>

* Class::Dynamic by Simon Cozens
<http://search.cpan.org/dist/Class-Dynamic>

* require
I suppose require in the Perl core deserves an honorable mention.

SEE ALSO
For further discussion of this topic, please comment at the Perl
software design mailing/GMANE list:

<http://www.metaperl.com/sw-design>

AUTHOR
Terrence Brannon, <***@metaperl.com>

COPYRIGHT AND LICENSE
Copyright (C) 2005 by Terrence Brannon

This library is free software; you can redistribute it and/or modify it
under the same terms as Perl itself, either Perl version 5.8.4 or, at
your option, any later version of Perl 5 you may have available.

--
Carter's Compass: I know I'm on the right track when,
by deleting something, I'm adding functionality.

Adam Kennedy

2005-01-19 00:59:54 UTC

Permalink

You can add another one of mine to this list.

* prefork by Adam Kennedy
L<http://search.cpan.org/~adamk/prefork-0.03/lib/prefork.pm>

This is more of a meta-module for this list.

It is (ultimately) intended to provide a central point of control across
all of the various module loaders, so that in normal cases you autoload,
but in preforking scenarios you load early to take full advantage of
copy-op-write.

Basically, I wanted the best memory use for the same code under both CGI
and mod_perl.

Simple example
--------------

use prefork 'CGI';

if ( $happens_ten_percent_of_the_time ) {
require CGI;
print CGI->something;
}

--------------

If you run that code as a CGI, the C<use prefork> line is a null-op.

If you run if under mod_perl (or something else triggers the preforking
scenario) the module is loaded in early.

And it can handle delayed preforking, you don't have to know at compile
time which way to go.

Eventually I want it to support ALL of the autoloading modules, but for
now is basically provides the following.

* Autodetect mod_perl and enable preforking

* preloads explicitly specified modules (the example)

* preloads any delayed Class::Autouse modules

* Extensible enough to hook your own autoloader thingies into it

Adam

Post by Terrence Brannon
I think a list of modules which dynamically load packages and/or
create objects at runtime would be useful. Please add any modules that
you know of that I overlooked... also feel free to write a
comprehensive assessment of these modules (grin).
NAME
Catalog::lazy_loaders - list of all CPAN object/package lazy loaders
DESCRIPTION
A lazy loader allows a program to dynamically (i.e., at run-time, on
demand) load a package. I have seen several such modules make their way
onto CPAN and thought it would be beneficial to have a list of all of
them somewhere.
Ideally the features of these modules would be evaluated, but cataloging
the modules is a valuable first step.
RELATED MODULES
* Object::RealizeLater by Mark Overmeer
<http://search.cpan.org/dist/Object-Realize-Later/>
* Class::Autouse by Adam Kennedy
<http://search.cpan.org/~adamk/Class-Autouse-1.14/>
* AnyLoader by Michael Schwern
<http://search.cpan.org/search?dist=anyloader>
* Class::Loader by Vipul Ved Prakash
<http://search.cpan.org/dist/Class-Loader/>
<http://cpanratings.perl.org/d/Class-Loader>
* load by Elizabeth Mattijsen
<http://search.cpan.org/dist/load>
* Module::Load::Conditional by Jos Boumans
<http://search.cpan.org/dist/Module-Load-Conditional>
* Class::Dynamic by Simon Cozens
<http://search.cpan.org/dist/Class-Dynamic>
* require
I suppose require in the Perl core deserves an honorable mention.
SEE ALSO
For further discussion of this topic, please comment at the Perl
<http://www.metaperl.com/sw-design>
AUTHOR
COPYRIGHT AND LICENSE
Copyright (C) 2005 by Terrence Brannon
This library is free software; you can redistribute it and/or modify it
under the same terms as Perl itself, either Perl version 5.8.4 or, at
your option, any later version of Perl 5 you may have available.

Chris Winters

2005-01-19 01:04:15 UTC

Permalink

Class::Factory

Chris

--
Chris Winters
Creating enterprise-capable snack systems since 1988

Stevan Little

2005-01-19 01:17:24 UTC

Permalink

Terrence,

There is also Class::LazyLoad
<http://search.cpan.org/~rkinyon/Class-LazyLoad/> which Rob Kinyon and
I wrote (actually 99% of it was Rob, I just wrote some of the test
suite and added a few features). It is basically meant to be a very
very lightweight alternative to things like Object::RealizeLater and
Class::LazyObject (speaking of which you should add Class::LazyObject
<http://search.cpan.org/~daxelrod/Class-LazyObject/> to your list).

BTW - What's with the Catalog::* namespace? You building something
interesting you want to share with the rest of the group? :)

- Steve

Adam Kennedy

2005-01-19 01:27:40 UTC

Permalink

Oh, and you sparked a memory.

LazyLoader.pm was an older one Schwern and I tried to build a couple of
years ago which in theory could turn ANY code into lazyloading code by
rewriting it on the fly at install time. :/

I believe he ran it on his entire perl install once and found an across
the board 30%+ improvement.

Unfortunately, it got bogged down with closures and we could find a way
to solve it that didn't involve parsing Perl without perl, which was
impossible at the time ;)

It might bet reborn at some point once PPI is done, but I've included an
old snapshot for posterity.

Adam

-----------------------------------------------

package LazyLoader;

$VERSION = 0.01;
@EXPORT = qw(lazify);
@ISA = qw(Exporter);

use strict;
require Exporter;

our $MAXIMUM_IGNORE_SIZE = 100; # Set this to something better

sub lazify {
my $file = shift;

# Does the file exist, and can we read and write it
return undef unless -f $file;
return undef unless -r $file;
return undef unless -w $file;

# Slurp in the file
my $code;
{ local $/ = undef;
open( LAZYFILE, $file ) or return undef;
$code = <LAZYFILE>;
close LAZYFILE;
}

sub lazify_content {
my $content = shift;

# Use a basic state filter
my @out = ();
my $in_lazy_sub = 0;
my @lazy_sub = ();
my $lazy_count = 0;
my $line_num = 0;
my $starting_line = 0;
foreach my $line ( split /(?:\015{1,2}\012|\015|\012)/, $code ) {
$line_num++;
if ( $in_lazy_sub ) {
# Is this the last line?
if ( $line =~ /^}\s*$/ ) {
push @lazy_sub, $line;

# Do we need to lazify this?
if ( length(join '', @lazy_sub) > $MAXIMUM_IGNORE_SIZE ) {
# If this is the first time we've lazified something,
# define the warning sub we need to use.
unless ( $lazy_count++ ) {
push @out, "my \$__lazy_warn = sub { warn \@_ unless join('',\@_)
=~ /^Subroutine \$LazyLoader::name redefined at/; };";
}

# Add the lazified code
_add_lazified_sub( \@out, \@lazy_sub,
$starting_line, $line_num )
or return undef;
} else {
push @out, @lazy_sub;
}

# Reset
$in_lazy_sub = 0;
@lazy_sub = ();
} else {
# Part of the lazy sub
push @lazy_sub, $line;
}

} else {
if (
$line =~ /^sub\b/ # The start of a sub?
and
$line =~ /{\s*$/ # It's not a prototype
and
$line !~ /}\s*$/ # It's not a one line sub
and
$line !~ s/\:\s*not_lazy\b// # We've been told to ignore it
) {
push @lazy_sub, $line;
$in_lazy_sub = 1;
$starting_line = $line_num;
} else {
# Normal line in the file
push @out, $line;
}
}
}

# Write the changed code back to the file
open ( LAZYFILE, ">$file" ) or return undef;
print LAZYFILE join "\n", @out;
close LAZYFILE;

1;
}

sub _add_lazified_sub {
my($out, $code, $starting_line, $ending_line) = @_;
return unless ref $out eq 'ARRAY';
return unless ref $code eq 'ARRAY';

# The first line of the lazy sub code contains the same sub definition,
# but we want to add our loading code to it.
my $firstline = $code->[0];
my ($name) = $firstline =~ /^sub.+?(\w+)/;
$firstline =~ s/\s+{.*//;
$firstline .= " { delete *$name; eval <<'LAZYLOAD_END'; goto &$name }";
push @$out, $firstline;

push @$out, "#line $starting_line";
push @$out, @$code;
push @$out, 'LAZYLOAD_END';
push @$out, "#line ". ($ending_line + 1);

return 1;
}

Post by Stevan Little
Terrence,
There is also Class::LazyLoad
<http://search.cpan.org/~rkinyon/Class-LazyLoad/> which Rob Kinyon and I
wrote (actually 99% of it was Rob, I just wrote some of the test suite
and added a few features). It is basically meant to be a very very
lightweight alternative to things like Object::RealizeLater and
Class::LazyObject (speaking of which you should add Class::LazyObject
<http://search.cpan.org/~daxelrod/Class-LazyObject/> to your list).
BTW - What's with the Catalog::* namespace? You building something
interesting you want to share with the rest of the group? :)
- Steve

_______________________________________________
sw-design mailing list
http://metaperl.com/cgi-bin/mailman/listinfo/sw-design

Terrence Brannon

2005-01-19 12:05:45 UTC

Permalink

Post by Stevan Little
Terrence,
There is also Class::LazyLoad
<http://search.cpan.org/~rkinyon/Class-LazyLoad/> which Rob Kinyon and

added

Post by Stevan Little
Class::LazyObject (speaking of which you should add
Class::LazyObject

added

Post by Stevan Little

Post by Stevan Little
<http://search.cpan.org/~daxelrod/Class-LazyObject/> to your list).

BTW - What's with the Catalog::* namespace? You building something
interesting you want to share with the rest of the group? :)

I suppose this is a catalog of lazy loaders so I named it as such. The
CPAN as magazne idea was shot down, but I still like using POD and
h2xs to cobble up articles. The archive of article-pod is here:

http://www.metaperl.com/article-pod/

--
Carter's Compass: I know I'm on the right track when,
by deleting something, I'm adding functionality.

Matthew Simon Cavalletto

2005-01-22 19:32:39 UTC

Permalink

Post by Terrence Brannon

Post by Stevan Little
BTW - What's with the Catalog::* namespace? You building something
interesting you want to share with the rest of the group? :)

I suppose this is a catalog of lazy loaders so I named it as such. The
CPAN as magazne idea was shot down, but I still like using POD and
h2xs to cobble up articles.

As I said when this subject came up a couple of months ago, I really
like the idea of publishing these catalogs on CPAN for cross-indexing
purposes.

I think that finding relevant modules on CPAN and selecting between
them can be daunting, and this is an area where interested groups like
ours can really contribute to the community. Having catalogs like this
indexed by search.cpan.org should help people to find more of these
modules even if they're using different search terms.

Here are some similar existing documents which are candidates for being
converted into or replaced by Catalog::* POD files:

http://poop.sourceforge.net/
http://search.cpan.org/dist/DBIx-SQLEngine/SQLEngine/Docs/Related.pod
http://search.cpan.org/dist/Class-MakeMethods/MakeMethods/Docs/
RelatedModules.pod

If this does head towards CPAN, I would encourage you to simplify the
distribution directory, removing the tests and Exporter boilerplate;
all you really need in the tarball is a single directory with the
Makefile.PL and the LazyLoaders.pod.

-Simon

Stevan Little

2005-01-22 20:42:35 UTC

Permalink

Post by Matthew Simon Cavalletto

Post by Terrence Brannon

Post by Stevan Little
BTW - What's with the Catalog::* namespace? You building something
interesting you want to share with the rest of the group? :)

I suppose this is a catalog of lazy loaders so I named it as such. The
CPAN as magazne idea was shot down, but I still like using POD and
h2xs to cobble up articles.

As I said when this subject came up a couple of months ago, I really
like the idea of publishing these catalogs on CPAN for cross-indexing
purposes.

I second that, I really like the idea too.

Personally, every time I have a need for something, I first search CPAN
and maybe ask around on perlmonks, and if I don't find what I need, I
write it myself.

However this method is frought with problems. For instance, I spent
almost a whole day searching for some kind of mock DBI/DBD, and found
nothing because of the terms I was searching with (MockDB, MockDBI,
etc). For whatever reason, DBD::Mock never came up. I only found out
about it when I posted the question on perl monks and Chris Winters
replied. If Chris had not seen my post, I would have started writing it
myself.

Basically my point is that CPAN's search engine is good, but limited.
Short of asking module authors to add more meta-data to their module
distros or writing some large AI which can understand our modules and
the problem spaces they apply too, I think the next best thing might be
to write these CPAN catalog modules.

Should we start something official? Terrence already has a good start
going, and I would surely be interested in contributing.

- Steve

Gabriel Horner

2005-01-22 22:21:44 UTC

Permalink

Post by Stevan Little
Basically my point is that CPAN's search engine is good, but limited.
Short of asking module authors to add more meta-data to their module
distros or writing some large AI which can understand our modules and
the problem spaces they apply too, I think the next best thing might be
to write these CPAN catalog modules.

I'd recommend a search interface similar to http://del.icio.us for cataloguing/indexing modules.
One could associate multiple keywords or tags with a given module.
Currently rjbs has an open-source implementation of del.icio,
http://search.cpan.org/perldoc?Rubric .

If you all plan on indexing many modules, I would hope that the meta-deta (
module name,keywords,module descriptions ...) would be database-backended.
This way the meta-data could be used both to generate browseable catalog
modules or to provide a search interface.

I've been using my own tag-based module, Fry::Lib::CDBI::Tags, for keeping
track of cpan modules for a while. I'd be interested in helping
with a tag-based search interface.

Gabriel

--
my looovely website -- http://www.chwhat.com
BTW, IF chwhat.com goes down email me at ***@cern.ch

Ed Summers

2005-01-19 15:34:54 UTC

Permalink

Apologies if this has already been covered: Module::Pluggable

Provides a simple (a couple hundred lines of clean code) and platform
independent way of discovering and instantiating modules aka plugins.

//Ed

Adam Kennedy

2005-01-27 07:44:50 UTC

Permalink

The Catalog:: name is probably a bad one, it is already occupied with an
actual cataloging system of some sort.

It would be interesting to see if there could be a relatively standard
namespace for doing pure documentation packages. There is already
thoughts traveling around a few people's heads about how to tie things
other than code into the namespace.

Doc::Catalog::DelayedLoading

... something like that comes to mind. Doc:: is at least currently
empty, unlike Catalog.

But I'd want to bounce the idea of using Doc:: off of modules@ first.

In other news, I'm also doing a little research into tying data to the
perl namespace.

Early days yet, I have a few other projects that are higher priority.
But you may be interested in it.

See Data::Package for my first stab at the very VERY base functionality.

Adam

Chris Winters

2005-01-27 13:11:45 UTC

Permalink

Post by Adam Kennedy
...
In other news, I'm also doing a little research into tying data to the
perl namespace.
Early days yet, I have a few other projects that are higher priority.
But you may be interested in it.
See Data::Package for my first stab at the very VERY base
functionality.

What a great coincidence. I was just in the process of coding something
like this up for OpenInteract2. Thanks much for pointing it out.

Chris

--
Chris Winters
Creating enterprise-capable snack systems since 1988

Terrence Brannon

2005-01-27 18:24:31 UTC

Permalink

Yet another addition. Again, this is a tangential module, but I think
it fits the idea of the others listed. The whole list is here:

http://www.metaperl.com/article-pod/Catalog-lazy_loaders/

=item * Class::Prototyped by Toby Ovod-Everett

This module supports prototype-based object-oriented programming. An
object in such a system consists of slots which may be retrieved from
anywhere instead of just a member of an inheritance hierarchy. Slots
in these objects can be marked C<autoload> which defers computation of
its value until requested.

In some cases, deferral of slot population may be a reasonable
alternative to deferring class loading or object creation.

--
Carter's Compass: I know I'm on the right track when,
by deleting something, I'm adding functionality.

Mark Stosberg

2005-08-16 03:31:23 UTC

Permalink

Post by Terrence Brannon
RELATED MODULES

Would Data::Lazy count here?

http://search.cpan.org/dist/Data-Lazy/

I recently found it handy when refactoring to avoid needless
database connections, by only making $DBH actually create the connection
if it was every used.

Mark

--
http://mark.stosberg.com/

Terrence Brannon

2005-08-16 16:53:18 UTC

Permalink

Post by Mark Stosberg

Post by Terrence Brannon
RELATED MODULES

Would Data::Lazy count here?

oh yes, good catch!

Post by Mark Stosberg
http://search.cpan.org/dist/Data-Lazy/
I recently found it handy when refactoring to avoid needless
database connections, by only making $DBH actually create the connection
if it was every used.

added to list:

http://www.metaperl.com/article-pod/Catalog-lazy_loaders/lazy_loaders.html#data__lazy

--
Carter's Compass: I know I'm on the right track when,
by deleting something, I'm adding functionality.