Discussion:
PPI gets Perl Foundation Funding
Stevan Little
2004-12-21 17:18:09 UTC
Permalink
Congratulations to our illustrious list member Adam on getting Perl
Foundation Funding for PPI.

(http://search.cpan.org/src/ADAMK/PPI-0.840_01/Changes)

Adam, please feel free to use the list as a sounding board if you like,
I for one have great hopes for this module.

- Steve
Adam Kennedy
2004-12-22 05:37:45 UTC
Permalink
/me bows and thanks the academy.

Sounding board eh?

The only problem with _that_ is that PPI is so frigging finicky that it
can be difficult to explain the context of the problems for the
decisions regarding the internals. :)

Of course, if anyone is interested in helping, there are about a zillion
unit tests that need to be written...

*crickets*

In all seriousness, part of the job will be moving everything into
SourceForge once I get 0.840 out the door. So if anyone wants to talk
about PPI or help out in any way, or start one of the 50 or so modules
that we can build on top of it, my doors are open to volunteers.

Todays issue:

- Making Element->location "Just Work" while taking into account
here-docs, and selectively caching so that it isn't a dog.

Upcoming issues:

- Detecting the use of source filtering modules, and knowing which ones
modify the grammar far enough to break the lexer (and thus we should
refuse to parse the file) and which ones we can tolerate.

- Normalising PPI::Token::Quote objects for the four quote types ('',
"", q, qq) into a single value for use in PPI::Compare.

(Coming up with a generic solution (just in written terms) would be
something that should be fairly approachable without knowing the internals)

- Making the Lexer/Tokenizer capable of "reflowing", so that you can
insert arbitrary text at some point in the PDOM tree and the document
will morph to take it into account.

(Yes, I'm aware that last one is going to chew a shitload of CPU) :)

Adam
Post by Stevan Little
Congratulations to our illustrious list member Adam on getting Perl
Foundation Funding for PPI.
(http://search.cpan.org/src/ADAMK/PPI-0.840_01/Changes)
Adam, please feel free to use the list as a sounding board if you like,
I for one have great hopes for this module.
- Steve
_______________________________________________
sw-design mailing list
http://metaperl.com/cgi-bin/mailman/listinfo/sw-design
Stevan Little
2004-12-22 13:49:35 UTC
Permalink
Adam,

On Dec 22, 2004, at 12:37 AM, Adam Kennedy wrote:\
Post by Adam Kennedy
/me bows and thanks the academy.
Sounding board eh?
The only problem with _that_ is that PPI is so frigging finicky that
it can be difficult to explain the context of the problems for the
decisions regarding the internals. :)
gotcha. That makes sense.
Post by Adam Kennedy
Of course, if anyone is interested in helping, there are about a
zillion unit tests that need to be written...
I will help out on that. Which module is most finished?

Steve
Adam Kennedy
2004-12-29 13:52:36 UTC
Permalink
Post by Stevan Little
Post by Adam Kennedy
Of course, if anyone is interested in helping, there are about a
zillion unit tests that need to be written...
I will help out on that. Which module is most finished?
"module" is a tricky word :)

http://search.cpan.org/~adamk/PPI-0.840/lib/PPI/Manual.pod#PPI_Data_Class_Full_Listing

As you can see from that URL, there are a *cough* LOT of modules in the
Perl Document Object Model. (like pretty much every other object model)

There are two or three tasks where people can help if they like

Firstly, I'm currently trying to finish up a sort of "PPI CPAN
Tinderbox". A module (PPI::Tinderbox) that will pull down all of CPAN,
suck out the perl modules and run them all through a series of stress tests.

Once this is working, I'll need people to help go through the logs, and
help find and isolate bugs (Tokenizer bugs, Lexer bugs, things being
recognised wrong, circular reference leaks, etc etc).

You should be fairly easily be able to install that module, and
configure some values in a 10 line launcher script. After it runs for
half a day or so and spits out the results, you can hunt stuff down.

PPI::Tester, the (broken till I sync it with 0.840) WxWindows-based
utility that shows the PDOM-dump for any code interactively as you type
it, should make this a lot easier.

Then just submit the code snippits to rt.cpan.org, I can do what needs
to be done to fix them much easier if others can help find them.

Second, there a zillion classes that need unit tests... but that's
pretty boring work :)

And third, well I need a tutorial. If someone wants to have a play
around with PPI in it's current state, which should be reasonably
stable, and then can write about their experiences, it's all fodder for
the POD-mill.

And fourth, if anyone REALLY wants to do unit testing, make a test
script for each of the documented PPI::Statement classes, generate some
simple Documents in it and make sure that the various methods in the
Statement classes generate what you expect them to.

Adam
Adam Kennedy
2005-01-18 16:38:50 UTC
Permalink
For those that are interested, PPI 0.900 just got uploaded.

This is a Beta 1 release. For the record, of the 38,000 non-Acme perl
modules in CPAN, PPI comfortably handles all but 20. (and 10 of those
are bugs in the modules)

It is now feature-complete and largely frozen, so if people were
interesting in having a bit of a dicker with PPI, from now on you should
be able to do so with a fairly reasonable assumption the API won't
change underneath you :)

Enjoy

Adam K
Post by Adam Kennedy
Post by Stevan Little
Post by Adam Kennedy
Of course, if anyone is interested in helping, there are about a
zillion unit tests that need to be written...
I will help out on that. Which module is most finished?
"module" is a tricky word :)
http://search.cpan.org/~adamk/PPI-0.840/lib/PPI/Manual.pod#PPI_Data_Class_Full_Listing
As you can see from that URL, there are a *cough* LOT of modules in the
Perl Document Object Model. (like pretty much every other object model)
There are two or three tasks where people can help if they like
Firstly, I'm currently trying to finish up a sort of "PPI CPAN
Tinderbox". A module (PPI::Tinderbox) that will pull down all of CPAN,
suck out the perl modules and run them all through a series of stress tests.
Once this is working, I'll need people to help go through the logs, and
help find and isolate bugs (Tokenizer bugs, Lexer bugs, things being
recognised wrong, circular reference leaks, etc etc).
You should be fairly easily be able to install that module, and
configure some values in a 10 line launcher script. After it runs for
half a day or so and spits out the results, you can hunt stuff down.
PPI::Tester, the (broken till I sync it with 0.840) WxWindows-based
utility that shows the PDOM-dump for any code interactively as you type
it, should make this a lot easier.
Then just submit the code snippits to rt.cpan.org, I can do what needs
to be done to fix them much easier if others can help find them.
Second, there a zillion classes that need unit tests... but that's
pretty boring work :)
And third, well I need a tutorial. If someone wants to have a play
around with PPI in it's current state, which should be reasonably
stable, and then can write about their experiences, it's all fodder for
the POD-mill.
And fourth, if anyone REALLY wants to do unit testing, make a test
script for each of the documented PPI::Statement classes, generate some
simple Documents in it and make sure that the various methods in the
Statement classes generate what you expect them to.
Adam
_______________________________________________
sw-design mailing list
http://metaperl.com/cgi-bin/mailman/listinfo/sw-design
Rob Kinyon
2005-01-18 16:56:27 UTC
Permalink
I'm curious - what bugs in what modules are causing you fits?

Rob
Post by Adam Kennedy
For those that are interested, PPI 0.900 just got uploaded.
This is a Beta 1 release. For the record, of the 38,000 non-Acme perl
modules in CPAN, PPI comfortably handles all but 20. (and 10 of those
are bugs in the modules)
It is now feature-complete and largely frozen, so if people were
interesting in having a bit of a dicker with PPI, from now on you should
be able to do so with a fairly reasonable assumption the API won't
change underneath you :)
Enjoy
Adam K
Post by Adam Kennedy
Post by Stevan Little
Post by Adam Kennedy
Of course, if anyone is interested in helping, there are about a
zillion unit tests that need to be written...
I will help out on that. Which module is most finished?
"module" is a tricky word :)
http://search.cpan.org/~adamk/PPI-0.840/lib/PPI/Manual.pod#PPI_Data_Class_Full_Listing
As you can see from that URL, there are a *cough* LOT of modules in the
Perl Document Object Model. (like pretty much every other object model)
There are two or three tasks where people can help if they like
Firstly, I'm currently trying to finish up a sort of "PPI CPAN
Tinderbox". A module (PPI::Tinderbox) that will pull down all of CPAN,
suck out the perl modules and run them all through a series of stress tests.
Once this is working, I'll need people to help go through the logs, and
help find and isolate bugs (Tokenizer bugs, Lexer bugs, things being
recognised wrong, circular reference leaks, etc etc).
You should be fairly easily be able to install that module, and
configure some values in a 10 line launcher script. After it runs for
half a day or so and spits out the results, you can hunt stuff down.
PPI::Tester, the (broken till I sync it with 0.840) WxWindows-based
utility that shows the PDOM-dump for any code interactively as you type
it, should make this a lot easier.
Then just submit the code snippits to rt.cpan.org, I can do what needs
to be done to fix them much easier if others can help find them.
Second, there a zillion classes that need unit tests... but that's
pretty boring work :)
And third, well I need a tutorial. If someone wants to have a play
around with PPI in it's current state, which should be reasonably
stable, and then can write about their experiences, it's all fodder for
the POD-mill.
And fourth, if anyone REALLY wants to do unit testing, make a test
script for each of the documented PPI::Statement classes, generate some
simple Documents in it and make sure that the various methods in the
Statement classes generate what you expect them to.
Adam
_______________________________________________
sw-design mailing list
http://metaperl.com/cgi-bin/mailman/listinfo/sw-design
_______________________________________________
sw-design mailing list
http://metaperl.com/cgi-bin/mailman/listinfo/sw-design
Adam Kennedy
2005-01-18 17:02:46 UTC
Permalink
Well... there's Zanas, which has

file.html.pm
-------
<html></html>
-------

And don't get me started on

file.gif.pm
------------------
(binary gif file)
------------------

Still haven't worked out what the hell they are for... :)

It's partly my fault for making the "what is a module" regex wrong.

Paper::Specs has used a code generator that doesn't escape properly, so
half a dozen Paper::Specs::Avery::8XXX label modules die

description => "Avery 5 1/2" blah blah labels",

Of the ones that are my fault, we have gems like

tr [\$|\%|\&|\*] //g;

And one that got fixed...

$|=1;$::|=1;

Or Damian's use of @0 <-- that's a zero

Or the fact that <<1HERE1 is illegal per the docs, but does actually work.

Since it takes 4 or 5 minutes to nail down the exact fault location for
each file, I haven't had time to get details on the rest, but it's
mostly insanely ugly quote/regexs or fairly insane code in general that
is causing the problems now, and I never said I'd support every single
thing.

I've been uploading some autogenerated reports as well, to

http://ali.as/PPI/tinderbox/

So you can see there are a large number of modules uploaded with
corrupted tarballs, or who have started using Build.PL without keeping
the wrapper Makefile.PL for compatibility.

Adam
Post by Rob Kinyon
I'm curious - what bugs in what modules are causing you fits?
Rob
Post by Adam Kennedy
For those that are interested, PPI 0.900 just got uploaded.
This is a Beta 1 release. For the record, of the 38,000 non-Acme perl
modules in CPAN, PPI comfortably handles all but 20. (and 10 of those
are bugs in the modules)
It is now feature-complete and largely frozen, so if people were
interesting in having a bit of a dicker with PPI, from now on you should
be able to do so with a fairly reasonable assumption the API won't
change underneath you :)
Enjoy
Adam K
Post by Adam Kennedy
Post by Stevan Little
Post by Adam Kennedy
Of course, if anyone is interested in helping, there are about a
zillion unit tests that need to be written...
I will help out on that. Which module is most finished?
"module" is a tricky word :)
http://search.cpan.org/~adamk/PPI-0.840/lib/PPI/Manual.pod#PPI_Data_Class_Full_Listing
As you can see from that URL, there are a *cough* LOT of modules in the
Perl Document Object Model. (like pretty much every other object model)
There are two or three tasks where people can help if they like
Firstly, I'm currently trying to finish up a sort of "PPI CPAN
Tinderbox". A module (PPI::Tinderbox) that will pull down all of CPAN,
suck out the perl modules and run them all through a series of stress tests.
Once this is working, I'll need people to help go through the logs, and
help find and isolate bugs (Tokenizer bugs, Lexer bugs, things being
recognised wrong, circular reference leaks, etc etc).
You should be fairly easily be able to install that module, and
configure some values in a 10 line launcher script. After it runs for
half a day or so and spits out the results, you can hunt stuff down.
PPI::Tester, the (broken till I sync it with 0.840) WxWindows-based
utility that shows the PDOM-dump for any code interactively as you type
it, should make this a lot easier.
Then just submit the code snippits to rt.cpan.org, I can do what needs
to be done to fix them much easier if others can help find them.
Second, there a zillion classes that need unit tests... but that's
pretty boring work :)
And third, well I need a tutorial. If someone wants to have a play
around with PPI in it's current state, which should be reasonably
stable, and then can write about their experiences, it's all fodder for
the POD-mill.
And fourth, if anyone REALLY wants to do unit testing, make a test
script for each of the documented PPI::Statement classes, generate some
simple Documents in it and make sure that the various methods in the
Statement classes generate what you expect them to.
Adam
_______________________________________________
sw-design mailing list
http://metaperl.com/cgi-bin/mailman/listinfo/sw-design
_______________________________________________
sw-design mailing list
http://metaperl.com/cgi-bin/mailman/listinfo/sw-design
_______________________________________________
sw-design mailing list
http://metaperl.com/cgi-bin/mailman/listinfo/sw-design
Adam Kennedy
2005-01-19 00:53:03 UTC
Permalink
I've already done one for the Avery bug, since it's an obvious typo.

From a design and testing point of view (to talk on the actual topic of
the mailing list) I think it also goes to show the benefit of doing an
initial unit test script which does nothing but load all the modules in
a distribution and make sure they compile and load cleanly.

Even if you do no other unit test scripts, at least you know there
aren't any killer syntax errors.

And in the other unit tests, you can just C< use Module; > normally,
without having to test it.

I've been doing this for about the last year now, and I've found it to
work quite well.

01_compile.t
------------

#!/usr/bin/perl -w

# Load testing for CGI::Capture

use strict;
use lib ();
use File::Spec::Functions ':ALL';
BEGIN {
$| = 1;
unless ( $ENV{HARNESS_ACTIVE} ) {
require FindBin;
chdir ($FindBin::Bin = $FindBin::Bin); # Avoid a warning
lib->import( catdir( updir(), updir(), 'modules') );
}
}

use Test::More tests => 2;

# Check their perl version
ok( $] >= 5.006, "Your perl is new enough" );

# Does the module load
use_ok('CGI::Capture');

exit(0);

----------------------------

That block at the top is to let the tests work inside my slightly
unusual CVS layout. (I have a build_module.pl script to assemble stuff
into the proper layout when releasing).

Adam
This is actually kinda cool. Are you planning on submitting RT
reports? *hint hint*
Rob
Post by Adam Kennedy
Well... there's Zanas, which has
file.html.pm
-------
<html></html>
-------
And don't get me started on
file.gif.pm
------------------
(binary gif file)
------------------
Still haven't worked out what the hell they are for... :)
It's partly my fault for making the "what is a module" regex wrong.
Paper::Specs has used a code generator that doesn't escape properly, so
half a dozen Paper::Specs::Avery::8XXX label modules die
description => "Avery 5 1/2" blah blah labels",
Of the ones that are my fault, we have gems like
tr [\$|\%|\&|\*] //g;
And one that got fixed...
$|=1;$::|=1;
Or the fact that <<1HERE1 is illegal per the docs, but does actually work.
Since it takes 4 or 5 minutes to nail down the exact fault location for
each file, I haven't had time to get details on the rest, but it's
mostly insanely ugly quote/regexs or fairly insane code in general that
is causing the problems now, and I never said I'd support every single
thing.
I've been uploading some autogenerated reports as well, to
http://ali.as/PPI/tinderbox/
So you can see there are a large number of modules uploaded with
corrupted tarballs, or who have started using Build.PL without keeping
the wrapper Makefile.PL for compatibility.
Adam
Stevan Little
2005-01-19 01:08:12 UTC
Permalink
Adam,
Post by Adam Kennedy
Even if you do no other unit test scripts, at least you know there
aren't any killer syntax errors.
I agree whole heartedly.

I have been using this small script for a while now to create my
'smoke' tests with, it only attempts to load the .pm files and skips
any CVS or svn directories too. It also formats the output, putting
comments for each folder, and indenting as well. I basically just say
'perl create_smoke_test.pl lib/ > t/001_smoke_test.t' and I am good. It
basically saves me from having to bother doing this by hand.

- Steve

#!/usr/bin/perl

use strict;
use warnings;

my $directory = shift(@ARGV);

(defined($directory))
|| die "You must specify a directory to create a smoke test
for\nusage: create_smoke_test <directory>\n";

sub map_dir {
my ($dir, $dir_func, $file_func, $depth) = @_;
$depth ||= 0;
return map {
if (-d $_ && !/CVS|\.svn/) {
$dir_func->($_, $depth + 1);
map_dir($_, $dir_func, $file_func, $depth + 1);
}
elsif (-f $_ && /\.pm/) {
$file_func->($_, $depth);
}
} sort { ((-d $a) ? 1 : (-d $b) ? -1 : ($a cmp $b)) } <$dir/*>;
}

sub handle_dir {
my ($dir, $depth) = @_;
$depth ||= 1;
$dir =~ s/^$directory\///;
print("\n" . (" " x $depth) . "# $dir\n");
}

sub handle_file {
my ($file, $depth) = @_;
$depth ||= 1;
$file =~ s/^$directory\///;
my ($module_name) = ($file =~ /(.*?)\.pm/);
$module_name =~ s/\//\:\:/g;
print((" " x $depth) . "use_ok('$module_name');\n");
}

print <<HEADER;
#!/usr/bin/perl

use strict;
use warnings;

use Test::More no_plan => 1;

BEGIN {

HEADER

map_dir($directory, \&handle_dir, \&handle_file);

print <<FOOTER;

};

FOOTER
Post by Adam Kennedy
And in the other unit tests, you can just C< use Module; > normally,
without having to test it.
I've been doing this for about the last year now, and I've found it to
work quite well.
01_compile.t
------------
#!/usr/bin/perl -w
# Load testing for CGI::Capture
use strict;
use lib ();
use File::Spec::Functions ':ALL';
BEGIN {
$| = 1;
unless ( $ENV{HARNESS_ACTIVE} ) {
require FindBin;
chdir ($FindBin::Bin = $FindBin::Bin); # Avoid a warning
lib->import( catdir( updir(), updir(), 'modules') );
}
}
use Test::More tests => 2;
# Check their perl version
ok( $] >= 5.006, "Your perl is new enough" );
# Does the module load
use_ok('CGI::Capture');
exit(0);
----------------------------
That block at the top is to let the tests work inside my slightly
unusual CVS layout. (I have a build_module.pl script to assemble stuff
into the proper layout when releasing).
Adam
This is actually kinda cool. Are you planning on submitting RT
reports? *hint hint*
Rob
Post by Adam Kennedy
Well... there's Zanas, which has
file.html.pm
-------
<html></html>
-------
And don't get me started on
file.gif.pm
------------------
(binary gif file)
------------------
Still haven't worked out what the hell they are for... :)
It's partly my fault for making the "what is a module" regex wrong.
Paper::Specs has used a code generator that doesn't escape properly, so
half a dozen Paper::Specs::Avery::8XXX label modules die
description => "Avery 5 1/2" blah blah labels",
Of the ones that are my fault, we have gems like
tr [\$|\%|\&|\*] //g;
And one that got fixed...
$|=1;$::|=1;
Or the fact that <<1HERE1 is illegal per the docs, but does actually work.
Since it takes 4 or 5 minutes to nail down the exact fault location for
each file, I haven't had time to get details on the rest, but it's
mostly insanely ugly quote/regexs or fairly insane code in general that
is causing the problems now, and I never said I'd support every single
thing.
I've been uploading some autogenerated reports as well, to
http://ali.as/PPI/tinderbox/
So you can see there are a large number of modules uploaded with
corrupted tarballs, or who have started using Build.PL without keeping
the wrapper Makefile.PL for compatibility.
Adam
_______________________________________________
sw-design mailing list
http://metaperl.com/cgi-bin/mailman/listinfo/sw-design
Loading...