Issue processing site?

[youradds]

Hi,

This module works well - apart from a few sites. For example:

https://ragnitrotta.uk

My code is:

use HTML::Restrict;
my %rules = (
    a       => [qw( href title )],
    b       => [],
    caption => [],
    center  => [],
    em      => [],
    i       => [],
    #img     => [qw( alt border height width src style )],
    li      => [],
    ol      => [],
    p       => [],
    span    => [],
    strong  => [],
    sub     => [],
    sup     => [],
    table   => [qw()],
    tbody   => [],
    td      => [],
    tr      => [],
    u       => [],
    ul      => [],
    title   => [],
    br      => [],
    head    => [],
    div     => [],
    meta    => [qw(name content property)],
    html    => [qw(lang)],
    iframe  => [qw(src)]
);

my $hr = HTML::Restrict->new( rules => \%rules, uri_schemes => [ undef, 'http', 'https', 'tel', 'mailto' ] );
print "before...\n";
$page = $hr->process($page);
print "after ..\n";

The text before shows, but the "after" bit doesn't. Any suggestions? Its been working well otherwise :/

[oalders]

Are you sure you're getting the page content? This works for me:

#!/usr/bin/env perl

use strict;
use warnings;

use HTML::Restrict ();
use LWP::UserAgent ();

my %rules = (
    a       => [qw( href title )],
    b       => [],
    caption => [],
    center  => [],
    em      => [],
    i       => [],
    #img     => [qw( alt border height width src style )],
    li      => [],
    ol      => [],
    p       => [],
    span    => [],
    strong  => [],
    sub     => [],
    sup     => [],
    table   => [qw()],
    tbody   => [],
    td      => [],
    tr      => [],
    u       => [],
    ul      => [],
    title   => [],
    br      => [],
    head    => [],
    div     => [],
    meta    => [qw(name content property)],
    html    => [qw(lang)],
    iframe  => [qw(src)]
);

my $ua = LWP::UserAgent->new( agent => 'Mozilla/5.0' );
my $content = $ua->get("https://ragnitrotta.uk/")->content;
my $hr = HTML::Restrict->new( rules => \%rules, uri_schemes => [ undef, 'http', 'https', 'tel', 'mailto' ] );
my $page = $hr->process($content);
print $page;

[youradds]

Sorry for the late reply. Hmm how odd - it does seem to work that way. I even tried it using the method I'm grabbing with:

my $content = `curl --insecure --compressed -A "PageThing.com Crawler" --silent --max-time 5 --location --connect-timeout 10 'https://ragnitrotta.uk/'`;
my $hr = HTML::Restrict->new( rules => \%rules, uri_schemes => [ undef, 'http', 'https', 'tel', 'mailto' ] );
my $page = $hr->process($content);
print $page;

..and that works as well now! Maybe there was a glitch with the server (just weird how it only seemed to be that domain)

Anyway - thanks for looking into it :)

[oalders]

Thanks for the follow-up. It didn't work for me until I changed the default UserAgent string, so there are some requests which may be getting blocked based on that by this particular site. Glad to hear you're back up and running. :)

[youradds]

@oalders hmm odd - its doing it again - this time with another domain

use strict;
use warnings;

use HTML::Restrict ();
use LWP::UserAgent ();

my %rules = (
    a       => [qw( href title )],
    b       => [],
    caption => [],
    center  => [],
    em      => [],
    i       => [],
    #img     => [qw( alt border height width src style )],
    li      => [],
    ol      => [],
    p       => [],
    span    => [],
    strong  => [],
    sub     => [],
    sup     => [],
    table   => [qw()],
    tbody   => [],
    td      => [],
    tr      => [],
    u       => [],
    ul      => [],
    title   => [],
    br      => [],
    head    => [],
    div     => [],
    meta    => [qw(name content property)],
    html    => [qw(lang)],
    iframe  => [qw(src)]
);

my $ua = LWP::UserAgent->new( agent => 'Mozilla/5.0' );
my $content = $ua->get("http://simple4xsystem.net/")->content;

print "Got $content \n";

my $hr = HTML::Restrict->new( rules => \%rules, uri_schemes => [ undef, 'http', 'https', 'tel', 'mailto' ] );
my $page = $hr->process($content);
print $page;

It prints out the html with print "Got $content \n";, but then locks up on process() . Just hangs there for ages (I've not left it to run out, but it was going for a good 10 mins)

Does that one work for you your end?

[oalders]

If you pass a debug => 1 to HTML::Restrict->new() then you can get a better idea of what's going on. It looks like it makes it to the end of the document, but it doesn't return. The debugging output doesn't make it clear what's going on after that. I'll need some time to poke at it.

[youradds]

No problem. I'm glad you could replicate it as well :) Its indeed a weird one! No rush though - for the moment I have just told my script to ignore that domain (it only seems to be a small number of sites that cause it)

[youradds]

Hi,

So for http://empoweringmindsmat.uk/ , it seems they just let their site get hammered with spam! If you download the file via wget, its 14mb! I had a look at it, and there are tons and tons of spammy comments. So what I've done as a hack is just to remove the comments id:

$page =~ s/\<div id=\"comments".*>.*?<\/div>//is; # get rid of stupid "comments" section, as seems lots of people let their sites get spammed to hell!

Still can't figure out http://simple4xsystem.net/, but at least that has helped trimmed the fat off some of the other bits :)

Cheers

Andy