Page title fetching is still failing in some cases

[julienCXX]

Hi,
I recently installed Shaarli (v0.6.5) on a private server (running PHP 7.0.5), in order to save a large amount of links from opened browser tabs. As I imported links manually, I noticed that the link’s title did not always appear properly (nothing, only partially or breaking the UI). That happened quite often (about 90 broken titles on more than 520 links). As these issues seemed to have been fixed in #410 and #512, I tried to reiterate the process with a local instance (same PHP version), using the most recent development version at that time (11609d9). The result was a bit better (no UI breakage), but about 50 links are still broken, in different ways.
These are the following:

No title appears:

• http://www.latitud-argentina.com/
• http://lesacados.com/
• http://www.a-contresens.net/
• http://www.leboncoin.fr/annonces/offres/ile_de_france/occasions/?th=1&q=test 404
• https://riunet.upv.es/
• http://pcsx2.net/ (blocked by cloudflare)
• https://www.linagora.com/
• http://coderstats.net/
• http://boilingsteam.com/ (308 redirect to https://boilingsteam.com/, using https://boilingsteam.com/ directly works)
• https://www.amazon.fr/gp/cart/view.html/ref=nav_cart binary output, not HTML
• http://www.lesnumeriques.com/carte-memoire/comparatif-cartes-micro-sd-sd-compact-flash-a340.html#filters/283c=0&283=433&pricex=10&pricey=383&pricen=0
• https://developer.apple.com/library/mac/documentation/UserExperience/Reference/ScreenSaver/ObjC_classic/index.html
• https://fr.atlassian.com/git/tutorials/comparing-workflows/forking-workflow
• https://www.facebook.com/
• http://ast2015.acroe-ica.org/ website down/timeout
• http://android.izzysoft.de/
• https://static.societegenerale.fr/ 404
• http://societegenerale.fr/ (301 redirect to https://www.sg.fr/ but still works)
• http://www.ldlc.com/
• https://www.hiawatha-webserver.org/
• https://www.occitech.fr/
• http://roulezelectrique.com/
• http://antix.mepis.org/ NXDOMAIN
• http://gagne.homedns.org/
• https://www.transip.eu/
• https://www.tilaa.com/
• http://www.quickvz.com/ NXDOMAIN
• https://mapzen.com/
• http://services.gisgraphy.com/
• http://www.pagesjaunes.fr/ (301 redirect to https://www.pagesjaunes.fr/, bookmarking it directly does not work either)
• http://www.yapaslefeuaulac.ch/
• http://geekfault.org/
• http://techtach.com/
• https://www.sqlite.org/
• https://sematext.com/blog/
• http://www.kris3d.com/ website down/timeout
• https://stackoverflow.com/jobs
• http://www.timschwartz.org/
• https://tuxicoman.jesuislibre.net/
• http://blog.mro.name/
• https://mro.name/foaf.rdf 404

A title appears, but is truncated/not the right one:

• http://expandedramblings.com/index.php/3-reasons-self-hosted-video-better-youtube/
• http://www.amazon.fr/s/ref=nb_sb_noss?__mk_fr_FR=%C3%85M%C3%85%C5%BD%C3%95%C3%91&url=search-alias%3Daps&field-keywords=test
• http://www.meetup.com/
• https://wiki.cyanogenmod.org/w/Install_CM_for_jfltexx
• http://host1.no/

A title appears, but non-ASCII characters are replaced with a question mark (encoding issue?):

• http://www.upv.es/
• http://fr.lolix.org/search/offre/offre.php?id=21082

Furthermore, title fetching fails on links to PDF files.

[alexisju]

I tried the 5 first links. With Firefox API, title, url and description are fetched correctly. However, with bookmarklet (or Shaarli-next firefox addon), description are not loaded.

[ArthurHoaro]

Thanks for the dataset! It'll be easier to fix this for good.

So far, here's what I got:

1. 30x HTTP redirection can contain a relative URL.
2. While some host requires request_fulluri attribute in the HTTP request, some others reject it.
3. It fails if the <title> HTML tag contain an attribute (eg. <title stuff="morestuff">.
4. Some host tells me that my request is HTTP 406 Not Acceptable, which is not very kind, because my browser is too old (Firefox 23 UA).
5. The apple one was tricky. They use simple quotes to define their <meta charset=, while we only expect double quotes or nothing, which leads to an invalid charset.

truncated/not the right one

• http://expandedramblings.com : point 2.
• amazon: the title is more likely changed in JS after page loading.
• meetup: same as amazon.
• https://wiki.cyanogenmod.org/w/Install_CM_for_jfltexx : Retrieved title is not escaped! That's very bad.
• http://host1.no/ returns a 404 page (probably because I'm a bot), but with a 200 HTTP code. Don't use this host.

Encoding issues: mb_convert_encoding parameters are messed up.

Not solved yet and/or probably won't fix:

• https://riunet.upv.es/ : it works if I remove the 4Mo download limit, but the page is 35ko. However, it takes me 5s to get a response from the host, and I can't even ping it.
• https://fr.atlassian.com/git/tutorials/comparing-workflows/forking-workflow : It's weird, we have this in the same response:

[ 
  0 => 'HTTP/1.1 301 Moved Permanently',
  1 => 'HTTP/1.1 301 Moved Permanently',
  2 => 'HTTP/1.1 200 OK',
]

Probably due to going through reverse proxies, but that's more likely a bad configuration on their side.

• https://static.societegenerale.fr/ : access is forbidden.
• A few of your links seems to have an anti-bot policy (custom 401/403 error codes). We won't fight it.
• http://mro.name/foaf.rdf doesn't have any title.

@alexisju: it doesn't work the same way. With bookmarklets, the page is already opened, and Shaarli grabs the title with JS (or Firefox API). In Shaarli, it has to make an external request from your server to the targeted URL.

[julienCXX]

OK, the explanation seems legit, but does not convince me completely.

amazon: the title is more likely changed in JS after page loading.

I do not agree with that. The correct title appears in a title tag, in the page’s source code. Have you looked at the URL encoding/escaping?
On another (desert) hand, this one does it.

meetup: same as amazon.

On this one, the retrieved title has an extra space between the “-” and the preceding word. It looks like the carriage return is interpreted as a space character. Other title differences could be related to an adaptation to the browser’s accepted language settings.

http://host1.no/ returns a 404 page (probably because I'm a bot), but with a 200 HTTP code. Don't use this host.

I had not got a 404 page. Strange. Yes, trough my local instance.

https://static.societegenerale.fr/ : access is forbidden.

Yes, I know. I expected to get “403 Forbidden” as page title.

http://mro.name/foaf.rdf doesn't have any title.

This is not a valid HTML page, but my browser (Firefox) still displays a title (Marcus Rohrmoser). It looks like it was extracted from the foaf:name tag.

@alexisju: it doesn't work the same way. With bookmarklets, the page is already opened, and Shaarli grabs the title with JS (or Firefox API). In Shaarli, it has to make an external request from your server to the targeted URL.

I almost only used the Shaarli Web interface for adding links.

[nodiscc]

amazon: the title is more likely changed in JS after page loading.

I ran wget on that page and the <title> is correct (Amazon.fr : test), so not generated by JS.

I expected to get “403 Forbidden”

The message seems to depend on the User Agent (curl returns <title>Erreur / Error</title>, firefox <title>403 Forbidden</title>

This is not a valid HTML page, but my browser (Firefox) still displays a title (Marcus Rohrmoser). It looks like it was extracted from the foaf:name tag.

I think Shaarli can live without RDF parsing.

[ArthurHoaro]

amazon: the title is more likely changed in JS after page loading.

My bad, I jumped on that conclusion too fast. I found another issue, which will probably fix some other issues: the URL is escaped too early, so Shaarli's trying to reach a non valid URL (& => &).

On this one, the retrieved title has an extra space between the “-” and the preceding word. It looks like the carriage return is interpreted as a space character. Other title differences could be related to an adaptation to the browser’s accepted language settings.

Yep, I thought you were talking about the translation. The replace function adds an additional space to replace new lines, which is unnecessary. Also, I guess adding the locale in the HTTP request wouldn't harm.

Yes, I know. I expected to get “403 Forbidden” as page title.

No, Shaarli only downloads the page/retrieves the title if it finds 200 OK HTTP code. I don't really see the point to set 403 Forbidden as a title.

This is not a valid HTML page, but my browser (Firefox) still displays a title (Marcus Rohrmoser). It looks like it was extracted from the foaf:name tag.

Right, NoScript was blocking the rendering. I agree with @nodiscc though.

Anyway, there is a bunch of things to fix, but that will be a great improvement. Thanks!

[ArthurHoaro]

Actually I found something else for fr.atlassian.com: they use location instead of Location redirection directive (notice the case). It doesn't cost much to support it.

EDIT: Another. Bad certificate can be ignored.

[julienCXX]

It looks like your commit fixed most problems, as I tested the problematic links with that version.
However, there are still issues with some of them (excluding the forbidden and won’t fix cases).
No title appears (maybe an anti-bot policy):

• http://coderstats.net/
• http://antix.mepis.org/
• https://www.transip.eu/
• https://www.tilaa.com/
• http://services.gisgraphy.com/

Title appears, but is not the expected one:

• http://expandedramblings.com/index.php/3-reasons-self-hosted-video-better-youtube/ (it gives me no title, instead of “DMR” with the versions used in my first comment; this page also has 2 title HTML tags and my browser shows the other one “3 Reasons Self-Hosted Video Is Better Than YouTube”)
• http://android.izzysoft.de/ (German title on Shaarli, English title in my browser)

host1.no has no 404 error anymore, title fetching works well here.

I found some new links causing other issues:

• http://news.zing.vn/ (no title)
• Unicode domains (e.g. http://www.académie-française.fr/ or http://ふそう.com/) no title appears, but non-unicode variant/converted versions work (respectively http://www.academie-francaise.fr/ and http://Xn--P8j2bxd.com/)
• http://www.paruvendu.fr/bateau-nautisme/ : it looks like the encoding is broken (browser: “Nautisme : bateaux occasion, voiliers, jet ski... Vente bateaux à moteur, voiliers d’occasion”, Shaarli’s new link page: “Nautisme : bateaux occasion, voiliers, jet ski... Vente bateaux à moteur, voiliers d�occasion” and the invalid character does not appear when browsing links)
• EDIT: http://www.ign.fr/ (no title and long to process)

[ArthurHoaro]

• http://coderstats.net/ is working on my local instance.
• http://antix.mepis.org/ returns 403 bad behaviour.
• https://www.transip.eu/ returns 403 forbidden.
• https://www.tilaa.com/ returns false.
• http://services.gisgraphy.com/ returns 401 unauthorized.
• http://expandedramblings.com/index.php/3-reasons-self-hosted-video-better-youtube/ it works here. :/
• http://android.izzysoft.de/ I fixed the Accept-Language header (it sends exactly the same thing as my browser do), but still no luck. I bet my 2 cents on this, which might be a browser feature:

<link rel="alternate" hreflang="de" href="/articles.php?lang=de">

• http://news.zing.vn/ If you try to wget it, you'll get the same crap as Shaarli does: a 40kB file full of unknown characters. No idea here.
• Unicode domains are not supported by filter_var($url, FILTER_VALIDATE_URL)...
• http://www.paruvendu.fr/bateau-nautisme/ They use the old charset definition in their HTML.
• http://www.ign.fr/ also works.

So there are 2 fixable things here: unicode domains and supporting pre-HTML5 charset.

For the links that work for me, it might be because I have a better connection than yours. Although a proper server should have a better connection than mine.

[julienCXX]

For the links that work for me, it might be because I have a better connection than yours. Although a proper server should have a better connection than mine.

They now work for me, after another try.

http://news.zing.vn/ If you try to wget it, you'll get the same crap as Shaarli does: a 40kB file full of unknown characters. No idea here.

wget also gave me that. Using file, I discovered that the document is actually a gzip-compressed HTML page. After gunzipping it, the title appears clearly, surrounded by 2 carriage return characters (0x0D or ^M).