Exclude URLs based on HTML tags

[MichaIng]

Currently lychee seems to check each an every URL found in the HTML document.

Generally there is nothing bad about it. We previously used liche, which only checks specific tags and attributes.

While generally checking URLs from text is not bad, IMO, it would be great to be able to exclude certain elements, not just based on the URL itself. As HTML files are parsed already as such, it should be cheaply possible to exclude specific tags, e.g. <code> and <pre> tags. Allowing to specify URL excludes based on tags via an option, would be awesome.

[mre]

For HTML that would work as the tags have names. Not sure about Markdown or Plaintext. If we add an option like --exclude-tags=pre,code, what would be the behavior for Markdown? Simply ignore it (which might be confusing) or throw an error (which might be frustrating) or something else?

In general, I wonder if it makes a difference whether a link occurs inside or outside a pre tag for example. Do you have an example for links that predominantly occur in special tags that you couldn't exclude with a regex? I just don't want to make the project unnecessarily complex.

[MichaIng]

I personally check links in Markdown only after building the HTML pages from it. But I agree it should be made clear, when adding such an option, that it can and does only apply for HTML input files, e.g. start with naming it --exclude-html-tags, or so. In Markdown things are much more complicated, due to the many extensions and ways to create code blocks and inline code, effective also based on indentation etc. So I guess practically it can only be used on HTML indeed, but that should cover the majority of scanned documents, especially as websites can be scanned wie URLs as well, and as this is currently the only way to scan internal links etc.

In general, I wonder if it makes a difference whether a link occurs inside or outside a pre tag for example.

We have no "links" in any pre or code tag, and you're right that this is not even possible, when you mean clickable links by this. Both tags take all content literally. But we have URLs inside of them, e.g. when telling users how to access a web interface of their locally running application, using loopback IP, localhost, LAN hostnames or IPs. In case of MkDocs, code tags are nice, as they allow easy copy&paste. But also when having a pre tag to show a scripts content, filled with LAN IPs/hostnames or example URLs.

I know the -E option, to exclude them all together (aside of invalid example URLs), but we cannot use it as me must check internal links via locally running webserver and use hence the loopback IP as base URL, which is then excluded as well. Actually, an alternative for at least our case, would be if -E excludes would not be applied to any URL that was generated for internal links with base-url. That actually makes sense anyway, as noone would define a base URL, to then exclude it from checks 😄.

[lebensterben]

lychee should first distinguish strict and fuzzy link detections.

strict means a string is a link if and only if it's rendered as a hyperlink in a HTML file. This naturally exclude tags like code or pre.

fuzzy means a string is a link if it's a valid URI of the scheme we support.

after implementing this, it's easier to offer finer control to users.

[mre]

Agree with both of your comments. I guess your ideas would fit together quite nicely.
The collector could return links with a tag field, that would be an HTML tag. We'd classify those as strict or fuzzy, so a pre would be fuzzy for example.
We could think about using different words than strict/fuzzy like hyperlink/plaintext or just link/text, but I like the proposal in general.

[MichaIng]

While checking our forum for outdated links, I recognised that also shortened URL text within anchor tags is checked. phpBB shortens long URLs. so that such an element is produced:

<a href="https://example.com/loong/looong/url/that/is/shorteneed/index.html" class="postlink">https://example.com/loong/looong/url/th ... index.html</a>

lychee now checks the URL itself but as well the text, where it identifies https://example.com/loong/looong/url/th as URL, which of course fails.

So once a basic functionality exists to exclude URLs/text based on HTML tags, the text content of tags/elements which have an explicit src/href/... attribute could be excluded by default, as AFAIK such shortening is quite common.

[mre]

That's odd. If this is a file with a .html extension, it should have skipped that shortened version. Can you double check that this is the case?

[MichaIng]

Ah this was when scanning URLs (with .php extensions and query string) and no local files. In case of URLs as input, it's of course harder to derive/guess the type 🤔. content-type: text/html response header could be used to interpret it as HTML, but I'm not sure how reliable this is, compared to a file extension. Otherwise additionally a contained <html> tag? But then it's getting complicated when currently the type/parser is obtained/chosen based on the extension only. Interpreting .php files as HTML in general also doesn't make sense as those could be PHP source scripts as well.

[MichaIng]

Adding this idea here: CSS selector syntax based excludes (and exclusive includes, if it turns out there is a wider use case for such) would be awesome. But I guess that's hard to achieve by hand, so only if the underlying HTML parser or other library supports such already.

[mre]

As a workaround for HTML tags, I wonder if lychee could be combined with htmlq or something similar.

htmlq --invert 'pre' | htmlq --invert 'code' | lychee

The --invert flag doesn't exist yet, but we could create an issue for that.

There's also pup, which has a PR open for that for a long time: ericchiang/pup#81.

[MichaIng]

But that would not work when using lychee's regular input arguments, especially complicated when wanting to scan URLs instead of local files.

It is not a that big issue for us. The -E flag + a small amount of --exclude's does the trick. It was more what we were used to from liche and code tags were hence a simple way to exclude example URLs or emails for web application access and login documentations, like (with Markdown `` converted to HTML code tags):

Access the web interface at: `http://<your.IP>:1234`
Username: `admin@example.org`

We have at least one web application where indeed an invalid example email is required for initial login, from where it can be changed 😉.

[mre]

Wait, you say liche supported this? Can you post an example that you used before? Can't find much about it in their repo.

[mre]

We have at least one web application where indeed an invalid example email is required for initial login, from where it can be changed 😉.

That sounds more like a bug / weird feature on the site no? You could exclude that with --exclude-mail right now? (There is no way to exclude a single mail address yet, but that might be an easier thing to add.)