Crawl https sitemap with http urls

[SaschaHeyer]

Hi Pascal,

how can I crawl a sitemap with is reachable with https but contains urls with http. Norconex is not identifying any startURLs in that case.

I tried already setting lenient to true

<sitemapResolverFactory ignore="false" lenient="true" class="com.norconex.collector.http.sitemap.impl.StandardSitemapResolverFactory">
      <path></path>
</sitemapResolverFactory>

As well as stayOnProtocol to false

<startURLs stayOnDomain="true" stayOnPort="true" stayOnProtocol="false">
        #parse("./shared/startUrls-sitemap.xml")
</startURLs>

Any other recommendations?

Best regards
Sascha

[essiembre]

I would try stayOnPort="false" given https is on 443.

[SaschaHeyer]

No difference with stayOnPort="false"

[essiembre]

Have you tried using GenericURLNormalizer and convert the said URLs to https with the secureScheme normalization?

[jetnet]

same issue here.
secureScheme helps with such sites, but unfortunately it cannot be used globally, as we have to crawl many unsecured sites too.
Any suggestions? Thanks!

[essiembre]

That is a tough one because any URLs, even within the same site, might be supporting different schemes: http, https, or both.

We could investigate a new feature that always tries the "preferred" scheme first, and if failing, falling back to the other. But for some sites, it could pretty much double the number of "hits" on the server.

My preference would be to encourage identifying those sites instead and handle them separately, in their own crawler config let's say. It may not always be the most realistic, but preferable when possible.

If you have better options, I would like to hear them.