Issues

• Can this be used as a validator?

  chrisakroyd/robots-txt-parser#19 · Filter by author stevenvachon opened on Sep 23, 2025

• sitemaps links are not returned for some websites (ex: https://www.sainsburys.co.uk)

  chrisakroyd/robots-txt-parser#17 · Filter by author kvamsij opened on Aug 15, 2024

• canCrawl stalls in nodejs but not browser

  chrisakroyd/robots-txt-parser#16 · Filter by author salivity opened on Jun 16, 2024
  1

• canCrawl() returns incorrect value for middle paths

  chrisakroyd/robots-txt-parser#15 · Filter by author ssaumya1711 opened on Jan 14, 2024

• Valid URL results in error

  chrisakroyd/robots-txt-parser#12 · Filter by author dskvr opened on Feb 19, 2023

• canCrawl() returns incorrect result when matching middle of path

  chrisakroyd/robots-txt-parser#8 · Filter by author Trott opened on Feb 1, 2023

• Bug (in useRobotsFor()?) causes canCrawl() to sometimes return incorrect result

  chrisakroyd/robots-txt-parser#5 · Filter by author Trott opened on Nov 5, 2021