Quick pass to reduce reliance on Selenium

[PaulBrack]

I went through the test input list and targeted Selenium based scrapers to see if there were any I could quickly find that coalesced on GET endpoints where data could be looked up without doing anything like mocking authentication. Of the exactly 100 Selenium based scrapers, I was able to immediately identify 11 with such endpoints.

I am posting them here before re-writing any parsers using them to double-check my thinking before rewriting anything.

Notes:

1. I avoided anything that relied on cookies except where the cookie passed the UPRN/postcode as plain text - this is just another request header at this point. There are 2 of these and I trimmed them down to their most basic working format.
2. I avoided anything with authorisation, however two of these endpoints required an API token / subscription key.
   • I believe Leeds is fine because it's an openly issued subscription key
   • I believe using East Riding Council's API key would be misuse as-is. However, this is a proper API endpoint returning structured JSON and I have included this on the basis that I suspect a team who made such an endpoint would be open to issuing an open key that they could rate limit, if someone were to ask nicely?
3. There where a bunch of oldschool .aspx endpoints where one can, Selenium-free, use a scraper-based approach to pass the viewstate and other fields back in a valid request. Having looked at other discussions, I do not know if this would be considered an acceptable approach? This would be robust and I do not believe it's particularly hacky.

GET Endpoints:

• BarkingDagenham
  • GET https://www.lbbd.gov.uk/rest/bin/<UPRN>
• BroadlandDistrictCouncil
  • GET https://area.southnorfolkandbroadland.gov.uk/
  • Needs cookie MyArea.Data={"Uprn":"<UPRN>","Address":"<POSTCODE>"}
• EastRidingCouncil
  • GET https://wasterecyclingapi.eastriding.gov.uk/api/RecyclingData/CollectionsData?APIKey=ekBWR8tSiv6qwMo31REEeTZ5FAiMNB&Licensee=BinCollectionWebTeam&Postcode=<POSTCODE>
  • As per discussion above, this is potentially a misuse of an API key?
• EnfieldCouncil
  • GET https://www.enfield.gov.uk/_design/integrations/bartec/find-my-collection/rest/schedule?uprn=<UPRN>
• LeedsCityCouncil
  • GET https://api.leeds.gov.uk/public/waste/v1/BinsDays?uprn=<UPRN>&startDate=2026-01-12&endDate=2026-06-29
  • Needs header Ocp-Apim-Subscription-Key: ad8dd80444fe45fcad376f82cf9a5ab4
  • You can do a GET of the script first to get this, it's an open subscription key issued to all website users so I think it's fine?
    • GET https://shared-frontends.leeds.gov.uk/CheckYourBinDay/dist/dist/assets/index.js
    • Extract the API key using regex: key = re.search(r'apiKey\s*:\s*["\']([a-f0-9]{32})["\']', js, re.I).group(1) (you'll want to null check though)
  • The date range will need to be added to this scraper
• LondonBoroughOfRichmondUponThames
  • GET https://www.richmond.gov.uk/my_richmond?pid=<UPRN>
• LondonBoroughRedbridge
  • GET https://my.redbridge.gov.uk/RecycleRefuse
  • Needs cookie UPRN=<UPRN>
• MidUlsterDistrictCouncil
  • GET https://midulsterbincalendar.azurewebsites.net/api/collectiondates/<UPRN>
• NorthWestLeicestershire
  • GET https://my.nwleics.gov.uk/my-property-finder?address_id=nwl<UPRN>
• PeterboroughCityCouncil
  • GET https://report.peterborough.gov.uk/waste/<POSTCODE>:<UPRN>
• RenfrewshireCouncil
  • GET https://www.renfrewshire.gov.uk/bins-and-recycling/bin-collection/bin-collection-calendar/check-your-bin-collection-day/view/<UPRN>

POST Endpoints:

• ArunCouncil
  • POST https://www1.arun.gov.uk/when-are-my-bins-collected/select
  • request data data={"address": <UPRN>}
  • This requires capturing the auth cookie earlier in the flow

What now?

Put in issues for these tagged as 'improvement' maybe?

[dp247]

I originally wrote the Leeds parser (and some others, I forget), but the reason Selenium was introduced was to get around firewalls that councils (like Leeds) run when you try emulate browser behaviour repeatedly.

[PaulBrack]

Understood - however I suspect this is a new API? There was a new website introduced last year

[robbrad]

@PaulBrack nice work.

I never really wanted selenium in the project as it adds a big layer of setup for the user, however there are some that need temp creds and api keys which only make sense if accessing like a human does.

I'm cool with reducing the need for selenium if we can provide a stable way to get the data that won't use a cred or key that will expire.

[PaulBrack]

@robbrad thanks :)

I've edited my original post to include the following insurance against key expiry for Leeds:

• GET https://shared-frontends.leeds.gov.uk/CheckYourBinDay/dist/dist/assets/index.js
• Extract the API key using regex, e.g. : key = re.search(r'apiKey\s*:\s*["\']([a-f0-9]{32})["\']', js, re.I).group(1) (you'll want to null check though)

[PaulBrack]

@robbrad I've done 4/12 of these - I'll await comments before tackling the others