Is your feature request related to a problem? Please describe.
At the moment the list of supported stores is hard coded in the crawler's code. So editing it is not practical as we will need to edit the code and commit the changes to the repo.
Describe the solution you'd like
We should move the supported stores list to the database by adding a boolean support_scraping column to the vendors column. Then the scraper will fetch all vendors where the value is set to True.
Describe alternatives you've considered
None
Additional context
We could also set a column for the parser location like this the script can fetch everything in one go, or leave it as is, and the script will link the hard coded parser's location to the vendor table it would fetch from the database.
This could also make the scraper fetch the url of the vendor in the same way. It will allow to modify it easily to.
This could be paired with a dashboard on the website for the staff members to directly modify which vendors are supported.
Is your feature request related to a problem? Please describe.
At the moment the list of supported stores is hard coded in the crawler's code. So editing it is not practical as we will need to edit the code and commit the changes to the repo.
Describe the solution you'd like
We should move the supported stores list to the database by adding a boolean support_scraping column to the vendors column. Then the scraper will fetch all vendors where the value is set to True.
Describe alternatives you've considered
None
Additional context
We could also set a column for the parser location like this the script can fetch everything in one go, or leave it as is, and the script will link the hard coded parser's location to the vendor table it would fetch from the database.
This could also make the scraper fetch the url of the vendor in the same way. It will allow to modify it easily to.
This could be paired with a dashboard on the website for the staff members to directly modify which vendors are supported.