Improvements in the application architecture to improve performance

[sak007]

👋 Welcome Team!

@zedunaid @sak1997 @wizard-31 @rjain09

There seems to be a lot of issues with the current architecture of the application. The following are my findings and suggestions. Please share your thoughts and ideas in the comments.

• Mail is sent to the users every time the scrapper is executed. Mail should be sent only once.
• User information is not deleted after sending mail.
• Database is used as alternative for API which makes the application very slow and inefficient.
• There are lots of redundant code which needs to be refactored.
• Directory structure is not proper. There should be a code and test directory and the test files should be in proper path.
• Documentation is not proper.

[sak1997]

Also, the Right To Be Forgotten is not implemented. Users do not have the authority to delete their data from the application at any point.

We should not be storing the data at all if we're just going to be getting their emails and parsing their resume each time they run the application

[sak007]

Yeah, this is a major issue. This should be fixed.

[sak1997]

Agreed. No point in using a DB when a JSON file would do instead.

[sak007]

Instead of discarding the DB, can we utilize the existing DB infrastructure to create a scheduled job notifier?
We can also add a feature to delete user details which will enforce Right To Be Forgotten.

[sak1997]

RE: Scheduled job notifier - That is a good idea, since scraping can take time and also since the user would need to run the application each time. Also, it can take a while each time to scrape the data, so this way, we can save time and send to users by filtering after the scraping is done

[sak1997]

RE: Right to be Forgotten - Yes, we need to either not store their info, or we can implement a delete data feature.

[sak1997]

Additionally, a lot of the docs seem to be a mix of PDFs and markdown files. Let's get a standard set up to use ONLY markdown files since that would make the most sense on Github.

[wizard-31]

I'll take a look into it and see where we could optimize.

[sak1997]

We can wait on this until we're done with the architectural changes - I believe this would make sense.

[sak007]

Looks like basic selenium scripts are used for web scrapping. We should use BeautifulSoup instead.