Skip to content

Added multithreading for 4x quicker applying - #21

Closed
BigweldIndustries wants to merge 5 commits into
SeanDaBlack:mainfrom
BigweldIndustries:main
Closed

Added multithreading for 4x quicker applying#21
BigweldIndustries wants to merge 5 commits into
SeanDaBlack:mainfrom
BigweldIndustries:main

Conversation

@BigweldIndustries

Copy link
Copy Markdown
Contributor

Hey all! I recently came across from r/antiwork and thought I would give this script a try. Thanks to the awesome work of SeanDaBlack and other contributors, I found it does. However, as the selenium snob I am, I felt it had a bit to be desired. First of all, it is now supported on linux, and requires no webdriver installation. This was done via a pip dependency called webdriver_manager. I then sped up the program to send about three at a time (not too many, chrome windows can be laggy). This was done through the futures dependency, which allows the simple creation of threadpools. Best of luck, comrades!

@tommmmyb

Copy link
Copy Markdown

Nicely done. I did a similar thing with my fork, 4 threads at a time for a few hours - 1500+ applications. Now I'm wishing I'd used futures instead of multiprocessing - looks a lot cleaner. Too bad the reCaptcha ruined my fun

@BigweldIndustries

Copy link
Copy Markdown
Contributor Author

Yes, futures is incredible. Makes multithreading so much easier, and isn't that hard to implement. Whenever manually working with threads outside of threadpools my brain just dies.

Comment thread README.md
# Setup

On mac/pc:
# Setup (Mac/Windows/Linux)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great that you've been able to streamline the process for chromedriver as such. Code related to this from this PR should be merged to main immediately.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created a seperate pull request due to this one becoming a bit buggy due to many different purposes.

@pws1453 pws1453 mentioned this pull request Dec 11, 2021
Comment thread req.py
driver.close()
continue
def main():
global processes

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the proper way to run multiple things simultaneously. This functionality mirrors what I believe is the intent of #16 , but uses the proper library.

@pws1453 pws1453 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code is written well, and is readable. Functionality improves tremendously. Can be pulled into main, but preferably merged with the readability changes made in #26

@BigweldIndustries

Copy link
Copy Markdown
Contributor Author

Sorry all, this is behind many commits and I am meaning to rework lots of this. I have created a seperate request for webdriver-manager. I will create one later for multiproccessing after captcha and webdriver-manager are implemented. For now, defer to #27

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants