Bot isn't working

[JaxzKyd]

The bot just opens scrcpy and does nothing, when I solve a word it keeps clicking the dictionary button in the top left corner

bash arg: com.peoplefun.wordcross
bash arg: 1
args: [-p, com.peoplefun.wordcross, 1]
 arg: "-p"
 arg: "com.peoplefun.wordcross"
 arg: "1"
data="com.peoplefun.wordcross"
Events injected: 1
## Network stats: elapsed time=22ms (0ms mobile 0ms wifi, 22ms not connected)

[asarandi]

hi,
my guess is that the script is not able to capture a proper screenshot of scrcpy window
it takes a screenshot of area 0, 0 to 768 H, 353 W

if you have window borders, top menu bars, or different device aspect ratio (mine is 19.5:9) then it might fail to capture the proper screenshot.

that being said, i will try to help you debug

---

clone and install requirements:

cd /tmp
git clone https://github.com/asarandi/wordscapes-bot
cd wordscapes-bot/
python3 -m venv .
source bin/activate
pip3 install -r requirements.txt

run the script:

./script.py

the script should launch scrcpy and it should open a window that is 768x353 in your desktop's top left corner

the initial screen capture happens here:

wordscapes-bot/script.py

Line 202 in fc6e1ad

capture = pyautogui.screenshot(region=region).convert("RGB")

right after that line, you can add this debug code to see what the script is able to capture:

        capture.show(); sys.exit()

this is what it looks like for me:

please confirm that you're getting similar results.
thank you

[JaxzKyd]

okay, after doing install -r requirements.txt command, it couldn't find Pillow==9.2.1, the best it could do was install 9.2.0
And after adding the capture.show thing to the script it only opened scrcpy and displayed an error

not the object's thread (0x55cccb8af0a0).

Cannot move to target thread (0x55cccb8a8080)



qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/jaxz/.local/lib/python3.10/site-packages/cv2/qt/plugins" even though it was found.

This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.



Available platform plugins are: xcb, eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx.



/usr/bin/xdg-open: line 613: 31916 Aborted                 (core dumped) kde-open${KDE_SESSION_VERSION} "$1"

[asarandi]

hi,
i think the error is due to capture.show(),
please change the debug code to

        capture.save("/tmp/capture.png"); sys.exit()

then try and open the file with some viewer, maybe even with your browser

[JaxzKyd]

This is the imagine it made

[asarandi]

as i suspected that is not what it should "see",
does it not launch the scrcpy program? or was it running already?
please close scrcpy and try again, it should launch it and place it in top-left corner

[JaxzKyd]

It did launch scrcpy, and it did appear in the top left corner

[asarandi]

ok, assuming scrcpy is running in top left corner and has proper window height and width;
what happens when you run the script again? can you get a proper capture file now ?

[JaxzKyd]

Yes! I can get a proper capture file this time (which is strange since I didn't change anything since the first time I ran it

[JaxzKyd]

Okay now running it with capture.save("/tmp/capture.png"); sys.exit() it just keeps hitting the shuffle letters button

[asarandi]

i see some empty space on the right side of the capture image
it seems that your device's aspect ratio is a little different than mine,
the script might need some fine tuning, please look at the "config" object:

wordscapes-bot/script.py

Line 21 in fc6e1ad

config = {

based on the screenshot you posted, please try these values:

config = {
    "window_max_size": 768,
    "window_height": 768,
    "screenshot_region": (0, 0, 344, 768 + y_offset),
    "circle_center": (172, 596 + y_offset),
    "shuffle_button": (32, 507 + y_offset),
    "close_piggybank": (285, 210 + y_offset),
    "circle_radius": 109,
    "rearranged_box_size": 64,
    "rearranged_padding": 8,
    "forbid_three_circle": (43, 43 + 22, 685 + y_offset, 685 + 22 + y_offset),
    "letter_contour_dim": (30, 39, 6, 50),
}

beware, these are rough and might need to be further adjusted,
i would recommend you open the screenshot in an image editor (i use gimp)
then you can figure out X, Y offsets and positions
(gimp shows the coordinates of mouse cursor over image in bottom left corner)

---

don't forget to remove the debug line you inserted earlier: capture.save(); sys.exit()

[JaxzKyd]

I ran it and received this error

Traceback (most recent call last):

  File "/home/jaxz/.local/lib/python3.10/site-packages/pytesseract/pytesseract.py", line 254, in run_tesseract

    proc = subprocess.Popen(cmd_args, **subprocess_args())

  File "/usr/lib/python3.10/subprocess.py", line 969, in __init__

    self._execute_child(args, executable, preexec_fn, close_fds,

  File "/usr/lib/python3.10/subprocess.py", line 1845, in _execute_child

    raise child_exception_type(errno_num, err_msg, err_filename)

FileNotFoundError: [Errno 2] No such file or directory: 'tesseract'



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "/home/jaxz/wordscapes-bot/./script.py", line 204, in <module>

    success, detected = detect_letters(capture)

  File "/home/jaxz/wordscapes-bot/./script.py", line 135, in detect_letters

    tes = pytesseract.image_to_string(rearranged, config="--psm 13")

  File "/home/jaxz/.local/lib/python3.10/site-packages/pytesseract/pytesseract.py", line 416, in image_to_string

    return {

  File "/home/jaxz/.local/lib/python3.10/site-packages/pytesseract/pytesseract.py", line 419, in <lambda>

    Output.STRING: lambda: run_and_get_output(*args),

  File "/home/jaxz/.local/lib/python3.10/site-packages/pytesseract/pytesseract.py", line 286, in run_and_get_output

    run_tesseract(**kwargs)

  File "/home/jaxz/.local/lib/python3.10/site-packages/pytesseract/pytesseract.py", line 258, in run_tesseract

    raise TesseractNotFoundError()

pytesseract.pytesseract.TesseractNotFoundError: tesseract is not installed or it's not in your PATH. See README file for more information.

which is very strange cause I did do pip install -r requirements.txt and it did say it installed pytesseract, strange

[asarandi]

have a look at the detect_letters function, there are 3 debug locations that are commented out,
you can un-comment those one by one and compare them to these reference images:

initial capture:

here:

wordscapes-bot/script.py

Line 87 in fc6e1ad

# cv.imshow("debug 1", otsu); cv.waitKey(); print(otsu.shape); sys.exit()

here:

wordscapes-bot/script.py

Line 99 in fc6e1ad

# cv.imshow("debug 2", otsu); cv.waitKey(); sys.exit()

and here:

wordscapes-bot/script.py

Line 133 in fc6e1ad

# cv.imshow("debug 3", rearranged); cv.waitKey(); sys.exit()

[asarandi]

regarding tesseract on a debian based linux system do:

sudo apt install tesseract-ocr tesseract-ocr-eng

otherwise try to search for "tesseract" in your package manager,
or consult the docs:
https://tesseract-ocr.github.io/tessdoc/Home.html

[JaxzKyd]

Okay after installing tesseract with my package manager the bot did start to work!
My last question is why does it keep opening the piggy bank on the right side of the screen after it finishes a level? Is there a way to make it so it hits the next level button?

[asarandi]

i'm glad it's working now,
you can try and comment out lines 206, 207 see if that makes a difference:

wordscapes-bot/script.py

Line 207 in fc6e1ad

click(config["close_piggybank"])

the developers of the app keep changing things and newer versions of the app keep bringing up different windows and modals that were not there back when i wrote the script, same goes with X, Y positions of things on screen, when they do that the script will break and will need fine-tuning