tests are failing on 3.0 branch

[jeffkowalski]

While some of the initial functionality of geeknote appears to work, a number of the unit tests are failing.
Most of the failures are due to string encoding, apparently. Encoding with evernote is quite finicky.

============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/jeff/Dropbox/workspace/repos/geeknote
plugins: anyio-3.4.0
collected 114 items

tests/test_argparser.py .............. [ 12%]
tests/test_editor.py FF..... [ 18%]
tests/test_gclient.py .. [ 20%]
tests/test_geeknote.py ...F.. [ 25%]
tests/test_gnsync.py FFFFF [ 29%]
tests/test_out.py ................. [ 44%]
tests/test_sandbox.py ssssssssssssssss [ 58%]
tests/test_storage.py .....F...................F...F..... [ 89%]
tests/test_tools.py ...F........ [100%]

=================================== FAILURES ===================================
__________________________ testEditor.test_ENMLToText __________________________

self = <test_editor.testEditor testMethod=test_ENMLToText>

def test_ENMLToText(self):
    wrapped = Editor.wrapENML(self.HTML_TEXT)

  self.assertEqual(Editor.ENMLtoText(wrapped), self.MD_TEXT)

E AssertionError: b'# Header 1\n\n## Header 2\n\nLine 1\n\n_Line 2_\n\nLine 3\n\n' != '# Header 1\n\n## Header 2\n\nLine 1\n\n_Line 2_\n\nLine 3\n\n'

tests/test_editor.py:34: AssertionError
_____________________________ testEditor.test_TODO _____________________________

self = <test_editor.testEditor testMethod=test_TODO>

def test_TODO(self):
    MD_TODO = "\n* [ ]item 1\n\n* [x]item 2\n\n* [ ]item 3\n\n"
    HTML_TODO = "<div><en-todo></en-todo>item 1</div><div><en-todo checked=\"true\"></en-todo>item 2</div><div><en-todo></en-todo>item 3</div>\n"
    self.assertEqual(Editor.textToENML(MD_TODO),
                     Editor.wrapENML(HTML_TODO))

    wrapped = Editor.wrapENML(HTML_TODO)
    text = Editor.ENMLtoText(wrapped)

  self.assertEqual(text, MD_TODO)

E AssertionError: b'\n* [ ]item 1\n\n* [x]item 2\n\n* [ ]item 3\n\n' != '\n* [ ]item 1\n\n* [x]item 2\n\n* [ ]item 3\n\n'

tests/test_editor.py:44: AssertionError
____________________ testNotes.test_editWithEditorInThread2 ____________________

self = <test_geeknote.testNotes testMethod=test_editWithEditorInThread2>

def test_editWithEditorInThread2(self):
    txt = "Please delete this line, save, and quit the editor"

  self.check_editWithEditorInThread(txt, "\n\n")

tests/test_geeknote.py:98:

---

tests/test_geeknote.py:90: in check_editWithEditorInThread
self.assertEqual(Editor.ENMLtoText(testNote.content), expected)
E AssertionError: b'\n\n' != '\n\n'
----------------------------- Captured stderr call -----------------------------
Edited note could not be saved, so it remains in /tmp/tmps7egj5cr.markdown
_______________ testGnsync.test_create_file_with_non_ascii_chars _______________

self = <test_gnsync.testGnsync testMethod=test_create_file_with_non_ascii_chars>
mock_storage =
mock_geeknote =
mock_logger =

@patch('geeknote.gnsync.logger', autospec=True)
@patch('geeknote.gnsync.GeekNote', autospec=True)
@patch('geeknote.gnsync.Storage', autospec=True)
def test_create_file_with_non_ascii_chars(self, mock_storage, mock_geeknote, mock_logger):

    # Mock GeekNote#loadNoteContent to provide some content with non-ascii
    def mock_note_load(note):
        with open('tests/fixtures/Test Note with non-ascii.xml', 'r') as f:
            note.content = f.read()
        note.notebookName = 'testNotebook'
    mock_geeknote.return_value.loadNoteContent.side_effect = mock_note_load

    # Mock Storage().getUserToken() so the GNSync constructor doesn't throw
    # an exception
    mock_storage.return_value.getUserToken.return_value = True

    subject = GNSync('test_notebook', self.test_dir, '*.*',
                     'plain', download_only=True)

    mock_note = Mock()
    mock_note.guid = '123abc'
    mock_note.title = 'Test Note'

    subject._create_file(mock_note)

    # Verify that a file could be created with non-ascii content
    mock_logger.exception.assert_not_called_with(
        ANY,
        AnyStringWith("codec can't decode byte")
    )

    with open(self.test_dir + "/Test Note.txt", 'r') as f:

      self.assertIn("œ ž © µ ¶ å õ ý þ ß Ü", f.read())

E AssertionError: 'œ ž © µ ¶ å õ ý þ ß Ü' not found in ''

tests/test_gnsync.py:90: AssertionError
__________________________ testGnsync.test_strip_chn ___________________________

self = <test_gnsync.testGnsync testMethod=test_strip_chn>

def test_strip_chn(self):

  self.assertEqual(remove_control_characters(self.given_chn.decode('utf-8')).encode('utf-8'),

                     self.expected_chn)

E AttributeError: 'str' object has no attribute 'decode'

tests/test_gnsync.py:51: AttributeError
__________________________ testGnsync.test_strip_eng ___________________________

self = <test_gnsync.testGnsync testMethod=test_strip_eng>

def test_strip_eng(self):

  self.assertEqual(remove_control_characters(self.given_eng.decode('utf-8')).encode('utf-8'),

                     self.expected_eng)

E AttributeError: 'str' object has no attribute 'decode'

tests/test_gnsync.py:43: AttributeError
__________________________ testGnsync.test_strip_kor ___________________________

self = <test_gnsync.testGnsync testMethod=test_strip_kor>

def test_strip_kor(self):

  self.assertEqual(remove_control_characters(self.given_kor.decode('utf-8')).encode('utf-8'),

                     self.expected_kor)

E AttributeError: 'str' object has no attribute 'decode'

tests/test_gnsync.py:47: AttributeError
________________________ testGnsync.test_strip_nochange ________________________

self = <test_gnsync.testGnsync testMethod=test_strip_nochange>

def test_strip_nochange(self):

  self.assertEqual(remove_control_characters(self.given_combined.decode('utf-8')).encode('utf-8'),

                     self.expected_combined)

E AttributeError: 'str' object has no attribute 'decode'

tests/test_gnsync.py:55: AttributeError
__________________ storageTest.test_get_setting_exist_success __________________

self = <test_storage.storageTest testMethod=test_get_setting_exist_success>

def test_get_setting_exist_success(self):
    self.storage.setSettings({'editor': 'vim'})
    editor = self.storage.getSetting('editor')

  self.assertEqual(pickle.loads(editor), 'vim')

E TypeError: a bytes-like object is required, not 'str'

tests/test_storage.py:91: TypeError
____________________ storageTest.test_set_settings_success _____________________

self = <test_storage.storageTest testMethod=test_set_settings_success>

def test_set_settings_success(self):
    self.storage.setSettings({'editor': 'vim'})

  self.assertEqual(self.storage.getSettings(),

                      {'editor': "S'vim'\np0\n."})

E AssertionError: {'editor': b'\x80\x04\x95\x07\x00\x00\x00\x00\x00\x00\x00\x8c\x03vim\x94.'} != {'editor': "S'vim'\np0\n."}
E - {'editor': b'\x80\x04\x95\x07\x00\x00\x00\x00\x00\x00\x00\x8c\x03vim\x94.'}
E + {'editor': "S'vim'\np0\n."}

tests/test_storage.py:73: AssertionError
____________________ storageTest.test_update_settings_fail _____________________

self = <test_storage.storageTest testMethod=test_update_settings_fail>

def test_update_settings_fail(self):
    self.storage.setSettings({'editor': 'vim'})
    self.assertTrue(self.storage.setSettings({'editor': 'nano'}))

  self.assertEqual(self.storage.getSettings(),

                      {'editor': "S'nano'\np0\n."})

E AssertionError: {'editor': b'\x80\x04\x95\x08\x00\x00\x00\x00\x00\x00\x00\x8c\x04nano\x94.'} != {'editor': "S'nano'\np0\n."}
E - {'editor': b'\x80\x04\x95\x08\x00\x00\x00\x00\x00\x00\x00\x8c\x04nano\x94.'}
E + {'editor': "S'nano'\np0\n."}

tests/test_storage.py:85: AssertionError
______________________ testTools.test_decode_args_success ______________________

self = <test_tools.testTools testMethod=test_decode_args_success>

def test_decode_args_success(self):
    result = [1, '2', 'test', '\xc2\xae',
              '\xd1\x82\xd0\xb5\xd1\x81\xd1\x82']

  self.assertEqual(decodeArgs([1, '2', 'test', '®', 'тест']), result)

E AssertionError: Lists differ: [1, '2', 'test', '®', 'тест'] != [1, '2', 'test', '®', 'Ñ\x82еÑ\x81Ñ\x82']
E
E First differing element 3:
E '®'
E '®'
E
E - [1, '2', 'test', '®', 'тест']
E + [1, '2', 'test', '®', 'Ñ\x82еÑ\x81Ñ\x82']

tests/test_tools.py:44: AssertionError
=============================== warnings summary ===============================
../../../../../../usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3
/usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
from collections import Mapping

tests/test_geeknote.py::testNotes::test_editWithEditorInThread2
tests/test_geeknote.py::testNotes::test_editWithEditorInThread2
/home/jeff/.local/lib/python3.8/site-packages/geeknote/geeknote.py:1006: DeprecationWarning: isAlive() is deprecated, use is_alive() instead
if not thread.isAlive():

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_editor.py::testEditor::test_ENMLToText - AssertionError: b'...
FAILED tests/test_editor.py::testEditor::test_TODO - AssertionError: b'\n* [ ...
FAILED tests/test_geeknote.py::testNotes::test_editWithEditorInThread2 - Asse...
FAILED tests/test_gnsync.py::testGnsync::test_create_file_with_non_ascii_chars
FAILED tests/test_gnsync.py::testGnsync::test_strip_chn - AttributeError: 'st...
FAILED tests/test_gnsync.py::testGnsync::test_strip_eng - AttributeError: 'st...
FAILED tests/test_gnsync.py::testGnsync::test_strip_kor - AttributeError: 'st...
FAILED tests/test_gnsync.py::testGnsync::test_strip_nochange - AttributeError...
FAILED tests/test_storage.py::storageTest::test_get_setting_exist_success - T...
FAILED tests/test_storage.py::storageTest::test_set_settings_success - Assert...
FAILED tests/test_storage.py::storageTest::test_update_settings_fail - Assert...
FAILED tests/test_tools.py::testTools::test_decode_args_success - AssertionEr...
============ 12 failed, 86 passed, 16 skipped, 3 warnings in 1.03s =============

[alperenkose]

Hi Jeff,

Thanks a lot for pointing out this. Haven't focused on the test cases yet. Would you be kind to check when I update them to make sure everything is fine.
Btw I am using it with no issues but will definitely check the tests.

Thanks,

[jeffkowalski]

Sure. You can futurize the tests, and even run them yourself if you pip3 install mock and pip3 install pytest, then simply run pytest from the top directory. Nice job taking the initiative on the upgrade, btw.

…

On Wed, Jan 26, 2022 at 2:53 PM Alp Eren Kose ***@***.***> wrote: Hi Jeff, Thanks a lot for pointing out this. Haven't focused on the test cases yet. Would you be kind to check when I update them to make sure everything is fine. Btw I am using it with no issues but will definitely check the tests. Thanks, — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAN4MGD4HVBQOZ7DC7WA2LUYB3PZANCNFSM5M35YJBQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[alperenkose]

@jeffkowalski There is the following part failing in the test cases, which comes down to stdinEncode method.
As far as I understand we no longer need to worry about decoding stdin and encoding to utf-8 in Python 3 right, could you please confirm that?

def test_decode_args_success(self):
    result = [1, '2', 'test', '\xc2\xae',
              '\xd1\x82\xd0\xb5\xd1\x81\xd1\x82']
    self.assertEqual(decodeArgs([1, '2', 'test', '®', 'тест']), result)

def stdinEncode(data):
    try:
        return data.decode(sys.stdin.encoding).encode("utf8")
    except:
        return data

[jeffkowalski]

Thanks for trying it out. I am afraid I don't understand the encoding differences between Python2.7 and Python3. That's precisely why I never upgraded geeknote. But if you are confident that different encodings for note content and title make it into and out of the system, then please remove the obsolete tests and create one to exercise the case.

…

On Sun, Jan 30, 2022 at 12:02 PM Alp Eren Kose ***@***.***> wrote: @jeffkowalski <https://github.com/jeffkowalski> There is the following part failing in the test cases, which comes down to stdinEncode method. As far as I understand we no longer need to worry about decoding stdin and encoding to utf-8 in Python 3 right, could you please confirm that? def test_decode_args_success(self): result = [1, '2', 'test', '\xc2\xae', '\xd1\x82\xd0\xb5\xd1\x81\xd1\x82'] self.assertEqual(decodeArgs([1, '2', 'test', '®', 'тест']), result) def stdinEncode(data): try: return data.decode(sys.stdin.encoding).encode("utf8") except: return data — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAN4MA4T76RAPDYKSLTVB3UYWKOXANCNFSM5M35YJBQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[alperenkose]

Thanks Jeff. The purpose of this code piece in Python 2 is to convert any input from user to utf-8 encoded byte string right? Could you please provide information on that in Python 2 to help me understand how to port it to Python 3.

[jeffkowalski]

Very sorry, Alp, but I inherited the code (it had been unmaintained - I fixed bugs, integrated pr's, etc., but couldn't call myself an author). I don't know the reason behind the utf-8 encoding, other than it worked. Perhaps it's no longer relevant in Python 3. I've been disappointed in the challenge in maintaining geeknote. Recently, I created a very lightweight version in Ruby only 300 lines long. I've been more interested in propelling that rather than continuing with the broadly-scoped project that geeknote has become. I'd be happy to share that project with you, but it'll be several months until I might be able to devote any time to maintaining the Python geeknote I no longer use.

…

On Mon, Jan 31, 2022 at 9:56 PM Alp Eren Kose ***@***.***> wrote: Thanks Jeff. The purpose of this code piece in Python 2 is to convert any input from user to utf-8 encoded byte string right? Could you please provide information on that in Python 2 to help me understand how to port it to Python 3. — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAN4MDZVPUKLTT66JYP63TUY5Y2RANCNFSM5M35YJBQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[alperenkose]

Hi Jeff,
Thank you very much, actually I am more of a sysadmin guy I do not have expertise with Ruby but thanks for offering that.

I have done some further testing with this when default locale is other than utf-8 - made a few changes to make sure it works when the default sys.stdin.encoding is different as well.

Looks like it is working as expected. I couldn't mock sys.stdin.encoding for the test code though which needs work. Apart from this all the tests are passing now.

I will close this now, let us track other issues separately.
Thanks,