[R3gardless]

PostgreSQL gives the position of a syntax error as a 1-based character index, but the parser functions treat it as an offset in the UTF-8 bytes and convert it once more, so with multibyte characters before the error the reported index is too small:

>>> parse_sql("SELECT '한글', FROM x")
pglast.parser.ParseError: syntax error at or near "FROM", at index 9

where FROM is at index 13. This uses the position as it comes, so the reported index changes only when there are non-ASCII characters before the error. For plain ASCII statements it stays the same.

Since v7 has the same problem, I based this on v7 to save you the transplant. The regenerated parser.c is in a separate commit.

[lelit]

Thank you!

I think the failure on CI is completely unrelated to this and due to a regression I introduced in commit 44abbff, using a feature of IntFlag absent in old snakes.
I pushed a possible fix on v7 (commit b41caab), if that goes green please rebase this on top of v7.

[lelit]

Sigh, no, that is not enough...

[lelit]

Ok, I think the problem is gone.

[lelit]

I'm surprised by your finding 🤔 ... are you saying that the whole Displacement machinery is wrong?

[R3gardless]

Thanks for the quick fix, I rebased this on top of v7.

Sorry for the confusion, I should have explained it better. No, Displacements is fine: node locations, scan() tokens and split() slices are byte offsets, and it converts them correctly. Only the error position is different, because PostgreSQL already reports it in characters (scanner_errposition() turns the byte offset into a character number with pg_mbstrlen_with_len()), so it was converted twice.

While rebasing I also removed a leftover Displacements(stmts) call in split() that I had missed.