Lexer hangs on bad PHP file from Codeception

[silbinarywolf]

Was looking at this project to look into making a fast PHP autocomplete daemon. Out of curiosity, any reason why the lexer is running in a Goroutine? Wouldn't it be the job of the Parser / user-code to run the lexer in a go-routine (if at all?)

File:
https://github.com/Codeception/Codeception/blob/2.2/tests/data/Invalid.php

File Contents:

<?php
$I do nothing here

My Golang loop:

for {
	token := lexer.Next()
	if token.Typ == PHPToken.EOF {
		break
	}
	// ... more
}

[stephens2424]

It's designed after Rob Pike's talk on the text/template lexer. Possible I got it wrong, but the idea is that the lexer is concurrent with its client (typically the parser), and lexes a token ahead of the parser, limited by the channel it emits tokens onto. It means the lexer is typically a step ahead of the parser, but never more.

I'm not able to reproduce the hang, though. I'm just trying it with the parserdebug command in the package and also on phpconsole.stephensearles.com. Feel free to upload a branch with a failing test of what you see is wrong.

[silbinarywolf]

Not sure I have time to, but if it helps this was occurring on a Windows 10 machine and I think I had the Lexer inside a Goroutine.