ØMQ (zeromq) isn't quite working

[remy]

TL;DR - please have a look at my zeromq setup - it's wrong and needs fixing - help me Obi-Wan....

---

I'm using zeromq to post off the job of parsing all the CSS in a url, and then once it's complete, it passes the message back to the the main app.js node process.

Ideally, lots of requests could come in, and I could be running 2-4 worker processes (node prefix.js) and requests would queue up, get processed, and returned to the correct waiting http client.

My problem (aside from having never used zeromq or any queuing system for that matter) is that the job is passed off to the worker, but because (I think) I have an odd one-to-one queue set up, the same job gets returned to all of the clients.

It's tricky to replicate, because you need concurrent request - but here's an example:

curl "http://cssprefix.com/check.json?http://remysharp.com"; \
curl "http://cssprefix.com/check.json?http://jsconsole.com"; \
curl "http://cssprefix.com/check.json?http://remote-tilt.com"

{"pass":true}
{"pass":false,"lint":{"type":"warning","line":834,"col":3,"message":"The property -ms-transition is compatible with -webkit-transition, -moz-transition, -o-transition and should be included as well.","evidence":"  -webkit-transition: opacity 200ms;\r","rule":{"id":"compatible-vendor-prefixes","name":"Require compatible vendor prefixes","desc":"Include all compatible vendor prefixes to reach a wider range of users.","browsers":"All"}}}
{"pass":false,"lint":{"type":"warning","line":834,"col":3,"message":"The property -ms-transition is compatible with -webkit-transition, -moz-transition, -o-transition and should be included as well.","evidence":"  -webkit-transition: opacity 200ms;\r","rule":{"id":"compatible-vendor-prefixes","name":"Require compatible vendor prefixes","desc":"Include all compatible vendor prefixes to reach a wider range of users.","browsers":"All"}}}

Note that the last two return checks are exactly the same, even though the CSS is different for all three sites - which says to me that the worker returned to two clients - hence the screw up.

I'd love for someone with experience to dive in an have a decent crack at getting the parent / worker set up correct - then we'll have a web service that doesn't blow up my machine when concurrent requests come in!

[hardyjohnson]

Not sure about the return issue, but is there a reason you wouldn't use valid URIs in your API (i.e. http://cssprefix.com/check.json?url=URLENCODEDURL)?

Thanks,
Hardy

[mathiasbynens]

I haven’t looked at the code, but this sounds like it could be a scoping issue.

[remy]

@hardyjohnson no - there's not. But it's still in development and the scaling issue already hit my server hard enough to take the thing out. So yeah, the url should be validated first because I end up with junk files in the job directory.

@mathiasbynens the jobs are passed back and forth via sockets, and only a single socket is setup (with workers listening and grabbing). I think it needs an intermediate process to join requests up to responses and pipe them back and forth correctly, but I'm not savvy enough with zeromq to know how to go about doing that (or even if it's correct).

[lennym]

I'm learning about ZMQ sockets as I went on this, but even with just binding on('message', ...) in app.js then I still wasn't even getting the messages I was expecting from the prefix worker. Changing the socket type to 'pair' seemed to emit all the events I would expect, and swapping the per-request bindings to 'message' for a single listener which emits job specific events seems to fix your issue.

[remy]

@lennym so send me a pull request! It's utterly new to me and I couldn't find any decent examples that I could get what I needed on the web (though the docs are superb, I needed the starting thread).