Deleting queues doesn't work

[StabbyCutyou]

Calling the delete endpoint did not clear queues out of active memory, and upon server restart the queues were still detected for processing.

Likely something simple with the delete action, but needs looking into.

[StabbyCutyou]

Woops - looks like we don't even have a proper route for deleting queues alone - just queues from a topic. That's likely the brunt of the issue here.

[lumost]

heh oversight :)

for the delete operation do we want to simply delete the config option or delete all messages in the queue?

--James

[StabbyCutyou]

All messages, I'd wager.

Orphaned data is data we'll never get rid of

On Fri, Dec 19, 2014 at 10:18 AM, James Moore notifications@github.com
wrote:

heh oversight :)

for the delete operation do we want to simply delete the config option or
delete all messages in the queue?

--James

—
Reply to this email directly or view it on GitHub
#40 (comment).

[lumost]

agreed I'll get something up on this task later today,

one problem that I can forsee is removing the queue from any subscribed
topics.

--James

On Fri, Dec 19, 2014 at 10:19 AM, StabbyCutyou notifications@github.com
wrote:

All messages, I'd wager.

Orphaned data is data we'll never get rid of

On Fri, Dec 19, 2014 at 10:18 AM, James Moore notifications@github.com
wrote:

heh oversight :)

for the delete operation do we want to simply delete the config option
or
delete all messages in the queue?

--James

—
Reply to this email directly or view it on GitHub
#40 (comment).

—
Reply to this email directly or view it on GitHub
#40 (comment).

[StabbyCutyou]

I was thinking about that too....

Removing queues is a rare operation, realistically. It's not crazy to think
it's a slow op, which takes time to complete - ergo, it could be ok to do
the dumb thing at first and just cycle through topics and remove the queue.

We could also likely do a reverse queue-to-topic map, which would remove
the time spent on the delete, but add another piece of data to track...

On Fri, Dec 19, 2014 at 10:28 AM, James Moore notifications@github.com
wrote:

agreed I'll get something up on this task later today,

one problem that I can forsee is removing the queue from any subscribed
topics.

--James

On Fri, Dec 19, 2014 at 10:19 AM, StabbyCutyou notifications@github.com
wrote:

All messages, I'd wager.

Orphaned data is data we'll never get rid of

On Fri, Dec 19, 2014 at 10:18 AM, James Moore notifications@github.com

wrote:

heh oversight :)

for the delete operation do we want to simply delete the config option
or
delete all messages in the queue?

--James

—
Reply to this email directly or view it on GitHub
#40 (comment).

—
Reply to this email directly or view it on GitHub
#40 (comment).

—
Reply to this email directly or view it on GitHub
#40 (comment).

[lumost]

by far the slowest op will be deleting the data, so cycling through the
topics isn't the biggest of deals

the part that I think will be sketchy is the deletion of the queue ( as
once its deleted we won't be able to call delete again )

1. I'm thinking the way of doing this is delete the queue from all topics.
2. kick off a potentially rate limited delete thread which will iterate
   through the queue and delete messages
3. when a full 2i query returns no messages the queue should be deleted
   from the config.

this would allow at the minimum for a user to call delete again should the
thread fail.

one other question is whether we should flag the queue as being deleted and
block writes?

--James

On Fri, Dec 19, 2014 at 10:32 AM, StabbyCutyou notifications@github.com
wrote:

I was thinking about that too....

Removing queues is a rare operation, realistically. It's not crazy to
think
it's a slow op, which takes time to complete - ergo, it could be ok to do
the dumb thing at first and just cycle through topics and remove the
queue.

We could also likely do a reverse queue-to-topic map, which would remove
the time spent on the delete, but add another piece of data to track...

On Fri, Dec 19, 2014 at 10:28 AM, James Moore notifications@github.com
wrote:

agreed I'll get something up on this task later today,

one problem that I can forsee is removing the queue from any subscribed
topics.

--James

On Fri, Dec 19, 2014 at 10:19 AM, StabbyCutyou notifications@github.com

wrote:

All messages, I'd wager.

Orphaned data is data we'll never get rid of

On Fri, Dec 19, 2014 at 10:18 AM, James Moore <
notifications@github.com>

wrote:

heh oversight :)

for the delete operation do we want to simply delete the config
option
or
delete all messages in the queue?

--James

—
Reply to this email directly or view it on GitHub
#40 (comment).

—
Reply to this email directly or view it on GitHub
#40 (comment).

—
Reply to this email directly or view it on GitHub
#40 (comment).

—
Reply to this email directly or view it on GitHub
#40 (comment).

[lumost]

@StabbyCutyou @mattcarbone

[StabbyCutyou]

I think that approach makes sense...

1. Make sure no new messages get in (cut queue out from topics, maybe add a "disabled" flag to prevent direct publishing?)
2. Slowly start deleting messages
3. Once we can't find anymore for that queue, delete the queue itself from the queue list, save to riak.

There is definitely the chance we can still get messages into the queue while this happens (eventually consistent info about the state of topics/queues), but not sure if we can do much about it.

[lumost]

we can def flag the queue as disabled, I'm thinking for simplicity we block direct PUT/GET/ operations with a flag, then sleep for the sync interval to ensure that we don't get new messages, then begin burning down the queue.

I'm tempted to DRY it by iterating the GET call, but I don't think that's really the behavior that we want.

I'll get cookin.

--James

[StabbyCutyou]

We've got this covered from an application perspective (can now delete queues), but they still leave behind old messages, which is potentially a problem.

Not closing this issue yet until that is solved for