Persisting worklist using CubQ #46
Conversation
jfcloutier
force-pushed
the
cubq
branch
2 times, most recently
from
fcf7f42 to
231a044
Compare
jfcloutier
force-pushed
the
cubq
branch
11 times, most recently
from
a4c9741 to
6c549aa
Compare
jfcloutier
force-pushed
the
cubq
branch
7 times, most recently
from
ee35399 to
3453e99
Compare
jfcloutier
force-pushed
the
cubq
branch
9 times, most recently
from
4b9dee1 to
0e54ec6
Compare
|
Protocol is now used instead of behaviour for "pluggable work list". The persistent work list stores system monotonic time every 10 minutes and on terminate. Expiration is now simply a monotonic time. In the tests, Jackalope is now started supervised and is no longer explicitly stopped. |
jfcloutier
force-pushed
the
cubq
branch
4 times, most recently
from
84ae713 to
f67856e
Compare
fhunleth
force-pushed
the
cubq
branch
2 times, most recently
from
ef96d37 to
d6fb5ff
Compare
| db = | ||
| case CubDB.start_link(data_dir: data_dir, name: :work_list, auto_compact: true) do | ||
| {:ok, pid} -> | ||
| pid | ||
|
|
||
| {:error, {:already_started, pid}} -> | ||
| pid | ||
|
|
||
| other -> | ||
| Logger.warn("[Jackalope] Corrupted DB : #{inspect(other)}. Erasing it.") | ||
| _ = File.rmdir(data_dir) | ||
| raise "Corrupted work list DB" | ||
| end |
There was a problem hiding this comment.
How about something like this? The GenServer should never return already_started. It would be nice to more specifically know that the DB is corrupt rather than always erasing, but I didn't look to see if that was possible. Crashing after fixing the corruption just to get init called again seemed like it might have ramifications on sibling GenServers. I didn't want to think about documenting the behavior if the caller used :all_for_one.
| db = | |
| case CubDB.start_link(data_dir: data_dir, name: :work_list, auto_compact: true) do | |
| {:ok, pid} -> | |
| pid | |
| {:error, {:already_started, pid}} -> | |
| pid | |
| other -> | |
| Logger.warn("[Jackalope] Corrupted DB : #{inspect(other)}. Erasing it.") | |
| _ = File.rmdir(data_dir) | |
| raise "Corrupted work list DB" | |
| end | |
| cubdb_opts = [data_dir: data_dir, name: :work_list, auto_compact: true] | |
| {:ok, db} = | |
| case CubDB.start_link(cubdb_opts) do | |
| {:error, reason} -> | |
| Logger.warn("[Jackalope] Corrupted DB : #{inspect(reason)}. Erasing it.") | |
| _ = File.rmdir(data_dir) | |
| CubDB.start_link(cubdb_opts) | |
| success -> success | |
| end |
There was a problem hiding this comment.
What if start_link fails for reasons other than DB corruption?
There was a problem hiding this comment.
The same thing as before is fine. I.e., crash.
There was a problem hiding this comment.
Committed and pushed
| CubDB.set_auto_file_sync(db, true) | ||
|
|
||
| queue = | ||
| case CubQ.start_link(db: db, queue: queue_name) do |
There was a problem hiding this comment.
I think things could be shuffled around and the CubDB and CubQ start_links combined in a helper function that would make testing recovery easier.
| def new(expiration_fn, update_expiration_fn, max_size \\ @default_max_size, opts \\ []) do | ||
| args = [ | ||
| expiration_fn: expiration_fn, | ||
| update_expiration_fn: update_expiration_fn, | ||
| max_size: max_size, | ||
| opts: opts | ||
| ] |
There was a problem hiding this comment.
If the first three arguments are going to be added to a new argument list anyway, how about passing them in opts from the beginning?
There was a problem hiding this comment.
Committed and pushed
Support pluggable implementations of a work list, one transient (the default) and the other persistent (via CubQ).