Custom Aggregation Pipeline Operators & OTHERS

[SemAiSoft]

First, thanks for this most anticipated project - very promising indeed.

That being said, I think the followings are invaluable features and would really make this project a FIRE!

1. Ability to register custom pipeline operators, for cases where their is not explicit support for it (yet) in the project. For example, $lookup operator which if I understand correctly, there is no plan support for.
2. Leveraging on and adapting the battle tested mingo npm lib, to support even more operators and features, including window and cross collection lookup

[AnkanSaha]

@SemAiSoft Thanks, this is a well-aimed issue.

On $lookup: we are aware, and it is actively being engineered. It will land. The "No JOIN operations" line on the limitations page reflects where AxioDB is today, not where it is going, and that page will be updated when the work ships.

The reason it is taking design time rather than being a quick add: AxioDB uses file-per-document storage and only auto-indexes documentId, so a naive join on any other field means a full scan of the foreign collection per input document. At the 10K to 500K range AxioDB targets, that is disk-bound and would be a performance trap for users. So $lookup will require an index on the join field rather than silently degrading. The second piece is that encryption is per collection and keys can differ, so a cross-collection read needs a proper key-handling path. Both are solvable, they just need to be solved once and correctly.

Your point about a plugin surface stands separately. The pipeline is currently 7 stages ($match, $group, $sort, $project, $limit, $skip, $count) and users have no way to add anything without forking. That gap is real regardless of $lookup. Registered operators over raw documents are straightforward; the design work is in what happens when an operator wants to touch another collection, which runs into the same key handling above.

On mingo: worth a serious look, and thanks for the pointer. AxioDB already materialises documents to run $group, so mingo working over in-memory arrays costs nothing we are not already paying, and it would cover the long tail of expression operators and window functions without hand-writing each one. Storage, indexing, caching and encryption stay ours.

No version number yet, but this is tracked and being worked on. If you want to sketch the registration interface you would want to write against as a consumer, that input would genuinely help.