Token Bayesian Network (TokenBN) Parser

Encoding Approach Overview

TODO: Update the readme with the current design

Create Storage Objects:

• Create dictionaries for:
  • Tokens mapping them to their token numbers
  • Token numbers that map to dictionaries of all token numbers that map to token frequencies (for tracking frequency of current node relative to its composing nodes)
  • Tokens that map to their frequencies
• Instantiate ordered lists for:
  • Tokens (their positions in the list represent their token numbers)
  • Edges (originating token number, destination token number)

Build a Modified Suffix Tree

1. Start with an alphabet of 1-grams for the first layer.
2. Reset the suffix parsing process at the end of every clause, doc, etc.
3. Build on the previous suffix tree.
4. For each new suffix read:
   • Update the counts for all component nodes/tokens as the suffix path is traversed in the:
     • Dictionary of token obje
     • Token frequencies dict
   • If it leads to a new node or an edge split, add new entries to:
     • Token list
     • Edge list
     • Token: number mapping dictionary
     • 2D dictionary
     • Token frequencies dict

Future Work

• Add token composition probabilities
  • start with depth of 1, refactor later
  • tokens' posteriors can be determined via their parents in the associated markov blanket
• Replace base set of primitive tokens with word tokens and their POS tags
  • Create PCFG by modelling POS tag/word pair composition likelihoods

Sauces

• Wordpiece