Document Tokenizer

Preprocessing of documents into sentences

Guidelines

To run the project locally initialize the environment and run:

uvicorn main:run --reload --app-dir=src

Environment

Uses poetry for everything needed to run the application Poetry

Poetry config file (pyproject.toml) is located in the root folder.

Install env

poetry install --with dev

Activate env

poetry shell

Installing packages

When the environment is activated, packages can be installed using poetry directly.

Example: pip

poetry add [package]

Testing

poetry run pytest

Linting

poetry pylint src/

Formatting

poetry black src/

Migrate

python src/migrate.py