Redact DB credentials on SQLAlchemy error

[bram-vdberg]

At the moment some SQLAlchemy errors are displaying sensitive database credentials in the logs. For example, on a connection error the logs will show:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at [db url] ... failed

This error will display the database url, database user, and database. These details should be redacted to prevent leaking sensitive information in the logs.

[bh2smith]

Great find. ONe option here would be not to re-raise caught exceptions like we have been doing. This would only capture those errors which are "known/caught". Open to other suggestions.

Assigning to @mooster531

[bram-vdberg]

I think that might make debugging harder in the future, it is nice to be able to see the exceptions that were raised.

Maybe a better option would be to filter the logs to replace these credentials with a placeholder. We could filter out the db url quite easy with something like postgresql://*:5432. If we can find something similar to filter the database user and database name that would be quite a nice and easy solution.

Otherwise we could also patch these credentials after using them so that the script can connect to the database, but afterwards only logs the patched credentials. But that would be a bit more invasive and might be harder to implement nicely.

[mooster531]

Hi @bram-vdberg,
Could you please provide some steps to reproduce this?

1. I tried running main.py while having a running postgresql server using a wrong password and got this log:

log_incorrect_password.txt

2. I tried running main.py with correct credentials in the config, but the server was offline, got this:

log_server_offline.txt

3. Also tried running with correct credentials, with a valid and online server, but wrong schema (replaced cow with bow in the task named cow-solvers):

log_invalid_schema.txt

As you can see, in neither case is the password printed to stdout or stderr. I'm very curious to see a case I can reproduce so I could work on it. Thank you!