App load failure silently deletes dynamic PersistentStoreDatabaseSettings via TethysApp cascade

[romer8]

Summary

An exception while loading an app's url_patterns or handler_patterns causes Tethys to delete
the app's TethysApp row. That row cascades to TethysAppSetting, so every dynamically created
PersistentStoreDatabaseSetting is destroyed — and nothing can rebuild them, because a harvest
only recreates the settings declared in app.py.

With DEBUG = False, which is the normal production setting, this happens silently and without a
prompt. The harvester runs at app boot and on every tethys CLI invocation, so a single bad
import anywhere in an app or an extension is enough to trigger it.

For an app that creates a database per user-uploaded model, the result is that every pointer to
every model database is gone while the databases themselves are untouched. The portal cannot find
them again on its own.

Why this looks like a bug rather than intended behaviour

TethysApp.sync_settings goes out of its way to protect exactly these rows
(tethys_apps/models.py:112-118 on main):

for setting in existing_settings:
    # Do not remove dynamically craeted settings
    if getattr(setting, "dynamic", False) and setting.dynamic:
        continue

    if setting.name not in setting_names:
        setting.delete()

So the invariant "dynamic settings must not be removed automatically" is already stated in the
codebase. The remove_from_db cascade quietly violates it.

The path

tethys_apps/harvester.py:292-316 — the app is synced to the database first, then removed if its
URLs or handlers fail to load:

app_instance.sync_with_tethys_db()

try:
    app_instance.url_patterns
except Exception:
    tethys_log.exception(
        "App {0} not loaded because of an issue with loading urls:".format(app_package)
    )
    app_instance.remove_from_db()
    continue

try:
    app_instance.handler_patterns
except Exception:
    tethys_log.exception(...)
    app_instance.remove_from_db()
    continue

tethys_apps/base/app_base.py:1959-1985 — with DEBUG = False there is no prompt:

proceed = None if settings.DEBUG else True
if proceed is None:
    ...  # interactive y/n, only when DEBUG is True

if proceed:
    try:
        TethysApp.objects.filter(package__exact=self.package).delete()
    except Exception as e:
        tethys_log.error(e)

tethys_apps/models.py:219-221 — the cascade:

tethys_app = models.ForeignKey(
    TethysApp, on_delete=models.CASCADE, related_name="settings_set"
)

PersistentStoreDatabaseSetting.dynamic (tethys_apps/models.py:955) is what marks the rows that
no harvest will ever recreate.

Reproduction

1. Install an app that creates dynamic persistent store database settings (via
   app.create_persistent_store(..., ) at runtime rather than declaring them in app.py).
2. Confirm the rows exist:
   select count(*) from tethys_apps_persistentstoredatabasesetting where dynamic;
3. Introduce an ImportError anywhere that the app's URL or handler loading reaches — for us this
   was a version mismatch between the app image and an installed extension, where the extension's
   controller imported a name its own helpers module did not yet export.
4. Run any tethys command, or restart the app.
5. tethys_apps_tethysapp, tethys_apps_tethysappsetting, and every dynamic
   PersistentStoreDatabaseSetting are now empty. The <app>_<uuid> databases still exist.

Observed on our deployment: tethys_apps_tethysapp 1 → 0 and settings 8 → 0, with all 8 model
databases intact. Fixing the import and running tethys db sync brought the app back with only the
app.py-declared settings; the dynamic ones stayed gone, and the app row came back with a new
id, which is how we identified the cascade after the fact.

We had also seen this in production earlier, where roughly 430 dynamic settings disappeared while
all 434 model databases survived. Recovering it took an app-specific command that re-derives the
settings by scanning the database server for <app>_<uuid> databases. An app without such a
command would have no way back.

Impact

• Silent, unprompted destruction of database rows in production, triggered by an unrelated import
  error.
• The data destroyed is precisely the data that sync_settings is written to preserve.
• Not recoverable by any Tethys mechanism — recovery requires app-specific knowledge of how the
  dynamic settings were derived, or a database backup.
• The trigger is any version skew between an app and an extension, which is a routine deployment
  hazard rather than an exotic failure.

Suggested direction

Roughly in order of how much they would have helped us:

1. Do not let a load failure delete dynamic settings. Either detach or preserve rows with
   dynamic=True before deleting the app, or delete the app row only when it has no dynamic
   settings. The existing check in sync_settings is the precedent.
2. Do not delete on a transient load error at all. An app failing to import is usually a broken
   deploy, not an uninstall. Leaving the row in place and logging loudly is recoverable; deleting is
   not. Removal could be reserved for an explicit tethys uninstall.
3. If removal stays, make it opt-in in production. proceed = None if settings.DEBUG else True
   inverts the safer default: the non-interactive environment is the one that destroys data without
   asking. A setting such as REMOVE_UNLOADABLE_APPS_FROM_DB defaulting to False would preserve
   the behaviour for those who want it.
4. Log at error level naming what was deleted, including the dynamic setting count, so the
   cause is visible in logs. Ours recorded the load exception but nothing about the deletion, which
   is why the connection was not obvious.

Happy to put up a PR for whichever direction you prefer.

Environment

• Tethys main (paths and line numbers above are from origin/main)
• PostgreSQL, DEBUG = False
• App with per-model dynamic persistent store databases, plus a Tethys extension