please adapt download_embeddings to support 'file://' url scheme

[EricDeveaud]

Hello,

can you please adapt download_embeddings to also support file://.... url schema.

this will allow us to provide centralized embedings files on our cluster, avoiding the download time.

something in this spirit.

    log.info("Downloading embeddings from %s �~F~R %s", url, dest)

    if url.startswith("file://"):
        from urllib.request import urlopen
        with urlopen(url) as resp:
            content = resp.read()
            with open(dest, "wb")as f:
                f.write(content)
        return dest
    else:
        #else download from given URL
        with requests.get(url, stream=True, timeout=timeout) as resp:
            resp.raise_for_status()

problem, this will duplicate the file for each user // base_directory

I would prefer to main.py to directly handle embeddings data file

something like:

    # Nuevo: obtener nombre del archivo desde la URL
    URL=conf["embeddings_url"]
    if URL.startswith('file://'):
        file_path=urllib.parse.urlparse(URL).path
        if os.path.exists(file_path):
            tar_path=file_path
    else:
        filename = os.path.basename(urllib.parse.urlparse(conf["embeddings_url"]).path)
        tar_path = os.path.join(embeddings_dir, filename)

        logger.info(f"Downloading reference embeddings to {tar_path}...")
        download_embeddings(conf["embeddings_url"], tar_path)

    logger.info("Loading embeddings into the database...")
    load_dump_to_db(tar_path, conf)

maybeed there is something I missed regarding the embeddings file needed in base_directory

lmk if this sounds acceptable and which method you prefer. I will then propose a PR

!hasta luego¡

Eric

[frapercan]

Hi Eric,

To be honest, when I need to switch from one information system to another, I usually handle it manually rather than through initialize.

More recently, I have been competing in CAFA6, and I have had to switch constantly between different information systems: sometimes working with one set of annotations at a given point in time, and other times with different snapshots and different annotations for validation and score optimization purposes.

The lookup table we provide is intended to represent the set of all proteins with strictly experimental annotations. I would work directly with Layer 0, unless you specifically want to study how the different layers affect the ranking.

I normally connect to the target database, reset the schema completely:

DROP SCHEMA public CASCADE;
CREATE SCHEMA public;

CREATE EXTENSION vector;

and then restore the backup with pg_restore:

PGPASSWORD=clave pg_restore \
  -h localhost \
  -U usuario \
  -d BioData \
  BioData_backup.tar

So in practice, that is usually the workflow I follow.

As I understand it, initialize is intended to automate the download and loading steps, but I have to admit it could be improved quite easily. That part of the code has been there since the beginning of the project.

The solution you propose seems quite straight forward to me, assuming it works as expected, since I have not tested that specific use case myself.

Best regards,
Francisco

[EricDeveaud]

Hello,

FYI modification of main.py to accept file:// URL scheme as described above is functional.
I ran succesfully few runs using it

regards

Eric