[Question] Appetite for read header (column labels) only method?

[xflow-ben]

I'm working on 2 separate software tools - I'd like to leverage weio for them if possible. For set up of analyses (before running them) it is useful to know what data is in the file. For very large files it slow to load the whole file just to see the data headers.

Would adding a method for reading only the header be of interest? I could take a crack at it.
Cheers,
Ben

[ebranlard]

Hi Ben,

Is there an existing weio file format you have in mind?

For very large files, I agree that reading the entire content can be inefficient.
The simplest would be to add a flag headerOnly=True to init and read() which is False by default.
If after figuring out the header, the user decides to read more, the you can open the file again with headerOnly=False.

If we want to keep the file open between calls, and avoid reading the header multiple times, and adding a function real_all() and read_chunk(), then we would need to implement a context manager interface to the class. mayeb the keyword headerOnly should be replaced like something like streaming, to indicate that we allow the file to be read chunk by chunk.
Ideally, we would ensure that the user uses a context manager. The parent class could look something like that:

def __init__(self, filename=None, streaming=False, **kwargs):
        self.filename = filename
        self.streaming= streaming # if True, read the headers only, but allow chunks to be read later. Needs to be in a context manager
        self.data = None
        self._in_context = False  # True only if in 'with' block
        self._fid = None  # For explicit file handle if needed
        if filename is not None:
            self.read(streaming=streaming, **kwargs)

    def __enter__(self):
        self._in_context = True
        if self.filename:
            self.read(streaming=self.streaming) 
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        if self._fid:
            self._fid.close()
            self._fid = None
        if not self.headerOnly:
            self.data = None  # Optional: free full data on exit
        return False  # Propagate exceptions

    def _enforce_context_if_needed(self):
        """Private helper: Raise if headerOnly and not in context."""
        if self.streamingand not self._in_context:
            raise RuntimeError(
                "headerOnly=True requires using a context manager ('with' statement) "
                "to ensure the file is closed. Use: `with File(...) as reader:`"
            )

    def read(self, filename=None, streaming=False, **kwargs):
        if filename is not None:
            self.filename = filename
        self.streaming= streaming
        
        self._enforce_context_if_needed()  # Enforce here (before any open)

        self._read(**kwargs) # To be implemented by child class

One small problem, is that in the past, all classes relied on code from the File class (its parent class), and therefore it was easy to add generic code to the parent class (for instance context managers, additional flags). This was great, true object oriented programming. But at some point I figured it was more difficult to share a given file format to a colleague, so I tried to make the readers more or less standandalone (if the parent class is not there, they survive...). Maybe it's time to roll back on that. Afterall, I can just ship File.py with another fileformat if needed.

A lot of the current children class currently overide the read() method of the parent, but that's okay, we can change the child class if/when needed.

[xflow-ben]

I gave the streaming thing a shot. LMK what you think

[ebranlard]

It looks pretty good! Thanks a lot for doing that! I've added some comments,
Cheers!

[xflow-ben]

Ok I gave addressing your comments a shot!

[ebranlard]

Thanks a lot, I have a couple of suggestions, if they are a pain to address let me know, I don't mind taking care of them