Remove examples from repo

[jimboid]

This means deleting the files and directory as well as removing the history to compact the repository.

[jimboid]

Ok so the practicalities of removing these files totally contain some side effects that we should be aware of.

1. This involved rewriting history, this means that the commit hashes will all change and as a result means that we can't be mid change when we do this.
2. We will have to coordinate so that we have no active open branches from the current repo, and all local repos will need to pull the changed history. So you can't have unpushed local history either.

This means then, our options are to simply remove the files and not interfere with history. This means that the repo stays bloated by files you can't see in future. Or we do the full history rewrite to remove any trace of them and shrinking the repo size to that or a normal software only repo.

Anyone have any thoughts either way?

I was initially going to do this whilst you are all away at the conference. But we also have large files used in tests. So we would end up doing it twice, we should avoid this. Perhaps we do this:

1. we decide what we are doing, either the full history removal or the lighter deletion.

2. Then about the tests if we go for full rewrite of history then we implement this change to tests before doing that so we hit them all in one pass. Minimising disruption, if we take the lighter option then it doesn't matter when the tests are considered.

[harryswift01]

I've reviewed the test data and found that only 9 test cases use the following files:

• md_A4_dna.tpr - 84 KB
• md_A4_dna_xf.trr - 916 KB

These are relatively small and actively used, so I suggest we keep them.

The rest of the files in tests/data/ and everything in Example/, especially Example/data at 257 MB, are unused and can be removed. This would reduce the repo size significantly and help to avoid long-term bloat.

I think that the best course of action is to go ahead with a full history rewrite to clean this up properly.

[jimboid]

Agreed. We should try to schedule this where there are no active branches so that we don't get into a mess. We can talk over it tomorrow at the group meeting. I suspect it will be fast, and I will search the history for hidden files that have been deleted previously that are large and delete those too.

[jimboid]

I think the operation will take less than an hour though, but we just need to be careful nobody pushes historical branches and make sure history settings prevent it on the repo

[jimboid]

Here is how the history was rewritten.

• Firstly a seperate mirror clone was taken as a backup, and repo also forked to my own user and left the fork network
• installed git-filter-repo
• run "git-filter-repo --analyze"
• created the following file /tmp/files-to-delete

Example/
paper/
CodeEntropy/tests/
tests/data/1AKI_prod_60.trr
tests/data/1AKI_prod_60.tpr
tests/data/poseidon_example.trr
tests/data/poseidon_example.prmtop
tests/data/1AKI_ws.trr
tests/data/1AKI_ws.tpr
tests/data/.1AKI_ws.trr_offsets.lock
tests/data/.md_A4_dna_xf.trr_offsets.lock
tests/data/.poseidon_example.trr_offsets.lock

• ran the filter on history "git filter-repo --force --invert-paths --paths-from-file /tmp/file-to-remove.txt"
• checked the files and they looked ok
• disabled branch protection rules on origin
• added origin back as the tools deletes as a safety precaution "git remote add origin git@github.com:CCPBioSim/CodeEntropy.git"
• force pushed rewritten history "git push --force --mirror origin"
• reinstated the rules on main branch on GH
• did a fresh local git clone to discard any remaining old history