Find and flag names within code as a pre-commit hook

[SimonBiggs]

The idea here is to create a dictionary of first and last names (and other names too). Then, have a CLI that is designed to be used by https://pre-commit.com/ to check and flag if any new about to be committed files have a name in them somewhere.

Make it so that names are allowed on a line if that line ends in names allowed so that you can flag a line to be allowed to have a name in it by writing # names allowed. Can also have a configuration file that can be used to flag whether or not to ignore the first lines of files if that line contains the text Copyright.

Will design to search for names even if there are no spaces around them, so that names in paths etc. are still flagged. As such only the shortest version of a name need be searched for initially.

Here is an example public database of names that I have found:

https://www.openacademic.ai/oag/

Need to also have a way to verify that all names within the downloaded database catch all names within your clinic. It would be helpful to flag whether or not more public databases need to be found by highlighting names that are within Mosaiq, that are not within the public database. These names are NOT to be used. However, more public databases of names can be found and added until the coverage is complete.

@pchlap, keen for your thoughts. If something like this was running, and it was verified that the public database contained all the names currently in use in your clinic, if all code commits were checked locally before being committed to not contain these names within them, might that be a step in the right direction to allowing public code submissions?

[SimonBiggs]

A list of APIs that might be useful to find public databases of names:

https://guides.lib.berkeley.edu/information-studies/apis

[SimonBiggs]

@Matthew-Jennings might this be something you might be interested working on with me? This risk of a privacy breach when allowing public code submissions is quite the road block I believe to allowing people to submit to public code repositories...

I have searched around and I wasn't able to find a tool like this. It would be something that would be very nice to have as an extra safety harness for people to wear when they are being introduced to submitting to GitHub.

[SimonBiggs]

Will likely also need the ability to whitelist certain names within a config file. Given names also overlap with common dictionary words.

[SimonBiggs]

Okay, a first pass name list can be downloaded from https://zenodo.org/record/3831205/files/all-names.txt?download=1

[pchlap]

Hi @SimonBiggs,
thats an interesting idea. A step in the right direction, but I think patient names is only one small part of private data that could slip through. I know our management are concerned about any data that could be identifiable: addresses, phone numbers, patient demographics and even image data where a 3d rendering of a patient's face could be reproduced.

So while this would be a good first step, I'm not sure how we could solve the other concerns...

[Matthew-Jennings]

@SimonBiggs, I think it's a cool idea. It's helpful to be able to narrow down the scope of identifiable information that can slip through, even if absolutely everything can't be covered.

I suspect that patient names are some of the biggest culprits, because they tend to find their way into filenames and directory names.

@pchlap does make a good point. Although, maybe my imagination is lacking, but it seems to me that almost all of the other culprits that @pchlap mentioned are likely to be in labelled fields (as in DICOM, or similar), and so are anonymisable by their labels. In any case, it also seems to me that the concept can be extended to phone numbers, demographics, and addresses. 3D renderings are a little trickier...

Direction over destination. :)

[pchlap]

@Matthew-Jennings my concern is the all too common:
git add *
git commit
git push

in which case, who knows whats getting in there. Some less experienced contributors might not have such a clean working directory and all sorts of unexpected things could slip through like that (dicom, tar/zip files, xlsx files, log files etc). I wonder if an extremely restrictive .gitignore would be a solution to this sort of thing.

But I do agree that this would be a useful tool, I am just talking about the more extreme cases here. More realistically contributors will be aware of concerns surrounding submitting private data, and a tool like this could definitely help flag bits and pieces that may have inadvertently missed.

[Matthew-Jennings]

@pchlap, this is a pre-commit hook. The checks happen automatically when you attempt git commit. If the check fails, the commit fails. The point is, developers can't commit files containing patient names with this hook in place. It takes an existing risk and helps to mitigate it.

Not to put words in @SimonBiggs' mouth, but I'm sure he's not suggesting that developers become complacent and blindly trust it. He's more likely suggesting that something like this lessens the consequences of mistakes.

This is certainly not a substitute for education and careful practice, just like airbags in a car are no substitute for careful driving.

[pchlap]

@pchlap, this is a pre-commit hook. The checks happen automatically when you attempt git commit. If the check fails, the commit fails. The point is, developers can't commit files containing patient names with this hook in place. It takes an existing risk and helps to mitigate it.

Yep, but the pre-commit hook would only check text files? I could imagine something sneaking through in a tar.gz file.

This is certainly not a substitute for education and careful practice, just like airbags in a car are no substitute for careful driving.

For sure, I agree :)

[SimonBiggs]

Yep, but the pre-commit hook would only check text files? I could imagine something sneaking through in a tar.gz file.

@pchlap have you looked into .gitignore whitelisting patterns?

https://jasonstitt.com/gitignore-whitelisting-patterns

I could imagine, for example, a repo only whitelisting *.py, and then having a tool like this to cover identification information that can appear in there might be a good combination. I am certainly keen in finding ways to adequately address the risks and barriers for physicists to being able to contribute code publicly.

[pchlap]

Yep, but the pre-commit hook would only check text files? I could imagine something sneaking through in a tar.gz file.

@pchlap have you looked into .gitignore whitelisting patterns?

https://jasonstitt.com/gitignore-whitelisting-patterns

I could imagine, for example, a repo only whitelisting *.py, and then having a tool like this to cover identification information that can appear in there might be a good combination. I am certainly keen in finding ways to adequately address the risks and barriers for physicists to being able to contribute code publicly.

Yeah I like that, I think this would be useful for some of our repos.