Replacing temporary file within a context manager causes failure

[tashrifbillah]

os.rename did not work when we changed the TMPDIR to /mnt/ProNET/newtmp. So we replaced it with os.replace

lochness/lochness/__init__.py

Line 594 in fa04f6d

os.rename(tmp.name, filename)

If this solution is found stable, we shall commit it to kcho/pronet branch.

---

Edit:

os.replace was found to be ineffectual.

[tashrifbillah]

Within this context manager, a temporary file is created:

lochness/lochness/redcap/__init__.py

Lines 121 to 124 in fa04f6d

	with tf.NamedTemporaryFile(suffix='tmp.json') as tmpfilename:
	lochness.atomic_write(tmpfilename.name, content)
	with open(tmpfilename.name, 'r') as f:
	data = json.load(f)

Then this tries to replace that temporary file with a second temporary file:

lochness/lochness/__init__.py

Lines 580 to 594 in fa04f6d

	dirname = os.path.dirname(filename)
	with tf.NamedTemporaryFile(dir=dirname, prefix='.', delete=False) as tmp:
	if isinstance(content, six.string_types):
	tmp.write(content.decode(encoding))
	else:
	tmp.write(content)
	tmp.flush()
	os.fsync(tmp.fileno())
	try: #
	os.chmod(tmp.name, permissions)
	except PermissionError: # when admin forces output file permissions
	pass
	os.rename(tmp.name, filename)

This cannot be a good design. After we set TMPDIR environment variable, the Lochness program fails at these two lines.

[tashrifbillah]

So we removed dependency on the nested context manager/temporary file and wrote content directly to the first temporary file:

 with tf.NamedTemporaryFile(suffix='tmp.json') as tmpfilename: 
     # lochness.atomic_write(tmpfilename.name, content)
     with open(tmpfilename.name, 'w') as f: 
         f.write(content.decode('utf-8')
         f.flush()

     with open(tmpfilename.name, 'r') as f: 
         data = json.load(f) 

This 'direct' approach solved our current issue and is running Lochness now.

[dheshanm]

Traceback showing FileNotFoundError: [Errno 2]

Traceback (most recent call last):                                                                                                                                                                                 
  File "/opt/software/lochness/lochness/redcap/__init__.py", line 122, in initialize_metadata                                                                                                                      
    lochness.atomic_write(tmpfilename.name, content)                                                                                                                                                               
  File "/opt/software/lochness/lochness/__init__.py", line 599, in atomic_write                                                                                                                                    
    os.replace(tmp.name, filename)                                                                                                                                                                                 
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/ProNET/newtmp/.7fh0thez' -> '/mnt/ProNET/newtmp/tmploo_5t34tmp.json'                                                                                 
                                                                                                                                                                                                                   
During handling of the above exception, another exception occurred:                                                                                                                                                
                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "/opt/software/lochness/scripts/sync.py", line 289, in <module>                                                                                                                                             
    main()                                                                                                                                                                                                         
  File "/opt/software/lochness/scripts/sync.py", line 159, in main                                                                                                                                                 
    do(args, Lochness)                                                                                                                                                                                             
  File "/opt/software/lochness/scripts/sync.py", line 222, in do                                                                                                                                                   
    lochness.initialize_metadata(Lochness, args,                                                                                                                                                                   
  File "/opt/software/lochness/lochness/__init__.py", line 104, in initialize_metadata                                                                                                                             
    REDCap.initialize_metadata(                                                                                                                                                                                    
  File "/opt/software/lochness/lochness/redcap/__init__.py", line 124, in initialize_metadata                                                                                                                      
    data = json.load(f)                                                                                                                                                                                            
  File "/opt/software/miniconda3/lib/python3.8/tempfile.py", line 492, in __exit__                                                                                                                                 
    self.close()                                                                                                                                                                                                   
  File "/opt/software/miniconda3/lib/python3.8/tempfile.py", line 499, in close                                                                                                                                    
    self._closer.close()                                                                                                                                                                                           
  File "/opt/software/miniconda3/lib/python3.8/tempfile.py", line 436, in close                                                                                                                                    
    unlink(self.name)                                                                                                                                                                                              
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/ProNET/newtmp/tmploo_5t34tmp.json'

[tashrifbillah]

This solves the error in only one place. But atomic_write() is used in other places in Lochness too. To cover all of them, Tashrif commented out this block and inserted:

with open(filename,'w') as f:
  f.write(content.decoding(encoding))
  f.flush()