Remote submit issues

[jungerm2]

I'm trying to follow the remote-submit example and keep running into an issue while connecting to the remote pool. Specifically, I get a "HTCondorException: Unable to locate local daemon." error.

I assumed this was because I didn't have condor installed locally and went ahead and installed it. This guide, as well as the docs here don't really make it clear if you need to install condor on your system to use the python library, so I'm unsure about this.

I'm on linux so I just followed the install procedure here which led to a "Error: Unsupported distribution Fedora". I'm not sure what makes Fedora incompatible with this install method, especially since redhat seems to be supported. Instead, I installed it via my package manager, and while the install succeded condor_schedd gave the following error:

ERROR "Cannot open log file '/var/log/condor/SchedLog'" at line 173 in file /builddir/build/BUILD/condor-23.9.6-build/condor-23.9.6/src/condor_utils/dprintf_setup.cpp

In the master log file (in /var/log/condor/MasterLog) the daemon was quitting because error: couldn't open file "/var/log/condor/ProcLog" for logging: Permission denied and same for SchedLog. I fixed this by manually creating these log files and ensuring proper permissions. I also had to create/chown /var/lock/condor, /etc/condor, /var/lib/condor and /var/run/condor which seems like a horrible idea and something that should be done on install, but now the daemon starts correctly but the collector host fails to connect:

# From /var/log/condor/MasterLog: 
04/22/26 14:29:14 Daemons::StartAllDaemons all daemons were started
04/22/26 14:29:16 Setting ready state 'Ready' for STARTD
04/22/26 14:29:19 attempt to connect to <128.105.68.172:9618> failed: Invalid argument (connect errno = 22).  Will keep trying for 20 total seconds (20 to go).
04/22/26 14:29:39 attempt to connect to <128.105.68.172:9618> failed: Invalid argument (connect errno = 22).
04/22/26 14:29:39 ERROR: SECMAN:2003:TCP connection to collector cm.chtc.wisc.edu failed.
04/22/26 14:29:39 Failed to start non-blocking update to <128.105.68.172:9618>.

I've tried both with/without the university VPN (which also seems to be a requirement, since cm.chtc.wisc.edu doesn't resolve without it), but still cannot connect. Is this correct, or did I misconfigure something? Any help would be greatly appreciated.

[jungerm2]

Let me also mention that I tried compiling everything from source as detailed here, and while that did work, I was unable to actually install the generated RPM package due to missing dependencies that I wasn't able to find. If my current install is indeed the issue, then I'll push on this a bit more but for the time being, I gave up. These build scripts work well for Fedora, so I'm not sure why the curl install script doesn't support it.

[jungerm2]

@xamberl any ideas this this?

[xamberl]

@jungerm2 You shouldn't have to install a local condor instance, just the Python bindings. At which step are you getting the HTCondorException: Unable to locate local daemon. error?

[jungerm2]

Here's a minimum reproducible example:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "htcondor",
# ]
# ///


import htcondor2 as htcondor

htcondor.enable_debug()
access_point = htcondor.Schedd()

Running this with uv run (which creates a fresh ephemeral venv) yields:

/home/sjung/.cache/uv/environments-v2/condor-test-0338f8e50b10a870/lib/python3.14/site-packages/htcondor2/__init__.py:27: UserWarning: The environment variable CONDOR_CONFIG is unset and none of the default locations contain a condor_config file.  Using /dev/null, instead.
  _warnings.warn(message)
05/14/26 09:03:53 attempt to connect to <128.105.68.172:9618> failed: timed out after 60 seconds.
Traceback (most recent call last):
  File "/home/sjung/Downloads/condor_test.py", line 12, in <module>
    access_point = htcondor.Schedd()
  File "/home/sjung/.cache/uv/environments-v2/condor-test-0338f8e50b10a870/lib/python3.14/site-packages/htcondor2/_schedd.py", line 145, in __init__
    location = c.locate(DaemonType.Schedd)
  File "/home/sjung/.cache/uv/environments-v2/condor-test-0338f8e50b10a870/lib/python3.14/site-packages/htcondor2/_collector.py", line 205, in locate
    return _collector_locate_local(self, self._handle, int(daemon_type))
htcondor2_impl.HTCondorException: Unable to locate local daemon.

If I don't have condor installed locally, then what local daemon is it trying to connect to?

[xamberl]

Did you create a ~/.condor/user_config file? https://github.com/CHTC/recipes/tree/remote-submit-htc/workflows-htc/remote-submit#create-user-configuration

[jungerm2]

Yes:

[xamberl]

@jungerm2 Do you mind stopping by office hours today? May be faster to figure this out over Zoom.

[xamberl]

@jungerm2 Can you set this environment variable and paste the new outputs you get when rerunning the script? It'll help us get some better debug information.

_CONDOR_TOOL_DEBUG=D_FULLDEBUG,D_HOSTNAME

[jungerm2]

Ah, nice! That's a bit more verbose:

❯ _CONDOR_TOOL_DEBUG=D_FULLDEBUG,D_HOSTNAME uv run condor_test.py 
/home/sjung/.cache/uv/environments-v2/condor-test-0338f8e50b10a870/lib/python3.14/site-packages/htcondor2/__init__.py:27: UserWarning: The environment variable CONDOR_CONFIG is unset and none of the default locations contain a condor_config file.  Using /dev/null, instead.
  _warnings.warn(message)
05/14/26 13:23:48 NETWORK_INTERFACE=* matches lo 127.0.0.1, wlp8s0 192.168.1.156, gpd0 10.130.181.234, docker0 172.17.0.1, br-677c9d233ffb 172.18.0.1, lo ::1, wlp8s0 fe80::ef63:ab35:9bb8:eb89, choosing IP 192.168.1.156
05/14/26 13:23:48 hostname: fedora
05/14/26 13:23:48 I am: hostname: fedora, fully qualified doman name: fedora, IP: 192.168.1.156, IPv4: 192.168.1.156, IPv6: 
05/14/26 13:23:48 Trying to getting network interface information after reading config
05/14/26 13:23:48 NETWORK_INTERFACE=* matches lo 127.0.0.1, wlp8s0 192.168.1.156, gpd0 10.130.181.234, docker0 172.17.0.1, br-677c9d233ffb 172.18.0.1, lo ::1, wlp8s0 fe80::ef63:ab35:9bb8:eb89, choosing IP 192.168.1.156
05/14/26 13:23:48 NETWORK_INTERFACE=* matches lo 127.0.0.1, wlp8s0 192.168.1.156, gpd0 10.130.181.234, docker0 172.17.0.1, br-677c9d233ffb 172.18.0.1, lo ::1, wlp8s0 fe80::ef63:ab35:9bb8:eb89, choosing IP 192.168.1.156
05/14/26 13:23:48 hostname: fedora
05/14/26 13:23:48 I am: hostname: fedora, fully qualified doman name: fedora, IP: 192.168.1.156, IPv4: 192.168.1.156, IPv6: 
05/14/26 13:23:48 COLLECTOR_HOST is set to "cm.chtc.wisc.edu"
05/14/26 13:23:48 Checking if cm.chtc.wisc.edu is a sinful address
05/14/26 13:23:48 cm.chtc.wisc.edu is not a sinful address: does not begin with "<"
05/14/26 13:23:48 New Daemon obj (collector) name: "cm.chtc.wisc.edu", pool: "", addr: ""
05/14/26 13:23:48 Using name "cm.chtc.wisc.edu" to find daemon
05/14/26 13:23:48 Port not specified, using default (9618)
05/14/26 13:23:48 Host info "cm.chtc.wisc.edu" is a hostname, finding IP address
05/14/26 13:23:48 Found CM IP address and port <128.105.68.172:9618?alias=cm.chtc.wisc.edu>
05/14/26 13:23:48 Daemon client (collector) address determined: name: "cm.chtc.wisc.edu", pool: "cm.chtc.wisc.edu", alias: "cm.chtc.wisc.edu", addr: "<128.105.68.172:9618?alias=cm.chtc.wisc.edu>"
05/14/26 13:23:48 Will use TCP to update collector cm2000.chtc.wisc.edu <128.105.68.172:9618?alias=cm.chtc.wisc.edu>
05/14/26 13:23:48 New Daemon obj (schedd) name: "", pool: "", addr: ""
05/14/26 13:23:48 No name given, but SCHEDD_HOST defined to "ap2002.chtc.wisc.edu"
05/14/26 13:23:48 Finding proper daemon name for "ap2002.chtc.wisc.edu"
05/14/26 13:23:48 Daemon name contains no '@', treating as a regular hostname
05/14/26 13:23:48 Returning daemon name: "ap2002.chtc.wisc.edu"
05/14/26 13:23:48 Using "ap2002.chtc.wisc.edu" for name in Daemon object
05/14/26 13:23:48 Using "ap2002.chtc.wisc.edu" for full hostname in Daemon object
05/14/26 13:23:48 Local daemon name would be "fedora"
05/14/26 13:23:48 COLLECTOR_HOST is set to "cm.chtc.wisc.edu"
05/14/26 13:23:48 Checking if cm.chtc.wisc.edu is a sinful address
05/14/26 13:23:48 cm.chtc.wisc.edu is not a sinful address: does not begin with "<"
05/14/26 13:23:48 New Daemon obj (collector) name: "cm.chtc.wisc.edu", pool: "", addr: ""
05/14/26 13:23:48 Using name "cm.chtc.wisc.edu" to find daemon
05/14/26 13:23:48 Port not specified, using default (9618)
05/14/26 13:23:48 Host info "cm.chtc.wisc.edu" is a hostname, finding IP address
05/14/26 13:23:48 Found CM IP address and port <128.105.68.172:9618?alias=cm.chtc.wisc.edu>
05/14/26 13:23:48 Daemon client (collector) address determined: name: "cm.chtc.wisc.edu", pool: "cm.chtc.wisc.edu", alias: "cm.chtc.wisc.edu", addr: "<128.105.68.172:9618?alias=cm.chtc.wisc.edu>"
05/14/26 13:23:48 Will use TCP to update collector cm2000.chtc.wisc.edu <128.105.68.172:9618?alias=cm.chtc.wisc.edu>
05/14/26 13:23:48 Trying to query collector <128.105.68.172:9618?alias=cm.chtc.wisc.edu>
05/14/26 13:23:48 Querying collector <128.105.68.172:9618?alias=cm.chtc.wisc.edu> (cm2000.chtc.wisc.edu) with classad:
LimitResults = 1
LocationQuery = "ap2002.chtc.wisc.edu"
MyType = "Query"
Projection = "CondorVersion CondorPlatform MyAddress AddressV1 Name Machine _condor_PrivRemoteAdminCapability ScheddIpAddr CredDIpAddr"
Requirements = ((Name == "ap2002.chtc.wisc.edu"))
TargetType = "Scheduler"
05/14/26 13:23:48  --- End of Query ClassAd ---
05/14/26 13:23:48 Guess address string for host = <128.105.68.172:9618?alias=cm.chtc.wisc.edu>, port = 0
05/14/26 13:23:48 it was sinful string. ip = 128.105.68.172, port = 9618
05/14/26 13:23:48 Directory::Rewind(): path "/var/lib/condor/passwords.d" does not exist (yet)
05/14/26 13:23:48 Directory::Rewind(): path "/var/lib/condor/passwords.d" does not exist (yet)
05/14/26 13:23:48 Destroying Daemon object:
05/14/26 13:23:48 Type: 5 (collector), Name: cm.chtc.wisc.edu, Addr: <128.105.68.172:9618?alias=cm.chtc.wisc.edu>
05/14/26 13:23:48 FullHost: cm2000.chtc.wisc.edu, Host: cm2000, Pool: cm.chtc.wisc.edu, Port: 9618
05/14/26 13:23:48 IsLocal: N, IdStr: collector cm.chtc.wisc.edu, Error: 
05/14/26 13:23:48  --- End of Daemon object info ---
05/14/26 13:23:48 Found Name in ClassAd, using "ap2002.chtc.wisc.edu"
05/14/26 13:23:48 Daemon client (schedd) address determined: name: "ap2002.chtc.wisc.edu", pool: "", alias: "ap2002.chtc.wisc.edu", addr: "<128.105.68.113:9618?addrs=128.105.68.113-9618+[2607-f388-2200-100-eaeb-d3ff-fea7-d05e]-9618&alias=ap2002.chtc.wisc.edu&noUDP&sock=schedd_7771_d68e>"
05/14/26 13:23:48 Found SCHEDDIpAddr in ClassAd, using "<128.105.68.113:9618?addrs=128.105.68.113-9618+[2607-f388-2200-100-eaeb-d3ff-fea7-d05e]-9618&alias=ap2002.chtc.wisc.edu&noUDP&sock=schedd_7771_d68e>"
05/14/26 13:23:48 Found CondorVersion in ClassAd, using "$CondorVersion: 25.10.0 2026-04-21 BuildID: 900018 PackageID: 25.10.0-0.900018 GitSHA: f6aba35b RC $"
05/14/26 13:23:48 Found CondorPlatform in ClassAd, using "$CondorPlatform: x86_64_AlmaLinux9 $"
05/14/26 13:23:48 Found Machine in ClassAd, using "ap2002.chtc.wisc.edu"
05/14/26 13:23:48 Found CondorVersion in ClassAd, using "$CondorVersion: 25.10.0 2026-04-21 BuildID: 900018 PackageID: 25.10.0-0.900018 GitSHA: f6aba35b RC $"
05/14/26 13:23:48 Found CondorPlatform in ClassAd, using "$CondorPlatform: x86_64_AlmaLinux9 $"
05/14/26 13:23:48 Checking if <128.105.68.113:9618?addrs=128.105.68.113-9618+[2607-f388-2200-100-eaeb-d3ff-fea7-d05e]-9618&alias=ap2002.chtc.wisc.edu&noUDP&sock=schedd_7771_d68e> is a sinful address
05/14/26 13:23:48 <128.105.68.113:9618?addrs=128.105.68.113-9618+[2607-f388-2200-100-eaeb-d3ff-fea7-d05e]-9618&alias=ap2002.chtc.wisc.edu&noUDP&sock=schedd_7771_d68e> is a sinful address!
05/14/26 13:23:48 Using port 9618 based on address "<128.105.68.113:9618?addrs=128.105.68.113-9618+[2607-f388-2200-100-eaeb-d3ff-fea7-d05e]-9618&alias=ap2002.chtc.wisc.edu&noUDP&sock=schedd_7771_d68e>"
05/14/26 13:23:48 Destroying Daemon object:
05/14/26 13:23:48 Type: 3 (schedd), Name: ap2002.chtc.wisc.edu, Addr: <128.105.68.113:9618?addrs=128.105.68.113-9618+[2607-f388-2200-100-eaeb-d3ff-fea7-d05e]-9618&alias=ap2002.chtc.wisc.edu&noUDP&sock=schedd_7771_d68e>
05/14/26 13:23:48 FullHost: ap2002.chtc.wisc.edu, Host: ap2002, Pool: , Port: 9618
05/14/26 13:23:48 IsLocal: N, IdStr: , Error: 
05/14/26 13:23:48  --- End of Daemon object info ---
05/14/26 13:23:48 Destroying Daemon object:
05/14/26 13:23:48 Type: 5 (collector), Name: cm.chtc.wisc.edu, Addr: <128.105.68.172:9618?alias=cm.chtc.wisc.edu>
05/14/26 13:23:48 FullHost: cm2000.chtc.wisc.edu, Host: cm2000, Pool: cm.chtc.wisc.edu, Port: 9618
05/14/26 13:23:48 IsLocal: N, IdStr: , Error: 
05/14/26 13:23:48  --- End of Daemon object info ---

It seems it finds it, then gets killed?

[jungerm2]

Is path "/var/lib/condor/passwords.d" does not exist (yet) the issue?

[jungerm2]

I'm not sure what suddenly changed, but I can't reproduce this error anymore. I tried a few different Python versions too.

One small issue I ran into was the following error:

ERROR: read_secure_file(/home/sjung/.condor/tokens.d/ap2002): file must not be readable by others, had perms 100640

Which is easily fixed by chmod 600 ~/.condor/tokens.d/ap2002. Might be worth adding a note in the recipe about this.
I'll update this thread if I encounter further issues.

[xamberl]

Noted! I'll put it on my to-do list to revise the docs so that users

1. know they don't need to install the actual htcondor package, just the bindings
2. make sure to chmod the tokens

[jungerm2]

Does remote submit not work with containers that are in staging? I have a sample job (just runs ls) which works fine when container_image = docker://debian:trixie-slim but not with container_image = osdf:///chtc/staging/sjungerman/containers/trixie.sif. Instead, my jobs are held with reason:

Transfer input files failure at execution point
slot1_4@e2462.chtc.wisc.edu using protocol osdf. Details: failed to get namespace information                         
for remote URL osdf:///chtc/staging/sjungerman/containers/trixie.sif: error while querying the                        
director at https://osdf-director.osg-htc.org: Contact.Director Error: Error code 3001: 404:                          
No sources found for the requested path: no origins found for the requested namespace                                 
'/chtc/staging/sjungerman/containers/trixie.sif': Request ID:                                                         
5cce7986-f358-4483-a43d-0fd2773ead19 ( URL file =                                                                     
osdf:///chtc/staging/sjungerman/containers/trixie.sif )

Here's the full log too

000 (5078464.000.000) 2026-05-17 15:41:54 Job submitted from host: <128.105.68.113:9618?addrs=128.105.68.113-9618+[2607-f388-2200-100-eaeb-d3ff-fea7-d05e]-9618&alias=ap2002.chtc.wisc.edu&noUDP&sock=schedd_7771_d68e>
...
040 (5078464.000.000) 2026-05-17 15:42:05 Started transferring input files
	Transferring to host: <128.105.68.154:9618?addrs=128.105.68.154-9618+[2607-f388-2200-100-b226-28ff-fe51-60cc]-9618&alias=e2013.chtc.wisc.edu&noUDP&sock=slot1_22_12411_6c12_40245>
...
040 (5078464.000.000) 2026-05-17 15:42:07 Finished transferring input files
...
040 (5078464.000.000) 2026-05-17 15:42:09 Started transferring output files
...
021 (5078464.000.000) 2026-05-17 15:42:09 Error from starter on slot1_22@e2013.chtc.wisc.edu:
	Failed to transfer files: FILETRANSFER:1:non-zero exit (1) from /usr/libexec/condor/stash_plugin. |Error: failed to get namespace information for remote URL osdf:///chtc/staging/sjungerman/containers/trixie.sif: error while querying the director at https://osdf-director.osg-htc.org: Contact.Director Error: Error code 3001: 404: No sources found for the requested path: no origins found for the requested namespace '/chtc/staging/sjungerman/containers/trixie.sif': Request ID: 99186a1f-81d8-43ec-ad3e-bb9d85b2252e ( URL file = osdf:///chtc/staging/sjungerman/containers/trixie.sif )|
	Code 12 Subcode 256
...
004 (5078464.000.000) 2026-05-17 15:42:10 Job was evicted. Code 13 Subcode 256
	(0) CPU times
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Reason: Transfer input files failure at execution point slot1_22@e2013.chtc.wisc.edu using protocol osdf. Details: failed to get namespace information for remote URL osdf:///chtc/staging/sjungerman/containers/trixie.sif: error while querying the director at https://osdf-director.osg-htc.org: Contact.Director Error: Error code 3001: 404: No sources found for the requested path: no origins found for the requested namespace '/chtc/staging/sjungerman/containers/trixie.sif': Request ID: 99186a1f-81d8-43ec-ad3e-bb9d85b2252e ( URL file = osdf:///chtc/staging/sjungerman/containers/trixie.sif )|
	Partitionable Resources :    Usage  Request Allocated 
	   Cpus                 :        0        8         8 
	   Disk (KB)            :     1607  8388608   8388608 
	   GPUs                 :                           0 
	   IoHeavy              :                           0 
	   Memory (MB)          :              8192      8192 
...
012 (5078464.000.000) 2026-05-17 15:42:10 Job was held.
	Transfer input files failure at execution point slot1_22@e2013.chtc.wisc.edu using protocol osdf. Details: failed to get namespace information for remote URL osdf:///chtc/staging/sjungerman/containers/trixie.sif: error while querying the director at https://osdf-director.osg-htc.org: Contact.Director Error: Error code 3001: 404: No sources found for the requested path: no origins found for the requested namespace '/chtc/staging/sjungerman/containers/trixie.sif': Request ID: 99186a1f-81d8-43ec-ad3e-bb9d85b2252e ( URL file = osdf:///chtc/staging/sjungerman/containers/trixie.sif )|
	Code 13 Subcode 256
...

Note: I got the sif image by running apptainer pull /staging/sjungerman/containers/trixie.sif docker://debian:trixie-slim on the AP.