Having created my own node-allocation class, I pass it to the JobCluster-constructor:
class O2Allocate(dispy.NodeAllocate):
def allocate(self, cluster, ip_addr, name, cpus, avail_info = None,
platform = '*'):
global settings, logger, value_date
logger.info('Allocating node %s (%s), with %d cores (plus 20%)',
name, ip_addr, cpus)
self.setup_args = (settings.settings_dict['vendorLib'],
datetime.strptime(value_date, '%Y%m%d'))
return int(cpus * 1.2)
...
cluster = dispy.JobCluster(
oneScenario,
setup = prep,
callback = done,
loglevel = dispy.logger.DEBUG,
reentrant = True,
depends = Depends,
dispy_port = settings.settings_dict.get('dispynodeport', 51347),
secret = settings.settings_dict.get('dispySecret', ''),
pulse_interval = 11,
ip_addr = list(util.my_ip_addresses()),
nodes = [O2Allocate('*')]
)The nodes here have 16 or 18 cores, so the int(cpus * 1.2) should result in 19 or 21 respectively -- and that's, what we want for better CPU-saturation, because the jobs spend some time reading data from NAS (cannot be helped).
However, the setting does not seem to have an effect -- the client reports node-discoveries citing their actual core-counts:
2025-10-28 15:10:53.282 dispy - Discovered 10.92.161.138:21349 (node1) with 18 cpus
2025-10-28 15:10:53.283 dispy - Discovered 10.92.161.124:21349 (node3) with 18 cpus
2025-10-28 15:10:53.287 dispy - Discovered 10.92.161.131:21349 (node2) with 18 cpus
and then it continues to treat each node as having that same 18 cores -- instead of the increased number:
2025-10-28 15:10:53.605 dispy - Running job ('xxx', -1) / 140063137885784 on 10.92.161.126 (busy: 18 / 18)
2025-10-28 15:10:53.606 dispy - Running job ('yyy', -1) / 140063135756720 on 10.92.161.126 (busy: 18 / 18)
2025-10-28 15:10:53.607 dispy - Running job ('zzz', -1) / 140063137885184 on 10.92.161.126 (busy: 18 / 18)
How should I raise the core-count perceived by the client?
Separately, how do I pass the actual list of nodes to JobCluster -- instead of relying on it automatically discovering them?
Using:
- dispy==4.15.2
- pycos==4.12.2
Thank you!
Having created my own node-allocation class, I pass it to the
JobCluster-constructor:The nodes here have 16 or 18 cores, so the
int(cpus * 1.2)should result in 19 or 21 respectively -- and that's, what we want for better CPU-saturation, because the jobs spend some time reading data from NAS (cannot be helped).However, the setting does not seem to have an effect -- the client reports node-discoveries citing their actual core-counts:
and then it continues to treat each node as having that same 18 cores -- instead of the increased number:
How should I raise the core-count perceived by the client?
Separately, how do I pass the actual list of nodes to
JobCluster-- instead of relying on it automatically discovering them?Using:
Thank you!