Speed up device query - #852
ben-grande wants to merge 8 commits into
Conversation
7374fee to
753c7ef
Compare
753c7ef to
7de7cf3
Compare
7de7cf3 to
fe99eab
Compare
fe99eab to
acdbe4a
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #852 +/- ##
==========================================
+ Coverage 70.51% 70.86% +0.35%
==========================================
Files 61 61
Lines 14371 14275 -96
==========================================
- Hits 10133 10116 -17
+ Misses 4238 4159 -79
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| for disk in xml.xpath("//domain/devices/disk"): | ||
| if disk.find("backenddomain") is not None: | ||
| pool_name = "p_%s" % disk.find("backenddomain").get("name") | ||
| pool = self.app.pools[pool_name] |
There was a problem hiding this comment.
This code was already wrong... I attached a block device to a qube:
<disk type='block' device='disk'>
<driver name='phy' type='raw'/>
<source dev='/dev/sda1'/>
<backenddomain name='sys-usb'/>
<script path='/etc/xen/scripts/qubes-block'/>
<target dev='xvdi' bus='xen'/>
</disk>But this method is not used anywhere: https://github.com/search?q=org%3AQubesOS+attached_volumes&type=code.
As there can be a functional client for a broken server method, can I delete this method?
8b9c2ea to
8639b85
Compare
|
Mypy is trippin..., and I can't seem to make the linting fail locally even though the mypy version is the same from CI. |
137790d to
bdaf6a7
Compare
| if line.startswith("\t\t") and class_id and subclass_id: | ||
| if not line: | ||
| continue | ||
| first_char = line[0] |
There was a problem hiding this comment.
Would saving line[:2] work better here? All cases below look at first two chars, so comparing (with ==) will be nicer.
| line = line.rstrip() | ||
| if not line: | ||
| continue | ||
| first_char = line[0] |
Each call was recording 1ms, now the calls are below that.
This function reimports the same information over and over and is time consuming when looping through multiple domains. Reduce listing USB devices while there is no cache from 2.8s to 0.5s.
| ) -> Iterator[tuple[PCIDevice, dict]]: | ||
| # pylint: disable=unused-argument | ||
| if not vm.is_running() or isinstance(vm, qubes.vm.adminvm.AdminVM): | ||
| if not vm.is_running() or vm.qid == 0: |
There was a problem hiding this comment.
To avoid isinstance, one call is not slow, but multiple are. I don't have the benchmarks for this specifically, but if you'd like, I can generate it.
There was a problem hiding this comment.
Hum... this is weird, mostly because the time to access qid is too long.
class QubesAdminvmQid(QubesPerf):
def __init__(self, *args, **kwargs) -> None:
# pylint: disable=import-outside-toplevel,import-error
super().__init__(*args, **kwargs)
self.vm = self.app.domains[0]
def test(self, previous_value=None) -> bool:
return self.vm.qid == 0
class QubesAdminvmIsinstance(QubesPerf):
def __init__(self, *args, **kwargs) -> None:
# pylint: disable=import-outside-toplevel,import-error
super().__init__(*args, **kwargs)
self.vm = self.app.domains[0]
import qubes.vm.adminvm
self.klass = qubes.vm.adminvm.AdminVM
def test(self, previous_value=None) -> bool:
return isinstance(self.vm, self.klass)% ~/bench.py -r 500 -b QubesAdminvmQid
- Tot: 0.000613
- 1st: 0.000007
- Max: 0.000007
- Min: 0.000001
- Avg: 0.000001
- Med: 0.000001
[user@dom0 ~]
% ~/bench.py -r 500 -b QubesAdminvmIsinstance
- Tot: 0.000530
- 1st: 0.000006
- Max: 0.000022
- Min: 0.000001
- Avg: 0.000001
- Med: 0.000001
[user@dom0 ~]
% ~/bench.py -r 500 -b QubesAdminvmQid
- Tot: 0.000504
- 1st: 0.000004
- Max: 0.000004
- Min: 0.000001
- Avg: 0.000001
- Med: 0.000001
[user@dom0 ~]
% ~/bench.py -r 5000 -b QubesAdminvmQid
- Tot: 0.005201
- 1st: 0.000007
- Max: 0.000031
- Min: 0.000001
- Avg: 0.000001
- Med: 0.000001
[user@dom0 ~]
% ~/bench.py -r 5000 -b QubesAdminvmIsinstance
- Tot: 0.000034
- 1st: 0.000002
- Max: 0.000029
- Min: 0.000000
- Avg: 0.000000
- Med: 0.000000
[user@dom0 ~]
% ~/bench.py -r 5000 -b QubesAdminvmQid
- Tot: 0.005200
- 1st: 0.000006
- Max: 0.000025
- Min: 0.000001
- Avg: 0.000001
- Med: 0.000001
[user@dom0 ~]
% ~/bench.py -r 5000 -b QubesAdminvmIsinstance
- Tot: 0.000046
- 1st: 0.000003
- Max: 0.000019
- Min: 0.000000
- Avg: 0.000000
- Med: 0.000000
% ~/bench.py -r 5000 -b QubesAdminvmIsinstance -p cProfile
cProfile
80005 function calls in 0.037 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
5000 0.007 0.000 0.037 0.000 bench.py:166(_test)
10001 0.004 0.000 0.016 0.000 {built-in method builtins.next}
5001 0.002 0.000 0.014 0.000 contextlib.py:145(__exit__)
10000 0.006 0.000 0.012 0.000 bench.py:93(_timeit)
5000 0.003 0.000 0.007 0.000 contextlib.py:303(helper)
5000 0.002 0.000 0.006 0.000 contextlib.py:136(__enter__)
5000 0.004 0.000 0.005 0.000 contextlib.py:108(__init__)
5000 0.003 0.000 0.003 0.000 {built-in method builtins.round}
5000 0.001 0.000 0.002 0.000 bench.py:258(test)
10000 0.002 0.000 0.002 0.000 {built-in method time.perf_counter_ns}
5000 0.001 0.000 0.001 0.000 {built-in method builtins.getattr}
5000 0.001 0.000 0.001 0.000 {method 'append' of 'list' objects}
5000 0.001 0.000 0.001 0.000 {built-in method builtins.isinstance}
1 0.000 0.000 0.000 0.000 bench.py:127(_profile)
1 0.000 0.000 0.000 0.000 bench.py:119(_end_prof)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
- Tot: 0.010176
- 1st: 0.000011
- Max: 0.000027
- Min: 0.000002
- Avg: 0.000002
- Med: 0.000002
[user@dom0 ~]
% ~/bench.py -r 5000 -b QubesAdminvmQid -p cProfile
cProfile
95005 function calls in 0.048 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
5000 0.008 0.000 0.048 0.000 bench.py:166(_test)
10001 0.004 0.000 0.016 0.000 {built-in method builtins.next}
5001 0.002 0.000 0.015 0.000 contextlib.py:145(__exit__)
10000 0.007 0.000 0.013 0.000 bench.py:93(_timeit)
5000 0.002 0.000 0.012 0.000 bench.py:246(test)
5000 0.005 0.000 0.010 0.000 __init__.py:233(__get__)
5000 0.003 0.000 0.007 0.000 contextlib.py:303(helper)
5000 0.002 0.000 0.006 0.000 contextlib.py:136(__enter__)
5000 0.004 0.000 0.005 0.000 contextlib.py:108(__init__)
10000 0.004 0.000 0.004 0.000 {built-in method builtins.getattr}
5000 0.003 0.000 0.003 0.000 {built-in method builtins.round}
10000 0.002 0.000 0.002 0.000 {built-in method time.perf_counter_ns}
5000 0.001 0.000 0.001 0.000 {built-in method builtins.isinstance}
5000 0.001 0.000 0.001 0.000 {method 'append' of 'list' objects}
5000 0.001 0.000 0.001 0.000 __init__.py:250(get_default)
1 0.000 0.000 0.000 0.000 bench.py:127(_profile)
1 0.000 0.000 0.000 0.000 bench.py:119(_end_prof)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
- Tot: 0.020359
- 1st: 0.000021
- Max: 0.000032
- Min: 0.000004
- Avg: 0.000004
- Med: 0.000004
[user@dom0 ~]
% ~/bench.py -r 5000 -b QubesAdminvmIsinstance -p cProfile
cProfile
80005 function calls in 0.030 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
5000 0.006 0.000 0.030 0.000 bench.py:166(_test)
10001 0.003 0.000 0.013 0.000 {built-in method builtins.next}
5001 0.002 0.000 0.011 0.000 contextlib.py:145(__exit__)
10000 0.005 0.000 0.009 0.000 bench.py:93(_timeit)
5000 0.002 0.000 0.006 0.000 contextlib.py:303(helper)
5000 0.002 0.000 0.005 0.000 contextlib.py:136(__enter__)
5000 0.003 0.000 0.004 0.000 contextlib.py:108(__init__)
5000 0.002 0.000 0.002 0.000 {built-in method builtins.round}
5000 0.001 0.000 0.002 0.000 bench.py:258(test)
10000 0.002 0.000 0.002 0.000 {built-in method time.perf_counter_ns}
5000 0.001 0.000 0.001 0.000 {method 'append' of 'list' objects}
5000 0.001 0.000 0.001 0.000 {built-in method builtins.getattr}
5000 0.001 0.000 0.001 0.000 {built-in method builtins.isinstance}
1 0.000 0.000 0.000 0.000 bench.py:127(_profile)
1 0.000 0.000 0.000 0.000 bench.py:119(_end_prof)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
- Tot: 0.009329
- 1st: 0.000010
- Max: 0.000045
- Min: 0.000001
- Avg: 0.000002
- Med: 0.000002
[user@dom0 ~]
% ~/bench.py -r 5000 -b QubesAdminvmQid -p cProfile
cProfile
95005 function calls in 0.049 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
5000 0.008 0.000 0.049 0.000 bench.py:166(_test)
10001 0.004 0.000 0.017 0.000 {built-in method builtins.next}
5001 0.002 0.000 0.015 0.000 contextlib.py:145(__exit__)
10000 0.007 0.000 0.013 0.000 bench.py:93(_timeit)
5000 0.002 0.000 0.013 0.000 bench.py:246(test)
5000 0.005 0.000 0.010 0.000 __init__.py:233(__get__)
5000 0.003 0.000 0.007 0.000 contextlib.py:303(helper)
5000 0.002 0.000 0.007 0.000 contextlib.py:136(__enter__)
5000 0.004 0.000 0.005 0.000 contextlib.py:108(__init__)
10000 0.004 0.000 0.004 0.000 {built-in method builtins.getattr}
5000 0.003 0.000 0.003 0.000 {built-in method builtins.round}
10000 0.002 0.000 0.002 0.000 {built-in method time.perf_counter_ns}
5000 0.001 0.000 0.001 0.000 {built-in method builtins.isinstance}
5000 0.001 0.000 0.001 0.000 __init__.py:250(get_default)
5000 0.001 0.000 0.001 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 bench.py:127(_profile)
1 0.000 0.000 0.000 0.000 bench.py:119(_end_prof)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
- Tot: 0.020597
- 1st: 0.000019
- Max: 0.000038
- Min: 0.000004
- Avg: 0.000004
- Med: 0.000004
% ~/bench.py -r 5000 -b QubesAdminvmQid -p pyinstrument -i 0.00000003
pyinstrument
_ ._ __/__ _ _ _ _ _/_ Recorded: 18:25:15 Samples: 190024
/_//_/// /_\ / //_// / //_'/ // Duration: 1.390 CPU time: 1.376
/ _/ v4.6.2
Program: /home/user/bench.py -r 5000 -b QubesAdminvmQid -p pyinstrument -i 0.00000003
1.390 run bench.py:170
├─ 1.361 _test bench.py:166
│ ├─ 0.439 __exit__ contextlib.py:145
│ │ ├─ 0.266 _timeit bench.py:93
│ │ │ ├─ 0.151 [self] bench.py
│ │ │ ├─ 0.039 round <built-in>
│ │ │ ├─ 0.038 list.append <built-in>
│ │ │ └─ 0.037 perf_counter_ns <built-in>
│ │ ├─ 0.136 [self] contextlib.py
│ │ └─ 0.037 next <built-in>
│ ├─ 0.339 test bench.py:246
│ │ ├─ 0.276 __get__ qubes/__init__.py:233
│ │ │ ├─ 0.161 [self] qubes/__init__.py
│ │ │ ├─ 0.040 getattr <built-in>
│ │ │ ├─ 0.037 isinstance <built-in>
│ │ │ └─ 0.037 get_default qubes/__init__.py:250
│ │ └─ 0.063 [self] bench.py
│ ├─ 0.242 __enter__ contextlib.py:136
│ │ ├─ 0.107 _timeit bench.py:93
│ │ │ ├─ 0.073 [self] bench.py
│ │ │ └─ 0.034 perf_counter_ns <built-in>
│ │ ├─ 0.099 [self] contextlib.py
│ │ └─ 0.036 next <built-in>
│ ├─ 0.181 helper contextlib.py:303
│ │ ├─ 0.115 __init__ contextlib.py:108
│ │ │ ├─ 0.077 [self] contextlib.py
│ │ │ └─ 0.038 getattr <built-in>
│ │ └─ 0.066 [self] contextlib.py
│ └─ 0.160 [self] bench.py
├─ 0.029 [self] bench.py
├─ 0.000 __exit__ contextlib.py:145
│ ├─ 0.000 _profile bench.py:127
│ │ ├─ 0.000 _end_prof bench.py:119
│ │ │ ├─ 0.000 stop pyinstrument/profiler.py:138
│ │ │ │ ├─ 0.000 unsubscribe pyinstrument/stack_sampler.py:77
│ │ │ │ │ ├─ 0.000 [self] pyinstrument/stack_sampler.py
│ │ │ │ │ ├─ 0.000 ContextVar.set <built-in>
│ │ │ │ │ ├─ 0.000 <genexpr> pyinstrument/stack_sampler.py:79
│ │ │ │ │ └─ 0.000 next <built-in>
│ │ │ │ ├─ 0.000 get_stack_sampler pyinstrument/stack_sampler.py:154
│ │ │ │ │ ├─ 0.000 hasattr <built-in>
│ │ │ │ │ └─ 0.000 [self] pyinstrument/stack_sampler.py
│ │ │ │ └─ 0.000 [self] pyinstrument/profiler.py
│ │ │ └─ 0.000 [self] bench.py
│ │ └─ 0.000 [self] bench.py
│ └─ 0.000 [self] contextlib.py
└─ 0.000 __enter__ contextlib.py:136
├─ 0.000 _profile bench.py:127
│ ├─ 0.000 _start_prof bench.py:106
│ │ ├─ 0.000 start pyinstrument/profiler.py:103
│ │ │ ├─ 0.000 subscribe pyinstrument/stack_sampler.py:54
│ │ │ │ ├─ 0.000 _update pyinstrument/stack_sampler.py:92
│ │ │ │ │ ├─ 0.000 [self] pyinstrument/stack_sampler.py
│ │ │ │ │ └─ 0.000 _start_sampling pyinstrument/stack_sampler.py:102
│ │ │ │ └─ 0.000 [self] pyinstrument/stack_sampler.py
│ │ │ └─ 0.000 [self] pyinstrument/profiler.py
│ │ └─ 0.000 [self] bench.py
│ └─ 0.000 [self] bench.py
└─ 0.000 [self] contextlib.py
- Tot: 0.683171
- 1st: 0.000219
- Max: 0.028014
- Min: 0.000081
- Avg: 0.000137
- Med: 0.000111
[user@dom0 ~]
% ~/bench.py -r 5000 -b QubesAdminvmIsinstance -p pyinstrument -i 0.00000003
pyinstrument
_ ._ __/__ _ _ _ _ _/_ Recorded: 18:25:58 Samples: 160024
/_//_/// /_\ / //_// / //_'/ // Duration: 1.901 CPU time: 1.876
/ _/ v4.6.2
Program: /home/user/bench.py -r 5000 -b QubesAdminvmIsinstance -p pyinstrument -i 0.00000003
1.901 run bench.py:170
├─ 1.854 _test bench.py:166
│ ├─ 0.752 __exit__ contextlib.py:145
│ │ ├─ 0.420 _timeit bench.py:93
│ │ │ ├─ 0.243 [self] bench.py
│ │ │ ├─ 0.063 round <built-in>
│ │ │ ├─ 0.057 list.append <built-in>
│ │ │ └─ 0.057 perf_counter_ns <built-in>
│ │ ├─ 0.270 [self] contextlib.py
│ │ └─ 0.063 next <built-in>
│ ├─ 0.415 __enter__ contextlib.py:136
│ │ ├─ 0.206 _timeit bench.py:93
│ │ │ ├─ 0.151 [self] bench.py
│ │ │ └─ 0.055 perf_counter_ns <built-in>
│ │ ├─ 0.155 [self] contextlib.py
│ │ └─ 0.055 next <built-in>
│ ├─ 0.285 helper contextlib.py:303
│ │ ├─ 0.180 __init__ contextlib.py:108
│ │ │ ├─ 0.122 [self] contextlib.py
│ │ │ └─ 0.058 getattr <built-in>
│ │ └─ 0.105 [self] contextlib.py
│ ├─ 0.251 [self] bench.py
│ └─ 0.150 test bench.py:258
│ ├─ 0.099 [self] bench.py
│ └─ 0.051 isinstance <built-in>
├─ 0.046 [self] bench.py
├─ 0.000 __exit__ contextlib.py:145
│ ├─ 0.000 _profile bench.py:127
│ │ ├─ 0.000 _end_prof bench.py:119
│ │ │ ├─ 0.000 stop pyinstrument/profiler.py:138
│ │ │ │ ├─ 0.000 unsubscribe pyinstrument/stack_sampler.py:77
│ │ │ │ │ ├─ 0.000 [self] pyinstrument/stack_sampler.py
│ │ │ │ │ ├─ 0.000 ContextVar.set <built-in>
│ │ │ │ │ ├─ 0.000 <genexpr> pyinstrument/stack_sampler.py:79
│ │ │ │ │ └─ 0.000 next <built-in>
│ │ │ │ ├─ 0.000 get_stack_sampler pyinstrument/stack_sampler.py:154
│ │ │ │ │ ├─ 0.000 [self] pyinstrument/stack_sampler.py
│ │ │ │ │ └─ 0.000 hasattr <built-in>
│ │ │ │ └─ 0.000 [self] pyinstrument/profiler.py
│ │ │ └─ 0.000 [self] bench.py
│ │ └─ 0.000 [self] bench.py
│ └─ 0.000 [self] contextlib.py
└─ 0.000 __enter__ contextlib.py:136
├─ 0.000 _profile bench.py:127
│ ├─ 0.000 _start_prof bench.py:106
│ │ ├─ 0.000 start pyinstrument/profiler.py:103
│ │ │ ├─ 0.000 subscribe pyinstrument/stack_sampler.py:54
│ │ │ │ ├─ 0.000 _update pyinstrument/stack_sampler.py:92
│ │ │ │ │ ├─ 0.000 [self] pyinstrument/stack_sampler.py
│ │ │ │ │ └─ 0.000 _start_sampling pyinstrument/stack_sampler.py:102
│ │ │ │ └─ 0.000 [self] pyinstrument/stack_sampler.py
│ │ │ └─ 0.000 [self] pyinstrument/profiler.py
│ │ └─ 0.000 [self] bench.py
│ └─ 0.000 [self] bench.py
└─ 0.000 [self] contextlib.py
- Tot: 0.778388
- 1st: 0.000073
- Max: 0.080601
- Min: 0.000047
- Avg: 0.000156
- Med: 0.000128
Method 'startwith' adds considerable overhead.
bdaf6a7 to
e8d9c51
Compare
|
PipelineRetry |
Libvirt API can be quite slow on the methods for "listAllDevices" and "XMLDesc", aggravated in loops. Tests had to use instantiate AdminVM class so "isinstance" is happy.
Although it doesn't help reduce the time to get the XML, as XMLDesc() is super slow, this helps cleanup the code a bit.
There is no client using them, they are broken and have been for a long time. The backeenddomain name is a qube name, not a pool name.
e8d9c51 to
ef612ad
Compare
This PR was split from #819 and it does not contain commits that should be problematic (more tested), those were left out to #855.