-
Notifications
You must be signed in to change notification settings - Fork 4
Expand file tree
/
Copy pathservices.html
More file actions
286 lines (267 loc) · 14.1 KB
/
services.html
File metadata and controls
286 lines (267 loc) · 14.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
<!DOCTYPE html>
<html lang="en">
<head>
<link rel="stylesheet" type="text/css" href="styles/style.css">
<title>OpenFAM: A library for programming Fabric-Attached Memory</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<header>
<h1>OpenFAM Reference Implementation</h1>
</header>
<section>
<nav>
<ul>
<li><a href="index.html">Home</a></li>
<li><a href="release_notes.html">Release Notes</a></li>
<li><a href="limitations.html">Design Choices</a></li>
<li><a href="errors.html">Exceptions and Error Codes</a></li>
<li><a href="services.html">Services</a></li>
<li><a href="config_files.html">Configuration Files</a></li>
</ul>
<hr>
Initialization and Finalization
<ul>
<li><a href="fam_initialize.html">fam_initialize</a></li>
<li><a href="fam_finalize.html">fam_finalize</a></li>
<li><a href="fam_abort.html">fam_abort</a></li>
</ul>
<hr>
Options & Query
<ul>
<li><a href="fam_list_options.html">fam_list_options</a></li>
<li><a href="fam_get_option.html">fam_get_option</a></li>
<li><a href="fam_lookup.html">fam_lookup</a></li>
<li><a href="fam_stat.html">fam_stat</a></li>
</ul>
<hr>
Memory Allocation
<ul>
<li><a href="fam_create_region.html">fam_create_region</a></li>
<li><a href="fam_destroy_region.html">fam_destroy_region</a></li>
<li><a href="fam_resize_region.html">fam_resize_region</a></li>
<li><a href="fam_allocate.html">fam_allocate</a></li>
<li><a href="fam_deallocate.html">fam_deallocate</a></li>
<li><a href="fam_change_permissions.html">fam_change_permissions</a></li>
<li><a href="fam_close.html">fam_close</a></li>
</ul>
<hr>
Memory Map
<ul>
<li><a href="fam_map.html">fam_map</a></li>
<li><a href="fam_unmap.html">fam_unmap</a></li>
</ul>
<hr>
Data Path Operations
<ul>
<li><a href="fam_get.html">fam_get</a></li>
<li><a href="fam_put.html">fam_put</a></li>
<li><a href="fam_gather.html">fam_gather</a></li>
<li><a href="fam_scatter.html">fam_scatter</a></li>
<li><a href="fam_copy.html">fam_copy</a></li>
<li><a href="fam_copy_wait.html">fam_copy_wait</a></li>
<li><a href="fam_backup.html">fam_backup</a></li>
<li><a href="fam_backup_wait.html">fam_backup_wait</a></li>
<li><a href="fam_restore.html">fam_restore</a></li>
<li><a href="fam_restore_wait.html">fam_restore_wait</a></li>
<li><a href="fam_delete_backup.html">fam_delete_backup</a></li>
<li><a href="fam_delete_backup_wait.html">fam_delete_backup_wait</a></li>
</ul>
<hr>
Atomics
<ul>
<li><a href="fam_set.html">fam_set</a></li>
<li><a href="fam_add.html">fam_add</a></li>
<li><a href="fam_subtract.html">fam_subtract</a></li>
<li><a href="fam_min.html">fam_min</a></li>
<li><a href="fam_max.html">fam_max</a></li>
<li><a href="fam_and.html">fam_and</a></li>
<li><a href="fam_or.html">fam_or</a></li>
<li><a href="fam_xor.html">fam_xor</a></li>
<li><a href="fam_fetch_TYPE.html">fam_fetch_TYPE</a></li>
<li><a href="fam_swap.html">fam_swap</a></li>
<li><a href="fam_compare_swap.html">fam_compare_swap</a></li>
<li><a href="fam_fetch_add.html">fam_fetch_add</a></li>
<li><a href="fam_fetch_subtract.html">fam_fetch_subtract</a></li>
<li><a href="fam_fetch_min.html">fam_fetch_min</a></li>
<li><a href="fam_fetch_and.html">fam_fetch_and</a></li>
<li><a href="fam_fetch_max.html">fam_fetch_max</a></li>
<li><a href="fam_fetch_or.html">fam_fetch_or</a></li>
<li><a href="fam_fetch_xor.html">fam_fetch_xor</a></li>
</ul>
<hr>
Ordering
<ul>
<li><a href="fam_barrier_all.html">fam_barrier_all</a></li>
<li><a href="fam_fence.html">fam_fence</a></li>
<li><a href="fam_quiet.html">fam_quiet</a></li>
<li><a href="fam_progress.html">fam_progress</a></li>
<li><a href="fam_context_open.html">fam_context_open</a></li>
<li><a href="fam_context_close.html">fam_context_close</a></li>
</ul>
<hr>
</nav>
<article>
<h1>OpenFAM 3.1 Services</h1>
<p> OpenFAM version 2.0 introduced the concept of supporting large pool of memory using multiple memory servers.
OpenFAM version 3.1 contains significant design changes to increase scalability and improve performance.
<ul>
<li>The design supports a larger pool of memory by supporting multiple memory servers. This memory pool can be either volatile or persistent.</li>
<li>Data can be interleaved as data items can now span across multiple memory servers, thus reducing performance bottlenecks.</li>
<li>Datapath operations can now be partitioned into multiple contexts.</li>
<li>Multi-threaded data path and atomic operations are supported.</li>
<li>Additional APIs have been added to the facilitate data movement between memory pool and archival storage.</li>
</ul>
<figure>
<img src="images/openfam2.0memoryserver.png" width="450px">
<figcaption>Figure 1: OpenFAM Memory server components</figcaption>
</figure>
<p> Figure 1 shows the three main components in the design.
<dl>
<dt>The client interface service (CIS):</dt><dd>All PEs talk to the CIS for memory server operations.</dd>
<dt>The metadata management service:</dt><dd>Region and data item metadata is hosted by the metadata management service.
The service can be co-located in the same node as a CIS server or a memory server, or it can be run in dedicated metadata management nodes.</dd>
<dt>The memory management service:</dt><dd> The actual memory is served from memory servers.
For data path (<code>get</code>, <code>put</code> and <code>atomic</code>) operations PEs talk to the memory servers directly.
For memory management and metadata operations, e.g., <code>allocate</code>, <code>lookup</code>, <code>create_region</code>, etc.,
the CIS communicates with memory servers via an RPC service or via direct functions calls.</dd>
</dl>
The services communicate with one another using grpc calls. However, as an optimization, if the components are compiled into the same executable, they bypass grpc and use direct function calls for communication.
Yaml-based configuration files determine how the services are configured.
<h1>Starting Services</h1>
<p> All the services and the PE read the respective configuration file from "config" directory placed at the path mentioned
by <code>OPENFAM_ROOT</code> environment variable.</p>
<p> Or by default, <code>OPENFAM_ROOT</code> will be <code>/opt/hpe/OpenFAM</code>.</p>
<p> User needs to update this environment variable to select config folder created by them.</p>
<p> A tool by name <code>openfam_adm</code> has been introduced which helps to perform the basic tasks useful to an OpenFAM user such as </p>
<ol>
<li>creating configuration files </li>
<li>starting OpenFAM services </li>
<li>terminating OpenFAM services</li>
<li>running tests and </li>
<li>cleanup FAM and metadata. </li>
</ol>
<p>To access the tool, the openfam module must be loaded. That can be done using following command: </p>
<code>module load openfam</code>
<p>The detailed description and available options are given below.</p>
<p>The details of available command line options can be found using <code>-h</code> option of each of the service executable.
For Example:</p>
<pre>#<text style="background-color:LightGrey;">openfam_adm -h</text>
Usage :
openfam_adm [-h] [--create_config_files] [--start_service]
[--stop_service] [--clean] [--install_path]
[--config_file_path] [--launcher]
[--global_launcher_options] [--cisinterface]
[--memserverinterface] [--metaserverinterface]
[--cisserver id:{comma separated attributes}]
[--memservers id:{comma seperated attributes}]
[--metaservers id:{comma seperated attributes}]
[--test_args] [--model] [--thrdmodel] [--provider]
[--provider] [--runtime] [--kvstype]
[--libfabricport] [--metapath] [--fampath]
[--fam_backup_path] [--atlthreads] [--atlqsize]
[--atldatasize] [--disableregionspanning]
[--regionspanningsize] [--interleaveblocksize]
[--runtests] [--sleep] [--rpc_framework]
[--disable_resource_release]
Options :
optional arguments:
-h, --help show this help message and exit
--create_config_files
Generates all configuration files
--start_service Start all OpenFAM services
--stop_service Stop all OpenFAM services
--clean Clean all FAMs and Metadata
--install_path Path to OpenFAM installation dir
--config_file_path Path to create configuration files
--launcher Launcher to be used to launch processes(mpi/slurm)
--global_launcher_options
Launcher command options which can be used commonly
across all services. Specify all options within
quotes(") eg : --global_launcher_options="-N" "1" "--
mpi=pmix"
--cisinterface CIS interface type(rpc/direct)
--memserverinterface
Memory Service interface type(rpc/direct)
--metaserverinterface
Metadata Service interface type(rpc/direct)
--cisserver CIS parameters, eg : ,0:{rpc_interface:127.0.0.1,rpc_p
ort:8795,launcher_options: "-n" "1"}. Note : Specify
all launcher options within quotes(") eg :
launcher_options:"-N" "1" "--mpi=pmix"
--memservers id:{comma seperated attributes}
Add a new memory server, eg : 0:{memory_type:volatile,
fam_path:/dev/shm/vol/,rpc_interface:127.0.0.1,rpc_por
t:8795,libfabric_port:7500,if_device:eth0,launcher_opt
ions: '-n' '1'}. Note : Specify all launcher options
within quotes(") eg : launcher_options: "-N" "1" "--
mpi=pmix"
--metaservers id:{comma seperated attributes}
Add a metadata server, eg : 0:{rpc_interface:127.0.0.1
,rpc_port:8795,launcher_options: '-n' '1'}. Note :
Specify all launcher options within quotes(") eg :
launcher_options: "-N" "1" "--mpi=pmix"
--test_args Test configuration parameters, eg :
,0:{num_pes:1,launcher_options: '-n' '1'}. Note :
Specify all launcher options within quotes(") eg :
launcher_options: "-N" "1" "--mpi=pmix"
--model OpenFAM model(memory_server/shared_memory)
--thrdmodel Thread model(single/multiple)
--provider Libfabric provider to be used for data path operations
--runtime Runtime process management interface(PMI) to be used
--kvstype Type of key value store to be used for metadata
management
--libfabricport Libafabric port for data path operation
--metapath path to store metadata
--fampath path where data is stored
--fam_backup_path path where data backup is stored(on shared filesystem)
--atlthreads Atomic library threads count
--atlqsize ATL queue size
--atldatasize ATL data size per thread(MiB)
--disableregionspanning
Disable region spanning
--regionspanningsize
Region spanning size
--interleaveblocksize
Dataitem interleave block size
--runtests Run regression and unit tests
--sleep Number of seconds to sleep
--rpc_framework RPC Framework type(grpc/thallium)
--disable_resource_release
Disable the resource relinquishment in FAM
For Example, following commands are primarily used to create configuration files, start/stop OpenFAM services and cleanup the FAM and metadata:
- Generate Configuration Files (Only one time step)
$openfam_adm @arg_file --create_config_files --config_file_path=/path/to/generate/config/files
Content of sample "arg_file" :
--launcher=slurm
--model=memory_server
--cisinterface=rpc
--memserverinterface=rpc
--metaserverinterface=rpc
--provider=cxi
--cisserver={rpc_interface:node1,rpc_port:8787}
--memservers=0:{memory_type:volatile,fam_path:/dev/shm/vol,rpc_interface:node2,rpc_port:8793,libfab-
ric_port:7500,if_device:eth0}
--metaservers=0:{rpc_interface:node3,rpc_port:8788}
--global_launcher_options="-p" "nodes" "--mpi=pmix"
- Export path of config files dir
$export OPENFAM_ROOT=/path/to/generate/config/files
- Start/Stop OpenFAM Services
$openfam_adm --start_service
- Stop OpenFAM Services
$openfam_adm --stop_service
- Cleanup
$openfam_adm --clean
</pre>
<p>The configuration options passed to the services using command line arguments take precedence over
the options present in the respective configuration files.</p>
<p>Similarly, for the PE, configuration options passed using <code>Fam_Options</code> argument of <a href="fam_initialize.html">fam_initialize</a> API take precedence
over the options present in <code>fam_pe_config.yaml</code> configuration file.
</article>
</section>
<footer>
<p>Copyright 2021-23, Hewlett Packard Enterprise Development Co, LLP</p>
</footer>
</body>
</html>