Kerrighed: use cases

Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Kerrighed: use cases
Cyril Brulebois
[email protected]
Kerrighed
http://www.kerrighed.org/
Kerlabs
http://www.kerlabs.com/
1 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Introducing Kerrighed
What’s Kerrighed?
Single-System Image (SSI) cluster system
Patched Linux kernel, plus userland tools
Started at INRIA in 1999, collaboration with University of
Rennes 1 and EDF
Since 2006, mainly developed by Kerlabs, an INRIA spin-off
Released under the GPL
Last releases:
Kerrighed 2.4.4, based on Linux 2.6.20 (January 29th, 2010)
Kerrighed 3.0.0, based on Linux 2.6.30 (June 14th, 2010)
2 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
What’s new? What’s planned?
v2.4.4
Memory
CPU
IPC
Network
C/R
Misc
v3.0.0
in 2011
Sharing
Injection
Process migration
Thread migration
Pool of threads migration
Configurable scheduler
SysV
POSIX
Migratable streams
Cluster IP
High performance
Single process
Applications
Open files
File substitution
IPC
Node hotplug
3 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Managing nodes (1/2)
To manage cluster and nodes, a single command: krgadm.
# krgadm cluster status
status: up on 4 nodes
# krgadm nodes status
1:online
2:online
3:online
4:online
5:present
6:present
7:present
8:present
4 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Managing nodes (2/2)
Adding 2 nodes:
# krgadm nodes add --count 2
Waiting for 2 nodes to join... done
Adding nodes [5,6]... done
# krgadm cluster status
status: up on 6 nodes
Adding 2 nodes, in a different way:
# krgadm nodes add --total 8
Waiting for 2 nodes to join... done
Adding nodes [7,8]... done
# krgadm cluster status
status: up on 8 nodes
5 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Managing features
Use of capabilities to turn Kerrighed features on/off as
appropriate. Examples:
krgcapset -d +DISTANT_FORK
krgcapset --pid 6291711 -e +CAN_MIGRATE
Also possible through libkerrighed.
Most common capabilities:
DISTANT FORK: may fork remotely.
CAN MIGRATE: may be migrated while running.
CHECKPOINTABLE: may be checkpointed.
SEE LOCAL PROC STAT: only see local resources.
6 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
1. Introducing Kerrighed
What’s Kerrighed?
What’s new? What’s planned?
Managing nodes
Managing features
2. Load balancing
Use case 1: build platform
Use case 2: network computing
Use case 3: distributed rendering
Use case 4: webservers
Use case 5: parallel computing
Use case 6: LTSP
There’s more!
3. Checkpoint/restart
Use case 1: long running computations
Use case 2: playing with sockets
7 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Use case 1: build platform
Setup: $C cores
Capability: DISTANT FORK
Trivial:
make -j$C
8 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Use case 2: network computing
Capabilities: DISTANT FORK and/or CAN MIGRATE
Example: BOINC (Berkeley project), running ∗@home, PrimeGrid,
etc.
How: Just start BOINC! It runs as many children as there are
cores. It starts new children as they return.
9 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Use case 3: distributed rendering
Setup: $C cores
Capabilities: DISTANT FORK and/or CAN MIGRATE
Example: Blender, rendering a 1 → $N frame range.
How: Blender is able to render frames by batch, either a single
frame at once, or a frame range.
blender -b foo.blender -F PNG -o //render_####.png -f $i
10 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Naive approach
Trivial implementation:
Spawn $C processes.
Wait for all of them to return.
Back to spawning until the last frame is rendered.
Issue: if some frames are quicker to render than others, the global
wait will leave some cores idle.
11 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Smarter approach
A more efficient implementation:
Spawn $C processes.
Wait for one of them to return.
Spawn a new process unless the last frame has been reached.
Back to waiting.
That ensures $C processes running all the time until the end,
almost no idling.
Many other renderers and mostly anything scriptable can be run
this way, with this single and simple “job scheduler”. No need for
complex client/server solutions.
12 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Use case 4: webservers
Capabilities: DISTANT FORK and/or CAN MIGRATE
Example: Apache MPM worker (Multi-Processing Module), prefork
version (non-threaded, pre-forking).
Drawback: Socket handling.
Short-term solution: Enable an extra capability to have all sockets
listen on a given node, which acts as an entry point.
Long-term solution: Use cluster IP.
13 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Use case 5: parallel computing
Capabilities: DISTANT FORK and/or CAN MIGRATE
Example: R and its multicore package.
Code: Replace %do% with %dopar%
library("doMC")
registerDoMC()
x <- foreach(i=1:42) %dopar% svd(matrix(rnorm(1000*1000),ncol=1000))
Cores are automatically detected, but the worker count can be
tweaked by calling:
options(nodes=42)
14 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Use case 6: LTSP
Capabilities: DISTANT FORK and/or CAN MIGRATE
Example: Run one VNC server per user on the first node.
Launched applications get load-balanced over the whole cluster.
Possible issue with desktop environments: Heavy use of local
networking services (e.g. D-Bus).
Possible solutions:
Same as the web servers use case: use an extra capability to
direct all sockets to a given node.
Cluster IP?
Probably a bit more complex: loopback, global address space
for UNIX sockets, etc.
15 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
There’s more!
Schedulers for DISTANT FORK and CAN MIGRATE can be tweaked,
extended, or replaced. Configurable through configfs, thanks to:
Probes (e.g. free RAM).
Filters (e.g. reaching some threshold).
Policies.
Process sets and node sets.
Some possible policies, in addition to load balancing:
Swap avoidance
Disk I/O balancing
Slightly more complex: keep interactive applications local, move
others away, and welcome them back when there are no more
interactive applications (use case: Network of Workstations in
universities).
16 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
1. Introducing Kerrighed
What’s Kerrighed?
What’s new? What’s planned?
Managing nodes
Managing features
2. Load balancing
Use case 1: build platform
Use case 2: network computing
Use case 3: distributed rendering
Use case 4: webservers
Use case 5: parallel computing
Use case 6: LTSP
There’s more!
3. Checkpoint/restart
Use case 1: long running computations
Use case 2: playing with sockets
17 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Use case 1: long running computations
Why: Even with parallel algorithms running on powerful clusters,
computations can take hours, days, weeks, or more.
Checkpoint/Restart useful in case of hardware failures, system
errors, etc.
Example: R.
How: Either enable the CHECKPOINTABLE capability, or use a
wrapper which also creates a new session for the program to be
checkpointed:
krgcr-run -- R
Create a checkpoint every hour:
while :; do checkpoint $(pgrep R|head -1); sleep 3600; done
18 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Step by step (1/2)
Starting:
$ krgcr-run R
Running application 6291648
R version 2.11.1 (2010-05-31)
[...]
>
Checkpointing:
$ checkpoint 6291648
Freezing application in which process 6291648 is involved...
Checkpointing application in which process 6291648 is involved...
Identifier: 6291648
Version: 1
Description: No description
Date: Thu Jul 8 22:54:06 2010
Unfreezing application in which process 6291648 is involved...
19 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Step by step (2/2)
Contents of the checkpoint:
$ ls /var/chkpt/6291648/v1/
description.txt node_1.bin
global.bin
shared_obj_1.bin
task_6291648.bin
user_info_1.txt
Restarting:
$ restart 6291648 1
Restarting application 6291648 (v1) ...
Application 6291648 has been successfully restarted
20 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Use case 2: playing with sockets
Problem: checkpointing sockets isn’t supported yet.
Solution:
Force checkpointing: -i option for checkpoint.
Use file descriptor substitution.
Plugging a given file descriptor on a given checkpoint identifier:
$ cat /var/chkpt/6291711/v1/user_info_1.txt
tty
|0001FFFF88003ED8AC48|/dev/pts/1|6291711:0,6291711:1,6291711:2
socket |0001FFFF88007E5F3168|socket:[162646]|6291711:3
$ restart -s 0001FFFF88007E5F3168,0 6291711 1
Restarting application 6291711 (v1) ...
Application 6291711 has been successfully restarted
Future: Use that to restore the Unix socket to the X server?
21 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
Conclusion
Kerrighed’s strong features right now:
Stability
Flexibility
Can be configured/tweaked to suit specific needs
General solution for many common use cases
Kerrighed’s next features (short to mid-term):
Performance
More flexibility
Partial thread support
22 / 23
Introducing Kerrighed
Load balancing
Checkpoint/restart
Conclusion
The end
Thanks for your attention! Questions?
A few pointers:
Want to play?
→ http://www.kerrighed.org/
Want to talk?
→ kerrighed.{users,dev}@listes.irisa.fr
→ #kerrighed on Freenode
Want to apply for a nice job?
→ [email protected]
23 / 23