Using Parallel Magics¶

IPython has a few magics to make working with your engines a bit nicer in IPython, e.g. via a Jupyter notebook.

As always, first we start a cluster (or connect to an existing one):

import ipyparallel as ipp

rc = ipp.Cluster(n=4).start_and_connect_sync()
dv = rc[:]
rc.ids

Using existing profile dir: '/Users/benjaminrk/.ipython/profile_default'
Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

[0, 1, 2, 3]

Creating a Client registers the parallel magics %px, %%px, %pxresult, %pxconfig, and %autopx.
These magics are initially associated with a DirectView, always associated with all currently registered engines.

First, we can execute single lines remotely with %px:

%px a=5

%px print(a)

[stdout:0] 5

[stdout:1] 5

[stdout:2] 5

[stdout:3] 5

%px a

Out[2:3]: 5

Out[0:3]: 5

Out[3:3]: 5

Out[1:3]: 5

with dv.sync_imports():
    import sys

importing sys on engine(s)

%px from __future__ import print_function
%px print("ERROR", file=sys.stderr)

[stderr:0] ERROR

[stderr:1] ERROR

[stderr:3] ERROR

[stderr:2] ERROR

You don't have to wait for results. The %pxconfig magic lets you change the default blocking/targets for the %px magics:

%pxconfig --noblock

%px import time
%px time.sleep(1)
%px time.time()

<AsyncResult(%px): pending>

But you will notice that this didn't output the result of the last command. For this, we have %pxresult, which displays the output of the latest request:

%pxresult

Out[0:8]: 1635248507.7749069

Out[1:8]: 1635248507.7802722

Out[2:8]: 1635248507.780295

Out[3:8]: 1635248507.780204

Remember, an IPython engine is IPython, so you can do magics remotely as well!

%pxconfig --block
%px %matplotlib inline

%%px can be used to lower the priority of the engines to improve system performance under heavy CPU load.

%%px
import psutil
psutil.Process().nice(20 if psutil.POSIX else psutil.IDLE_PRIORITY_CLASS)

%%px
import numpy as np
import matplotlib.pyplot as plt

%%px can also be used as a cell magic, for submitting whole blocks. This one acceps --block and --noblock flags to specify the blocking behavior, though the default is unchanged.

dv.scatter('id', dv.targets, flatten=True)
dv['stride'] = len(dv)

%%px --noblock
x = np.linspace(0,np.pi,1000)
for n in range(id,12, stride):
    print(n)
    plt.plot(x,np.sin(n*x))
plt.title("Plot %i" % id)

<AsyncResult(%px): pending>

%pxresult

[stdout:0] 
0
4
8
[stdout:1] 
1
5
9
[stdout:2] 
2
6
10
[stdout:3] 
3
7
11

[output:0]

[output:1]

[output:2]

[output:3]

Out[0:13]: Text(0.5, 1.0, 'Plot 0')

Out[1:13]: Text(0.5, 1.0, 'Plot 1')

Out[2:13]: Text(0.5, 1.0, 'Plot 2')

Out[3:13]: Text(0.5, 1.0, 'Plot 3')

It also lets you choose some amount of the grouping of the outputs with --group-outputs:

The choices are:

engine - all of an engine's output is collected together
type - where stdout of each engine is grouped, etc. (the default)
order - same as type, but individual displaypub outputs are interleaved. That is, it will output the first plot from each engine, then the second from each, etc.

%%px --group-outputs=engine
x = np.linspace(0,np.pi,1000)
for n in range(id+1,12, stride):
    print(n)
    plt.figure()
    plt.plot(x,np.sin(n*x))
    plt.title("Plot %i" % n)

[stdout:0] 1
5
9

[stdout:1] 2
6
10

[stdout:2] 3
7
11

[stdout:3] 4
8

[output:3]

[output:0]

[output:1]

[output:2]

[output:3]

[output:0]

When you specify 'order', then individual display outputs (e.g. plots) will be interleaved.

%pxresult takes the same output-ordering arguments as %%px, so you can view the previous result in a variety of different ways with a few sequential calls to %pxresult:

%pxresult --group-outputs=order

[stdout:0] 
1
5
9
[stdout:1] 
2
6
10
[stdout:2] 
3
7
11
[stdout:3] 
4
8

[output:0]

[output:1]

[output:2]

[output:0]

[output:1]

[output:2]

[output:0]

[output:1]

[output:2]

Single-engine views¶

When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):

from __future__ import print_function

def generate_output():
    """function for testing output
    
    publishes two outputs of each type, and returns something
    """
    
    import sys,os
    from IPython.display import display, HTML, Math
    
    print("stdout")
    print("stderr", file=sys.stderr)
    
    display(HTML("<b>HTML</b>"))
    
    print("stdout2")
    print("stderr2", file=sys.stderr)
    
    display(Math(r"\alpha=\beta"))
    
    return os.getpid()

dv['generate_output'] = generate_output

You can also have more than one set of parallel magics registered at a time.

The View.activate() method takes a suffix argument, which is added to 'px'.

e0 = rc[-1]
e0.block = True
e0.activate('0')

%px0 generate_output()

[stdout:3] stdout
stdout2

[stderr:3] stderr
stderr2

[output:3]

[output:3]

Out[3:15]: 83115

%px generate_output()

[stdout:0] stdout
stdout2

[stdout:1] stdout
stdout2

[stdout:2] stdout
stdout2

[stderr:2] stderr
stderr2

[stdout:3] stdout
stdout2

[stderr:0] stderr
stderr2

[output:0]

[stderr:1] stderr
stderr2

[output:1]

[stderr:3] stderr
stderr2

As mentioned above, we can redisplay those same results with various grouping:

%pxresult --group-outputs order

[stdout:0] 
stdout
stdout2
[stdout:1] 
stdout
stdout2
[stdout:2] 
stdout
stdout2
[stdout:3] 
stdout
stdout2

[stderr:0] 
stderr
stderr2
[stderr:1] 
stderr
stderr2
[stderr:2] 
stderr
stderr2
[stderr:3] 
stderr
stderr2

[output:0]

[output:1]

[output:2]

[output:3]

[output:0]

[output:1]

[output:2]

[output:3]

%pxresult --group-outputs engine

[stdout:0] 
stdout
stdout2

[stderr:0] 
stderr
stderr2

[output:0]

Out[0:15]: 83112

[stdout:1] 
stdout
stdout2

[stderr:1] 
stderr
stderr2

[output:1]

Out[1:15]: 83113

[stdout:2] 
stdout
stdout2

[stderr:2] 
stderr
stderr2

Parallel Exceptions¶

When you raise exceptions with the parallel exception, the CompositeError raised locally will display your remote traceback.

%%px
from numpy.random import random
A = random((100, 100, 'invalid shape'))

[0:execute]: 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83112/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer
[2:execute]: 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83114/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer
[1:execute]: 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83113/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer
[3:execute]: 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83115/1064401740.py in <module>
      1 from numpy.random import random
----> 2 A = random((100, 100, 'invalid shape'))

mtrand.pyx in numpy.random.mtrand.RandomState.random()

mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

_common.pyx in numpy.random._common.double_fill()

TypeError: 'str' object cannot be interpreted as an integer

4 errors

Sometimes, an error will occur on just one engine, while the rest are still working.

When this happens, you will see errors immediately, and can interrupt the execution:

dv.scatter("rank", rc.ids, flatten=True)

<AsyncResult(scatter): pending>

%%px
import time
if rank == 0:
    raise RuntimeError("rank 0 failed!")
time.sleep(10)

[0:execute]: 
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83112/2811384868.py in <module>
      1 import time
      2 if rank == 0:
----> 3     raise RuntimeError("rank 0 failed!")
      4 time.sleep(10)
      5 

RuntimeError: rank 0 failed!

1 errors

Remote Cell Magics¶

Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic.

%%px
%%timeit
from numpy.random import random
from numpy.linalg import norm
A = random((100, 100))
norm(A, 2)

[stdout:1] 6.01 ms ± 596 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

[stdout:0] 6.02 ms ± 549 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

[stdout:3] 6.06 ms ± 580 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

[stdout:2] 6.08 ms ± 430 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Local Execution¶

You can instruct %%px to also execute the cell locally. This is useful for interactive definitions, or if you want to load a data source everywhere, not just on the engines.

%%px --local
import os
thispid = os.getpid()
print(thispid)

83088
[stdout:0] 
83112
[stdout:1] 
83113
[stdout:2] 
83114
[stdout:3] 
83115