
n = number of MPI processes (= number of BGL "nodes")
m = number of live blocks per process
k = number of dead blocks per process (unused spaces)
r = number of times to repeat


Each block will consist of 16K bytes worth of doubles.

Initialization:

The block in position (i,j) (i.e., the j-th block in the process of
rank i) will be filled with the double mi+j.  Rationale: this is just
to give each block a unique value in case we want to check the blocks
are moving around correctly.

Map:

(i,j) |-> ((mi+j)%n, (mi+j)/n)

Example: n=3, m=4

Map:

 | 0  1  2  3
-------------
0|00 10 20 01
1|11 21 02 12
2|22 03 13 23

Rationale for this map: this is a simple map but really moves things
around quite a bit.

For BGL I propose trying this with something like

m = 20000
k =  5000
n = 32, 64, 128, 256, 512, 1024.

The memory used by each process (for block memory) is then (m+k)*16KB
= 400MB which I hope is enough less than 512MB that there should be
enough memory for the kernel and application as well as this data.

Do this for each bred, and see what happens.

What do you think?

Question: should we repeat this redistribution, or is once enough?  If
we repeat, how many times?

-S

The initial value map is

(i,j) |-> i*m + j

The inverse map is

(x,y) |-> ((yn+x)/m, (yn+x)%m))

The initial value of this is

      |-> yn+x
