Python: shared memory: Sharing state between processes

What is shared memory?

Shared memory is a fastest inter process communication mechanism. Memory is shared by n number of processes. All process can able to write, read from shared memory without calling operating system functions.

Suppose process P1 wants to send an image image.png to process P2.

Without shared memory the flow looks like below.

1.   Process P1 reads the file image.png into memory, pass it to the network functions, that copy that memory to the OS's internal memory.

2.   Process P2 uses the network functions to copy the data from OS’s internal memory to P2’s local memory.

As you observe there are two copies maintained, one from memory to OS’s internal memory and other from OS’s internal memory to P2’s local memory.

By using shared memory the flow looks like below

1.   Process P1 gets exclusive access to shared memory and copies the file image.png to shared memory.

2.   Process P2 waits until P1 releases the exclusive access and reads image.png from shared memory

By using multiprocessing.Value, multiprocessing.Array we can store data in shared memory. You must enforce proper synchronization while working with shared memory. Updates to a Value or Array object will be immediately visible to other processes with access to that object.

multiprocessing.Value(typecode_or_type, *args, lock=True)

Returns an object allocated from shared memory. By default all the operations on this value are synchronized.

typecode_or_type: Determines the type of object returned. For example 'd' indicates a double precision float and 'i' indicates a signed integer.

lock: If lock is true, then all the operations on this object are synchronized internaly, else not.

from multiprocessing import Process, Value

def increment_value(val):
    val.value += 10

if __name__=="__main__":
    num = Value('i', 0)

    p = Process(target=increment_value, args=(num,))
    p.start()
    p.join()

    print("Value is ", num.value)

Output

Value is  10

multiprocessing.Array(typecode_or_type, size_or_initializer, *, lock=True)

Returns an array allocated from shared memory. By default the return value is actually a synchronized wrapper for the array.

typecode_or_type: Determines the type of the array returned. For example 'd' indicates a double precision float and 'i' indicates a signed integer.

lock: If lock is true, then all the operations on this object are synchronized internally, else not.

from multiprocessing import Process, Array

def increment_value(arr):
    for i in range(len(arr)):
        arr[i] = arr[i] + 1

if __name__=="__main__":
    arr = Array('i', [2, 3, 5, 7])

    p = Process(target=increment_value, args=(arr,))
    p.start()
    p.join()

    for i in arr:
        print(i)

Output

3

4

6

8

Previous                                                 Next                                                 Home