Programmer's Python: Async

Programmer's Python: Async - Shared Memory

Written by Mike James

Tuesday, 28 October 2025

Article Index
Programmer's Python: Async - Shared Memory
Locking
Raw Shared Memory
Shareable List

Page 4 of 4

As well as the raw shared memory class, there is also the ShareableList which is elaborated to allow you to treat the buffer as if it was a Python list, but with some restrictions. A ShareableList isn’t dynamic and it cannot change its size via slicing or appending. It is also limited to storing basic data types: int float, bool, string and bytes and each smaller than 10Mbytes.

This is just an implementation of the shared ctypes structure, but one that doesn’t need you to explicitly specify the format.

To create a ShareableList use:

multiprocessing.shared_memory.ShareableList(
                       sequence = None, *, name = None)

This creates a new shared memory block with the format of the specified sequence or, if sequence is None, attaches to the existing block with the specified name.

There are a number of useful attributes:

count(value) - returns the number of occurrences of value
index(value) - returns the index of value in the list and a ValueError exception is raised if it isn’t found
Format – The struct packing format that has be deduced from the initial list
Shm – The SharedMemory instance being used to store the list

As a simple example, the previous counter can be implemented without the worry of having to convert a bignum to a byte sequence:

import time
import multiprocessing
import multiprocessing.shared_memory

def myUpdate(mySharedList,lock):
    for i in range(1000): 
        time.sleep(0.005) 
        with lock:
            mySharedList[0]=mySharedList[0]+1
    mySharedList.shm.close()

if __name__ == '__main__':
    mylock=multiprocessing.Lock()
    mySharedList=multiprocessing.shared_memory.
        ShareableList(sequence=[1])
    p1=multiprocessing.Process(target=myUpdate,
                           args=(mySharedList,mylock))
    p2=multiprocessing.Process(target=myUpdate,
                           args=(mySharedList,mylock))  
    p2.start()
    p1.start()
    p1.join()
    p2.join()
    print(mySharedList[0])
    mySharedList.shm.close()
    mySharedList.shm.unlink()

In practice, you would generally want to use multiple elements of the list and they might well be of mixed type, but this serves to demonstrate the principle. Notice that we have to use the shm attribute to access some of the shared memory attributes.

Shared Memory Manager

Finally there is a shared memory manager which is only worth exploring if you plan to use lots of different shared memory blocks:

multiprocessing.managers.SharedMemoryManager(address,
                                               authkey)

where address and authkey are optional. This creates a SharedMemoryManager instance. If you call its start method a new child process is created which oversees the creation and destruction of all memory blocks – given that this involves an additional process you can appreciate why you would use it only if really needed. To terminate all of the shared memory blocks owned by the manager you would use its shutdown() method which calls the unlink method on all of the blocks.

The two methods SharedMemory(size) and ShareableList(sequence) create a SharedMemory and ShareableList object under the manager’s control.

One advantage of using SharedMemoryManager is that it can be used in a context manager to automatically start the server and dispose of all the shared memory blocks. For example, we can change the main program of the previous example to use a manager very easily:

if __name__ == '__main__':
    mylock=multiprocessing.Lock()
    with multiprocessing.managers.
                        SharedMemoryManager() as smm:
        mySharedList=smm.ShareableList([1])    
        p1=multiprocessing.Process(target=myUpdate,
                             args=(mySharedList,mylock))
        p2=multiprocessing.Process(target=myUpdate,
                             args=(mySharedList,mylock))  
        p2.start()
        p1.start()
        p1.join()
        p2.join()
        print(mySharedList[0])

Even though SharedMemoryManager is derived from BaseManager it doesn’t have any facilities for being used as a remote server as it doesn’t create proxies, see Chapter 9 for further explanation.

In chapter but not in this extract

Computing Pi

Summary

Processes don’t share data as a matter of course because they don’t share memory. Each process runs in its own allocated memory.
The multiprocessing Queue is the highest level way of sharing data. It is a FIFO stack and is very easy to use because it is buffered. Any number of processes can add or remove data from the queue.
A standard Queue data structure can be used to share data between threads.
The Pipe is a lower level implementation of sharing between processes and is a Python version of a facility provided by the operating system.
A Pipe is bi-directional and you can read and write at each end of the pipe using connection objects. Data written to one end of the pipe can be read from the other end.
Any number of processes can have access to either end of the Pipe.
An equally fundamental way of sharing data is shared memory and this is supported in hardware and by the operating system.
You can share a block of memory and use it to exchange data between any number of processes.
Python lets you use the ctypes module to transfer data using shared memory. You have to convert Python data structures to C data structures to transfer via shared memory.
There is also a more basic way to access shared memory in its raw form. The only problem is that the data takes the form of a byte sequence and so you have to perform the conversion between Python data and byte data manually.
A slightly easier to use, but still restricted, form of raw shared memory is ShareableList which lets you treat the buffers an array of limited types – int, float, string and byte.