How to share data like lists and dictionaries between Processes in Parallel Processing in Python

Source: Pythonforthelab

Introduction: In Python, the multiprocessing module allows us to create and manage multiple processes to achieve parallel execution. When working with multiple processes, you may encounter scenarios where you need to share data between these processes efficiently. In this blog post, we will explore different approaches to sharing data, such as lists and dictionaries, between processes in Python.

Before we dive into the code, let’s quickly understand the two main methods of inter-process communication (IPC) supported by the multiprocessing module: shared memory and message passing.

1. Shared Memory: Shared memory allows processes to access the same region of memory, enabling them to share data without the need for serialization and deserialization. Python’s multiprocessing module provides a useful class called Value for sharing a single value, and Array for sharing sequences like lists and arrays.
2. Message Passing: Message passing involves passing data between processes using message queues. Python’s multiprocessing module provides a class called Queue that allows processes to enqueue and dequeue messages.

Now, let’s see how to implement these approaches in Python code:

Using Shared Memory:

from multiprocessing import Process, Value, Array

def modify_shared_data(shared_list, shared_dict, shared_value):
    # Access and modify the shared list
    shared_list[0] = 42
    # Access and modify the shared dictionary
    shared_dict['key'] = 'value'
    # Access and modify the shared value
    shared_value.value = 100
if __name__ == '__main__':
    # Create shared list and dictionary
    shared_list = Array('i', [1, 2, 3])
    shared_dict = Array('i', {'key': 'initial_value'})
    # Create shared value
    shared_value = Value('i', 0)
    # Create a process and pass the shared data
    process = Process(target=modify_shared_data, args=(shared_list, shared_dict, shared_value))
    # Start the process
    process.start()
    # Wait for the process to finish
    process.join()
    # Print the modified shared data
    print(shared_list[:])  # Output: [42, 2, 3]
    print(shared_dict[:])  # Output: {'key': 'value'}
    print(shared_value.value)  # Output: 100

Using Message Passing:

from multiprocessing import Process, Queue

def modify_data(queue):
    # Receive the data from the queue
    data = queue.get()
    # Modify the data
    data[0] = 42
    data['key'] = 'value'
    # Put the modified data back into the queue
    queue.put(data)
if __name__ == '__main__':
    # Create a queue
    queue = Queue()
    # Create the data to be shared
    data = [1, 2, 3]
    # Put the data into the queue
    queue.put(data)
    # Create a process and pass the queue
    process = Process(target=modify_data, args=(queue,))
    # Start the process
    process.start()
    # Wait for the process to finish
    process.join()
    # Get the modified data from the queue
    modified_data = queue.get()
    # Print the modified data
    print(modified_data)  # Output: [42, 2, 3, 'key': 'value']

Source: Tenor

Conclusion: Sharing data between processes is a common requirement when working with multiprocessing in Python. In this blog post, we explored two approaches: shared memory and message passing. We learned how to share data like lists and dictionaries between processes using these techniques. By leveraging shared memory or message passing, you can efficiently exchange data and synchronize operations across multiple processes in your Python applications.