How to share data like lists and dictionaries between Processes in Parallel Processing in Python

Aditya Mangal
3 min readJun 28, 2023

--

Source: Pythonforthelab

Introduction: In Python, the multiprocessing module allows us to create and manage multiple processes to achieve parallel execution. When working with multiple processes, you may encounter scenarios where you need to share data between these processes efficiently. In this blog post, we will explore different approaches to sharing data, such as lists and dictionaries, between processes in Python.

Before we dive into the code, let’s quickly understand the two main methods of inter-process communication (IPC) supported by the multiprocessing module: shared memory and message passing.

  1. Shared Memory: Shared memory allows processes to access the same region of memory, enabling them to share data without the need for serialization and deserialization. Python’s multiprocessing module provides a useful class called Value for sharing a single value, and Array for sharing sequences like lists and arrays.
  2. Message Passing: Message passing involves passing data between processes using message queues. Python’s multiprocessing module provides a class called Queue that allows processes to enqueue and dequeue messages.

Now, let’s see how to implement these approaches in Python code:

Using Shared Memory:

from multiprocessing import Process, Value, Array

def modify_shared_data(shared_list, shared_dict, shared_value):
# Access and modify the shared list
shared_list[0] = 42
# Access and modify the shared dictionary
shared_dict['key'] = 'value'
# Access and modify the shared value
shared_value.value = 100
if __name__ == '__main__':
# Create shared list and dictionary
shared_list = Array('i', [1, 2, 3])
shared_dict = Array('i', {'key': 'initial_value'})
# Create shared value
shared_value = Value('i', 0)
# Create a process and pass the shared data
process = Process(target=modify_shared_data, args=(shared_list, shared_dict, shared_value))
# Start the process
process.start()
# Wait for the process to finish
process.join()
# Print the modified shared data
print(shared_list[:]) # Output: [42, 2, 3]
print(shared_dict[:]) # Output: {'key': 'value'}
print(shared_value.value) # Output: 100

Using Message Passing:

from multiprocessing import Process, Queue

def modify_data(queue):
# Receive the data from the queue
data = queue.get()
# Modify the data
data[0] = 42
data['key'] = 'value'
# Put the modified data back into the queue
queue.put(data)
if __name__ == '__main__':
# Create a queue
queue = Queue()
# Create the data to be shared
data = [1, 2, 3]
# Put the data into the queue
queue.put(data)
# Create a process and pass the queue
process = Process(target=modify_data, args=(queue,))
# Start the process
process.start()
# Wait for the process to finish
process.join()
# Get the modified data from the queue
modified_data = queue.get()
# Print the modified data
print(modified_data) # Output: [42, 2, 3, 'key': 'value']
Source: Tenor

Conclusion: Sharing data between processes is a common requirement when working with multiprocessing in Python. In this blog post, we explored two approaches: shared memory and message passing. We learned how to share data like lists and dictionaries between processes using these techniques. By leveraging shared memory or message passing, you can efficiently exchange data and synchronize operations across multiple processes in your Python applications.

--

--

Aditya Mangal

My Personal Quote to overcome problems and remove dependencies - "It's not the car, it's the driver who win the race".