How to Combine Map And Reduce in Python

1. CUSTOM MAPPING FUNCTION

  • As noted in How to Use Map Function in Python, the specific function doing the mapping contains as many arguments as there are collections passed into the common map function
  • It follows, that the custom mapping function below will allow only for a single collection to be passed into map()
  • The exemplary function counts words in a document that is passed as a string and returns dict of frequencies
RUBBERDUCK

2. COMMON MAP

  • now, we’ll map the custom count_words() function over a list of documents using the generic map() function
  • map is lazy and returns a single-use map object
  • let’s evaluate the map object and have it pretty printed — what we are getting is a list of dict
OUTPUT
  • As you can see, the result is a sequence of dictionaries with word frequencies for the individual documents

3. CUSTOM REDUCING FUNCTION

  • As written in How To Use Reduce Function In Python, a custom reducing function by definition needs 2 arguments and the custom reducing function is combining them into a single “product”
  • The custom reducing function for our use case combines the frequencies of 2 documents
RUBBERDUCK

4. COMMON REDUCE CONTAINING COMMON MAP: MAP-REDUCE

  • The final step is the combination of all of the components which can look like this
FINAL OUTPUT

--

--

Infrastructure Support Engineer/Technical Writer (Snowplow Analytics) with a passion for Python/writing documentation. More about me: https://pavol.kutaj.com

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Pavol Kutaj

Pavol Kutaj

Infrastructure Support Engineer/Technical Writer (Snowplow Analytics) with a passion for Python/writing documentation. More about me: https://pavol.kutaj.com