Skip to main content

Python MapReduce framework

Project description

https://github.com/Yelp/mrjob/raw/master/docs/logos/logo_medium.png

mrjob is a Python 2.7/3.4+ package that helps you write and run Hadoop Streaming jobs.

Stable version (v0.7.4) documentation

Development version documentation

https://travis-ci.org/Yelp/mrjob.png

mrjob fully supports Amazon’s Elastic MapReduce (EMR) service, which allows you to buy time on a Hadoop cluster on an hourly basis. mrjob has basic support for Google Cloud Dataproc (Dataproc) which allows you to buy time on a Hadoop cluster on a minute-by-minute basis. It also works with your own Hadoop cluster.

Some important features:

  • Run jobs on EMR, Google Cloud Dataproc, your own Hadoop cluster, or locally (for testing).

  • Write multi-step jobs (one map-reduce step feeds into the next)

  • Easily launch Spark jobs on EMR or your own Hadoop cluster

  • Duplicate your production environment inside Hadoop

    • Upload your source tree and put it in your job’s $PYTHONPATH

    • Run make and other setup scripts

    • Set environment variables (e.g. $TZ)

    • Easily install python packages from tarballs (EMR only)

    • Setup handled transparently by mrjob.conf config file

  • Automatically interpret error logs

  • SSH tunnel to hadoop job tracker (EMR only)

  • Minimal setup

    • To run on EMR, set $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY

    • To run on Dataproc, set $GOOGLE_APPLICATION_CREDENTIALS

    • No setup needed to use mrjob on your own Hadoop cluster

Installation

pip install mrjob

As of v0.7.0, Amazon Web Services and Google Cloud Services are optional depedencies. To use these, install with the aws and google targets, respectively. For example:

pip install mrjob[aws]

A Simple Map Reduce Job

Code for this example and more live in mrjob/examples.

"""The classic MapReduce job: count the frequency of words.
"""
from mrjob.job import MRJob
import re

WORD_RE = re.compile(r"[\w']+")


class MRWordFreqCount(MRJob):

    def mapper(self, _, line):
        for word in WORD_RE.findall(line):
            yield (word.lower(), 1)

    def combiner(self, word, counts):
        yield (word, sum(counts))

    def reducer(self, word, counts):
        yield (word, sum(counts))


if __name__ == '__main__':
     MRWordFreqCount.run()

Try It Out!

# locally
python mrjob/examples/mr_word_freq_count.py README.rst > counts
# on EMR
python mrjob/examples/mr_word_freq_count.py README.rst -r emr > counts
# on Dataproc
python mrjob/examples/mr_word_freq_count.py README.rst -r dataproc > counts
# on your Hadoop cluster
python mrjob/examples/mr_word_freq_count.py README.rst -r hadoop > counts

Setting up EMR on Amazon

Setting up Dataproc on Google

Advanced Configuration

To run in other AWS regions, upload your source tree, run make, and use other advanced mrjob features, you’ll need to set up mrjob.conf. mrjob looks for its conf file in:

  • The contents of $MRJOB_CONF

  • ~/.mrjob.conf

  • /etc/mrjob.conf

See the mrjob.conf documentation for more information.

Reference

More Information

Thanks to Greg Killion (ROMEO ECHO_DELTA) for the logo.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mrjob-0.7.4.tar.gz (652.4 kB view details)

Uploaded Source

Built Distribution

mrjob-0.7.4-py2.py3-none-any.whl (439.6 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file mrjob-0.7.4.tar.gz.

File metadata

  • Download URL: mrjob-0.7.4.tar.gz
  • Upload date:
  • Size: 652.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.1.1.post20200604 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.7.7

File hashes

Hashes for mrjob-0.7.4.tar.gz
Algorithm Hash digest
SHA256 2c8d8dc8aa4b354a97de18d0260f551f018693af74af104b3d41daf165eebdd4
MD5 690655e16432de9cbadbb4fc3afd6b87
BLAKE2b-256 2ced207853d1ebc6b549551d12db35e471289d26cac2cdae363419357294d3c5

See more details on using hashes here.

Provenance

File details

Details for the file mrjob-0.7.4-py2.py3-none-any.whl.

File metadata

  • Download URL: mrjob-0.7.4-py2.py3-none-any.whl
  • Upload date:
  • Size: 439.6 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.1.1.post20200604 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.7.7

File hashes

Hashes for mrjob-0.7.4-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 d8fa1bafcada0ffe3e7166896a27e996815e2cb835088aec025e3dd12c7146ce
MD5 1b5064f0faff0a0f5769c1a3be218299
BLAKE2b-256 8e58fc28ab743aba16e90736ad4e29694bd2adaf7b879376ff149306d50c4e90

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page