Python MapReduce framework

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for davidmarin from gravatar.com davidmarin Avatar for manpreet1 from gravatar.com manpreet1 Avatar for yelplabs from gravatar.com yelplabs

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache)
  • Author: David Marin
Classifiers

Project description

https://github.com/Yelp/mrjob/raw/master/docs/logos/logo_medium.png

mrjob is a Python 2.7/3.3+ package that helps you write and run Hadoop Streaming jobs.

Stable version (v0.6.1) documentation

Development version documentation

https://travis-ci.org/Yelp/mrjob.png

mrjob fully supports Amazon’s Elastic MapReduce (EMR) service, which allows you to buy time on a Hadoop cluster on an hourly basis. mrjob has basic support for Google Cloud Dataproc (Dataproc) which allows you to buy time on a Hadoop cluster on a minute-by-minute basis. It also works with your own Hadoop cluster.

Some important features:

  • Run jobs on EMR, Google Cloud Dataproc, your own Hadoop cluster, or locally (for testing).

  • Write multi-step jobs (one map-reduce step feeds into the next)

  • Easily launch Spark jobs on EMR or your own Hadoop cluster

  • Duplicate your production environment inside Hadoop

    • Upload your source tree and put it in your job’s $PYTHONPATH

    • Run make and other setup scripts

    • Set environment variables (e.g. $TZ)

    • Easily install python packages from tarballs (EMR only)

    • Setup handled transparently by mrjob.conf config file

  • Automatically interpret error logs

  • SSH tunnel to hadoop job tracker (EMR only)

  • Minimal setup

    • To run on EMR, set $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY

    • To run on Dataproc, set up your Google account and credentials (see Dataproc Quickstart).

    • To run on your Hadoop cluster, just make sure $HADOOP_HOME is set.

Installation

From PyPI:

pip install mrjob

From source:

python setup.py install

A Simple Map Reduce Job

Code for this example and more live in mrjob/examples.

"""The classic MapReduce job: count the frequency of words.
"""
from mrjob.job import MRJob
import re

WORD_RE = re.compile(r"[\w']+")


class MRWordFreqCount(MRJob):

    def mapper(self, _, line):
        for word in WORD_RE.findall(line):
            yield (word.lower(), 1)

    def combiner(self, word, counts):
        yield (word, sum(counts))

    def reducer(self, word, counts):
        yield (word, sum(counts))


if __name__ == '__main__':
     MRWordFreqCount.run()

Try It Out!

# locally
python mrjob/examples/mr_word_freq_count.py README.rst > counts
# on EMR
python mrjob/examples/mr_word_freq_count.py README.rst -r emr > counts
# on Dataproc
python mrjob/examples/mr_word_freq_count.py README.rst -r dataproc > counts
# on your Hadoop cluster
python mrjob/examples/mr_word_freq_count.py README.rst -r hadoop > counts

Setting up EMR on Amazon

Setting up Dataproc on Google

Advanced Configuration

To run in other AWS regions, upload your source tree, run make, and use other advanced mrjob features, you’ll need to set up mrjob.conf. mrjob looks for its conf file in:

  • The contents of $MRJOB_CONF

  • ~/.mrjob.conf

  • /etc/mrjob.conf

See the mrjob.conf documentation for more information.

Reference

More Information

Thanks to Greg Killion (ROMEO ECHO_DELTA) for the logo.

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for davidmarin from gravatar.com davidmarin Avatar for manpreet1 from gravatar.com manpreet1 Avatar for yelplabs from gravatar.com yelplabs

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache)
  • Author: David Marin
Classifiers

Release history Release notifications | RSS feed

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.12

0.6.11

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

This version

0.6.1

0.6.0

0.5.12

0.5.11

0.5.10

0.5.9

0.5.8

0.5.7

0.5.6

0.5.5

0.5.4

0.5.3

0.5.2

0.5.1

0.5.0

0.4.6

0.4.5

0.4.4

0.4.3

0.4.2

0.4.1

0.4

0.4.0

0.4-RC1 pre-release

0.4-dev pre-release

0.3.5

0.3.4.1

0.3.3.2

0.3.3.1

0.3.3

0.3.2

0.3.1

0.3.0

0.2.8

0.2.7

0.2.6

0.2.5

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

0.1.0

0.1.0-pre3 pre-release

0.1.0-pre2 pre-release

0.1.0-pre1 pre-release

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mrjob-0.6.1.tar.gz (438.1 kB view details)

Uploaded Source

Built Distribution

mrjob-0.6.1-py2.py3-none-any.whl (291.9 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file mrjob-0.6.1.tar.gz.

File metadata

  • Download URL: mrjob-0.6.1.tar.gz
  • Upload date:
  • Size: 438.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for mrjob-0.6.1.tar.gz
Algorithm Hash digest
SHA256 a1fd13528be1290b7ba2e13e1f26882a77170b4eaea2e58a945d766c0355df87
MD5 a458e1b917d85187e23bcbe3bc04bffc
BLAKE2b-256 5138ceecc94ac2d89b9677f75e5fd18bbedad0fdefd7d4720ef7bac2e4c23628

See more details on using hashes here.

File details

Details for the file mrjob-0.6.1-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for mrjob-0.6.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 308428d161efe852b02f948dab0b9349f6c648e8bfe8f7c6b534eb3b98b1c053
MD5 e36ce9b3c0192092f286edf72bca6511
BLAKE2b-256 0afc9bcb307b288817140e7b9bdc7953c2c09bd1e947f603cf9e23ea34ffa0b6

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page