Hadoop Project

On the hadoop cluster located at [login to view URL], in HDFS, is a directory called /share/spoilers. This directory contains 25000 text files containing "spoiler logs" for a randomized version of The Legend of Zelda: A Link to the Past. Yes, I am a huge nerd. Each of these spoiler logs is stored in JSON format. The sections in each file labeled "Light World", "Eastern Palace", "Desert Palace", "Death Mountain", "Tower Of Hera", "Castle Tower", "Dark World", "Dark Palace", "Swamp Palace", "Skull Woods", "Thieves Town", "Ice Palace", "Misery Mire", "Turtle Rock", and "Ganons Tower" contain information about which locations contain which items. For example, in the "Light World" section, you might see: "Graveyard Ledge:12948": "ProgressiveSword:12948", This means that the location called "Graveyard Ledge" has the item "Progressive Sword". (Note, the numbers after the colon mean precisely nothing). Note: some items, like "Bombos," might appear in other sections of the file. If the item is not in the listed sections above, then it is not an item location, and should not be considered. Your task for this project is to create a MapReduce job (either through vanilla MapReduce, or hadoop streaming with python) that will take the name of an item (such as "Bow" or "Lamp") and return the top 10 locations in which that item appears throughout all the spoiler files, along with the number of times it appears in each of those 10 locations. The item name should be part of the arguments passed to the job. The files are in JSON format, so you will either need to get a way to read JSON in a MapReduce job, or simply read the relevant lines from each file (they will be the same in every file). In order to cut down on reduce time, you could even only worry about the lines in which the requested item appears.