I have been looking up the information available on apache's site. I believe some of the information shall prove very useful with the setting up of the project, and it also helps me with the required tech document.
Firstly a server needs to be setup.
Since Hadoop is not properly tested windows is not suported as a prodiction platform - so the server shall run Linux as this doubles as a development and production platform.
Secondly, the server needs to have the appropriate software installed:
Java 1.6+
ssh
latest stable hadoop release.
I need to find out, or decide if I am going to set up a server to run in 'pseudo-distributed' mode, (one server that is programmed to split the data up and behave as if it were a number of seperate servers in a cluster) or if I want to assemble or find a way to operate a network cluster.
There are installation instructions on the apache website. Once the server is set up, map reduce needs to be installed. The commands for this are written in Java.
Showing posts with label Map reduce. Show all posts
Showing posts with label Map reduce. Show all posts
Wednesday, October 5, 2011
Introductory Post.
Welcome to my honours project blog!
I am in 4th year at Dundee university studying Applied Computing. As part of the honours year I am required to participate in an honours project. This blog shall be the documentation/journal of the development of said project.
So, what is the project?
To summarise my understanding of it at this point in time - it is to use Hadoop to perform map-reduce functions on a dimensional database. We do not know how well this is going to work, but my task is to setup the database and use the hadoop functionality to try to analyse and extract data from the database.
My first tasks are:
I am in 4th year at Dundee university studying Applied Computing. As part of the honours year I am required to participate in an honours project. This blog shall be the documentation/journal of the development of said project.
So, what is the project?
To summarise my understanding of it at this point in time - it is to use Hadoop to perform map-reduce functions on a dimensional database. We do not know how well this is going to work, but my task is to setup the database and use the hadoop functionality to try to analyse and extract data from the database.
My first tasks are:
- Start this blog.
- Write an overview of the project (Presumably this will gauge my understanding of the project so I can be steered on course.)
- Perform a risk analysis of the project.
- Slap together a Gantt chart. (Will involve a proper breakdown and understanding of whats involved, but this will be a rough overview of the primary tasks and estimated time scale initially)
More later.
Subscribe to:
Posts (Atom)