Skip to main content

Big Data & Hadoop

Since everything moves fast in the IT world, you have new terminologies entering their 3rd or 4th generation by the time you get a chance to get your hands dirty with them. Big Data has been one of them, an alluring technology allowing massive distributed power over large datasets using the famous map-reduce algorithm. Apache Hadoop allows scaling to massive proportions and has been in use with tech giants like Google and Facebook.

 I decided to start running a Hadoop cluster myself using the following guide as a started.

 https://github.com/GoogleCloudPlatform/solutions-google-compute-engine-cluster-for-hadoop/blob/master/README.md

This version installs Hadoop locally but uses the Google App Engine and Google Cloud Storage and allows basic scaling/clustering. I started running the pre-requisites on a VM Centos 6.4 and things were going ok. Then I realized that I needed to go deeper into Hadoop and maybe run a sample locally, without achieving the Cloud version first.  Then I went to the following:

http://tecadmin.net/steps-to-install-hadoop-on-centosrhel-6/#

It had simple enough steps to get it installed. Now I am reading Hadoop in Action by Alex Holmes.


Comments

Post a Comment

Popular posts from this blog

Employment Based Green Card Marathon

There are 3 most popular ways to get a Green Card to live and work in US: Through Marriage with a US Citizen, Employment Sponsorship and the Diversity Lottery. I would like to articulate on the unfairness of the process for the Employment based applicants and its repercussions. After getting a scholarship to study at Brandeis University, I arrived in the United States on August 28, 1998 on an F-1 Student Visa. After graduating with my Master's degree, I had 1 year of OPT - Optional Practical Training which allowed me to work for companies that were in fields similar to my concentration. Next chapter in my immigration story is the H1-B Work Visa which is frowned upon. This visa provides an entry point for skilled immigrants and it is one of the very few points of entry to the US based on merit. Scorn on this quota of about 85k is well deserved on an emotional level, especially when considering high unemployment of today's workplace. Yet, working in IT and being involved with...
3 comments
Read more
Today marks my 14th year in the USA. In 4 short years, I will have lived in the US more than Turkey (18 years), where I was born. Here is a brief summary: F1 Student Visa (Summer 1998 - Bachelors), OPT - Optional Practical Training (Summer 2002), F1 Student Visa (2003 - Masters), OPT (Summer 2003), H1-B non-Immigrant Visa (Late 2003), H1-B renewal (2004 August - Job Change), PERM application for Green Card (2005), I-140 (Oct 21, 2005), H1-B Renewal (2006), H1-B Renewal (new job), PERM application for Green Card (2008 - new job), I-140 (2009 new job), H1-B renewal (2011), I-485 (2011 Aug - Adjustment of Status), I-765 (2011, Application of Employment Authorization), I-131 (2011, Application of Travel Document), I-765 renewal (2012 June), I-131 renewal (2012 June), Service Request for I-485 (2012 June), Service Request for Service Request for I-485 (2012-August, not kidding), Service Request for I-765 and I-131 renewals (2012 August) ... still an alien...
Post a Comment
Read more

Desktop PC

Yes, I admit it, I am a geek, nerd (whatever you call it) and I do have a Desktop PC (and I am proud). I've been keeping its shell ( antec 180 ) the same for the last 10 years or so, but the interior has been changing ever since from P2 days up to Socket 1366. One of the things that remain the same is my old HDDs. I recently (and finally) decided to build a RAID. Reading about it, RAID 5 seemed to be a good idea, where you get x-1 hard disk worth of space. You have a decent tolerance for faults, unless 2 HDDs crash at the same time (which is unlikely). So I had a collection of HDDs separately living in the case. I designated 4 x 1.5 TB disks to become the RAID (called it RAIDer - cheezy I know). I also had another 1.5 TB and 0.5 TB as spare. I did a hardcore cleaning operation and filled the spares with all the data. Research lead me to using physical RAID cards for better performance but I used Intel's Storage Matrix solution. RAID formation reduced the 6 TB diskspace to 4.09...
Post a Comment
Read more