Dryrun in Pig for debugging purposes

It is hard to debug pig scripts as the debug mechanism is not friendly. To see the parameters that we are passing via command line or by executing another script (bash,perl) - we can use dryrun option. but, I always forget whether it is dry-run or dryrun. It is dryrun and the way to use it is

bin/pig -dryrun /scriptName

awesome

I just saw this new facility in blogger where I can write in any language

నేను తెలుగు లో రాయగలను , ಕನ್ನಡ ಗೊತ್ತಿಲ್ಲ . हिन्दी हमारा जातीय भाषा हैं। :)

Writing complex code

Ok. I came across one such complex code -

Requirement:
Given an output log from a process which contains temporal information. Based on this temporal information, data need to be split and kept it in temporal directory structure. So, essentially, you have the data which contains when each line is generated and the data needs to be split and kept in directory structure like year/month/day/hour/filename.1 for 15 mins filename.2 for 30 mins worth of data and so on.

here is complicated version:

Have three threads - one is main thread , second thread creates a file for every 15 mins in above structure and third closes files created by previous thread. And the main thread reads each line from input and call write method. the stream handler will be modified by second thread for every 15 mins to point to new file. and will move the previous handler to queue and that is taken by third thread and closes it!!!!

here is simpler version:

for each inputline
check out the date
if it is with in minute domain write to same file
else close the previous file if any opened and create a new one
write inputline to output

close finally if there is any opened stream handler

Getting lazy day by day-No more

After my parents had come to Bangalore, I am more relaxed and getting lazy. My mom thinks that I am working too hard and so she never let me cook anything when she is here. :) Yesterday my mom went to my brother's place and it became my brother's duty to get food for us. :) I am sure that I am getting lazy and It might take time to get adjusted again once they go back to my hometown. :(

Huh!!I don't know at what time I started writing this blog. I wrote the above lines on 27th. And just after that minute, I got lot of work from here and there and finally, there is no free time. But this time, it is office work!!

Two Birthdays in a year

Can it be possible for any one to have two birthdays a year. ok. Don't think too complex. It is possible. I have two. One is my actual birthday which is in January and one is official birthday which is today. :) And I always forget and I get mail from IITK Alumni wishing me then I realize "Oh. Today is my birthday. :)"

There is a long story behind this. In short, my DOB changed twice in my school once for writing some Govt entrance exam in 4th Std and once for writing 10th class exams. I am elder to myself if I compare my official birthday with my actual birthday.

Silly interviews

Some of older memories:

During Microsoft interview:

Interviewer: which operating system you like most
Me : Linux
:))

During Yahoo interview:

Interviewer: What is your dream company - this was not part of interview..this was some casual chat I had with a manager
Me: Google!! :)) :))

Define data mining expert

Studying data mining and Machine learning as one of my course in IIT, when some one says they are dataminig experts- I assume them that they are very good in coming up with trends using some of machine learning techniques. They know what Association Rule Mining, Clustering, Classification mean.

But, in some places where they are not exposed to above, they assume data mining expert mean doing ETL operations and database partitioning/sharding etc..etc..!!!
I feel like shouting at them that this is not what it meant!

installing mplayer in fedora 8

yum search mplayer didn't give me mplayer package to install.
Thanks to Vaiju for his help. and here are the instructions:

1. rpm -Uvh http://www.fedorafaq.org/f8/yum http://rpm.livna.org/livna-release-8.rpm
if it asks that yum-priorities are missing install yum-priorities by command
yum -y install yum-priorities
2. yum search mplayer gives you mplayer package

install it. :)

Source:http://www.fedorafaq.org/f8/#yumconf

Distributed copy from remote hdfs to local hdfs

The command DistCp works for copying from one hdfs cluster to other hdfs cluster. But, the problem is that it is not specified any where(As I aware of) what should be the source url and destination url.
Many places, it is specified that specify hdfs://namenode1:50070/path_to_file and many other places some other port number.
So, after a lot of debugging, I figured out that it should be
hadoop distcp hdfs://(fs.default.name)_property_specified_in_hadoop-site.xml/path_to_file

for ex:
hadoop distcp hdfs://remotehost:10000/opt/hadoop-name/foo/bar hdfs://localhost:54310/opt/hadoop-name/foo/bar

The important note here is the url should be picked up exactly how it is in fs.default.name in hadoop-site.xml in hadoop conf directory.

Way to get hdfc standard life insurance premium receipt

Important Step: Never ever call the customer care (toll free:1800 209 7777). They give all weird reasons every time.
1. According to their policies, they can't send it to a mail id but can Fax it!!!
2. The system is going for upgradation (day and night for two days!!) so need to call after few hours and when you call go to step(2)
3. send a mail to service@hdfcinsurance.com and they will send a soft copy to u with in 3 days!!!(for sending a soft copy, it takes 3 days!!!!) But, in realty, it never happens.

Working Step:
Call up ur local branch - Again, it may redirect you to some other customer service -so, keep on pressing 9 till u hear a voice from other side (not the automated one :) )
then, tell them that you need a premium receipt.
Also, send a mail to oobangalore@hdfcinsurance.com requesting a premium receipt.

It worked for me. So, if you face similar issues, follow the procedure.

Files starting with -

If you have problem in using files starting with - in command prompt, then the solution is to use ./ in front of it.
Courtesy is this

Converting EOLs in dos format to unix

I had a file which contains dos style EOLs in it which I couldn't see except in Vim. Being long time, I forgot how to convert them to unix style EOLs and figured out only after searching in Google. :( My memory power is decreasing day by day I can say.
any ways, the command to use is

dos2unix filename

That's it.

Quote-1

After every sad day
comes a glad day

Question is: How will you define which is sad and which is glad. :D

want to be a kid forever

How good is life of a kid.no need to bother about what some one say. What ever we do as a kid, no one gets angry as they think that "leave it. he/she is just a kid".You are free to do what ever you want to do. No restrictions. no tensions.no responsibilities.

Enthusiasm is degrading poorly

I was browsing through my older posts starting from the blog that I first wrote. And, I could see a lot of degradation in all ways-
The fresh ideas I had just after coming out from school.
The frequency of writing
The kind of topics I chose during those days

After all, old is getting old day by day.And you can see younger kids are smarter than old.

Moral of the story: Time is precious. Don't waste it.

Finally,

Finally, smoking is going to be banned in public places from tomorrow.Source is this

But, I am not sure how many people actually follows it. :(

Life is boring
Life is exciting
Life is tough

Is it possible to feel all at the same time? I am just wondering.

LLR Questions

I attended LLR Test this weekend.
Below are some of the questions. There are more tricky questions, but I don't remember them.

1. What is the first property a person looses when he consumed alcohol
a. Anticipation b.Patience c. Judgement

My Answer was: a
2. What is the thing that you take extra care when you are taking right turn

a. Parked vehicle b.Vehicle coming in opposite direction c.

My answer was: b

3. When there is a signal of "Overtaking prohibited". Is that info
a. a Notice b. caution c. compelled

My answer was: c

4."You can take U turn as long as you are not obstructing users from other side road"
Is this statement
a. True b. False c. 50:50

My answer was: b
5. is sign like this ---->Hosur
^
| Bangalore
<-------- Mysore

a. Advance destination designation b. Destination designation c. Direction Designation

My answer was: c

6. If there is a sign like "Right turn prohibited". Then sign like "U turn prohibited" is implicit?
a. True b. False c.

My answer was: a

7. Find odd man out in below list
a. Median b. Round about c. Cross roads
b. didn't know what median is
8.If yellow light is flashing, you will slow down and
a. stop b. yield traffic c.


9.When the vehicle infront of you gave a signal to you to over take. then, what will you do.

a. accelerate and increase speed and over take the vehicle and turn left
b. slow down
c.

my answer is: a


NOTE: Needless to say that I failed the exam. :)

Strange google search error

I got above error when I did simple Google search. I don't know why this happened? Does any one have any clue?

But I found it really weird and annoying.

Bored at work?

Here is the solution given by my colleague.

You got bored while working and want a quick relaxation with an entertainment.
No need to open orkut. No need to open youtube or what ever.
Just look at the faces of other people around you. and look at their expressions. It will be so funny. :) and so you will be relaxed. :)