Friday 10 July 2015

How to avoid _success file in Mapreduce Output Folder.

How to avoid _success and _log files in mapreduce output:

Hadoop produces _logs folder and creates logs in that. But, this has been avoided in next versions of hadoop.
But, just want to know.

we need to set hadoop.job.history.user.location value to none.
In your mapreduce program, you can simply set
conf.set("hadoop.job.history.user.location","none");

Please ensure this property is set before the job creation.

How to avoid _success file.?

We need to set mapreduce.fileoutputcommitter.marksuccessfuljobs property to false.

like this:
conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false");


No comments:

Post a Comment