Tags » Hadoop

Cloud Architectures (C)



As it was difficult to know how much time each phase would take to execute (e.g., the launch phase decides dynamically how many instances need to start based on the request and hence execution time is unknown) Amazon SQS helped in building asynchronous systems. 903 more words

Computers And Internet

How Pig and Hadoop fit in Data Processing Architecture?

Hadoop and its ecosystem has evolved from a narrow map-reduced architecture to a universal data platform set to dominate the data processing landscape in the future. 281 more words


Coordination of distributed applications through Zookeeper

When I started to work in scope of Big Data, one of the most challenging things for me was to understand the distributed nature of Hadoop applications. 1,561 more words


HDFS copy/move commands

  • Copying files from HDFS to Local file system:
    • hadoop fs -get <source hdfs location> <destination local location>
  • Copying files from Local file system to HDFS: …
  • 110 more words

HDFS replication commands

Making replication for HDFS files manually:

  • hadoop fs -setrep -R 3 /

HDFS ACL commands

  • Check ACL’s settings in a file/folder:
    • hadoop fs -getfacl hdfs://<path_to_file/folder>
  • Setting ACL’s:
    • hadoop fs -setfacl -m -R default:other:rwx hdfs://<path_to_file/folder>

Hadoop mostly used shell commands

  • List files or dir inside HDFS:
    • hadoop fs -ls hdfs://<path_to_file/folder>
  • View contains of a file:
    • hadoop fs -cat hdfs://<path_to_file/folder>
  • Deleting a file:
    • hadoop fs -rm hdfs://<path_to_file>
  • 33 more words