Differences

This shows you the differences between two versions of the page.

--- distributed_computing:data_processing:hadoop:hdfs:small_files [2019/10/25 19:05] – created phreazer
+++ distributed_computing:data_processing:hadoop:hdfs:small_files [2019/10/25 19:55] (current) – [Solutions] phreazer
@@ Line 2: / Line 2: @@
 Memory overhead
-  * File/Directory/Block is represented as object in namenode's memory and occupies 150 bytes. 10.000.000 small files => (10.000.000 blocks + 10.000.000 files) * 150 bytes => 3 Gb
+  * File/Directory/Block is represented as object in namenode's memory and occupies 150 bytes. 10.000.000 small files => (10.000.000 blocks * replication factor + 10.000.000 file inodes ) * 150 bytes => 3 Gb
   * Namenode is limited by main memory
 ===== Solutions =====
-* HAR: Hadoop archive; Layered file system on top of HDFS; Used for archiving (slow read)
+  * HAR: Hadoop archive; Layered file system on top of HDFS; Used for archiving (slow read)
-* Sequence file: File name as key, contents as the value
+  * Sequence file: File name as key, contents as the value
-* Consolidator
+  * Consolidator
-* HBase: Stores data in indexed SequenceFiles (HBase)
+  * HBase: Stores data in indexed SequenceFiles (HBase)
+  * Spark compaction: https://github.com/KeithSSmith/spark-compaction
+  * Filecrush: https://github.com/asdaraujo/filecrush