vlambda博客
学习文章列表

hdfs fsck /user/root/output/part-r-00000 -files -blocks -locations

BlockPlacementPolicyDefault类具体可参阅：https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /** * The class is responsible for choosing the desired number of targets * for placing block replicas. * The replica placement strategy is that if the writer is on a datanode, * the 1st replica is placed on the local machine by default * (By passing the {@link org.apache.hadoop.fs.CreateFlag#NO_LOCAL_WRITE} flag * the client can request not to put a block replica on the local datanode. * Subsequent replicas will still follow default block placement policy.). * If the writer is not on a datanode, the 1st replica is placed on a random * node. * The 2nd replica is placed on a datanode that is on a different rack. * The 3rd replica is placed on a datanode which is on a different node of the * rack as the second replica. */

客户端读取一个HDFS文件，首先获得文件的blocks，然后获取blocks的locationshttps://github.com/apache/hadoop/blob/release-2.6.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java 从release-2.8.0-RC0起，HDFS调整了源代码结构。https://github.com/apache/hadoop/blob/release-3.2.2-RC5/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java  /** * Get block location info about file * * getBlockLocations() returns a list of hostnames that store * data for a specific file region. It returns a set of hostnames * for every block within the indicated region. * * This function is very useful when writing code that considers * data-placement when performing operations. For example, the * MapReduce system tries to schedule tasks on the same machines * as the data-block the task processes. * * Please refer to * {@link FileSystem#getFileBlockLocations(FileStatus, long, long)} * for more details. */ public BlockLocation[] getBlockLocations(String src, long start, long length) throws IOException { checkOpen(); try (TraceScope ignored = newPathTraceScope("getBlockLocations", src)) { LocatedBlocks blocks = getLocatedBlocks(src, start, length); BlockLocation[] locations = DFSUtilClient.locatedBlocks2Locations(blocks); HdfsBlockLocation[] hdfsLocations = new HdfsBlockLocation[locations.length]; for (int i = 0; i < locations.length; i++) { hdfsLocations[i] = new HdfsBlockLocation(locations[i], blocks.get(i)); } return hdfsLocations; } }