flink中的記憶體劃分

一護發表於2019-01-10
記憶體

在設定完taskManager記憶體之後相當於向yarn申請這麼大記憶體的container，然後flink內部的記憶體大部分是由flink框架管理，在啟動container之前就會 預先計算各個記憶體塊的大小。

記憶體塊劃分


// 預設值0.25f
final float memoryCutoffRatio = config.getFloat(ResourceManagerOptions.CONTAINERIZED_HEAP_CUTOFF_RATIO);

// 最少預留大小預設600MB
final int minCutoff = config.getInteger(ResourceManagerOptions.CONTAINERIZED_HEAP_CUTOFF_MIN);
            
// 先減去一塊記憶體預留給jvm
long cutoff = (long) (containerMemoryMB * memoryCutoffRatio);
if (cutoff < minCutoff) {
    cutoff = minCutoff;
}

final long javaMemorySizeMB = containerMemoryMB - cutoff;

// (2) split the remaining Java memory between heap and off-heap
final long heapSizeMB = TaskManagerServices.calculateHeapSizeMB(javaMemorySizeMB, config);
// use the cut-off memory for off-heap (that was its intention)
// 計算得到堆外記憶體後，總記憶體減去得到堆內的大小
final long offHeapSize = javaMemorySizeMB == heapSizeMB ? -1L : containerMemoryMB - heapSizeMB;

// (3) obtain the additional environment variables from the configuration
final HashMap<String, String> envVars = new HashMap<>();
final String prefix = ResourceManagerOptions.CONTAINERIZED_TASK_MANAGER_ENV_PREFIX;

for (String key : config.keySet()) {
    if (key.startsWith(prefix) && key.length() > prefix.length()) {
        // remove prefix
        String envVarKey = key.substring(prefix.length());
        envVars.put(envVarKey, config.getString(key, null));
    }
}	
複製程式碼

計算堆內記憶體大小TaskManagerServices.calculateHeapSizeMB


// 預設線上是開啟堆外記憶體的，為了資料交換的過程只使用堆外記憶體，gc友好
if (useOffHeap) {
    // subtract the Java memory used for network buffers
    final long networkBufMB = calculateNetworkBufferMemory(totalJavaMemorySize, config) >> 20; // bytes to megabytes
    final long remainingJavaMemorySizeMB = totalJavaMemorySizeMB - networkBufMB;

    long offHeapSize = config.getLong(TaskManagerOptions.MANAGED_MEMORY_SIZE);

    if (offHeapSize <= 0) {
        // calculate off-heap section via fraction
        // 將劃去networkBuffer大小*一個堆外的係數（預設是0.7）得到其他的堆外記憶體
        double fraction = config.getFloat(TaskManagerOptions.MANAGED_MEMORY_FRACTION);
        offHeapSize = (long) (fraction * remainingJavaMemorySizeMB);
    }

    TaskManagerServicesConfiguration
        .checkConfigParameter(offHeapSize < remainingJavaMemorySizeMB, offHeapSize,
            TaskManagerOptions.MANAGED_MEMORY_SIZE.key(),
            "Managed memory size too large for " + networkBufMB +
                " MB network buffer memory and a total of " + totalJavaMemorySizeMB +
                " MB JVM memory");

    heapSizeMB = remainingJavaMemorySizeMB - offHeapSize;
} else {
    heapSizeMB = totalJavaMemorySizeMB;
}
複製程式碼

計算堆外記憶體大小calculateNetworkBufferMemory

Preconditions.checkArgument(totalJavaMemorySize > 0);

int segmentSize = config.getInteger(TaskManagerOptions.MEMORY_SEGMENT_SIZE);

final long networkBufBytes;
// 涉及新老兩個版本的引數，以前版本是直接設定networkbuffer的個數，但是比較難估計，新的版本是直接設定記憶體塊大小
if (TaskManagerServicesConfiguration.hasNewNetworkBufConf(config)) {
    // new configuration based on fractions of available memory with selectable min and max
    float networkBufFraction = config.getFloat(TaskManagerOptions.NETWORK_BUFFERS_MEMORY_FRACTION);
    long networkBufMin = config.getLong(TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MIN);
    long networkBufMax = config.getLong(TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MAX);

    TaskManagerServicesConfiguration
        .checkNetworkBufferConfig(segmentSize, networkBufFraction, networkBufMin, networkBufMax);

    // 通過networkFraction計算network的記憶體大小，這個fraction預設值是0.1，同時最小設定預設是64M
    networkBufBytes = Math.min(networkBufMax, Math.max(networkBufMin,
        (long) (networkBufFraction * totalJavaMemorySize)));

    TaskManagerServicesConfiguration
        .checkConfigParameter(networkBufBytes < totalJavaMemorySize,
            "(" + networkBufFraction + ", " + networkBufMin + ", " + networkBufMax + ")",
            "(" + TaskManagerOptions.NETWORK_BUFFERS_MEMORY_FRACTION.key() + ", " +
                TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MIN.key() + ", " +
                TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MAX.key() + ")",
            "Network buffer memory size too large: " + networkBufBytes + " >= " +
                totalJavaMemorySize + " (total JVM memory size)");
} else {
    // use old (deprecated) network buffers parameter
    int numNetworkBuffers = config.getInteger(TaskManagerOptions.NETWORK_NUM_BUFFERS);
    networkBufBytes = (long) numNetworkBuffers * (long) segmentSize;

    TaskManagerServicesConfiguration.checkNetworkConfigOld(numNetworkBuffers);

    TaskManagerServicesConfiguration
        .checkConfigParameter(networkBufBytes < totalJavaMemorySize,
            networkBufBytes, TaskManagerOptions.NETWORK_NUM_BUFFERS.key(),
            "Network buffer memory size too large: " + networkBufBytes + " >= " +
                totalJavaMemorySize + " (total JVM memory size)");
}

return networkBufBytes;
複製程式碼

小結

  1. JVM預留記憶體，總記憶體的20%，最小預留，600M
  2. 剩下的記憶體的10%作為networkBuffer的記憶體，最小64M
  3. 剩下記憶體30%設為堆內記憶體，總記憶體減去堆內記憶體設為directMemory，用於netty和rocksDB和networkBuffer以及JVM自身記憶體

舉例

一個啟動時設定TaskManager記憶體大小為1024MB

  1. 1024MB - (1024 * 0.2 < 600MB) -> 600MB = 424MB (cutoff)
  2. 424MB - (424MB * 0.1 < 64MB) -> 64MB = 360MB (networkbuffer)
  3. 360MB * (1 - 0.7) = 108MB -> （onHeap）
  4. 1024MB - 108MB = 916MB （maxDirectMemory）

最終啟動命令：

yarn      46218  46212  1 Jan08 ?        00:17:50
/home/yarn/java-current/bin/java 
-Xms109m -Xmx109m -XX:MaxDirectMemorySize=915m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps 
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=512M 
-Xloggc:/data1/hadoopdata/nodemanager/logdir/application_1545981373722_0172/container_e194_1545981373722_0172_01_000005/taskmanager_gc.log 
-XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=75 
-XX:+UseCMSInitiatingOccupancyOnly 
-XX:+AlwaysPreTouch -server 
-XX:+HeapDumpOnOutOfMemoryError 
-Dlog.file=/data1/hadoopdata/nodemanager/logdir/application_1545981373722_0172/container_e194_1545981373722_0172_01_000005/taskmanager.log 
-Dlogback.configurationFile=file:./logback.xml 
-Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskManager 
--configDir .
複製程式碼

相關文章