dotnet學習筆記-專題01-非同步與多執行緒-01

random_d發表於2024-11-22

專題01 非同步 多執行緒

1. Thread類

1.1 使用Thread建立執行緒

namespace ConsoleApp1;

internal class Program
{
    private static void Main(string[] Args)
    {
        var t = new Thread(WirteY);
        t.Name = "Y thread ...";
        t.Start();
        for (int i = 0; i < 1000; i++)
        {
            Console.Write("x");
        }
    }
    
    private static void WirteY()
    {
        for (int i = 0; i < 1000; i++)
        {
            Console.Write("y");
        }
    }
}
  • 執行緒的一些屬性
    • 執行緒一旦開始執行,IsAlive就是true,執行緒結束變為false
    • 執行緒結束的條件是:執行緒建構函式傳入的委託結束了執行
    • 執行緒一旦結束,就無法再重啟
    • 每個執行緒都有個Name屬性,通常用於除錯
      • 執行緒Name只能設定一次,以後更改會丟擲異常
    • 靜態的thread.CurrentThread屬性,會返回當前執行的執行緒

1.2 Thread.join()和Thread.Sleep()

namespace ConsoleApp1;

internal class Program
{
    private static void Main()
    {
        Thread.CurrentThread.Name = "Main Thread";
        var thread1 = new Thread(ThreadProc1)
        {
            Name = "Thread1"
        };
        var thread2 = new Thread(ThreadProc2)
        {
            Name = "Thread2"
        };
        thread1.Start();
        thread2.Start();
        // 等待thread2執行結束
        thread2.Join();
        var currentTime = DateTime.Now;
        Console.WriteLine($"\nCurrent thread " +
            $"{Thread.CurrentThread.Name}: {currentTime.Hour}:{currentTime.Minute}:{currentTime.Second}");
    }

    private static void ThreadProc1()
    {
        Thread.Sleep(5000);
        var currentTime = DateTime.Now;
        Console.WriteLine($"\nCurrent thread " +
            $"{Thread.CurrentThread.Name}: {currentTime.Hour}:{currentTime.Minute}:{currentTime.Second}");
    }

    private static void ThreadProc2()
    {
        Thread.Sleep(3000);
        var currentTime = DateTime.Now;
        Console.WriteLine($"\nCurrent thread " +
            $"{Thread.CurrentThread.Name}: {currentTime.Hour}:{currentTime.Minute}:{currentTime.Second}");
    }
}

1.3 Thread存在的問題

  • 雖然開始執行緒的時候可以方便的傳入資料,但當Join的時候很難從執行緒獲得返回值
    • 可能需要設定共享欄位
    • 如果操作丟擲異常,捕獲和傳播該異常都很麻煩
  • 無法告訴子執行緒在結束時在當前執行緒做另外的工作,而不是結束子執行緒
    • 這要求必須進行Join操作,這會在程序中阻塞主執行緒
  • 很難使用較小的併發(concurrent)來組建大型的併發
  • 導致了對手動同步的更大依賴以及隨之而來的問題

2. Task類

2.1 Task類的優勢

  • Task類是對Thread的高階抽象,能夠解決Thread存在的問題
  • Task代表了一個併發操作(concurrent)
    • 這個併發操作可能由Thread支援,也可以不由Thread支援
  • Task是可組合的
    • 可以使用Continuation把任務串聯起來,同時使用執行緒池減少了啟動延遲
    • TaskCompletionSource類使得Tasks可以利用回撥方法,在等待I/O繫結操作時完全避免使用執行緒

2.2 建立一個Task

  • 開始一個Task最簡單的辦法是使用Task.Run這個靜態方法
    • 傳入一個Action委託即可
  • Task預設使用執行緒池,也就是後臺執行緒
    • 當主執行緒結束時,建立的所有Tasks都會結束
  • Task.Run返回一個Task物件,可以用來監視其過程
    • 在Task.Run之後無需呼叫Start的原因在於,使用該方法建立的是hot task,熱任務可以立刻執行。當然,也可以透過Task的建構函式 建立“冷”任務,但很少這樣做。
  • 可以使用Task類的Status屬性來跟蹤task的執行狀態。
  • 可以使用Task.Wait()阻塞對應task執行緒,直到該執行緒完成
namespace ConsoleApp1;

internal class Program
{
    /*該函式沒有任何輸出的原因在於:Task預設使用後臺執行緒,
     *當主執行緒Main()結束時所有task都會結束,然而此時task1還未執行完成 */

    private static void Function1()
    {
        var task1 = Task.Run(() =>
        {
            Thread.Sleep(3000);
            Console.WriteLine("Function1");
        });
        // 輸出當前task的執行狀態
        Console.WriteLine($"task1執行完成? {task1.IsCompleted}");
    }

    private static void Function2()
    {
        var task2 = Task.Run(() =>
        {
            Thread.Sleep(1000);
            Console.WriteLine("Function2");
        });
        Console.WriteLine($"task2執行完成? {task2.IsCompleted}");
        // 等待task2執行完成
        task2.Wait();
        Console.WriteLine($"task2執行完成? {task2.IsCompleted}");
    }

    private static void Main()
    {
        Function1();
        Function2();
    }
}
  • 也可以使用Task.Wait()設定一個超時時間和一個取消令牌來提前結束等待。
/* 帶有時間限制的task.Wait() */
/* 結束等待的途徑:1.任務超時;2.任務完成 */
namespace ConsoleApp1;

internal class Program
{
    private static void TimeLimitExample()
    {
        // 在0到100之間生成500萬個隨機數,並計算其值
        var t = Task.Run(() =>
        {
            var rnd = new Random();
            long sum = 0;
            int n = 500 * (int)Math.Pow(10, 4);
            for (int ctr = 0; ctr < n; ctr++)
            {
                int number = rnd.Next(0, 101);
                sum += number;
            }
            Console.WriteLine("Total:   {0:N0}", sum);
            Console.WriteLine("Mean:    {0:N2}", sum / n);
            Console.WriteLine("N:       {0:N0}", n);
        });
        // 設定超時時間為150ms
        var ts = TimeSpan.FromMilliseconds(150);
        if (!t.Wait(ts))
        {
            Console.WriteLine("The timeout interval elapsed.");
        }
    }

    private static void TimeoutExample()
    {
        // 在0到100之間生成500萬個隨機數,並計算其值
        var t = Task.Run(() =>
        {
            var rnd = new Random();
            long sum = 0;
            int n = 500 * (int)Math.Pow(10, 4);
            for (int ctr = 0; ctr < n; ctr++)
            {
                int number = rnd.Next(0, 101);
                sum += number;
            }
            // 人為增加延時200ms
            Thread.Sleep(200);
            Console.WriteLine("Total:   {0:N0}", sum);
            Console.WriteLine("Mean:    {0:N2}", sum / n);
            Console.WriteLine("N:       {0:N0}", n);
        });
        // 設定超時時間為150ms
        var ts = TimeSpan.FromMilliseconds(150);
        if (!t.Wait(ts))
        {
            Console.WriteLine("The timeout interval elapsed.");
        }
    }

    private static void Main()
    {
        TimeLimitExample();
        Console.WriteLine(Environment.NewLine);
        TimeoutExample();
    }
}
/* 帶有取消令牌的task.Wait() */
/* 結束等待的途徑:1.透過取消令牌取消等待;2.任務完成 */
/* 取消令牌需要ts.Cancel();和Task.Wait(ts.Token); */
/* 其中觸發ts.Cancel()的條件要在Task.Run(func)的func方法中指定 */
namespace ConsoleApp1;

internal class Program
{
    private static void Main()
    {
        var ts = new CancellationTokenSource();
        var t = Task.Run(() =>
        {
            Console.WriteLine("Calling Cancel...");
            // 傳達取消請求
            ts.Cancel();
            Task.Delay(5000).Wait();
            Console.WriteLine("Task ended delay...");
        });
        try
        {
            Console.WriteLine("About to wait for the task to complete...");
            // 使用ts.Token獲取與此CancellationToken關聯的CancellationTokenSource
            t.Wait(ts.Token);
        }
        catch (OperationCanceledException e)
        {
            Console.WriteLine("{0}: The wait has been canceled. Task status: {1:G}",
                  e.GetType().Name, t.Status);
            Thread.Sleep(6000);
            Console.WriteLine("After sleeping, the task status:  {0:G}", t.Status);
        }
        // 釋放CancellationTokenSource類的當前例項所使用的所有資源
        ts.Dispose();
    }
}
  • 針對長時間執行的任務或者阻塞操作,可以不採用執行緒池。可以使用Task.Factory.StartNew和TaskCreationOptions.LongRunning定義,後臺排程時會為這一任務新開一個執行緒。
namespace ConsoleApp1;

internal class Program
{
    private static void Main()
    {
        var task = Task.Factory.StartNew(() =>
        {
            Thread.Sleep(1000);
            Console.WriteLine("Run");
        }, TaskCreationOptions.LongRunning);
        task.Wait();
    }
}
  • 如果同時執行多個long-running tasks,那麼效能將會受到很大影響
    • 如果任務是IO-Bound(受限於資料吞吐量),可以使用由TaskCompletionSource和非同步函式組成的回撥代替執行緒實現併發。
    • 如果任務是CPU-Bound(受限於CPU運算能力),使用 生產者/消費者 佇列對任務的併發進行限制,防止把其他的執行緒或程序“餓死”。
    • 注:絕大多數任務都是IO-Bound的,CPU的資料處理能力遠強於資料傳輸能力。

2.3 CancellationToken的使用

2.3.1 示例1

以非同步方式下載一個網站內容100次,設定超時時間1.5s,如果超時則使用CancellationToken終止非同步下載過程。

namespace ConsoleApp1ForTest;

internal static class Program
{
    private static async Task Main()
    {
        var cancellationTokenSource = new CancellationTokenSource();
        // 在多長時間後取消
        cancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(1.5));
        // 從cancellationTokenSource建立令牌
        var cancellationToken = cancellationTokenSource.Token;
        const string url = "https://www.youzack.com";
        await Download(url, 100, cancellationToken);
    }

    private static async Task Download(string url, int n, CancellationToken token)
    {
        using var httpclient = new HttpClient
        {
            Timeout = TimeSpan.FromSeconds(1)
        };
        for (var i = 0; i < n; i++)
        {
            try
            {
                var html = await httpclient.GetStringAsync(url, token);
                Console.WriteLine($"{DateTime.Now}:{html}");
            }
            catch (TaskCanceledException exception)
            {
                Console.WriteLine(exception.Message);
                Console.WriteLine("任務被取消");
            }
            
            // 如果沒有CancellationToken繼續下次迴圈,跳過Console.WriteLine
            // 如果有CancellationToken則終止迴圈
            if (!token.IsCancellationRequested) continue;
            Console.WriteLine("CancellationToken觸發,跳過剩餘迴圈");
            break;
        }
    }
}

2.3.2 示例2

以非同步方式下載一個網站內容10000次,在此過程中使用者可以手動設定Cancellation Token取消下載任務。

namespace ConsoleApp1ForTest;

internal static class Program
{
    private static void Main()
    {
        var cancellationTokenSource = new CancellationTokenSource();
        const string url = "https://www.youzack.com";
        var cancellationToken = cancellationTokenSource.Token;
        _ = Download(url, 10000, cancellationToken);
        while (!cancellationToken.IsCancellationRequested)
        {
            Console.WriteLine("按Q鍵取消任務");
            var cancelSignal = Console.ReadLine();
            if (cancelSignal == "Q") cancellationTokenSource.Cancel();
            else Console.WriteLine("指令輸入錯誤,請重試");
        }
    }

    private static async Task Download(string url, int n, CancellationToken token)
    {
        using var httpclient = new HttpClient
        {
            Timeout = TimeSpan.FromSeconds(1)
        };
        for (var i = 0; i < n; i++)
        {
            try
            {
                _ = await httpclient.GetStringAsync(url, token);
                Console.WriteLine($"{DateTime.Now}:{i}/{n}");
            }
            catch (TaskCanceledException exception)
            {
                Console.WriteLine(exception.Message);
                Console.WriteLine("任務被取消");
            }

            // 如果沒有CancellationToken繼續下次迴圈,跳過Console.WriteLine
            // 如果有CancellationToken則終止迴圈
            if (!token.IsCancellationRequested) continue;
            Console.WriteLine("CancellationToken觸發,跳過剩餘迴圈");
            break;
        }
    }
}

3. 示例

3.1 【示例】從多個網頁上下載內容

title: 示例來源

C\# Async/Await: 讓你的程式變身時間管理大師
https://www.bilibili.com/video/av846932409/

不過做了改進,不要介面,只用控制檯程式。

using System.Diagnostics;

namespace ConsoleApp1;

internal static class Program
{
    private static void Main(string[] args)
    {
        var time1 = Downloader.DownloadWebsitesSync();
        var time2 = Downloader.DownloadWebsitesAsync();

        Console.WriteLine($"同步方法下載用時:{time1}, 非同步方法下載用時:{time2}");
    }
}

internal static class Downloader
{
    private readonly static List<string> WebAddresses = new List<string>()
    {
        "https://docs.microsoft.com",
        "https://docs.microsoft.com/aspnet/core",
        "https://docs.microsoft.com/azure",
        "https://docs.microsoft.com/azure/devops",
        "https://docs.microsoft.com/dotnet",
        "https://docs.microsoft.com/dynamics365",
        "https://docs.microsoft.com/education",
        "https://docs.microsoft.com/enterprise-mobility-security",
        "https://docs.microsoft.com/gaming",
        "https://docs.microsoft.com/graph",
        "https://docs.microsoft.com/microsoft-365",
        "https://docs.microsoft.com/office",
        "https://docs.microsoft.com/powershell",
        "https://docs.microsoft.com/sql",
        "https://docs.microsoft.com/surface",
        "https://docs.microsoft.com/system-center",
        "https://docs.microsoft.com/visualstudio",
        "https://docs.microsoft.com/windows",
        "https://docs.microsoft.com/xamarin"
    };

    private readonly static HttpClient HttpClient = new HttpClient()
    {
        Timeout = TimeSpan.FromSeconds(5)
    };

    public static string DownloadWebsitesSync()
    {
        var stopwatch = Stopwatch.StartNew();
        var linqQuery = from url in WebAddresses
            let response = HttpClient.GetByteArrayAsync(url).Result
            select new
            {
                Url = url,
                ResponseLength = response.Length
            };
        foreach (var element in linqQuery)
        {
            var outputSting =
                $"Finish downloading data from {element.Url}. Total bytes returned {element.ResponseLength}.";
            Console.WriteLine(outputSting);
        }

        var totalTime = $"{stopwatch.Elapsed:g}";
        Console.WriteLine($"Total bytes downloaded: {totalTime}");
        stopwatch.Stop();
        return totalTime;
    }

    public static string DownloadWebsitesAsync()
    {
        var stopwatch = Stopwatch.StartNew();
        var downloadWebsiteTasks =
            WebAddresses.Select(site => Task.Run(() => new
            {
                Url = site,
                ResponseLength = HttpClient.GetByteArrayAsync(site).Result.Length
            })).ToList();
        var results = Task.WhenAll(downloadWebsiteTasks).Result;
        foreach (var element in results)
        {
            var outputSting =
                $"Finish downloading data from {element.Url}. Total bytes returned {element.ResponseLength}.";
            Console.WriteLine(outputSting);
        }

        var totalTime = $"{stopwatch.Elapsed:g}";
        Console.WriteLine($"Total bytes downloaded: {totalTime}");
        stopwatch.Stop();
        return totalTime;
    }
}

測試結果如下:

Finish downloading data from https://docs.microsoft.com. Total bytes returned 42574.
Finish downloading data from https://docs.microsoft.com/aspnet/core. Total bytes returned 86207.
Finish downloading data from https://docs.microsoft.com/azure. Total bytes returned 395639.
Finish downloading data from https://docs.microsoft.com/azure/devops. Total bytes returned 84141.
Finish downloading data from https://docs.microsoft.com/dotnet. Total bytes returned 89027.
Finish downloading data from https://docs.microsoft.com/dynamics365. Total bytes returned 59208.
Finish downloading data from https://docs.microsoft.com/education. Total bytes returned 38835.
Finish downloading data from https://docs.microsoft.com/enterprise-mobility-security. Total bytes returned 31118.
Finish downloading data from https://docs.microsoft.com/gaming. Total bytes returned 66954.
Finish downloading data from https://docs.microsoft.com/graph. Total bytes returned 51945.
Finish downloading data from https://docs.microsoft.com/microsoft-365. Total bytes returned 66549.
Finish downloading data from https://docs.microsoft.com/office. Total bytes returned 29537.
Finish downloading data from https://docs.microsoft.com/powershell. Total bytes returned 58100.
Finish downloading data from https://docs.microsoft.com/sql. Total bytes returned 60741.
Finish downloading data from https://docs.microsoft.com/surface. Total bytes returned 46145.
Finish downloading data from https://docs.microsoft.com/system-center. Total bytes returned 49409.
Finish downloading data from https://docs.microsoft.com/visualstudio. Total bytes returned 34358.
Finish downloading data from https://docs.microsoft.com/windows. Total bytes returned 29840.
Finish downloading data from https://docs.microsoft.com/xamarin. Total bytes returned 58138.
Total bytes downloaded: 0:00:11.323132
Finish downloading data from https://docs.microsoft.com. Total bytes returned 42574.
Finish downloading data from https://docs.microsoft.com/aspnet/core. Total bytes returned 86207.
Finish downloading data from https://docs.microsoft.com/azure. Total bytes returned 395639.
Finish downloading data from https://docs.microsoft.com/azure/devops. Total bytes returned 84141.
Finish downloading data from https://docs.microsoft.com/dotnet. Total bytes returned 89027.
Finish downloading data from https://docs.microsoft.com/dynamics365. Total bytes returned 59208.
Finish downloading data from https://docs.microsoft.com/education. Total bytes returned 38835.
Finish downloading data from https://docs.microsoft.com/enterprise-mobility-security. Total bytes returned 31118.
Finish downloading data from https://docs.microsoft.com/gaming. Total bytes returned 66954.
Finish downloading data from https://docs.microsoft.com/graph. Total bytes returned 51945.
Finish downloading data from https://docs.microsoft.com/microsoft-365. Total bytes returned 66549.
Finish downloading data from https://docs.microsoft.com/office. Total bytes returned 29537.
Finish downloading data from https://docs.microsoft.com/powershell. Total bytes returned 58100.
Finish downloading data from https://docs.microsoft.com/sql. Total bytes returned 60741.
Finish downloading data from https://docs.microsoft.com/surface. Total bytes returned 46145.
Finish downloading data from https://docs.microsoft.com/system-center. Total bytes returned 49409.
Finish downloading data from https://docs.microsoft.com/visualstudio. Total bytes returned 34358.
Finish downloading data from https://docs.microsoft.com/windows. Total bytes returned 29840.
Finish downloading data from https://docs.microsoft.com/xamarin. Total bytes returned 58138.
Total bytes downloaded: 0:00:01.8900202
同步方法下載用時:0:00:11.323132, 非同步方法下載用時:0:00:01.8900202

3.2 【示例】批次計算檔案hash值

首先先編寫配置檔案,命名為configuration.json

{
  "DirectoryPath": "D:\\pictures",
  "HashResultOutputConfig": {
    "ConnectionString": "Data Source=sql.nas.home;Database=LearnAspDotnet_CalculateHash;User ID=root;Password=Aa123456+;pooling=true;port=3306;CharSet=utf8mb4",
    "ExcelFilePath": "D:\\hashResult.xlsx",
    "OutputToMySql": false,
    "OutputToExcel": true
  },
  "ParallelHashCalculateConfig": {
    "InitialDegreeOfParallelism": 4,
    "MaxDegreeOfParallelism": 32,
    "MinDegreeOfParallelism": 1,
    "AdjustmentInterval_Seconds": 5,
    "TargetBatchSize_MByte": 100,
    "FileCountPreAdjust": 5,
    "AdjustByFileSize": true
  }
}

為了方便呼叫,將配置檔案和配置類繫結。現在新建配置類Configuration

namespace CalculateHash.Core;

public class Configuration
{
    public string DirectoryPath { get; set; } = string.Empty;
    public HashResultOutputConfig HashResultOutputConfig { get; set; } = new();
    public ParallelHashCalculateConfig ParallelHashCalculateConfig { get; set; } = new();
}

public class HashResultOutputConfig
{
    public string ConnectionString { get; set; } = string.Empty;
    public string ExcelFilePath { get; set; } = string.Empty;
    public bool OutputToMySql { get; set; } = false;
    public bool OutputToExcel { get; set; } = false;
}

public class ParallelHashCalculateConfig
{
    // 初始並行度
    public int InitialDegreeOfParallelism { get; set; }

    // 最大並行度
    public int MaxDegreeOfParallelism { get; set; }

    // 最小並行度
    public int MinDegreeOfParallelism { get; set; }

    // 調整時間間隔
    public int AdjustmentInterval_Seconds { get; set; }

    // 每次並行時計算時調整的總檔案大小
    public int TargetBatchSize_MByte { get; set; }

    // 是否按照總檔案大小調整(false:不動態調整)
    public bool AdjustByFileSize { get; set; } = false;
}

現在還需要一個靜態工具類完成配置檔案和配置類的繫結並承擔配置的讀寫工作,該工具類為ConfigurationJsonHelper

using System.Data;
using Newtonsoft.Json;

namespace CalculateHash.Core;

public static class ConfigurationJsonHelper
{
    public static string GetFullPathOfJsonConfiguration()
    {
        var fileInfo = new FileInfo("./configuration.json");
        return fileInfo.FullName;
    }

    public static Configuration GetConfiguration()
    {
        try
        {
            using var fileReader = new StreamReader("./configuration.json");
            var jsonContent = fileReader.ReadToEndAsync().GetAwaiter().GetResult();
            var configuration = JsonConvert.DeserializeObject<Configuration>(jsonContent);
            if (configuration is null) throw new NoNullAllowedException("無法反序列化");
            return configuration;
        }
        catch (Exception e) when (e is FileNotFoundException or IOException or DirectoryNotFoundException)
        {
            Console.WriteLine("路徑出錯: {0}", e.Message);
            throw;
        }
        catch (NoNullAllowedException e)
        {
            Console.WriteLine("反序列化出錯:{0}", e.Message);
            throw;
        }
    }

    public static Configuration GetConfiguration(string configurationPath)
    {
        try
        {
            using var fileReader = new StreamReader(configurationPath);
            var jsonContent = fileReader.ReadToEndAsync().GetAwaiter().GetResult();
            var configuration = JsonConvert.DeserializeObject<Configuration>(jsonContent);
            if (configuration is null) throw new NoNullAllowedException("無法反序列化");
            return configuration;
        }
        catch (Exception e) when (e is FileNotFoundException or IOException or DirectoryNotFoundException)
        {
            Console.WriteLine("路徑出錯: {0}", e.Message);
            throw;
        }
        catch (NoNullAllowedException e)
        {
            Console.WriteLine("反序列化出錯:{0}", e.Message);
            throw;
        }
    }
}

還需要一個工具類負責將Hash結果儲存起來,該工具類為SaveTo

using MySql.Data.MySqlClient;
using OfficeOpenXml;

namespace CalculateHash.Core;

public class SaveTo(Configuration configuration)
{
    private readonly static object ExcelLock = new object();
    private readonly static object DatabaseLock = new object();

    public void SaveHashToDatabase(HashResultOutputDto hashResult)
    {
        lock (DatabaseLock)
        
        {
            using var connection = new MySqlConnection(configuration.HashResultOutputConfig.ConnectionString);
            connection.Open();
            // 是否有相同檔案記錄,如果有相同檔案就不插入
            const string sqlCommand =
                "INSERT INTO FileHashTable (fileCounter, fileName, sha512, fileNameWithFullPath) " +
                "select @fileCounter,@fileName, @sha512, @fileNameWithFullPath " +
                // 只有檔名+檔案路徑、sha512值完全一致時才視為重複插入,只有檔名+檔案路徑相同或者只有sha512值相同,均視為不同檔案,允許插入
                "where exists(select fileNameWithFullPath from FileHashTable where fileNameWithFullPath=@fileNameWithFullPath)=0" +
		        "or exist(select sha512 from FileHashTable where sha512=@sha512)=0";
            using var command = new MySqlCommand(sqlCommand, connection);
            command.Parameters.AddWithValue("@fileCounter", hashResult.FileCounter);
            command.Parameters.AddWithValue("@fileName", hashResult.FileName);
            command.Parameters.AddWithValue("@sha512", hashResult.HashResult_Sha512);
            command.Parameters.AddWithValue("@fileNameWithFullPath", hashResult.FileFullName);
            command.ExecuteNonQuery();
        }
    }

    public void SaveHashToExcel(HashResultOutputDto hashResult)
    {
        lock (ExcelLock)
        {
            ExcelPackage.LicenseContext = LicenseContext.NonCommercial;
            using var excelPackage = new ExcelPackage(new FileInfo(configuration.HashResultOutputConfig.ExcelFilePath));
            var worksheet = excelPackage.Workbook.Worksheets.FirstOrDefault() ??
                            excelPackage.Workbook.Worksheets.Add("File Hash Table");

            var lastRow = worksheet.Dimension?.Rows ?? 0;
            worksheet.Cells[lastRow + 1, 1].Value = hashResult.FileCounter;
            worksheet.Cells[lastRow + 1, 2].Value = hashResult.FileName;
            worksheet.Cells[lastRow + 1, 3].Value = hashResult.HashResult_Sha512;
            worksheet.Cells[lastRow + 1, 4].Value = hashResult.FileFullName;

            excelPackage.SaveAsync().GetAwaiter().GetResult();
        }
    }
}

現在可以開始檔案Hash值的計算工作了,新建FilesHashCalculator類,該類的功能有:

  1. 讀取配置檔案
  2. 計算Hash(這裡用SHA512)
  3. 動態並行度調整(如果計算的快就適量增加並行度,如果計算的慢就適量減少並行度)
using System.Security.Cryptography;
using System.Text;

namespace CalculateHash.Core;

public class FilesHashCalculator
{
    private readonly Configuration _configuration;
    private readonly SaveTo _save;
    private long _fileCounter;
    private readonly ManualResetEventSlim _pauseEvent;

    public FilesHashCalculator(ManualResetEventSlim pauseEvent)
    {
        _configuration = ConfigurationJsonHelper.GetConfiguration();
        _save = new SaveTo(_configuration);
        _fileCounter = 0;
        _pauseEvent = pauseEvent;
    }

    public void DynamicParallelHashCalculation(Action<long, string, string, long>? updateProgress,
        CancellationToken cancellationToken)
    {
        var directoryPath = _configuration.DirectoryPath;
        var initialDegreeOfParallelism = _configuration.ParallelHashCalculateConfig.InitialDegreeOfParallelism;
        var maxDegreeOfParallelism = _configuration.ParallelHashCalculateConfig.MaxDegreeOfParallelism;
        var minDegreeOfParallelism = _configuration.ParallelHashCalculateConfig.MinDegreeOfParallelism;
        // 時間間隔:毫秒
        long adjustmentInterval = _configuration.ParallelHashCalculateConfig.AdjustmentInterval_Seconds * 1000;
        // 每次調整的大小:位元組
        long targetBatchSize = 1024 * 1024 * _configuration.ParallelHashCalculateConfig.TargetBatchSize_MByte;

        try
        {
            var files = Directory.GetFiles(directoryPath)
                .Select(filePath => new FileInfo(filePath))
                .OrderByDescending(file => file.Length) // 按檔案大小降序排列
                .ToList();

            var totalFilesCount = files.Count;

            var currentDegreeOfParallelism = initialDegreeOfParallelism;
            var batches = CreateBatches(files, targetBatchSize);

            foreach (var batch in batches)
            {
                var parallelOptions = new ParallelOptions
                {
                    MaxDegreeOfParallelism = currentDegreeOfParallelism,
                    CancellationToken = cancellationToken
                };

                var stopwatch = System.Diagnostics.Stopwatch.StartNew();

                Parallel.ForEach(batch, parallelOptions, fileInfo =>
                {
                    _pauseEvent.Wait(cancellationToken);
                    var hash = ComputeHash(fileInfo.FullName);
                    var fileNumber = Interlocked.Increment(ref _fileCounter);
                    var hashResult = new HashResultOutputDto()
                    {
                        FileCounter = fileNumber,
                        FileName = fileInfo.Name,
                        FileFullName = fileInfo.FullName,
                        HashResult_Sha512 = hash
                    };
                    if (_configuration.HashResultOutputConfig.OutputToMySql) _save.SaveHashToDatabase(hashResult);
                    if (_configuration.HashResultOutputConfig.OutputToExcel) _save.SaveHashToExcel(hashResult);
#if DEBUG
                    Console.WriteLine($"{hashResult.FileCounter}-{hashResult.FileName}-{hashResult.HashResult_Sha512}");
#endif
                    updateProgress?.Invoke(hashResult.FileCounter, hashResult.FileName, hashResult.HashResult_Sha512,
                        totalFilesCount);
                });

                stopwatch.Stop();
                Console.WriteLine(
                    $"Processed {batch.Count} files in {stopwatch.ElapsedMilliseconds} ms with degree of parallelism {currentDegreeOfParallelism}");

                // 是否動態調整並行度
                if (!_configuration.ParallelHashCalculateConfig.AdjustByFileSize) continue;
                // 動態調整並行度
                if (stopwatch.ElapsedMilliseconds < adjustmentInterval &&
                    currentDegreeOfParallelism < maxDegreeOfParallelism)
                {
                    currentDegreeOfParallelism++;
                }
                else if (stopwatch.ElapsedMilliseconds > adjustmentInterval &&
                         currentDegreeOfParallelism > minDegreeOfParallelism)
                {
                    currentDegreeOfParallelism--;
                }
            }
        }
        catch (AggregateException ae)
        {
            foreach (var ex in ae.InnerExceptions) Console.WriteLine($"An error occurred: {ex.Message}");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"An error occurred: {ex.Message}");
        }
    }

    private static List<List<FileInfo>> CreateBatches(List<FileInfo> files, long targetBatchSize)
    {
        var batches = new List<List<FileInfo>>();
        var currentBatch = new List<FileInfo>();
        long currentBatchSize = 0;

        foreach (var file in files)
        {
            if (currentBatchSize + file.Length > targetBatchSize && currentBatch.Count > 0)
            {
                batches.Add(currentBatch);
                currentBatch = [];
                currentBatchSize = 0;
            }

            currentBatch.Add(file);
            currentBatchSize += file.Length;
        }

        if (currentBatch.Count > 0) batches.Add(currentBatch);

        return batches;
    }

    private static string ComputeHash(string fileInfoFullName)
    {
        using var stream = File.OpenRead(fileInfoFullName);
        using var sha512 = SHA512.Create();
        var hashBytes = sha512.ComputeHash(stream);
        var hashString = new StringBuilder();
        foreach (var b in hashBytes) hashString.Append(b.ToString("x2"));

        return hashString.ToString();
    }
}

下面新增圖形化介面,圖形化介面使用WPF。

首先是主視窗設計:

<Window x:Class="CalculateHash.GUI.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:local="clr-namespace:CalculateHash.GUI"
        mc:Ignorable="d"
        Title="MainWindow" Height="450" Width="800">
    <Grid HorizontalAlignment="Center" VerticalAlignment="Center">
        
        <Button x:Name="BtnSettings" Content="設定" HorizontalAlignment="Left" VerticalAlignment="Top" Width="100"
                Margin="120,10,0,0" Click="BtnSettings_Click" />
        <Button x:Name="BtnStart" Content="開始" HorizontalAlignment="Left" VerticalAlignment="Top" Width="100"
                Margin="230,10,0,0" Click="BtnStart_Click" />
        <Button x:Name="BtnPause" Content="暫停" HorizontalAlignment="Left" VerticalAlignment="Top" Width="100"
                Margin="340,10,0,0" Click="BtnPause_Click" />
        <Button x:Name="BtnStop" Content="停止" HorizontalAlignment="Left" VerticalAlignment="Top" Width="100"
                Margin="450,10,0,0" Click="BtnStop_Click" />
        <Button x:Name="BtnClear" Content="清除" HorizontalAlignment="Left" VerticalAlignment="Top" Width="100"
                Margin="560,10,0,0" Click="BtnClear_Click" />
        <ProgressBar x:Name="ProgressBar" HorizontalAlignment="Left" VerticalAlignment="Top" Width="760" Height="30"
                     Margin="10,50,0,0" />
        <DataGrid x:Name="DataGridResults" HorizontalAlignment="Left" VerticalAlignment="Top" Height="250" Width="760"
                  Margin="10,90,0,0" AutoGenerateColumns="False">
            <DataGrid.Columns>
                <DataGridTextColumn Header="檔案編號" Binding="{Binding FileNumber}" Width="*" />
                <DataGridTextColumn Header="檔名" Binding="{Binding FileName}" Width="*" />
                <DataGridTextColumn Header="雜湊值" Binding="{Binding HashResult}" Width="*" />
            </DataGrid.Columns>
        </DataGrid>
        <TextBlock x:Name="TxtTimeElapsed" HorizontalAlignment="Left" VerticalAlignment="Top" Margin="10,350,0,0"
                   Width="200" Text="已用時間: 0s" />
        <TextBlock x:Name="TxtTimeRemaining" HorizontalAlignment="Left" VerticalAlignment="Top" Margin="220,350,0,0"
                   Width="200" Text="預計剩餘時間: 0s" />
    </Grid>
</Window>

主視窗對應的程式碼:

using System.Collections.ObjectModel;
using System.Windows;
using CalculateHash.Core;

namespace CalculateHash.GUI;

/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
    private CancellationTokenSource _cancellationTokenSource;
    private readonly ManualResetEventSlim _pauseEvent;
    private bool _isPaused;
    private readonly ObservableCollection<DataGridResult> _hashResults;
    private DateTime _startTime;

    public MainWindow()
    {
        InitializeComponent();
        _cancellationTokenSource = new CancellationTokenSource();
        _pauseEvent = new ManualResetEventSlim(initialState: true);
        _hashResults = new ObservableCollection<DataGridResult>();
        _isPaused = false;
        DataGridResults.ItemsSource = _hashResults;
        _startTime = DateTime.Now;
        BtnPause.IsEnabled = false;
        BtnClear.IsEnabled = false;
        BtnStop.IsEnabled = false;
        BtnStart.IsEnabled = false;
    }

    private void BtnSettings_Click(object sender, RoutedEventArgs e)
    {
        BtnStart.IsEnabled = true;
        var settingsWindow = new SettingsWindow(ConfigurationJsonHelper.GetConfiguration());
        settingsWindow.ShowDialog();
    }

    private void BtnStart_Click(object sender, RoutedEventArgs e)
    {
        _startTime = DateTime.Now;
        _cancellationTokenSource = new CancellationTokenSource();
        var filesHashCalculator = new FilesHashCalculator(_pauseEvent);
        try
        {
            BtnStart.IsEnabled = false;
            BtnPause.IsEnabled = true;
            BtnStop.IsEnabled = true;
            // 確保訊號狀態為:“有訊號”
            _pauseEvent.Set();
            Task.Run(() =>
            {
                filesHashCalculator.DynamicParallelHashCalculation(UpdateProgress,
                    _cancellationTokenSource.Token);
            });
        }
        catch (OperationCanceledException)
        {
            MessageBox.Show("計算已停止");
        }
    }

	// 回撥方法:每當一個檔案的Hash計算完成就會執行該方法更新UI
    private void UpdateProgress(long fileNumber, string fileName, string hashResult, long totalFilesCount)
    {
        // Dispatcher.Invoke:從其他執行緒呼叫UI執行緒上的元素
        // 該方法是回撥方法,不是在UI執行緒上執行的,如果想和UI執行緒互動,必須使用Dispatcher.Invoke才可以
        Dispatcher.Invoke(() =>
        {
            _hashResults.Add(new DataGridResult()
            {
                FileName = fileName,
                FileNumber = fileNumber,
                HashResult = hashResult
            });
            ProgressBar.Value = _hashResults.Count / (double)totalFilesCount * 100;
            TxtTimeElapsed.Text = $"已用時間: {(DateTime.Now - _startTime).TotalSeconds}s";
            TxtTimeRemaining.Text =
                $"預計剩餘時間: {(DateTime.Now - _startTime).TotalSeconds / _hashResults.Count * (totalFilesCount - _hashResults.Count)}s";
        });
    }

    private void BtnPause_Click(object sender, RoutedEventArgs e)
    {
        if (!_isPaused)
        {
            // 將事件設定為無訊號狀態,暫停操作
            _pauseEvent.Reset();
            _isPaused = true;
            MessageBox.Show("已暫停");
        }
        else
        {
            // 將事件設定為有訊號狀態,恢復操作
            _pauseEvent.Set();
            _isPaused = false;
            MessageBox.Show("已恢復");
        }
    }

    private void BtnStop_Click(object sender, RoutedEventArgs e)
    {
        _cancellationTokenSource.Cancel();
        BtnPause.IsEnabled = false;
        BtnClear.IsEnabled = true;
        _isPaused = false;
        MessageBox.Show("已停止");
    }

    private void BtnClear_Click(object sender, RoutedEventArgs e)
    {
        BtnStart.IsEnabled = true;
        _hashResults.Clear();
    }
}

public struct DataGridResult
{
    public long FileNumber { get; set; }
    public string FileName { get; set; }
    public string HashResult { get; set; }
}

主視窗中呼叫了SettingsWindow,但現在SettingsWindow這個檢視還未建立,現在建立SettingsWindow.xaml

<Window x:Class="CalculateHash.GUI.SettingsWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:local="clr-namespace:CalculateHash.GUI"
        mc:Ignorable="d"
        Title="SettingsWindow" Height="210" Width="650">
    <Grid>
        <TextBox x:Name="TxtDirectoryPath" HorizontalAlignment="Left" VerticalAlignment="Top" Width="300"
                 Margin="10,10,0,0" Text="Directory Path" />
        <TextBox x:Name="TxtConnectionString" HorizontalAlignment="Left" VerticalAlignment="Top" Width="300"
                 Margin="10,40,0,0" Text="Connection String" />
        <TextBox x:Name="TxtInitialDegree" HorizontalAlignment="Left" VerticalAlignment="Top" Width="300"
                 Margin="10,70,0,0" Text="Initial Degree" />
        <TextBox x:Name="TxtMaxDegree" HorizontalAlignment="Left" VerticalAlignment="Top" Width="300"
                 Margin="10,100,0,0" Text="Max Degree" />

        <TextBox x:Name="TxtAdjustmentInterval" HorizontalAlignment="Left" VerticalAlignment="Top" Width="300"
                 Margin="320,10,0,0" Text="Adjustment Interval" />
        <TextBox x:Name="TxtExcelFilePath" HorizontalAlignment="Left" VerticalAlignment="Top" Width="300"
                 Margin="320,40,0,0" Text="Excel File Path" />
        <TextBox x:Name="TxtTargetBatchSize" HorizontalAlignment="Left" VerticalAlignment="Top" Width="300"
                 Margin="320,70,0,0" Text="Target BatchSize" />
        <TextBox x:Name="TxtMinDegree" HorizontalAlignment="Left" VerticalAlignment="Top" Width="300"
                 Margin="320,100,0,0" Text="Min Degree" />

        <CheckBox x:Name="ChkOutputToMySql" Content="Output to MySQL" HorizontalAlignment="Left"
                  VerticalAlignment="Top" Margin="10,130,0,0" />
        <CheckBox x:Name="ChkOutputToExcel" Content="Output to Excel" HorizontalAlignment="Left"
                  VerticalAlignment="Top" Margin="170,130,0,0" />
        <Button x:Name="BtnSaveSettings" Content="Save" HorizontalAlignment="Left" VerticalAlignment="Top" Width="80"
                Margin="330,125,0,0" Click="BtnSaveSettings_Click" />
        <Button x:Name="BtnDefaultSettings" Content="Default" HorizontalAlignment="Left" VerticalAlignment="Top"
                Width="80"
                Margin="420,125,0,0" Click="BtnDefaultSettings_Click" />
        <Button x:Name="BtnClearSettings" Content="Clear" HorizontalAlignment="Left" VerticalAlignment="Top"
                Width="80" Margin="510,125,0,0" Click="BtnClearSettings_Click" />
        <Button x:Name="BtnSelectFolder" Content="選擇資料夾" HorizontalAlignment="Left" VerticalAlignment="Top" Width="100"
                Margin="250,170,0,0" Click="BtnSelectFolder_Click" />
    </Grid>
</Window>

SettingWindow.xaml對應的程式碼如下:

using System.IO;
using System.Windows;
using CalculateHash.Core;
using Microsoft.Win32;
using Newtonsoft.Json;

namespace CalculateHash.GUI;

public partial class SettingsWindow : Window
{
    private readonly Configuration _configuration;

    public SettingsWindow(Configuration configuration)
    {
        InitializeComponent();
        _configuration = configuration;
    }

    private void BtnSaveSettings_Click(object sender, RoutedEventArgs e)
    {
        try
        {
            _configuration.DirectoryPath = TxtDirectoryPath.Text;
            _configuration.HashResultOutputConfig.ConnectionString = TxtConnectionString.Text;
            _configuration.HashResultOutputConfig.ExcelFilePath = TxtExcelFilePath.Text;
            _configuration.HashResultOutputConfig.OutputToMySql = ChkOutputToMySql.IsChecked == true;
            _configuration.HashResultOutputConfig.OutputToExcel = ChkOutputToExcel.IsChecked == true;
            _configuration.ParallelHashCalculateConfig.InitialDegreeOfParallelism = int.Parse(TxtInitialDegree.Text);
            _configuration.ParallelHashCalculateConfig.MaxDegreeOfParallelism = int.Parse(TxtMaxDegree.Text);
            _configuration.ParallelHashCalculateConfig.MinDegreeOfParallelism = int.Parse(TxtMinDegree.Text);
            _configuration.ParallelHashCalculateConfig.AdjustmentInterval_Seconds =
                int.Parse(TxtAdjustmentInterval.Text);
            _configuration.ParallelHashCalculateConfig.TargetBatchSize_MByte = int.Parse(TxtTargetBatchSize.Text);

            var jsonContent = JsonConvert.SerializeObject(_configuration);
            File.WriteAllText(ConfigurationJsonHelper.GetFullPathOfJsonConfiguration(), jsonContent);
            MessageBox.Show("Settings saved successfully.");
            Close();
        }
        catch (Exception ex)
        {
            MessageBox.Show($"Error saving settings: {ex.Message}");
        }
    }

    private void BtnDefaultSettings_Click(object sender, RoutedEventArgs e)
    {
        var defaultConfiguration = ConfigurationJsonHelper.GetConfiguration("./configuration_default.json");

        TxtDirectoryPath.Text = defaultConfiguration.DirectoryPath;
        TxtConnectionString.Text = defaultConfiguration.HashResultOutputConfig.ConnectionString;
        TxtExcelFilePath.Text = defaultConfiguration.HashResultOutputConfig.ExcelFilePath;
        ChkOutputToMySql.IsChecked = defaultConfiguration.HashResultOutputConfig.OutputToMySql;
        ChkOutputToExcel.IsChecked = defaultConfiguration.HashResultOutputConfig.OutputToExcel;
        TxtInitialDegree.Text = defaultConfiguration.ParallelHashCalculateConfig.InitialDegreeOfParallelism.ToString();
        TxtMaxDegree.Text = defaultConfiguration.ParallelHashCalculateConfig.MaxDegreeOfParallelism.ToString();
        TxtMinDegree.Text = defaultConfiguration.ParallelHashCalculateConfig.MinDegreeOfParallelism.ToString();
        TxtAdjustmentInterval.Text =
            defaultConfiguration.ParallelHashCalculateConfig.AdjustmentInterval_Seconds.ToString();
        TxtTargetBatchSize.Text = defaultConfiguration.ParallelHashCalculateConfig.TargetBatchSize_MByte.ToString();
    }

    private void BtnClearSettings_Click(object sender, RoutedEventArgs e)
    {
        TxtDirectoryPath.Text = "";
        TxtConnectionString.Text = "";
        TxtExcelFilePath.Text = "";
        ChkOutputToMySql.IsChecked = false;
        ChkOutputToExcel.IsChecked = false;
        TxtInitialDegree.Text = "";
        TxtMaxDegree.Text = "";
        TxtMinDegree.Text = "";
        TxtAdjustmentInterval.Text = "";
        TxtTargetBatchSize.Text = "";
    }

    private void BtnSelectFolder_Click(object sender, RoutedEventArgs e)
    {
        var dialog = new OpenFileDialog
        {
            CheckFileExists = false,
            CheckPathExists = true,
            ValidateNames = false,
            FileName = "Folder Selection."
        };
        if (dialog.ShowDialog() != true) return;

        var directoryPath = Path.GetDirectoryName(dialog.FileName);
        TxtDirectoryPath.Text = directoryPath ?? "NULL";
    }
}

由於用到了預設設定這一選項,所以還要再編寫一個configuration_default.json,這個configuration_default.json和configuration.json中的內容一致:

{
  "DirectoryPath": "D:\\pictures",
  "HashResultOutputConfig": {
    "ConnectionString": "Data Source=sql.nas.home;Database=LearnAspDotnet_CalculateHash;User ID=root;Password=Aa123456+;pooling=true;port=3306;CharSet=utf8mb4",
    "ExcelFilePath": "D:\\hashResult.xlsx",
    "OutputToMySql": false,
    "OutputToExcel": true
  },
  "ParallelHashCalculateConfig": {
    "InitialDegreeOfParallelism": 4,
    "MaxDegreeOfParallelism": 32,
    "MinDegreeOfParallelism": 1,
    "AdjustmentInterval_Seconds": 5,
    "TargetBatchSize_MByte": 100,
    "FileCountPreAdjust": 5,
    "AdjustByFileSize": true
  }
}

至此,這一專案就完成了。

相關文章