WPF下使用FreeRedis操作RedisStream實現簡單的訊息佇列

Redis Stream簡介

Redis Stream是隨著5.0版本釋出的一種新的Redis資料型別：

高效消費者組：允許多個消費者組從同一資料流的不同部分消費資料，每個消費者組都能獨立地處理訊息，這樣可以並行處理和提高效率。

阻塞操作：消費者可以設定阻塞操作，這樣它們會在流中有新資料新增時被喚醒並開始處理，這有助於減少資源消耗並提高響應速度。

資料持久化：它可以將資料持久化到記憶體（配置本地持久化後會寫入到儲存裝置）中進行儲存，等待消費。

多生產者多消費者：Redis Streams能夠在多個生產者和消費者之間建立一個資料通道，使得資料的流動和處理更加靈活。

擴充套件性和非同步通訊：使用者可以透過應用程式輕鬆擴充套件消費者數量，並且生產者和消費者之間的通訊可以是非同步的，這有助於提高系統的整體效能。

滿足多樣化需求：Redis Streams滿足從實時資料處理到歷史資料訪問的各種需求，同時保持易於管理。

Redis Stream可以幹什麼

訊息佇列：Redis Stream可以用作一個可靠的訊息佇列系統，支援釋出/訂閱模式，生產者和消費者可以非同步地傳送和接收訊息。

任務排程：Redis Stream可以用於實現分散式任務排程系統，將任務分發到多個消費者進行處理，從而提高處理速度和系統可擴充套件性。

事件驅動架構：Redis Stream可以作為事件驅動架構中的核心元件，用於處理來自不同服務的事件，實現解耦和靈活性。

FreeRedis簡介

FreeRedis 的命名來自，“自由”、“免費”，它和名字與 FreeSql 是一個理念，簡易是他們一致的追尋方向，最低可支援 .NET Framework 4.0 執行環境，支援到 Redis-server 7.2。

github MIT開源協議

作者部落格園地址

官方介紹

基於 .NET 的 Redis 客戶端，支援 .NET Core 2.1+、.NET Framework 4.0+、Xamarin 以及 AOT。

• 🌈 所有方法名與 redis-cli 保持一致
• 🌌 支援 Redis 叢集（服務端要求 3.2 及以上版本）
• ⛳ 支援 Redis 哨兵模式
• 🎣 支援主從分離（Master-Slave）
• 📡 支援釋出訂閱（Pub-Sub）
• 📃 支援 Redis Lua 指令碼
• 💻 支援管道（Pipeline）
• 📰 支援事務
• 🌴 支援 GEO 命令（服務端要求 3.2 及以上版本）
• 🌲 支援 STREAM 型別命令（服務端要求 5.0 及以上版本）
• ⚡ 支援本地快取（Client-side-cahing，服務端要求 6.0 及以上版本）
• 🌳 支援 Redis 6 的 RESP3 協議

要實現的功能

1、生產者生產資料

2、消費者消費資料後確認

3、消費者消費資料後不確認

4、已消費但超時未確認的訊息監控

5、已消費但超時未確認的訊息二次消費

專案依賴

WPF
CommunityToolkit.Mvvm
FreeRedis
Newtonsoft.Json
NLog
redis-windows-7.2.5

業務場景程式碼

涉及到的Redis命令

建立消費者組 XGROUP CREATE key group <id | $> [MKSTREAM] [ENTRIESREAD entries-read]

查詢消費者組資訊 XINFO STREAM key [FULL [COUNT count]]

訊息佇列數量（長度） XLEN key

新增訊息到佇列尾部 XADD key [NOMKSTREAM] [<MAXLEN | MINID> [= | ~] threshold [LIMIT count]] <* | id> field value [field value ...]

消費組成員讀取訊息 XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] id [id ...]

確認訊息 XACK key group id [id ...]

刪除訊息 XDEL key id [id ...]

獲取消費未確認訊息的佇列資訊 XPENDING key group [[IDLE min-idle-time] start end count [consumer]]

把未消費的訊息消費者轉移到當前消費者名下 XAUTOCLAIM key group consumer min-idle-time start [COUNT count] [JUSTID]

程式碼

App.xaml.cs

    public partial class App : Application
    {
        public static Logger Logger = LogManager.GetCurrentClassLogger();        
        public static RedisClient RedisHelper;
        public static MainViewModel MainViewModel;
        private void App_OnStartup(object sender, StartupEventArgs e)
        {
            Current.DispatcherUnhandledException += Current_DispatcherUnhandledException;

            try
            {
                //redis6以上版本啟用了ACL使用者管理機制，預設使用者名稱是default，可以忽略密碼
                RedisHelper = new RedisClient("127.0.0.1:6379,user=defualt,defaultDatabase=13");
                RedisHelper.Serialize = JsonConvert.SerializeObject;//序列化
                RedisHelper.Deserialize = JsonConvert.DeserializeObject;//反序列化
                RedisHelper.Notice += (s, ee) => Console.WriteLine(ee.Log); //列印命令日誌
                MainViewModel = new MainViewModel();
            }
            catch (Exception exception)
            {
                MessageBox.Show(exception.Message);
                Current.Shutdown(-100);
            }
        }

        private void Current_DispatcherUnhandledException(object sender, System.Windows.Threading.DispatcherUnhandledExceptionEventArgs e)
        {
            e.Handled = true;
            Logger.Error($"未捕獲的錯誤：來源:{sender},錯誤：{e}");
        }

        private void App_OnExit(object sender, ExitEventArgs e)
        {
            RedisHelper.Dispose();
        }
    }

MainWindow.xaml的主要內容

    <StackPanel>
        <WrapPanel ItemHeight="40" Margin="10,10,0,0" VerticalAlignment="Center">
            <TextBlock Text="生產資料數：" VerticalAlignment="Center"></TextBlock>
            <TextBox Text="{Binding RecordCount}" Width="100" MaxLength="9" VerticalAlignment="Center"></TextBox>

            <TextBlock Text="生產者數量：" Margin="10,0,0,0" VerticalAlignment="Center"></TextBlock>
            <TextBox Text="{Binding TaskCount}" Width="100" MaxLength="6" VerticalAlignment="Center"></TextBox>

            <TextBlock Margin="10,0,0,0" VerticalAlignment="Center">
                <Run Text="正在產生第："></Run>
                <Run Text="{Binding ProducerIndex}"></Run>
                <Run Text="條資料"></Run>
            </TextBlock>

            <Button Content="生成資料" HorizontalAlignment="Left" Margin="10,0,0,0" VerticalAlignment="Center" Width="103" Command="{Binding ProducerRelayCommand}" />
        </WrapPanel>

        <WrapPanel ItemHeight="40" Margin="0,10,0,0" VerticalAlignment="Center">
            <TextBlock Text="消費者數量：" Margin="10,0,0,0" VerticalAlignment="Center"></TextBlock>
            <TextBox Text="{Binding ConsumerCount}" Width="100" MaxLength="6" VerticalAlignment="Center"></TextBox>

            <TextBlock Margin="10,0,0,0" VerticalAlignment="Center">
                <Run Text="剩餘未消費："></Run>
                <Run Text="{Binding ConsumeIndex}"></Run>
                <Run Text="條資料。"></Run>
            </TextBlock>
            <CheckBox IsChecked="{Binding IsAutoAck,Mode=TwoWay}" Content="自動確認" VerticalAlignment="Center" ></CheckBox>
            <Button Content="開始消費" HorizontalAlignment="Left" Margin="10,0,0,0" VerticalAlignment="Center" Width="103" Command="{Binding ConsumeRelayCommand}" />
            <Button Content="消費未確認消費佇列" HorizontalAlignment="Left" Margin="10,0,0,0" VerticalAlignment="Center" Width="103" Command="{Binding PendingRelayCommand}" />
        </WrapPanel>

        <WrapPanel>
            <StackPanel>
                <TextBlock Margin="10,0,0,0" Text="佇列資訊：" VerticalAlignment="Center"></TextBlock>
                <TextBox Margin="10,0,0,0" VerticalAlignment="Center" Height="310" Width="250" Text="{Binding StreamInfo}" VerticalScrollBarVisibility="Auto"></TextBox>
            </StackPanel>

            <StackPanel>
                <TextBlock Margin="13,0,0,0" Text="消費資訊：" VerticalAlignment="Center"></TextBlock>
                <TextBox Margin="13,0,0,0" VerticalAlignment="Center" Height="310" Width="250" Text="{Binding ConsumeInfo}" VerticalScrollBarVisibility="Auto" TextWrapping="WrapWithOverflow"></TextBox>
            </StackPanel>

            <StackPanel>
                <TextBlock Margin="13,0,0,0" Text="未確認消費資訊：" VerticalAlignment="Center"></TextBlock>
                <TextBox Margin="13,0,0,0" VerticalAlignment="Center" Height="310" Width="250" Text="{Binding PendingInfo}" VerticalScrollBarVisibility="Auto" TextWrapping="WrapWithOverflow"></TextBox>
            </StackPanel>

            <StackPanel>
                <TextBlock Margin="13,0,0,0" Text="消費未確認資訊：" VerticalAlignment="Center"></TextBlock>
                <TextBox Margin="13,0,0,0" VerticalAlignment="Center" Height="310" Width="250" Text="{Binding PendingConsumeInfo}" VerticalScrollBarVisibility="Visible" HorizontalScrollBarVisibility="Auto" TextWrapping="WrapWithOverflow"></TextBox>
            </StackPanel>
        </WrapPanel>
    </StackPanel>
</Grid>

MainWindow.xaml.cs

	public partial class MainWindow : Window
    {
        public MainWindow()
        {
            InitializeComponent();
            DataContext = App.MainViewModel;
        }

        private void MainWindow_OnLoaded(object sender, RoutedEventArgs e)
        {
            App.MainViewModel.RecordCount = 1000;
            App.MainViewModel.TaskCount = 5;
            App.MainViewModel.ConsumerCount = 1;
            App.MainViewModel.ConsumeInfo = "等待消費資訊……";
            App.MainViewModel.PendingInfo = "載入中……";
            App.MainViewModel.StreamInfo = "載入中……";
        }
    }

MainViewModel的宣告和變數定義

	public class MainViewModel : ObservableObject
	{
		#region 變數定義
		private readonly string _streamKey = "redisstream";
		private readonly string _consumeGroupName = "counsumeGroup";

		private DateTime _utcTime = new DateTime(1970, 1, 1, 0, 0, 0);
		/// <summary>
		/// 生成的訊息條數
		/// </summary>
		private static int _exchangeValue;
		/// <summary>
		/// 剩餘未消費條數
		/// </summary>
		private static int _consumeValue;
		/// <summary>
		/// 消費資訊展示佇列
		/// </summary>
		private static ConcurrentQueue<string> _consumedQueue = new ConcurrentQueue<string>();
		/// <summary>
		/// 消費未確認展示佇列
		/// </summary>
		private static ConcurrentQueue<string> _pendingConsumedQueue = new ConcurrentQueue<string>();
		/// <summary>
		/// 退出令牌
		/// </summary>
		private CancellationTokenSource _cancellationTokenSource;
		/// <summary>
		/// 生成訊息
		/// </summary>
		public RelayCommand ProducerRelayCommand { get; }
		/// <summary>
		/// 消費訊息
		/// </summary>
		public RelayCommand ConsumeRelayCommand { get; }
		/// <summary>
		/// 消費未確認資訊佇列消費
		/// </summary>
		public RelayCommand PendingRelayCommand { get; }
		private int _recordCount;
		/// <summary>
		/// 資料條數
		/// </summary>
		public int RecordCount
		{
			get => _recordCount;
			set => SetProperty(ref _recordCount, value);
		}

		private int _taskCount;
		/// <summary>
		/// 開啟後臺生產者數量
		/// </summary>
		public int TaskCount
		{
			get => _taskCount;
			set => SetProperty(ref _taskCount, value);
		}

		private int _consumerCount;
		/// <summary>
		/// 消費者數量
		/// </summary>
		public int ConsumerCount
		{
			get => _consumerCount;
			set => SetProperty(ref _consumerCount, value);
		}

		private int _producerIndex;
		/// <summary>
		/// 正在生產的序列號
		/// </summary>
		public int ProducerIndex
		{
			get => Interlocked.Exchange(ref _producerIndex, _exchangeValue);
			set
			{
				SetProperty(ref _producerIndex, _exchangeValue);
			}
		}

		private long _consumeIndex;
		/// <summary>
		/// 正在消費的序列號
		/// </summary>
		public long ConsumeIndex
		{
			get => Interlocked.Read(ref _consumeIndex);
			set => SetProperty(ref _consumeIndex, value);
		}

		private string _streamInfo;
		/// <summary>
		/// 佇列資訊展示
		/// </summary>
		public string StreamInfo
		{
			get => _streamInfo;
			set => SetProperty(ref _streamInfo, value);
		}

		private string _consumeInfo;
		/// <summary>
		/// 消費資訊展示
		/// </summary>
		public string ConsumeInfo
		{
			get => _consumeInfo;
			set
			{
				value = $"暫無消費訊息[{DateTime.Now:yyyy-MM-dd HH:mm:ss}]";
				if (_consumedQueue.TryDequeue(out var message))
				{
					value = _consumeInfo + Environment.NewLine + message;
				}
				SetProperty(ref _consumeInfo, value);
			}
		}

		private string _pendingInfo;
		/// <summary>
		/// 消費未確認佇列資訊展示
		/// </summary>
		public string PendingInfo
		{
			get => _pendingInfo;
			set => SetProperty(ref _pendingInfo, value);
		}

		private string _pedingConsumeInfo;
		/// <summary>
		/// 消費未確認佇列的展示
		/// </summary>
		public string PendingConsumeInfo
		{
			get => _pedingConsumeInfo;
			set
			{
				value = $"暫無未確認消費資訊[{DateTime.Now:yyyy-MM-dd HH:mm:ss}]";
				if (_pendingConsumedQueue.TryDequeue(out var message))
				{
					value = _pedingConsumeInfo + Environment.NewLine + message;
				}

				SetProperty(ref _pedingConsumeInfo, value);
			}
		}

		private bool _isProduceCanExec;
		/// <summary>
		/// 是否可以執行生成任務
		/// </summary>
		public bool IsProduceCanExec
		{
			get => _isProduceCanExec;
			set
			{
				SetProperty(ref _isProduceCanExec, value);
				ProducerRelayCommand.NotifyCanExecuteChanged();
			}
		}

		private bool _isAutoAck;
		/// <summary>
		/// 是否自動確認消費資訊
		/// </summary>
		public bool IsAutoAck
		{
			get => _isAutoAck;
			set
			{
				SetProperty(ref _isAutoAck, value);
				ProducerRelayCommand.NotifyCanExecuteChanged();
			}
		}
		#endregion

		public MainViewModel()
		{
			ProducerRelayCommand = new RelayCommand(async () => await DoProduce(), () => !_isProduceCanExec);

			ConsumeRelayCommand = new RelayCommand(async () => await DoConsume());

			PendingRelayCommand = new RelayCommand(async () => await DoPendingConsume());

			_cancellationTokenSource = new CancellationTokenSource();

			var exist = App.RedisHelper.Exists(_streamKey);
			if (!exist)
			{
				//建立消費組，同一個消費組可以有多個消費者，它們直接不會重複讀取到同一條訊息
				App.RedisHelper.XGroupCreate(_streamKey, _consumeGroupName, MkStream: true);
			}
			else
			{
				var groups = App.RedisHelper.XInfoGroups(_streamKey);

				if (groups == null || !groups.Any())
				{
					App.RedisHelper.XGroupCreate(_streamKey, _consumeGroupName);
				}
			}

			ConsumeIndex = App.RedisHelper.XLen(_streamKey);

			DoLoadStreamInfo();
			DoLoadPendingInfo();
		}
	}

訊息生成者

private async Task DoProduce()
{
	if (IsProduceCanExec)
	{
		return;
	}
	IsProduceCanExec = true;

	if (TaskCount > RecordCount)
	{
		TaskCount = RecordCount;
	}

	_exchangeValue = 0;
	ProducerIndex = 0;

	if (TaskCount > 0)
	{
		var pageSize = RecordCount / TaskCount;
		var tasks = Enumerable.Range(1, TaskCount).Select(x =>
		{
			return Task.Run(() =>
			{
				var internalPageSize = pageSize;

				if (TaskCount > 1 && x == TaskCount)
				{
					if (x * internalPageSize < RecordCount)
					{
						internalPageSize = RecordCount - (TaskCount - 1) * internalPageSize;
					}
				}
				
				for (var i = 1; i <= internalPageSize; i++)
				{
					ProducerIndex = Interlocked.Increment(ref _exchangeValue);
					ConsumeIndex = Interlocked.Increment(ref _consumeValue);

					var dic = new Dictionary<string, MessageModel> { { $"user_{x}", new MessageModel { Age = 16, Description = $"描述:{ProducerIndex}", Id = 1, Name = "wang", Status = 1 } } };
					App.RedisHelper.XAdd(_streamKey, 0, "*", dic);
				}
				return Task.CompletedTask;
			});
		});
		await Task.WhenAll(tasks);
	}
	IsProduceCanExec = false;
}

#endregion

訊息消費者

private Task DoConsume()
{
	var groups = App.RedisHelper.XInfoGroups(_streamKey);
	if (groups == null || !groups.Any())
	{
		App.RedisHelper.XGroupCreate(_streamKey, _consumeGroupName);
	}

	//新增消費者
	var tasks = Enumerable.Range(1, _consumerCount).Select(c =>
	{
		var task = Task.Run(async () =>
		{
			//從消費組中讀取訊息，同一個組內的成員不會重複獲取同一條訊息。
			var streamRead = App.RedisHelper.XReadGroup(_consumeGroupName, $"consumer{c}", 0, _streamKey, ">");
			if (streamRead != null)
			{
				//取得訊息
				var id = streamRead.id;
				var model = new Dictionary<string, MessageModel>(1)
				{
					{ streamRead.fieldValues[0].ToString(), JsonConvert.DeserializeObject<MessageModel>(streamRead.fieldValues[1].ToString()) }
				};
				_consumedQueue.Enqueue($"consumer{c}取到了訊息{id},{DateTime.Now:yyyy-MM-dd HH:mm:ss}");
				ConsumeInfo = "";
				await Task.Delay(100);//模擬業務邏輯耗時
				
				if (IsAutoAck)
				{
					//ACK
					var success = App.RedisHelper.XAck(_streamKey, _consumeGroupName, id);
					if (success > 0)
					{
						//xdel
						App.RedisHelper.XDel(_streamKey, id);
						_consumedQueue.Enqueue($"consumer{c}成功消費了訊息{id},{DateTime.Now:yyyy-MM-dd HH:mm:ss}");
					}
					else
					{
						_consumedQueue.Enqueue($"consumer{c}的訊息{id}加入了未確認佇列,{DateTime.Now:yyyy-MM-dd HH:mm:ss}");
					}
					ConsumeInfo = "";
				}
			}
		});
		return task;
	});
	Task.WhenAll(tasks);

	return Task.CompletedTask;
}
#endregion

未消費資訊佇列監控

private Task DoLoadStreamInfo()
{
	Task.Factory.StartNew(async () =>
	{
		while (!_cancellationTokenSource.IsCancellationRequested)
		{
			StreamInfo = "正在查詢佇列資訊……";
			try
			{
				if (!App.RedisHelper.Exists(_streamKey))
				{
					StreamInfo = "佇列尚未建立";
					await Task.Delay(3000);
					continue;
				}
				var info = App.RedisHelper.XInfoStream(_streamKey);
				StreamInfo = info?.first_entry == null ? "佇列資料為空。" : $"佇列長度：{info.length}{Environment.NewLine}第一個編號：{info.first_entry.id}{Environment.NewLine}最後一個編號：{info.last_entry.id}{Environment.NewLine}更新時間：{DateTime.Now:yyyy-MM-dd HH:mm:ss}";
				ConsumeIndex = info?.first_entry == null ? 0 : (int)info.length;
			}
			catch (Exception e)
			{
				StreamInfo = $"獲取佇列資訊失敗：{e.Message}";
			}
			await Task.Delay(3000);
		}
	}, _cancellationTokenSource.Token, TaskCreationOptions.LongRunning, TaskScheduler.Default);
	return Task.CompletedTask;
}
#endregion

未消費成功（超時或業務邏輯執行失敗）的訊息佇列訊息展示

private Task DoLoadPendingInfo()
{
	Task.Run(async () =>
	{
		while (!_cancellationTokenSource.IsCancellationRequested)
		{
			PendingInfo = "正在查詢未確認佇列……";
			if (!App.RedisHelper.Exists(_streamKey))
			{
				PendingInfo = "佇列尚未建立";
				await Task.Delay(3000);
				continue;
			}
			try
			{
				var info = App.RedisHelper.XPending(_streamKey, _consumeGroupName);
				if (info == null || info.count == 0)
				{
					PendingInfo = $"暫無未確認資訊。[{DateTime.Now:yyyy-MM-dd HH:mm:ss}]";
					await Task.Delay(3000);

					continue;
				}
				var infoTxt = $"未消費數量：{info.count}{Environment.NewLine}涉及{info.consumers.Length}個消費者{Environment.NewLine}最小編號：{info.minId}{Environment.NewLine}最大編號：{info.maxId}{Environment.NewLine}更新時間：[{DateTime.Now:yyyy-MM-dd HH:mm:ss}]";
				PendingInfo = infoTxt;
			}
			catch (Exception e)
			{
				PendingInfo = $"獲取佇列資訊失敗：{e.Message}";
			}

			await Task.Delay(3000);
		}
	}, _cancellationTokenSource.Token);
	return Task.CompletedTask;
}
#endregion

未消費成功的訊息重新消費

private Task DoPendingConsume()
{
	Task.Run(async () =>
	{
		while (!_cancellationTokenSource.IsCancellationRequested)
		{
			_pendingConsumedQueue.Enqueue($"正在查詢未確認佇列……[{DateTime.Now:yyyy-MM-dd HH:mm:ss}]");
			PendingConsumeInfo = "";
			if (!App.RedisHelper.Exists(_streamKey))
			{
				_pendingConsumedQueue.Enqueue($"佇列尚未建立……[{DateTime.Now:yyyy-MM-dd HH:mm:ss}]");
				PendingConsumeInfo = "";
				await Task.Delay(3000);
				continue;
			}
			try
			{
				//從stream佇列的頭部（0-0的位置）獲取2條已讀取時間超過2分鐘且未確認的訊息，修改所有者為pendingUser重新消費並確認。
				var info = App.RedisHelper.XAutoClaim(_streamKey, _consumeGroupName, "pendingUser", 120000, "0-0", 2);
				if (info == null || info.entries == null || info.entries.Length == 0)
				{
					_pendingConsumedQueue.Enqueue("未確認佇列中暫無資訊。[{DateTime.Now:yyyy-MM-dd HH:mm:ss}]");
					await Task.Delay(3000);
					continue;
				}

				foreach (var entry in info.entries)
				{
					if (entry == null) continue;
					_pendingConsumedQueue.Enqueue($"未確認消費資訊：{entry.id}[{DateTime.Now:yyyy-MM-dd HH:mm:ss}]");

					Debug.WriteLine(JsonConvert.DeserializeObject<MessageModel>(entry.fieldValues[1].ToString()));
					PendingConsumeInfo = "";
					//ACK
					await Task.Delay(100);//模擬業務邏輯執行時間
					var success = App.RedisHelper.XAck(_streamKey, _consumeGroupName, entry.id);
					if (success > 0)
					{
						//xdel
						if (App.RedisHelper.XDel(_streamKey, entry.id) > 0)
						{
							_pendingConsumedQueue.Enqueue($"consumer[pendingUser]成功消費了訊息{entry.id},{DateTime.Now:yyyy-MM-dd HH:mm:ss}");
						}
						else
						{
							_pendingConsumedQueue.Enqueue($"[pendingUser]刪除{entry.id}[失敗],{DateTime.Now:yyyy-MM-dd HH:mm:ss}");
						}
					}
					else
					{
						_pendingConsumedQueue.Enqueue($"[pendingUser]消費{entry.id}[失敗],{DateTime.Now:yyyy-MM-dd HH:mm:ss}");
					}
				}
			}
			catch (Exception e)
			{
				PendingConsumeInfo = $"獲取佇列資訊失敗：{e.Message}";
			}
			PendingConsumeInfo = "";

			await Task.Delay(3000);
		}
	}, _cancellationTokenSource.Token);
	return Task.CompletedTask;
}

#endregion

總結

本次我們透過Redis的Stream資料型別實現了部署簡單、高效能、高可用性的訊息佇列，在中小型專案上可適用於需要處理資料流轉的場景。

參考資料

① Redis Streams

②Redis Commands