[玩具程式碼]swoole多程式匯出大量資料

minororange發表於2020-10-13

說明

針對大量資料匯出採用 Swoole 多程式進行 csv 檔案寫入，程式碼僅供參考，此方法匯出不能保證資料的順序性。

控制程式碼類

class FileHandler
{
    private $fp;

    public function __construct(string $filePath)
    {
        $this->setFp($filePath);
    }

    /**
     * @return resource|false
     */
    public function getFp()
    {
        return $this->fp;
    }

    /**
     * @param string $filePath
     */
    public function setFp(string $filePath): void
    {
        $this->fp = fopen($filePath, 'w');
    }

    /**
     * @param array $row
     * @return false|int
     * @date 2020/10/13
     */
    public function putCsv(array $row)
    {
        return fputcsv($this->getFp(), $row);
    }
}

Swoole 匯出類

class SwooleExporter
{
    /**
     * 每個程式處理的資料條數
     */
    const PER_PROCESS_LIMIT = 1000;

    /**
     * @var FileHandler
     */
    private $writer;

    /**
     * SwooleExporter constructor.
     * @param string $filePath
     */
    public function __construct(string $filePath)
    {
        $this->writer = new FileHandler($filePath);
    }

    /**
     * @param array $head
     * @return SwooleExporter
     * @date 2020/10/13
     */
    public function writeHead(array $head)
    {
        $this->writer->putCsv($head);
        return $this;
    }


    /**
     * @param int $count
     * @param \Closure $write
     * @date 2020/10/13
     */
    public function writeBody(int $count, \Closure $write)
    {
        $groupQty = ceil($count / static::PER_PROCESS_LIMIT);
        for ($i = 0; $i < $groupQty; $i++) {
            $process = new \swoole_process(
                function (\swoole_process $worker) use ($write, $i) {
                    $worker->write($write($i, static::PER_PROCESS_LIMIT, $this->writer));
            }, true);
            $process->start();

            // 將子程式弄成非同步
            \swoole_event_add($process->pipe, function ($pipe) use ($process, $i) {
                echo "{$i} process start\n";
                echo $process->read() . PHP_EOL;

                swoole_event_del($pipe);//socket處理完成後，從epoll事件中移除管道
            });
        }

        while ($swooleResult = \swoole_process::wait()) {
            $pid = $swooleResult['pid'];

            echo "process {$pid} exit\n";
        }
    }
}

外部呼叫

$swooleExporter = new SwooleExporter('test.csv');
$swooleExporter->writeHead(['A1'])
    ->writeBody(10000, function ($currentPage, $pageSize, FileHandler $writer) {
        // 模擬資料庫查詢
        sleep(1);
        // 根據 $currentPage,$pageSize 進行資料庫分頁查詢
        // 查詢完畢後遍歷資料寫入
        for ($i = 0; $i < 1000; $i++) {
            $writer->putCsv(['data' . ($currentPage * $pageSize + $i)]);
        }

        return $currentPage . ' success';
    });
本作品採用《CC 協議》，轉載必須註明作者和本文連結

相關文章