高效能分散式記憶體佇列系統beanstalkd(轉)

zchd發表於2014-05-14
分散式記憶體佇列Bean

beanstalkd 一個高效能、輕量級的分散式記憶體佇列系統，最初設計的目的是想通過後臺非同步執行耗時的任務來降低高容量Web應用系統的頁面訪問延遲,支援過有9.5 million使用者的Facebook Causes應用。後來開源，之後有PostRank大規模部署和使用,每天處理百萬級任務.

安裝略了PHP客戶端可使用pheanstalk. 可以去github上在看

require_once(`pheanstalk/pheanstalk_init.php`);
$pheanstalk = new Pheanstalk(`127.0.0.1:11300`);

// Pushing things into the queue
for($i=0; $i<1000; $i++) {   $job = new stdClass();   $job->envelope_id = rand();
  $job->date = date(`Y-m-d H:i:s`);
  $job_data = json_encode($job);
  $pheanstalk->useTube(`test`)->put($job_data);
  echo "pushed: " . $job_data . "
";
}

server.php 

class Worker {

  private $path;

  public function __construct($path) {
    $this->setBasePath($path);
    $this->log(`starting`);
    require_once(`pheanstalk/pheanstalk_init.php`);
    $this->pheanstalk = new Pheanstalk(`127.0.0.1:11300`);
  }

  public function __destruct() {
    $this->log(`ending`);
  }

  private function setBasePath($path) {
    $this->path = $path;
  }

  public function run() {
    $this->log(`starting to run`);
    $cnt = 0;
    $done_jobs = array();

    while(1) {
      $job = $this->pheanstalk->watch(`test`)->ignore(`default`)->reserve();
      $job_encoded = json_decode($job->getData(), false);
      $done_jobs[] = $job_encoded;
      $this->log(`job:`.print_r($job_encoded, 1));
      $this->pheanstalk->delete($job);
      $cnt++;

      $memory = memory_get_usage();

      $this->log(`memory:` . $memory);

      if($memory > 1000000) {
        $this->log(`exiting run due to memory limit`);
        exit;
      }

      usleep(10);
    }
  }

  private function log($txt) {
    file_put_contents($this->path . `/log/worker.txt`, $txt . "
", FILE_APPEND);
  }
}

Picking up things from the queue
$worker = new Worker(dirname($argv[0]));
$worker->run();

注:
1)可使用supervisor或deamontools等將php worker.php變為守護程式.
2)其它語言類庫參考

worker端不一定用php來寫,可以用Python 或是 nodejs ,go寫,都比PHP強


相關文章