cephmonitor功能的相容性管理

fibercode發表於2017-06-08

軟體需要相容舊版本

ceph是在一個不斷進化的軟體,會增加功能也會取消舊的功能,如何在ceph monitor的不同版本間保持相容或者防止不相容而產生錯誤,是需要認真思考的問題。而我們閱讀程式碼時往往忽略了相容性管理的程式碼,而重點關心它的業務程式碼,所以這裡特別寫一篇文章談談相容性的管理。

ceph monitor相容主要包括兩方面:

  • 通訊時檢查對方的相容性
  • 程式碼訪問本機資料時的相容性

Messenger的建立和功能位features的設定

ceph使用原生的Messenger的方式來通訊,在通訊開始前需要建立一個Messenger, 例如ceph monitor建立monitor之間通訊的messenger:

Messenger *msgr = Messenger::create(g_ceph_context, g_conf->ms_type,
                       entity_name_t::MON(rank),
                       "mon",
                       0);

ceph在建立連線時會告訴對方自己是什麼型別的節點,主要有幾種型別: monitor, osd, mds, client,上面的程式碼就說明自己是一個Monitor,
而任何一個Messenger都會準對某一種節點設定一個Policy, Policy的內容與相容有關的是:

 /// Specify features supported locally by the endpoint.
uint64_t features_supported;
/// Specify features any remotes must have to talk to this endpoint.
int64_t features_required;

其中features_support表示本節點支援的功能,features_required表示對方必須具備的功能,每個功能一個bit位。預設的Policy把featuires_supported設定成當前程式碼支援的全部功能。即:CEPH_FEATURES_ALL。

而ceph為monitor之間設定的Policy:

msgr->set_policy(entity_name_t::TYPE_MON,
                    Messenger::Policy::lossless_peer_reuse(                      
                        supported,                                               
                        CEPH_FEATURE_UID |                                       
                        CEPH_FEATURE_MON_SINGLE_PAXOS)); 

初始設定的features_required僅僅包含了CEPH_FEATUIRE_UID和CEPH_FEATURE_MON_SINGLE_PAXOS,而features_supported則是全部功能, monitor之間的相容性是在隨後的通訊過程中逐漸被檢測的。

通訊時的相容性檢查

在連線建立時就檢查對方的功能位設定,在ceph messenger通訊協議中,雙方提供給對方支援的功能集,並且與本地Policy中設定的必需的功能位進行比較,例如:

ceph的simple messenger中,connect發起方會提供supported features:

    while (1) {
         delete authorizer;                                                          
         authorizer = msgr->get_authorizer(peer_type, false);                        
         bufferlist authorizer_reply;                                                
                                                                                 
         ceph_msg_connect connect;                                                   
         connect.features = policy.features_supported;   

當接收到應答後,又會驗證對方支援的功能:

     if (reply.tag == CEPH_MSGR_TAG_READY ||
        reply.tag == CEPH_MSGR_TAG_SEQ) {                                       
        uint64_t feat_missing = policy.features_required & ~(uint64_t)reply.features;
        if (feat_missing) {                                                       
             ldout(msgr->cct,1) << "missing required features " << std::hex << feat_missing << std::dec << dendl;
             goto fail_locked;                                                           
       } 

檢視reply的feature與本地必需的功能,如果缺少就會失敗。

一些內部相容性的表示方法

ceph使用一個CompatSet的資料結構來表示功能集合:

    struct CompatSet {
        struct Feature {
            uint64_t id;
            string name;
    
            Feature(uint64_t _id, const char *_name) : id(_id), name(_name) {}
            Feature(uint64_t _id, const string& _name) : id(_id), name(_name) {}
        };
    
        struct FeatureSet {
            uint64_t mask;                                                              
            map <uint64_t,string> names;  
        };

        FeatureSet compat;
          FeatureSet ro_compat; 
          FeatureSet incompat;
    };

mask中的每一位代表代表一個功能, 相容測試主要判斷是否可讀可寫。
測試是否可讀是通過readable成員函式來實現:

bool readable(CompatSet const& other) const {
    return !((other.incompat.mask ^ incompat.mask) & other.incompat.mask);
}

這個函式的意思是如果我的incompat不能全部包含對方的位域,我就無法讀取對方資料。

測試是否可寫是用writable成員函式來實現:

bool writeable(CompatSet const& other) const {
   return readable(other) &&
        !((other.ro_compat.mask ^ ro_compat.mask) & other.ro_compat.mask);        
}

這個函式的意思就是:除了readable,我在ro_compat全部的位域包含了對方的位域才能寫資料。

ceph monitor的內部相容的保護

  • 每個Monitor被建立時,都需要初始化本地資料,其中mkfs函式是被呼叫的重要一環。首先Monitor::mkfs會在本地寫入一個相容性集合,用以記錄用當前程式碼生成這些資料庫的時候,具備什麼功能:
int Monitor::mkfs()
{
   MonitorDBStore::TransactionRef t(new MonitorDBStore::Transaction);
                                                                                 
   // verify cluster fsid
   int r = check_fsid();
   if (r < 0 && r != -ENOENT)
     return r;
                                                                                 
   bufferlist magicbl;
   magicbl.append(CEPH_MON_ONDISK_MAGIC);
   magicbl.append("
");
   t->put(MONITOR_NAME, "magic", magicbl);
                                                                                                                                                        
   features = get_initial_supported_features();    <<<<==============
   write_features(t); 
  • 當Monitor啟動時,程式碼檢查本地檔案系統上的資料結構是否相容當前程式碼,注意因為本地檔案不被修改的情況下,
    ceph程式依然可以被升級或者用其他方法替換,所以程式啟動檢查本地資料是否相容是必須的, ceph_mon.cc會呼叫check_features(),它檢查本地資料檔案格式是否和當前程式碼相容:
int Monitor::check_features(MonitorDBStore *store)                              
{                                                                               
   CompatSet required = get_supported_features();                                
   CompatSet ondisk;                                                             
                                                                                  
    read_features_off_disk(store, &ondisk);                                       
                                                                                  
    if (!required.writeable(ondisk)) {                                            
      CompatSet diff = required.unsupported(ondisk);                              
      generic_derr << "ERROR: on disk data includes unsupported features: " << diff << dendl;
      return -EPERM;                                                              
    }                                                                             
                                                                                  
    return 0;                                                                     
}               
 

而get_supported_features()就是當前Monitor程式碼能支援的所有功能,read_features_off_disk()則把write_features()的資料讀出來,我們看到它用writable()測試當前程式碼是否有能力可以寫本地檔案系統上的資料。

read_features_off_disk讀出write_feature()生成的資料:

void Monitor::read_features_off_disk(MonitorDBStore *store, CompatSet *features)
{                                                                               
  bufferlist featuresbl;                                                        
  store->get(MONITOR_NAME, COMPAT_SET_LOC, featuresbl);                         
  if (featuresbl.length() == 0) {                                               
    generic_dout(0) << "WARNING: mon fs missing feature list.
"                
            << "Assuming it is old-style and introducing one." << dendl;        
    //we only want the baseline ~v.18 features assumed to be on disk.           
    //If new features are introduced this code needs to disappear or            
    //be made smarter.                                                          
    *features = get_legacy_features();                                          
                                                                                
    bufferlist bl;                                                              
    features->encode(bl);                                                       
    MonitorDBStore::TransactionRef t(new MonitorDBStore::Transaction);          
    t->put(MONITOR_NAME, COMPAT_SET_LOC, bl);                                   
    store->apply_transaction(t);                                                
  } else {                                                                      
    bufferlist::iterator it = featuresbl.begin();                               
    features->decode(it);                                                       
  }                                                                             
}                                                  

作為一種特殊情況,如果資料是舊版的ceph monitor生成的,因為舊版沒有寫features到本地檔案系統,所以read_features_off_disk會呼叫get_legacy_features()函式得到舊版本ceph monitor的功能集,這只是一個簡單的構造:

CompatSet Monitor::get_legacy_features()                                        
{                                                                               
   CompatSet::FeatureSet ceph_mon_feature_compat;                                
   CompatSet::FeatureSet ceph_mon_feature_ro_compat;                             
   CompatSet::FeatureSet ceph_mon_feature_incompat;                              
   ceph_mon_feature_incompat.insert(CEPH_MON_FEATURE_INCOMPAT_BASE);             
   return CompatSet(ceph_mon_feature_compat, ceph_mon_feature_ro_compat,         
            ceph_mon_feature_incompat);                                          
}  
  • Monitor執行時檢查和設定features

一旦ceph_mon.cc決定執行Monitor, 首先會呼叫成員函式preinit(), 而preinit的一項工作就時呼叫read_features()把
本地檔案中記錄的feature讀入到成員變數features中:

void Monitor::read_features()
{
   read_features_off_disk(store, &features);
   dout(10) << "features " << features << dendl;
                                                                                 
   apply_compatset_features_to_quorum_requirements();
   dout(10) << "required_features " << required_features << dendl;
 }

當然它不會忘記按照本地資料中儲存的功能位,要求monitor paxos叢集的所有法人必須有對應的功能位:

void Monitor::apply_compatset_features_to_quorum_requirements()                 
{                                                                               
  required_features = 0;                                                        
  if (features.incompat.contains(CEPH_MON_FEATURE_INCOMPAT_OSD_ERASURE_CODES)) {
    required_features |= CEPH_FEATURE_OSD_ERASURE_CODES;                        
  }                                                                             
  if (features.incompat.contains(CEPH_MON_FEATURE_INCOMPAT_OSDMAP_ENC)) {       
    required_features |= CEPH_FEATURE_OSDMAP_ENC;                               
  }                                                                             
  if (features.incompat.contains(CEPH_MON_FEATURE_INCOMPAT_ERASURE_CODE_PLUGINS_V2)) {
    required_features |= CEPH_FEATURE_ERASURE_CODE_PLUGINS_V2;                  
  }                                                                             
  dout(10) << __func__ << " required_features " << required_features << dendl;  
}         

required_features的設定,可以防止不相容的Monitor構成一個paxos叢集,有幾個地方通過required_features阻斷這些不相容的monitor之間的通訊:

收到一個探測包,發現對方不能提供相關的功能位,則阻斷通訊:

void Monitor::handle_probe_probe(MMonProbe *m)                                  
{                                                                               
   MMonProbe *r;                                                                 
                                                                                 
   dout(10) << "handle_probe_probe " << m->get_source_inst() << *m               
        << " features " << m->get_connection()->get_features() << dendl;         
   uint64_t missing = required_features & ~m->get_connection()->get_features();  
   if (missing) {                                                                
       dout(1) << " peer " << m->get_source_addr() << " missing features "         
            << missing << dendl;                                                    
       if (m->get_connection()->has_feature(CEPH_FEATURE_OSD_PRIMARY_AFFINITY)) {  
           MMonProbe *r = new MMonProbe(monmap->fsid, MMonProbe::OP_MISSING_FEATURES,
                    name, has_ever_joined);                                      
           m->required_features = required_features;                                 
           m->get_connection()->send_message(r);                                     
     }
     goto out;
}

獲取資料複製的cookie時的,發現對方不能提供相關的功能位,阻斷通訊:

void Monitor::handle_sync_get_cookie(MMonSync *m)                               
{                                                                               
   if (is_synchronizing()) {                                                     
     _sync_reply_no_cookie(m);                                                   
     return;                                                                     
   }                                                                             
                                                                                 
   assert(g_conf->mon_sync_provider_kill_at != 1);                               
                                                                                 
   // make sure they can understand us.                                          
   if ((required_features ^ m->get_connection()->get_features()) &               
       required_features) {              <<=======================
       dout(5) << " ignoring peer mon." << m->get_source().num()                   
          << " has features " << std::hex                                         
          << m->get_connection()->get_features()                                  
          << " but we require " << required_features << std::dec << dendl;        
     return;                                                                     
   }           
  • paxos叢集形成時的功能集相容

一個接收到要求投票的請求的選舉器,檢查功能位是否相容:通過獲得當前Monitor對相容性的要求和對方能支援的功能集的比較來決定是否還要繼續:

void Elector::handle_propose(MMonElection *m)                                   
{                                                                               
...     
   uint64_t required_features = mon->get_required_features();                    
   dout(10) << __func__ << " required features " << required_features            
            << ", peer features " << m->get_connection()->get_features()         
            << dendl;                                                            
   if ((required_features ^ m->get_connection()->get_features()) &               
       required_features) {                                                      
     dout(5) << " ignoring propose from mon" << from                             
         << " without required features" << dendl;                               
     nak_old_peer(m);                                                            
     return;      

一個選舉器在接收到選舉應答時,檢查功能位是否相容:通過獲得當前Monitor對相容性的要求和對方能支援的功能集的比較來決定是否還要繼續:

void Elector::handle_ack(MMonElection *m)                                       
{                                                                               
  ...
  uint64_t required_features = mon->get_required_features();                    
  if ((required_features ^ m->get_connection()->get_features()) &               
      required_features) {                                                      
    dout(5) << " ignoring ack from mon" << from                                 
        << " without required features" << dendl;                               
    m->put();                                                                   
    return;                                                                     
  }         

一個提出選舉的Monitor,會在選舉過程中收集與各個monitor連線時對方提供的features, 記錄在案,在獲勝後,會求出這些Monitor共同支援的功能集:

void Elector::victory()                                                         
{                                                                               
  leader_acked = -1;                                                            
  electing_me = false;                                                          
                                                                                
  uint64_t features = CEPH_FEATURES_ALL;                                        
  set<int> quorum;                                                              
  for (map<int, uint64_t>::iterator p = acked_me.begin(); p != acked_me.end();  
       ++p) {                                                                   
    quorum.insert(p->first);                                                    
    features &= p->second;                                                      
  }
  
  ...
  mon->win_election(epoch, quorum, features, cmds, cmdsize, &copy_classic_mons);                 

最後得到的features變數包含這些monitor共同支援的集合,同時把這個features傳給Monitor類記錄在案。

而Monitor::win_election把features儲存在qurum_features後呼叫finish_election, finish_election呼叫
apply_quorum_to_compatset_features(), apply_quorum_to_compatset_features就是把paxos叢集中的monitor的共同的功能集合儲存在本地檔案中,以備下次ceph mon啟動時通過read_features讀回來:

void Monitor::apply_quorum_to_compatset_features()                              
{                                                                               
   CompatSet new_features(features);                                             
   if (quorum_features & CEPH_FEATURE_OSD_ERASURE_CODES) {                       
    new_features.incompat.insert(CEPH_MON_FEATURE_INCOMPAT_OSD_ERASURE_CODES);  
   }                                                                             
   if (quorum_features & CEPH_FEATURE_OSDMAP_ENC) {                              
     new_features.incompat.insert(CEPH_MON_FEATURE_INCOMPAT_OSDMAP_ENC);         
   }                                                                             
   if (quorum_features & CEPH_FEATURE_ERASURE_CODE_PLUGINS_V2) {                 
     new_features.incompat.insert(CEPH_MON_FEATURE_INCOMPAT_ERASURE_CODE_PLUGINS_V2);
   }                                                                             
   if (new_features.compare(features) != 0) {                                    
     CompatSet diff = features.unsupported(new_features);                        
     dout(1) << __func__ << " enabling new quorum features: " << diff << dendl;  
     features = new_features;                                                    
                                                                                 
     MonitorDBStore::TransactionRef t(new MonitorDBStore::Transaction);          
     write_features(t);     <<<<<<<<<<<<<<儲存
     store->apply_transaction(t);                                                
                                                                                 
    apply_compatset_features_to_quorum_requirements();                          
   }                                                                             
}                    

總結

ceph monitor在通訊初始化時,宣告需要最小的功能集,它繞開了Messenger中對required_features的過分依賴,而是在通訊建立後動態地檢查是否相容。


相關文章