python logging模組註冊流程(以logging.config.dictConfig流程為例)
最近在檢視python logging相關的模組，用到了dictConfig這一函式，嘗試著跟蹤了一下，捋一捋在呼叫dictConfig之後，這些都發生了什麼,我想，用fileConfig的流程也差不多，先記載dictConfig的載入流程，後面跟蹤原始碼了之後再做更新。
logging.config.dictconfig原始碼：:
dictConfigClass = DictConfigurator
def dictConfig(config):
"""Configure logging using a dictionary."""
dictConfigClass(config).configure()
複製程式碼1. 程式碼中可以看到執行config操作的實際是DictConfigurator這一例項的configure方法.那麼就涉及到DictConfigurator例項的初始化：
class DictConfigurator(BaseConfigurator):
...
複製程式碼1.1 DictConfigurator繼承自BaseConfigurator,檢視其__init__方法:
class BaseConfigurator(object):
....
def __init__(self, config):
self.config = ConvertingDict(config)
self.config.configurator = self
複製程式碼程式碼中有兩點: ConvertingDict的例項self.config, 以及self.config.configurator引用了當前的Baseconfigurator例項,
那麼， 程式碼中的ConvertingDict(config)是個什麼？繼續跟蹤:
1.2 ConvertingDict解析
class ConvertingDict(dict, ConvertingMixin):
"""A converting dictionary wrapper."""
def __getitem__(self, key):
value = dict.__getitem__(self, key)
return self.convert_with_key(key, value)
def get(self, key, default=None):
value = dict.get(self, key, default)
return self.convert_with_key(key, value)
def pop(self, key, default=None):
value = dict.pop(self, key, default)
return self.convert_with_key(key, value, replace=False)
複製程式碼可以看到，Convertingdict繼承自dict, 重寫了dict中的__getitem__, get, pop三個方法，先呼叫dict原有的方法，然後對獲取的值使用self.convert_with_key進行轉換,其程式碼在ConvertingMixin類中。
1.3 ConvertingMixin類
class ConvertingMixin(object):
"""For ConvertingXXX's, this mixin class provides common functions"""
def convert_with_key(self, key, value, replace=True):
result = self.configurator.convert(value)
#If the converted value is different, save for next time
if value is not result:
if replace:
self[key] = result
if type(result) in (ConvertingDict, ConvertingList,
ConvertingTuple):
result.parent = self
result.key = key
return result
def convert(self, value):
result = self.configurator.convert(value)
if value is not result:
if type(result) in (ConvertingDict, ConvertingList,
ConvertingTuple):
result.parent = self
return result
複製程式碼看到convert_with_key方法中，每次都會呼叫它所引用的self.configurator的convert方法, 在這個流程中，self.configurator就是當前的Baseconfigurator例項.
1.4 Baseconfigurator的convert方法：
def convert(self, value):
"""
Convert values to an appropriate type. dicts, lists and tuples are
replaced by their converting alternatives. Strings are checked to
see if they have a conversion format and are converted if they do.
"""
if not isinstance(value, ConvertingDict) and isinstance(value, dict):
value = ConvertingDict(value)
value.configurator = self
elif not isinstance(value, ConvertingList) and isinstance(value, list):
value = ConvertingList(value)
value.configurator = self
elif not isinstance(value, ConvertingTuple) and\
isinstance(value, tuple):
value = ConvertingTuple(value)
value.configurator = self
elif isinstance(value, str): # str for py3k
m = self.CONVERT_PATTERN.match(value)
if m:
d = m.groupdict()
prefix = d['prefix']
converter = self.value_converters.get(prefix, None)
if converter:
suffix = d['suffix']
converter = getattr(self, converter)
value = converter(suffix)
return value
複製程式碼前面都是在遞迴，只有當value型別為str型別時，才會去進行轉換，其他都是遞迴轉換型別
1.4.1 str資料的轉化
CONVERT_PATTERN = re.compile(r'^(?P<prefix>[a-z]+)://(?P<suffix>.*)$')
複製程式碼CONVERT_PATTERN是一個正則，匹配類似於一個協議，如：prefix://suffix.
接著value_converters：
value_converters = {
'ext' : 'ext_convert',
'cfg' : 'cfg_convert',
}
複製程式碼可以看到，這對應了兩種協議，ext和cfg. 看ext://協議對應的ext_convert方法：
def ext_convert(self, value):
"""Default converter for the ext:// protocol."""
return self.resolve(value)
複製程式碼它其實呼叫了self.resolve(value), self是當前的的Baseconfigurator, 檢視resolve方法：
1.4.1 Baseconfigurator的resolve方法:
def resolve(self, s):
"""
Resolve strings to objects using standard import and attribute
syntax.
"""
name = s.split('.')
used = name.pop(0)
try:
found = self.importer(used)
for frag in name:
used += '.' + frag
try:
found = getattr(found, frag)
except AttributeError:
self.importer(used)
found = getattr(found, frag)
return found
except ImportError:
e, tb = sys.exc_info()[1:]
v = ValueError('Cannot resolve %r: %s' % (s, e))
v.__cause__, v.__traceback__ = e, tb
raise v
複製程式碼根據官方文件，ext協議主要是用來引用其他的模組，例如自定義模組ext://a.b.c, 分析原始碼：
先通過.號分割出各個小段，self.importer其實就是__import__函式, resolve方法會一直去迴圈載入模組,最後把模組返回。
1.5 總結一下：
Baseconfigurator初始化過程是：初始化一個ConvertingDict例項, 並將該例項的configurator指向Baseconfigurator自身,ConvertingDict的作用: 以ext為例，在獲取對應的key:value值時，轉化為協議對應的模組，以供使用。
2. DictConfigurator的configure方法：
原始碼如下：
"""Do the configuration."""
config = self.config
if 'version' not in config:
raise ValueError("dictionary doesn't specify a version")
if config['version'] != 1:
raise ValueError("Unsupported version: %s" % config['version'])
incremental = config.pop('incremental', False)
EMPTY_DICT = {}
logging._acquireLock()
try:
if incremental:
handlers = config.get('handlers', EMPTY_DICT)
for name in handlers:
if name not in logging._handlers:
raise ValueError('No handler found with '
'name %r' % name)
else:
try:
handler = logging._handlers[name]
handler_config = handlers[name]
level = handler_config.get('level', None)
if level:
handler.setLevel(logging._checkLevel(level))
except Exception as e:
raise ValueError('Unable to configure handler '
'%r: %s' % (name, e))
loggers = config.get('loggers', EMPTY_DICT)
for name in loggers:
try:
self.configure_logger(name, loggers[name], True)
except Exception as e:
raise ValueError('Unable to configure logger '
'%r: %s' % (name, e))
root = config.get('root', None)
if root:
try:
self.configure_root(root, True)
except Exception as e:
raise ValueError('Unable to configure root '
'logger: %s' % e)
else:
disable_existing = config.pop('disable_existing_loggers', True)
logging._handlers.clear()
del logging._handlerList[:]
# Do formatters first - they don't refer to anything else
formatters = config.get('formatters', EMPTY_DICT)
for name in formatters:
try:
formatters[name] = self.configure_formatter(
formatters[name])
except Exception as e:
raise ValueError('Unable to configure '
'formatter %r: %s' % (name, e))
# Next, do filters - they don't refer to anything else, either
filters = config.get('filters', EMPTY_DICT)
for name in filters:
try:
filters[name] = self.configure_filter(filters[name])
except Exception as e:
raise ValueError('Unable to configure '
'filter %r: %s' % (name, e))
# Next, do handlers - they refer to formatters and filters
# As handlers can refer to other handlers, sort the keys
# to allow a deterministic order of configuration
handlers = config.get('handlers', EMPTY_DICT)
deferred = []
for name in sorted(handlers):
try:
handler = self.configure_handler(handlers[name])
handler.name = name
handlers[name] = handler
except Exception as e:
if 'target not configured yet' in str(e):
deferred.append(name)
else:
raise ValueError('Unable to configure handler '
'%r: %s' % (name, e))
# Now do any that were deferred
for name in deferred:
try:
handler = self.configure_handler(handlers[name])
handler.name = name
handlers[name] = handler
except Exception as e:
raise ValueError('Unable to configure handler '
'%r: %s' % (name, e))
# Next, do loggers - they refer to handlers and filters
#we don't want to lose the existing loggers,
#since other threads may have pointers to them.
#existing is set to contain all existing loggers,
#and as we go through the new configuration we
#remove any which are configured. At the end,
#what's left in existing is the set of loggers
#which were in the previous configuration but
#which are not in the new configuration.
root = logging.root
existing = list(root.manager.loggerDict.keys())
#The list needs to be sorted so that we can
#avoid disabling child loggers of explicitly
#named loggers. With a sorted list it is easier
#to find the child loggers.
existing.sort()
#We'll keep the list of existing loggers
#which are children of named loggers here...
child_loggers = []
#now set up the new ones...
loggers = config.get('loggers', EMPTY_DICT)
for name in loggers:
if name in existing:
i = existing.index(name) + 1 # look after name
prefixed = name + "."
pflen = len(prefixed)
num_existing = len(existing)
while i < num_existing:
if existing[i][:pflen] == prefixed:
child_loggers.append(existing[i])
i += 1
existing.remove(name)
try:
self.configure_logger(name, loggers[name])
except Exception as e:
raise ValueError('Unable to configure logger '
'%r: %s' % (name, e))
#Disable any old loggers. There's no point deleting
#them as other threads may continue to hold references
#and by disabling them, you stop them doing any logging.
#However, don't disable children of named loggers, as that's
#probably not what was intended by the user.
#for log in existing:
# logger = root.manager.loggerDict[log]
# if log in child_loggers:
# logger.level = logging.NOTSET
# logger.handlers = []
# logger.propagate = True
# elif disable_existing:
# logger.disabled = True
_handle_existing_loggers(existing, child_loggers,
disable_existing)
# And finally, do the root logger
root = config.get('root', None)
if root:
try:
self.configure_root(root)
except Exception as e:
raise ValueError('Unable to configure root '
'logger: %s' % e)
finally:
logging._releaseLock()
複製程式碼2.1 incremental
從程式碼中可一個看到有個incremental引數:
incremental = config.pop('incremental', False)
複製程式碼這個引數的作用是什麼?
2.1.1 假設incremental引數為True其核心在這裡:
for name in loggers:
try:
self.configure_logger(name, loggers[name], True)
except Exception as e:
raise ValueError('Unable to configure logger '
複製程式碼這個例子中， configure_logger在DictConfigurator中， 程式碼如下:
def common_logger_config(self, logger, config, incremental=False):
"""
Perform configuration which is common to root and non-root loggers.
"""
level = config.get('level', None)
if level is not None:
logger.setLevel(logging._checkLevel(level))
if not incremental:
#Remove any existing handlers
for h in logger.handlers[:]:
logger.removeHandler(h)
handlers = config.get('handlers', None)
if handlers:
self.add_handlers(logger, handlers)
filters = config.get('filters', None)
if filters:
self.add_filters(logger, filters)
def configure_logger(self, name, config, incremental=False):
"""Configure a non-root logger from a dictionary."""
logger = logging.getLogger(name)
self.common_logger_config(logger, config, incremental)
propagate = config.get('propagate', None)
if propagate is not None:
logger.propagate = propagate
複製程式碼從以上程式碼中可以看到， 當incremental為True時，只會將日誌的等級level修改.
2.2 incremental引數為False時：
2.2.1 清除logging中的_handler和_handlerList資訊
logging._handlers.clear()
del logging._handlerList[:]
複製程式碼2.2.2 初始化formmaters, 原始碼如下
formatters = config.get('formatters', EMPTY_DICT)
for name in formatters:
try:
formatters[name] = self.configure_formatter(
formatters[name])
except Exception as e:
raise ValueError('Unable to configure '
'formatter %r: %s' % (name, e))
複製程式碼先會獲取所有的formatter,然後通過configure_formatter方法來初始化.
def configure_formatter(self, config):
"""Configure a formatter from a dictionary."""
if '()' in config:
factory = config['()'] # for use in exception handler
try:
result = self.configure_custom(config)
except TypeError as te:
if "'format'" not in str(te):
raise
#Name of parameter changed from fmt to format.
#Retry with old name.
#This is so that code can be used with older Python versions
#(e.g. by Django)
config['fmt'] = config.pop('format')
config['()'] = factory
result = self.configure_custom(config)
else:
fmt = config.get('format', None)
dfmt = config.get('datefmt', None)
style = config.get('style', '%')
cname = config.get('class', None)
if not cname:
c = logging.Formatter
else:
c = _resolve(cname)
result = c(fmt, dfmt, style)
return result
複製程式碼可以看到，引數當formatter配置中，有兩種情況：
- 有名為
()的key
此時，呼叫configure_custom方法:
def configure_custom(self, config):
"""Configure an object with a user-supplied factory."""
c = config.pop('()')
if not callable(c):
c = self.resolve(c)
props = config.pop('.', None)
# Check for valid identifiers
kwargs = dict([(k, config[k]) for k in config if valid_ident(k)])
result = c(**kwargs)
if props:
for name, value in props.items():
setattr(result, name, value)
return result
複製程式碼從程式碼中可以看到，config.pop('()')的呼叫，在上文中有提到：
def pop(self, key, default=None):
value = dict.pop(self, key, default)
return self.convert_with_key(key, value, replace=False)
複製程式碼這就是來獲取需要呼叫的模組, 最後呼叫對應的模組，生成formatter,
注意一點： props = config.pop('.', None), 配置.屬性可以設定對應例項的自身屬性，如:
"console": {
"class": "logging.StreamHandler",
"level": "DEBUG",
"formatter": "simple",
".": {
"hei": 1
}
}
複製程式碼這個配置加上了.號屬性的設定，初始化之後，對應的Streamhandler就會有屬性hei
2. 正常的key
直接呼叫對應的class模組，生成formatter
2.2.3 生成filter、Handler，與formatter類似, Handler最後會加上對應的formmaters以及filters
2.2.4 生成logger
這些就是logging.config.dictConfig的初始化過程，一步步解析，通過自帶協議：ext://和cfg://協議來引用外部模組或是已有的配置,初始化你自己需求的logger，也可自定義相關模組，完成特殊化需求,粗糙之問，有點亂。有錯誤，麻煩之處，好更正。