log4j+kafka+storm+mongodb+mysql 日誌處理

百聯達發表於2016-01-26
KafkaORMMongoDBMySql
一：背景
隨著系統的分散式部署，日誌的集中處理顯得極為重要。本文整合log4j+kafka+storm+mongodb+mysql介紹一下日誌的集中處理過程。本方案已應用到線上的正式環境中。

log4j版本：log4j-1.2.17.jar
kafka版本：kafka_2.10-0.8.2.1
zookeeper版本：zookeeper-3.4.7.tar.gz
storm版本：apache-storm-0.9.5.tar.gz
mongodb 版本：mongodb-linux-x86_64-2.6.11.tgz
MySql版本：mysql-5.5.31.tar.gz

二：軟體安裝
1.kafka安裝  請參照http://blog.itpub.net/28624388/viewspace-1966761/
2.storm安裝  請參照http://blog.itpub.net/28624388/viewspace-1973943/
3.mongodb安裝 請參照http://blog.itpub.net/28624388/viewspace-1974449/
4.mysql安裝  請參照http://blog.itpub.net/28624388/viewspace-764718/

三：log4j與kafka整合
請參照http://blog.itpub.net/28624388/viewspace-1972027/
日誌格式：
logger.warn("處理請求|"+((null==userId || "".equals(userId.toString()))?"0":userId.toString())+"|"+system+"|"+version+"|" + request.getRequestURI() +"|"+GbdDateUtils.format(Calendar.getInstance().getTime(),"yyyy-MM-dd HH:mm:ss")+ "|" + (System.currentTimeMillis() - accessTime) + "|毫秒！");

四：kafka+storm+mongodb+mysql整合
1.程式碼結構




2.gmap-logs-deal-center模組介紹
該模組實現消費kafka收集到的日誌資訊，並通過storm簡單的處理儲存到Mongodb資料庫內，通過schedule將mongodb裡面的日誌資料進行統計分析儲存到Mysql資料庫內。
a)pom.xml配置檔案

點選(此處)摺疊或開啟

  1. <?xml version="1.0"?>
  2. <project
  3.     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
  4.     xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  5.     <modelVersion>4.0.0</modelVersion>
  6.     <parent>
  7.         <groupId>com.gemdale.gmap</groupId>
  8.         <artifactId>GMap</artifactId>
  9.         <version>0.0.1-SNAPSHOT</version>
  10.     </parent>
  11.     <groupId>com.gemdale.gmap</groupId>
  12.     <artifactId>gmap-logs-deal-center</artifactId>
  13.     <version>0.0.1-SNAPSHOT</version>
  14.     <name>gmap-logs-deal-center</name>
  15.     <dependencies>
  16.         <dependency>
  17.             <groupId>${project.groupId}</groupId>
  18.             <artifactId>gmap-common</artifactId>
  19.             <version>${project.version}</version>
  20.         </dependency>
  21.         
  22.                 <dependency>
  23.             <groupId>${project.groupId}</groupId>
  24.             <artifactId>gmap-common-logs-bo</artifactId>
  25.             <version>${project.version}</version>
  26.         </dependency>

  28.         <dependency>
  29.             <groupId>${project.groupId}</groupId>
  30.             <artifactId>gmap-common-redis-support</artifactId>
  31.             <version>${project.version}</version>
  32.         </dependency>

  34.         <dependency>
  35.             <groupId>org.slf4j</groupId>
  36.             <artifactId>slf4j-log4j12</artifactId>
  37.         </dependency>

  39.         <dependency>
  40.             <groupId>org.apache.kafka</groupId>
  41.             <artifactId>kafka_2.10</artifactId>
  42.         </dependency>

  44.         <dependency>
  45.             <groupId>org.apache.kafka</groupId>
  46.             <artifactId>kafka-clients</artifactId>
  47.         </dependency>

  49.         <dependency>
  50.             <groupId>com.google.guava</groupId>
  51.             <artifactId>guava</artifactId>
  52.             <version>18.0</version>
  53.         </dependency>

  55.         <dependency>
  56.             <groupId>com.101tec</groupId>
  57.             <artifactId>zkclient</artifactId>
  58.         </dependency>

  60.         <dependency>
  61.             <groupId>org.apache.storm</groupId>
  62.             <artifactId>storm-kafka</artifactId>
  63.         </dependency>


  66.         <dependency>
  67.             <groupId>org.apache.storm</groupId>
  68.             <artifactId>storm-core</artifactId>
  69.         </dependency>


  72.         <dependency>
  73.             <groupId>javax.persistence</groupId>
  74.             <artifactId>persistence-api</artifactId>
  75.         </dependency>

  77.         <dependency>
  78.             <groupId>junit</groupId>
  79.             <artifactId>junit</artifactId>
  80.         </dependency>

  82.         <dependency>
  83.             <groupId>org.springframework</groupId>
  84.             <artifactId>spring-context</artifactId>
  85.         </dependency>

  87.         <dependency>
  88.             <groupId>org.springframework</groupId>
  89.             <artifactId>spring-test</artifactId>
  90.         </dependency>

  92.         <dependency>
  93.             <groupId>org.springframework</groupId>
  94.             <artifactId>spring-core</artifactId>
  95.         </dependency>

  97.         <dependency>
  98.             <groupId>org.springframework</groupId>
  99.             <artifactId>spring-beans</artifactId>
  100.         </dependency>

  102.         <dependency>
  103.             <groupId>org.springframework.data</groupId>
  104.             <artifactId>spring-data-mongodb</artifactId>
  105.         </dependency>
  106.         
  107.                 <dependency>
  108.             <groupId>${project.groupId}</groupId>
  109.             <artifactId>gmap-common-dao-support</artifactId>
  110.             <version>${project.version}</version>
  111.         </dependency>

  113.         <dependency>
  114.             <groupId>org.quartz-scheduler</groupId>
  115.             <artifactId>quartz</artifactId>
  116.         </dependency>


  119.     </dependencies>
  120. </project>
b)Start類

點選(此處)摺疊或開啟

  1. package com.enjoylink.gmap.logs.deal.center;

  3. import java.util.ArrayList;
  4. import java.util.List;

  6. import org.apache.log4j.Logger;
  7. import org.springframework.context.support.ClassPathXmlApplicationContext;
  8. import org.springframework.data.mongodb.core.MongoTemplate;
  9. import org.springframework.data.redis.core.RedisTemplate;

  11. import com.enjoyslink.gmap.logs.deal.center.stormkafka.StormKafkaBolt;
  12. import com.gemdale.gmap.common.redis.support.RedisUtil;
  13. import com.gemdale.gmap.common.util.ConfigureUtil;

  15. import backtype.storm.Config;
  16. import backtype.storm.LocalCluster;
  17. import backtype.storm.spout.SchemeAsMultiScheme;
  18. import backtype.storm.topology.TopologyBuilder;
  19. import storm.kafka.BrokerHosts;
  20. import storm.kafka.KafkaSpout;
  21. import storm.kafka.SpoutConfig;
  22. import storm.kafka.StringScheme;
  23. import storm.kafka.ZkHosts;

  25. /**
  26.  * @ClassName: Start
  27.  * @Description: TODO（用一句話描述這個類的用途）
  28.  * @author 資訊管理部-gengchong
  29.  * @date 2015年10月15日 下午5:42:52
  30.  *
  31.  */
  32. public class Start {
  33.     private static Logger logger = Logger.getLogger(Start.class);

  35.     @SuppressWarnings("resource")
  36.     public static void main(String[] args) {
  37.         // 進行spring的系統初始化化工作,進行必要的資料初始化
  38.         ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext("applicationContext.xml");

  40.         // 獲取介面,並提供服務
  41.         // 初始化REDIS服務
  42.         logger.info("初始化REDIS服務...");
  43.         RedisUtil.initRedisTemplate((RedisTemplate) context.getBean("redisTemplate"));
  44.       
  45.         logger.info("初始化storm-kafka......");
  46.         //配置kafka topic
  47.         String topic = ConfigureUtil.getProperty("topic", "gmap");
  48.         //配置kafka zookeeper
  49.         BrokerHosts hosts = new ZkHosts(ConfigureUtil.getProperty("kafka.zk"));
  50.         //配置storm spout
  51.         SpoutConfig spoutConfig = new SpoutConfig(hosts, topic, "/kafkastorm", "gmapKafka");
  52.         spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
  53.         //配置storm zookeeper
  54.         List<String> zkServers = new ArrayList<>();
  55.         zkServers.add(ConfigureUtil.getProperty("storm.zk.host"));
  56.         spoutConfig.zkServers = zkServers;
  57.         spoutConfig.zkPort = Integer.valueOf(ConfigureUtil.getProperty("storm.zk.port"));

  59.         spoutConfig.forceFromStart = false;
  60.         spoutConfig.socketTimeoutMs = 60 * 1000;
  61.         //配置topology
  62.         TopologyBuilder builder = new TopologyBuilder();
  63.         builder.setSpout("spout", new KafkaSpout(spoutConfig),
  64.                 Integer.valueOf(ConfigureUtil.getProperty("storm.max.thread")));
  65.         //配置storm bolt
  66.         builder.setBolt("bolt",new StormKafkaBolt(), Integer.valueOf(ConfigureUtil.getProperty("storm.max.thread")))
  67.                 .shuffleGrouping("spout");
  68.         //開啟topology
  69.         Config config = new Config();
  70.         config.setDebug(false);
  71.         new LocalCluster().submitTopology("topology", config, builder.createTopology());

  73.         logger.info("Gmap日誌處理中心啟動成功！");

  75.     }
  76. }
c).StormKafkaBolt類


d).transaction-context.xml

點選(此處)摺疊或開啟

  1. <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:aop="http://www.springframework.org/schema/aop" xmlns:tx="http://www.springframework.org/schema/tx" xmlns:mongo="http://www.springframework.org/schema/data/mongo" xsi:schemaLocation="http://www.springframework.org/schema/beans
  2.     http://www.springframework.org/schema/beans/spring-beans-4.0.xsd
  3.     http://www.springframework.org/schema/context
  4.     http://www.springframework.org/schema/context/spring-context-4.0.xsd
  5.     http://www.springframework.org/schema/aop
  6.     http://www.springframework.org/schema/aop/spring-aop-4.0.xsd
  7.     http://www.springframework.org/schema/data/mongo
  8.     http://www.springframework.org/schema/data/mongo/spring-mongo-1.8.xsd
  9.     http://www.springframework.org/schema/tx
  10.     http://www.springframework.org/schema/tx/spring-tx-4.0.xsd">

  12.     <bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
  13.         <property name="location">
  14.             <description>datasource config</description>
  15.             <value>classpath:context-datasource.properties</value>
  16.         </property>
  17.     </bean>

  19.     <mongo:db-factory host="${mongodb.host}" port="${mongodb.port}" dbname="${mongodb.database}" username="${mongodb.username}" password="${mongodb.password}" />


  22.     <bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
  23.         <constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
  24.     </bean>

  26.     <mongo:mapping-converter id="converter" db-factory-ref="mongoDbFactory" />
  27.     <bean id="gridFsTemplate" class="org.springframework.data.mongodb.gridfs.GridFsTemplate">
  28.         <constructor-arg ref="mongoDbFactory" />
  29.         <constructor-arg ref="converter" />
  30.     </bean>

  32.      <bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
  33.         <property name="dataSource" ref="platformTomcat" />
  34.     </bean>
  35.     
  36.       <bean id="jdbcReadTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
  37.         <property name="dataSource" ref="platformReadTomcat" />
  38.     </bean>
  39.     
  40.     <bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
  41.         <property name="dataSource" ref="platformTomcat"></property>
  42.     </bean>

  44.     <tx:annotation-driven transaction-manager="transactionManager" proxy-target-class="true"/>
  45. </beans>
e).GmapSystemLogsCollectSchedule類


3.gmap-common-logs-bo程式碼
GmapSystemLogBo類

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/28624388/viewspace-1982560/，如需轉載，請註明出處，否則將追究法律責任。

相關文章