大資料使用Excel匯出

schy_hqh發表於2013-07-17
[b]
需求描述:
1.匯出資料量約30-50W
2.提供手動即時生成資料和自動生成資料(直接提供下載)
[/b]
頁面
一個“手動匯出”按鈕,點選後臺執行程式碼生成資料並通過outputstream返回給瀏覽器。
缺點在於生成資料非常耗時(查詢資料庫,過濾不合法的資料,匯率轉換,查詢關聯表資訊等),最耗時的地方:比如,查詢到10W條資料,過濾掉2W條,之後會迴圈剩下的8W條資料,每次迴圈都去查詢資料庫獲取某些資訊,效率很低。【一次性將資料都查詢出來放記憶體中?分幾次查詢資料,第一次查詢一批,需要時從這批從獲取,如果沒有再查詢一批資料出來,這樣能夠提高效率呢?】

一個“自動匯出”按鈕,點選將從資料庫中查詢出已經由定時任務生成好的資料,直接下載。
涉及到幾個知識點:定時任務採用quartZ實現任務的促發。經過資料查詢、過濾、計算等操作後,將資料寫入Sheet,此時Workbook物件已經算是完成資料填充了。這是將Workbook物件寫入到一個流中,再將此流轉換為位元組陣列,再將位元組陣列轉換為字串,這時就需要注意了,操作字串難免不涉及編碼的問題。最開始沒有對字串進行編碼,直接存入資料庫,下載的時候直接從資料庫中查詢出來,轉換為流,結果下載的資料使用excel無法開啟。最後,通過設定字串的編碼解決了問題。存入資料庫時轉為UTF-8存入,讀取後再使用UTF-8解碼,再轉為ISO8859-1編碼,再轉為流,向瀏覽器輸出,這樣才解決了excel的讀取問題。【為什麼不直接存物件呢?由於資料庫中某張日誌表有一個大文字欄位足夠儲存資料,所以沒有使用BLOB來存放大資料,也沒有以物件的形式儲存Workbook物件。其實在第一次下載後excel無法開啟後,就考慮儲存物件的方式完成資料儲存,但是沒成功,取出來的物件和本地物件的sid總是不一致,於是放棄了存物件的實現方式,專案時間緊,只好以最能想到的辦法解決問題】

專案中遇到的另一個問題時,專案原來使用的是poi3.1-jar,版本較老了,而且不支援xlsx格式,而當前需求是資料量大,excel必須能容納足夠大的資料,所以引入了poi3.9版本。
這裡想說的是,當專案中出現新老jar包的替換時,必須小心!引入poi3.9後發現,雖然能實現大資料的儲存(50W),約20M,但是原來的excel匯出全部用的poi3.1實現的,由於3.1中某些方法在3.9中根本就沒有了,只好回去修改原來的程式碼,專案中一共10多個匯出,改得我痛苦。。。
而且,一旦確定要進行jar包的升級,選擇一個既能支援當前需求,又能最大限度相容老版本的版本是最合適的,不然,將陡然增加工作量!

以下是POI3.9匯出的部分程式碼和quartz的配置

<bean name="quartzScheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<list>
<ref bean="cronTrigger_07"/>
</list>
</property>
<property name="configLocation" value="classpath:config/quartz.properties"/>
</bean>

<bean id="cronTrigger_07" class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="jobDetail_07"/>
<!--每月1號0點開始每隔5分鐘觸發一次任務-->
<property name="cronExpression" value="0 0/5 0 1 * ?"/>
</bean>
<bean id="jobDetail_07" class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
<property name="targetObject" ref="xxxService"/>
<property name="targetMethod" value="generateExcelDatas"/>
</bean>
<bean id="xxxService" class="xxx.xxx.serviceImpl"
autowire="byName" lazy-init="true">
</bean>



建立Workbook物件,要匯出xlsx格式,需要使用XSSFWorkbook,而資料量大又不想讓記憶體溢位的話,就用Workbook workbook = new SXSSFWorkbook(1000);指定一定大小的緩衝區,當容量慢時,將清除一些物件出去,始終維持一定的記憶體暫用空間,不會發生溢位。

//**建立工作薄,工作表*//*
Workbook workbook = new SXSSFWorkbook(1000);
Sheet sheet = createSheet(workbook,businessType);



public Sheet createSheet(Workbook workbook,String businessType) {
Sheet sheet = workbook.createSheet();
if("0".equals(businessType) || "1".equals(businessType)){
if("0".equals(businessType)) {
workbook.setSheetName(0, "XXX清單");
} else {
workbook.setSheetName(0, "YYY清單");
}
sheet.setColumnWidth((short) 0, (short) (35.7*50));//
sheet.setColumnWidth((short) 1, (short) (35.7*120));//
sheet.setColumnWidth((short) 2, (short) (35.7*120));//
sheet.setColumnWidth((short) 3, (short) (35.7*120));//
sheet.setColumnWidth((short) 4, (short) (35.7*160));//
sheet.setColumnWidth((short) 5, (short) (35.7*160));//
sheet.setColumnWidth((short) 6, (short) (35.7*180));//
sheet.setColumnWidth((short) 7, (short) (35.7*120));//
sheet.setColumnWidth((short) 8, (short) (35.7*180));//
sheet.setColumnWidth((short) 9, (short) (35.7*90));//
sheet.setColumnWidth((short) 10, (short) (35.7*100));//
sheet.setColumnWidth((short) 11, (short) (35.7*100));//
sheet.setColumnWidth((short) 12, (short) (35.7*100));//
}
else if("9".equals(businessType)){
workbook.setSheetName(0, "ZZZ清單");
sheet.setColumnWidth((short) 0, (short) (35.7*50));//
sheet.setColumnWidth((short) 1, (short) (35.7*280));//
sheet.setColumnWidth((short) 2, (short) (35.7*120));//
sheet.setColumnWidth((short) 3, (short) (35.7*120));//
sheet.setColumnWidth((short) 4, (short) (35.7*100));//
sheet.setColumnWidth((short) 5, (short) (35.7*180));//
sheet.setColumnWidth((short) 6, (short) (35.7*120));//
sheet.setColumnWidth((short) 7, (short) (35.7*180));//
sheet.setColumnWidth((short) 8, (short) (35.7*90));//
sheet.setColumnWidth((short) 9, (short) (35.7*100));//
sheet.setColumnWidth((short) 10, (short) (35.7*100));//
sheet.setColumnWidth((short) 11, (short) (35.7*100));//
}
return sheet;
}


[b]表格樣式[/b]

Map<String,CellStyle> styleMap = getPaymentDatasStyles(workbook);
public Map<String, CellStyle> getPaymentDatasStyles(Workbook workbook) {

Map<String,CellStyle> styles = new HashMap<String,CellStyle>();

Font fontHead = workbook.createFont();
fontHead.setFontName("宋體"); // 字型
fontHead.setFontHeightInPoints((short)16); //字號
fontHead.setBoldweight(HSSFFont.BOLDWEIGHT_BOLD); //加粗

Font fontBody = workbook.createFont();
fontBody.setFontName("宋體"); // 字型
fontBody.setFontHeightInPoints((short)10); //字號

Font fontMark = workbook.createFont();
fontMark.setFontName("宋體"); // 字型
fontMark.setFontHeightInPoints((short)12); //字號

/**首行樣式*/
CellStyle cellStyleHead = workbook.createCellStyle();
cellStyleHead.setAlignment(CellStyle.ALIGN_CENTER);
cellStyleHead.setVerticalAlignment(CellStyle.VERTICAL_CENTER);
cellStyleHead.setFont(fontHead);

/**第2行樣式*/
CellStyle cellStyleUnit = workbook.createCellStyle();
cellStyleUnit.setAlignment(CellStyle.ALIGN_CENTER);
cellStyleUnit.setBorderBottom(CellStyle.BORDER_MEDIUM);
cellStyleUnit.setFont(fontBody);

/**列標題樣式*/
CellStyle cellStyleTitle = workbook.createCellStyle();
cellStyleTitle.setBorderBottom(CellStyle.BORDER_THIN);
cellStyleTitle.setBorderLeft(CellStyle.BORDER_THIN);
cellStyleTitle.setBorderRight(CellStyle.BORDER_THIN);
cellStyleTitle.setBorderTop(CellStyle.BORDER_THIN);
cellStyleTitle.setAlignment(CellStyle.ALIGN_CENTER);
cellStyleTitle.setVerticalAlignment(CellStyle.VERTICAL_CENTER);
cellStyleTitle.setFont(fontBody);
cellStyleTitle.setWrapText(true);

/**正文樣式*/
CellStyle cellStyleBody = workbook.createCellStyle();
cellStyleBody.setBorderBottom(CellStyle.BORDER_THIN);
cellStyleBody.setBorderLeft(CellStyle.BORDER_THIN);
cellStyleBody.setBorderRight(CellStyle.BORDER_THIN);
cellStyleBody.setBorderTop(CellStyle.BORDER_THIN);
cellStyleBody.setAlignment(CellStyle.ALIGN_CENTER);
cellStyleBody.setFont(fontBody);

/**正文樣式二*/
CellStyle cellStyleBodyDecimal = workbook.createCellStyle();
cellStyleBodyDecimal.setBorderBottom(CellStyle.BORDER_THIN);
cellStyleBodyDecimal.setBorderLeft(CellStyle.BORDER_THIN);
cellStyleBodyDecimal.setBorderRight(CellStyle.BORDER_THIN);
cellStyleBodyDecimal.setBorderTop(CellStyle.BORDER_THIN);
cellStyleBodyDecimal.setAlignment(CellStyle.ALIGN_RIGHT);
cellStyleBodyDecimal.setDataFormat(HSSFDataFormat.getBuiltinFormat("0.00"));
cellStyleBodyDecimal.setFont(fontBody);

/**正文樣式三*/
CellStyle cellStyleBodyInt = workbook.createCellStyle();
cellStyleBodyInt.setBorderBottom(CellStyle.BORDER_THIN);
cellStyleBodyInt.setBorderLeft(CellStyle.BORDER_THIN);
cellStyleBodyInt.setBorderRight(CellStyle.BORDER_THIN);
cellStyleBodyInt.setBorderTop(CellStyle.BORDER_THIN);
cellStyleBodyInt.setAlignment(CellStyle.ALIGN_RIGHT);
cellStyleBodyInt.setDataFormat(HSSFDataFormat.getBuiltinFormat("0"));
cellStyleBodyInt.setFont(fontBody);

/**備註*/
CellStyle cellStyleMark = workbook.createCellStyle();
cellStyleMark.setFont(fontMark);

styles.put("HEAD", cellStyleHead);
styles.put("UNIT", cellStyleUnit);
styles.put("TITLE", cellStyleTitle);
styles.put("BODY", cellStyleBody);
styles.put("BODY_DECIMAL", cellStyleBodyDecimal);
styles.put("BODY_INT", cellStyleBodyInt);
styles.put("MARK", cellStyleMark);
return styles;
}


[b]將資料往sheet中寫[/b]

putDatas2Xlsx(sheet,styleMap,list,type,locations,allExchRate);
public void putDatas2Xlsx(Sheet sheet, Map<String, CellStyle> styleMap,List<ScmsDocFeeDetailVo> detailList,String businessType,
Map<String,String> locations,Map<String,BigDecimal> allExchRate) throws Exception {


final String agentCodeForGD = "aaa";

//**********************列標題***************************//*
String headName = "";
String [] headList = null;
headName = "XXX清單(03)";
if("0".equals(businessType)){
headList = new String[]{};
}else
if("1".equals(businessType)){
headName = "XXX清單(02)";
headList = new String[]{};
} else
if("9".equals(businessType)){
headName = "XXX清單(01)";
headList = new String[]{};
}

ScmsDocFeeDetailVo vo = null;
String risk = "";
String handler1CodeName = "";

Row row = null;
Cell cell = null;

CellStyle cellStyleHead = styleMap.get("HEAD");
CellStyle cellStyleUnit = styleMap.get("UNIT");
CellStyle cellStyleTitle = styleMap.get("TITLE");
CellStyle cellStyleBody = styleMap.get("BODY");
CellStyle cellStyleBodyDecimal = styleMap.get("BODY_DECIMAL");
CellStyle cellStyleBodyInt = styleMap.get("BODY_INT");
CellStyle cellStyleMark = styleMap.get("MARK");

//建立第一行 標題
int rowNum = 0;
row = sheet.createRow(rowNum++);//建立行
cell = row.createCell(0);
cell.setCellValue(headName);
cell.setCellStyle(cellStyleHead);
//合併單元格
if("0".equals(businessType) || "1".equals(businessType)) {
sheet.addMergedRegion(new CellRangeAddress(0, 0, 0, 8));
} else {
sheet.addMergedRegion(new CellRangeAddress(0, 0, 0, 6));
}

//建立第二行 單位
row = sheet.createRow(rowNum++);
row.setHeight((short)400);
for(int i=0;i<headList.length;i++){
cell = row.createCell(i);
cell.setCellStyle(cellStyleUnit);
if(("0".equals(businessType) && i==10) || ("1".equals(businessType) && i==10) ||("9".equals(businessType) && i==9)){
cell.setCellValue("單位:元");
}
}
if("0".equals(businessType)) {
sheet.addMergedRegion(new CellRangeAddress(1, 1, 0, 7));
} else if("1".equals(businessType)) {
sheet.addMergedRegion(new CellRangeAddress(1, 1, 0, 8));
}

//第三行 列標題
row = sheet.createRow(rowNum++);
for(int i=0;i<headList.length;i++){
cell = row.createCell(i);
cell.setCellStyle(cellStyleTitle);
cell.setCellValue(headList[i]);
}
row.setHeightInPoints((short)33);//行高

//** 列印主體資訊列表------------------------------------------*//*
int detailCount = detailList.size();
for(int i = 0;i<detailCount;i++){
vo = detailList.get(i);
if("0".equals(businessType) || "1".equals(businessType)) {
handler1CodeName =codeService.translateCode("UserCode", vo.getHandler1Code(), "", "");
vo.setInsuredName(handler1CodeName);
vo.setAgentCode(agentCodeForGD);
}

/**匯率轉換*/
if(vo.getCurrency()!=null && !"".equals(vo.getCurrency().trim())) {
BigDecimal exchRate = allExchRate.get(vo.getCurrency().trim());
vo.setCostFee(vo.getCostFee().multiply(exchRate).setScale(2, RoundingMode.HALF_UP));
}


if(vo.getPaidPremium()==null || vo.getPaidPremium().equals("") || vo.getPaidPremium().compareTo(BigDecimal.ZERO)==0) {
vo.setCostRate(BigDecimal.ZERO);
} else {
vo.setCostRate(vo.getCostFee().divide(vo.getPaidPremium(), 2, RoundingMode.HALF_UP));
}
//開始迴圈生成EXCEL行資料
short colNum = 0;//初始化CELL下標
row = sheet.createRow(rowNum++);//建立行
row.setHeight((short)400);

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_NUMERIC);
cell.setCellStyle(cellStyleBodyInt);
cell.setCellValue((i+1));

//清單
if("0".equals(businessType) || "1".equals(businessType)){

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
String preComCode = vo.getComCode().substring(0, 4);
String addressName = locations.get(preComCode);
if(addressName!=null) {
cell.setCellValue(addressName);
} else {
cell.setCellValue(preComCode);
}

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(vo.getComCode());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
handler1CodeName=codeService.translateCode("UserCode", vo.getHandler1Code(), "PUB", "");
cell.setCellValue(handler1CodeName);

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(agentCodeForGD);

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(vo.getSellerNo());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(vo.getCertiNo());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
if(vo.getStartDate()!=null && !vo.getStartDate().equals("")) {
cell.setCellValue((new SimpleDateFormat("yyyy-MM-dd")).format(vo.getStartDate()));
} else {
cell.setCellValue("");
}

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
risk=codeService.translateCode("RiskCode", vo.getRiskCode(), "", "");
cell.setCellValue(risk);

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(vo.getRiskCode());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_NUMERIC);
cell.setCellStyle(cellStyleBodyDecimal);
cell.setCellValue(vo.getPaidPremium().doubleValue());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_NUMERIC);
cell.setCellStyle(cellStyleBodyDecimal);
cell.setCellValue(vo.getCostFee().doubleValue());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_NUMERIC);
cell.setCellStyle(cellStyleBodyDecimal);
cell.setCellValue(vo.getCostRate().doubleValue());
}else
//手續費清單
if("9".equals(businessType)){

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(vo.getAgentName());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(vo.getAgentCode());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
String preComCode = vo.getComCode().substring(0, 4);
String addressName = locations.get(preComCode);
if(addressName!=null) {
cell.setCellValue(addressName);
} else {
cell.setCellValue(preComCode);
}

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(vo.getComCode());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(vo.getCertiNo());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
if(vo.getStartDate()!=null) {
cell.setCellValue((new SimpleDateFormat("yyyy-MM-dd")).format(vo.getStartDate()));
} else{
cell.setCellValue("");
}
cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
//呼叫服務轉換程式碼為姓名【在迴圈中每次呼叫服務進行查詢,導致效能下降很多,這裡必須進行優化處理。可以一次性查詢所有需要轉換的資料到記憶體中!!!】
risk=codeService.translateCode("RiskCode", vo.getRiskCode(), "", "");
cell.setCellValue(risk);

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_STRING);
cell.setCellStyle(cellStyleBody);
cell.setCellValue(vo.getRiskCode());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_NUMERIC);
cell.setCellStyle(cellStyleBodyDecimal);
cell.setCellValue(vo.getPaidPremium().doubleValue());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_NUMERIC);
cell.setCellStyle(cellStyleBodyDecimal);
cell.setCellValue(vo.getCostFee().doubleValue());

cell = row.createCell(colNum++);
cell.setCellType(Cell.CELL_TYPE_NUMERIC);
cell.setCellStyle(cellStyleBodyDecimal);
cell.setCellValue(vo.getCostRate().doubleValue());

}
}
//備註
row = sheet.createRow(rowNum++);//建立行
row.setHeight((short)400);
cell = row.createCell(0);
cell.setCellStyle(cellStyleMark);
cell.setCellValue("備註:"); //序號

}



[b]儲存資料到資料庫[/b]

OutputStream os = null;
ByteArrayOutputStream baos = null;
try{
os = new ByteArrayOutputStream();
workbook.write(os);
baos = (ByteArrayOutputStream) os;
//編碼轉換-非常必須一定要這樣做!
String encodeStr = baos.toString("ISO-8859-1");
encodeStr = java.net.URLEncoder.encode(encodeStr, "UTF-8");
//現在可以直接儲存encodeStr 字串到資料庫中了!
//.......


[b]從資料庫中取資料並返回給瀏覽器下載[/b]

public void exportXlsxFromDB()throws Exception {
/**基於頁面輸入資訊查詢資料*/
String comCode = ServletActionContext.getRequest().getParameter("comCode");
String paymentYear = ServletActionContext.getRequest().getParameter("paymentYear");
String paymentMonth = ServletActionContext.getRequest().getParameter("paymentMonth");
String paymentDate = paymentYear+"-"+paymentMonth;
String riskType = ServletActionContext.getRequest().getParameter("risk");
String businessType = ServletActionContext.getRequest().getParameter("businessType");

long start = System.currentTimeMillis();
logger.info("開始匯出機構"+comCode+"及其下級"+paymentDate+"月的實收資料,匯出險類:"+riskType+",業務型別:"+businessType);
//查詢資料
Table table = tableService.getExcelDataFromDB(paymentDate, comCode, riskType, businessType);
if(table==null) {
//資料庫無資料
throw new BusinessException(paymentDate+"資料尚未定時生成,請手動生成本次資料。【提示:生成資料時,如果資料量較大,將比較耗時,請耐心等待...】");
}
String sheetName = interfaceLog.getResponsetype();
String excelDatas = interfaceLog.getResponsexml();

//還原資料---非常關鍵&&重要加必須
excelDatas = java.net.URLDecoder.decode(excelDatas, "UTF-8");
ByteArrayOutputStream baos = new ByteArrayOutputStream();
baos.write(excelDatas.getBytes("ISO-8859-1"));

//設定返回內容型別
this.getResponse().reset();
this.getResponse().setContentType("application/x-msdownload");
this.getResponse().setHeader("Content-Disposition","attachment; filename="+new String(sheetName.getBytes("gb2312"),"ISO-8859-1")+paymentDate+".xlsx");
OutputStream outStream = null;
try{
outStream = this.getResponse().getOutputStream();
baos.writeTo(outStream);
outStream.flush();
}catch(Exception e){
log.debug(e.getMessage());
e.printStackTrace();
throw new ScmsBusinessException("匯出異常,請聯絡管理員!",e);
}finally{
if(outStream!=null) {
outStream.close();
}
if(baos!=null) {
baos.close();
}
}
}

相關文章