odps dship客戶端使用
1、下載DShip外掛
下載地址：最下面的“開發者資源”，“資料上傳下載工具”，下載檔案為odps-dship.zip。
odps-dship\dship.bat即為windows下的執行檔案。
2、配置DShip
odps-dship\odps.conf為DShip的配置檔案，（需要jdk1.6以上），配置檔案原來是這樣的：
#odps dship config
endpoint=
id=
key=
project=
配置好專案名稱，access ID，access Key。
正確配置請參照文件下面部分的odps.conf配置
3、開啟客戶端
注意：要到命令列cd到相應的目錄下面，執行命令dfish可以看到dfish的幫助資訊。
C:\Users\yangswa>cd C:\Users\yangswa\Desktop\odps-dship
C:\Users\yangswa\Desktop\odps-dship>dship
Usage: dship
Type 'dship help
Type 'dship --version' to see the program version.
Available subcommands:
upload (u)
download (d)
resume (r)
show (s)
config (c)
purge (p)
help (h)
Valid global options:
-c [--charset] ARG : file charset
-dfp [--date-format-pattern] ARG : date format pattern
-e [--endpoint] ARG : specify endpoint
-fd [--field-delimiter] ARG : field delimiter
-i [--id] ARG : access id
-k [--key] ARG : access key
-ni [--null-indicator] ARG : null indicator string
-p [--project] ARG : default project
-rd [--record-delimiter] ARG : record delimiter
-te [--tunnel-endpoint] ARG : specify tunnel endpoint
dship is tool for odps data transfer.
4、在odps的yangsw_test專案中有一個dual表
表結構：
id，型別為bigint。
Name, 型別為string
Dt,型別為string，該欄位為分割槽欄位
odps:sql:yangsw_test> desc dual;
+------------------------------------------------------------------------------------+
| Table: dual |
| Owner: ALIYUN$************** | Project: yangsw_test |
| TableComment: |
+------------------------------------------------------------------------------------+
| CreatedTime: 2014-11-11 10:14:25 |
| LastMetaModifiedTime: 2014-11-11 10:14:25 |
| LastDataModifiedTime: 1970-01-01 08:00:00 |
+------------------------------------------------------------------------------------+
| Type : Table | Size: 0 Bytes |
+------------------------------------------------------------------------------------+
| Native Columns: |
+------------------------------------------------------------------------------------+
| Field | Type | Comment |
+------------------------------------------------------------------------------------+
| id | BIGINT | |
| name | STRING | |
+------------------------------------------------------------------------------------+
| Partition Columns: |
+------------------------------------------------------------------------------------+
| dt | STRING | |
+------------------------------------------------------------------------------------+
我們給dual表加一個分割槽
odps:sql:yangsw_test> alter table dual add partition(dt='2014');
InstanceId: 20141111021906223gw514il
OK
5、準備資料檔案
Dual_2014.txt
1,tom
2,jacky
3,jet
4,rose
5,marry
6,obama
6、用dship上傳資料
C:\Users\yangswa\Desktop\odps-dship> dship upload -fd "," -rd "\r\n" C:\Users\yangswa\Desktop\dual_2014.txt dual/dt="2014";
ERROR: Unknown error -
405 Not Allowed
檢視日誌檔案odps-dship\sessions\.sidnull\log.txt
2014-11-11 11:10:56 - java.lang.Exception: ERROR: Unknown error -
405 Not Allowed
at com.aliyun.odps.ship.DShip.parseCommandLine(DShip.java:101)
at com.aliyun.odps.ship.DShip.main(DShip.java:35)
Caused by: java.lang.RuntimeException:
405 Not Allowed
at com.aliyun.odps.tunnel.TunnelException.
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.initiate(TableTunnel.java:462)
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.
at com.aliyun.odps.tunnel.TableTunnel.createUploadSession(TableTunnel.java:95)
at com.aliyun.odps.ship.upload.TunnelUploadSession.
at com.aliyun.odps.ship.DShip.parseCommandLine(DShip.java:62)
... 1 more
Caused by: Parse response failed
at com.aliyun.odps.tunnel.TunnelException.loadFromJson(TunnelException.java:138)
at com.aliyun.odps.tunnel.TunnelException.
... 6 more
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character ('
at [Source: ; line: 1, column: 2]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:442)
at org.codehaus.jackson.impl.Utf8StreamParser._handleUnexpectedValue(Utf8StreamParser.java:2090)
at org.codehaus.jackson.impl.Utf8StreamParser._nextTokenNotInObject(Utf8StreamParser.java:606)
at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:492)
at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2770)
at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2718)
at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1507)
at com.aliyun.odps.tunnel.TunnelException.loadFromJson(TunnelException.java:127)
... 7 more
難道是多個分號？？？？？
C:\Users\yangswa\Desktop\odps-dship>dship upload -fd "," -rd "\r\n" C:\Users\yangswa\Desktop\dual_2014.txt dual/dt="2014"
ERROR: TunnelException – null
檢視日誌檔案odps-dship\sessions\.sidnull\log.txt
2014-11-11 14:41:38 - null
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.initiate(TableTunnel.java:467)
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.
at com.aliyun.odps.tunnel.TableTunnel.createUploadSession(TableTunnel.java:95)
at com.aliyun.odps.ship.upload.TunnelUploadSession.
at com.aliyun.odps.ship.DShip.parseCommandLine(DShip.java:62)
at com.aliyun.odps.ship.DShip.main(DShip.java:35)
Caused by: java.net.UnknownHostException: null
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.net.NetworkClient.doConnect(Unknown Source)
at sun.net.(Unknown Source)
at sun.net.(Unknown Source)
at sun.net.(Unknown Source)
at sun.net.(Unknown Source)
at sun.net.(Unknown Source)
at sun.net.(Unknown Source)
at sun.net.(Unknown Source)
at sun.net.(Unknown Source)
at sun.net.(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at com.aliyun.odps.commons.transport.DefaultConnection.getResponse(DefaultConnection.java:101)
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.initiate(TableTunnel.java:457)
... 5 more
還是報錯。。。 。。。
這位大神的配置起作用：
最終在windows環境下我得odps.conf改為下面就可以上傳資料了（注意上面兩行）
\u00EF\u00BB\u00BF\#odps=dship config
tunnel-endpoint=http\://dt.odps.aliyun.com
id=*******
key=********
project=yangsw_test
最終的執行結果如下：
C:\Users\yangswa\Desktop\odps-dship>dship upload -fd "," -rd "\r\n" C:\Users\yangswa\Desktop\dual_2014.txt yangsw_test.dual/dt="2014"
Upload session: 201411111456542581870a0024a407
2014-11-11 14:56:25 scanning file: 'dual_2014.txt'
2014-11-11 14:56:26 uploading file: 'dual_2014.txt'
2014-11-11 14:56:28 'dual_2014.txt' uploaded
OK
7、用odps sql查一下dual表的資料
odps:sql:yangsw_test> select * from dual;
InstanceId: 20141111070328172gre5pdx5
+------------+------+----+
| id | name | dt |
+------------+------+----+
| 1 | tom | 2014 |
| 2 | jacky | 2014 |
| 3 | jet | 2014 |
| 4 | rose | 2014 |
| 5 | marry | 2014 |
| 6 | obama | 2014 |
至此，使用dship上傳資料成功了。
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26613085/viewspace-1328434/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- FTP 客戶端使用教程FTP客戶端
- ftp客戶端，ftp客戶端軟體具體怎麼使用？FTP客戶端
- dbus客戶端使用指南客戶端
- Spring Cloud Config客戶端使用SpringCloud客戶端
- 使用 Webix 建立 Email 客戶端WebAI客戶端
- NEO-GUI 客戶端使用GUI客戶端
- keycloak~scope客戶端模板的使用客戶端
- 郵件客戶端的配置使用客戶端
- 命令列客戶端MySQL如何使用命令列客戶端MySql
- Curator（ZoooKeeper客戶端）使用詳解客戶端
- [jaeger] 二、客戶端使用 (Java版本)客戶端Java
- 中文版vnc客戶端，中文版vnc客戶端下載，中文版vnc客戶端如何使用？VNC客戶端
- Kubernetes客戶端認證（三）—— Kubernetes使用CertificateSigningRequest方式簽發客戶端證書客戶端
- ftp客戶端軟體，ftp客戶端軟體哪個好用，使用方法FTP客戶端
- vnc windows客戶端，vnc windows客戶端下載，具體使用教程。VNCWindows客戶端
- mqtt 客戶端MQQT客戶端
- dubbo客戶端客戶端
- Pulsar客戶端客戶端
- 使用 .NET MAUI 開發 ChatGPT 客戶端UIChatGPT客戶端
- Zookeeper客戶端使用與叢集原理客戶端
- kubernetes 客戶端KubeClient使用及常用api客戶端clientAPI
- C#之使用CefSharp建立客戶端C#客戶端
- ETH官方客戶端Geth的使用(一)客戶端
- 服務端，客戶端服務端客戶端
- 客戶端，服務端客戶端服務端
- Nacos - 客戶端心跳續約及客戶端總結客戶端
- [Redis 客戶端整合] Java 中常用Redis客戶端比較Redis客戶端Java
- 物理DataGuard客戶端無縫切換--客戶端TAF 配置客戶端
- netty系列之:使用netty搭建websocket客戶端NettyWeb客戶端
- 「Oracle」客戶端 PL/SQL DEVELOPER 安裝使用Oracle客戶端SQLDeveloper
- 使用測試客戶端「玩轉」MQTT 5.0客戶端MQQT
- Tomcat配置Gizp 客戶端使用okHttp3Tomcat客戶端HTTP
- [轉載] 使用Redis的Java客戶端JedisRedisJava客戶端
- .net socket.io客戶端使用過程客戶端
- validation客戶端驗證框架使用手冊客戶端框架
- Tower：GIt客戶端Git客戶端
- Zookeeper 客戶端 API客戶端API
- 客戶端加解密客戶端解密
- Redis-客戶端Redis客戶端