- 博客(0)
- 资源 (10)
- 收藏
- 关注
RSS订阅spark sftp 2.11
spark 读取 linux sftp上的文本文件，原jar只支持josn,csv等，增加bcp,txt文件的支持
下面是例子：
public static void main(String[] args) throws Exception {
SparkConf conf = new SparkConf().setMaster("local").setAppName("SparkDataFrame");
JavaSparkContext javacontext = new JavaSparkContext(conf);
SQLContext sqlContext = new SQLContext(javacontext);
Dataset<Row> df = sqlContext.read().
format("com.springml.spark.sftp").
option("host", "192.168.1.3").
option("username", "root").
option("password", "111111").
option("fileType", "bcp").
load("/sparktest/sparkfile0.bcp");
/*List<Row> list = df.collectAsList();
for(Row row:list){
String[] words = new String(row.getString(0).getBytes(),0,row.getString(0).length(),"UTF-8").split(" ",-1);
for(int i=0;i<words.length;i++){
System.out.println("words==="+words[i]);
}
}*/
JavaRDD<Row> rowRdd = df.javaRDD();
JavaRDD<Row> words_bcp= rowRdd.map(new Function<Row, Row>() {
@Override
public Row call(Row row) throws Exception {
// TODO Auto-generated method stub
String line = row.getString(0);
String[] words = new String(line.getBytes(),0,line.getBytes().length,"utf-8").split(" ",-1);
return RowFactory.create(words);
}
});
List<Row> list = words_bcp.collect();
for(Row row:list){
System.out.println("row1=="+row.getString(0));
}
df.write().format("com.springml.spark.sftp").
option("host", "192.168.1.3").
option("username", "root").
option("password", "111111").
option("fileType", "bcp").
save("/sparktest/luozhao.bcp");
df.show();
javacontext.close();
}
2018-06-29
spark sftp
spark 读取sftp文件
下面是用法
// Construct Spark dataframe using file in FTP server
DataFrame df = spark.read().
format("com.springml.spark.sftp").
option("host", "SFTP_HOST").
option("username", "SFTP_USER").
option("password", "****").
option("fileType", "json").
load("/ftp/files/sample.json")
// Write dataframe as CSV file to FTP server
df.write().
format("com.springml.spark.sftp").
option("host", "SFTP_HOST").
option("username", "SFTP_USER").
option("password", "****").
option("fileType", "json").
save("/ftp/files/sample.json");
2018-06-20
发送彩信如何生成smil文件
发送彩信如何生成smil文件,压缩包里面有两个文件，都可以，方法大同小异，可以参考下Ambulant-2.2-win32.exe可以播放smil文件
2011-08-02
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人