- 博客(60)
- 问答 (4)
- 收藏
- 关注
RSS订阅原创 更新 macOS Ventura ssh堡垒机报错：no matching host key type found. Their offer: ssh-rsa,ssh-dss
解决方法：增加两行即可。
2023-04-17 10:26:55
185
原创 hive表直接select * 可以查出结果，但是join查询的字段中文变乱码
hive表直接select * 可以查出结果，但是join查询的字段中文变乱码的解决办法和原因
2023-01-12 17:12:13
297
原创 The basic of Java-Extends,Encapsulation,Abstract,Object,Singleton,Static(Day04)
1. We cannot create local variables in 'static' method, since local variables are existed in stack, and they will be destroyed when the method are down, conversely, 'static' stored in method area.2.
2021-08-19 11:09:08
109
原创 The advance of Java -- Set, LinkedList, Map(Day03)
1. Comparable:A class implements Comparable, which means it own rule of comparison, like TreeSet. It need the elements we put must implements comparable interface and remember add generics.2. Com
2021-08-19 11:08:57
85
原创 The advance of Java -- Set, LinkedList, Map(Day03)
1. Comparable:A class implements Comparable, which means it own rule of comparison, like TreeSet. It need the elements we put must implements comparable interface and remember add generics.2. Com
2021-08-19 11:08:46
75
原创 The advance of Java -- Genericity, Exception, IO(Day04)
1. Genericity:①: only T typepublic class Dog { X str; int age; void test(){ System.out.println(str); }}public static void main(String[] args) { Dog s = new Dog<>(); s.str = "pd"; s.
2021-08-19 11:08:36
46
原创 The advance of Java -- IO, Thread(Day05)
1. IO①Serialize: When we need to transfer a oject from a computer to another computer's disk, we use serialize. ObjectOutputStream oosDog d = new Dog("H",12,"sssssssssss"); try ( ObjectOutpu
2021-08-19 11:08:22
71
原创 The advance of Java -- IO, Thread(Day05)
1. IO①Serialize: When we need to transfer a oject from a computer to another computer's disk, we use serialize. ObjectOutputStream oosDog d = new Dog("H",12,"sssssssssss"); try ( ObjectOutpu
2021-08-19 11:08:08
70
原创 The advance of Java -- JDBC(Day09)
JDBC(Java Database Connectivity)①JDBC only define interface, the detailed realization is responsible for database company. So we only need to call interface.
2021-08-19 11:07:52
41
原创 Hadoop(day01) -- Tomcat(Linux), Jdk(Linux) (), JVM
(First: VMWARE-LINUX(CENTOS/6.5/6.6))一.Jdk:1.Delete old jdk:2.add tomcat.tar.jar.gz:3.tar and mv:4.configure environment variables:(the bottom line)export JAVA_HOME=/usr/java
2021-08-19 11:07:38
67
原创 Hadoop(Day12) -- Hadoop-HA
一.Cluster Design IP HostnameNamenodeDatanodeZOOKEEPERHIVEHBASE192.168.16.100 name1 Y N Y NY192.168.16.101 name2 Y N Y NY1
2021-08-19 11:06:52
95
原创 解决SonarQube安装问题
无论你安装的什么版本，一定一定首先要看官方文档。1.比如我是7.6，官方说要创建一个空的schema和sonarqube用户，结果我创了个sonar用户，死活起不来，最后根据官方文档，终于成功了。2.如果MySQL的my.ini文件以前加了 innodb_force_recovery = 1 类似的一定要删掉，删掉重启mySQL也能继续跑，否则你在错误日志中能看到如下信息3.也是最...
2019-03-19 00:12:15
343
原创 解决mysql Server1067问题
1.查看my,ini设置的datadir: F:\mysql-5.6\data下的.err文件2.找到错误报告如图3.根据日志和提示知道是innoDB的问题，于是在my.ini下增加innodb_force_recovery = 1即可...
2019-03-18 05:47:04
134
原创 ElasticSearch
一.Overviewhttps://en.wikipedia.org/wiki/Elasticsearch二.Installation#We need an none-root user：useradd bigdataecho 123456 | passwd --stdin bigdata#Add bigdata into sudoersecho "bigdata ALL = (root) NOP...
2018-04-02 20:37:38
162
原创 Spark(day10) -- MLlib(2)
一.Principle of decision tree classification algorithm.1）OverviewDecision tree is a widely used classification algorithm.Compared with bayesian algorithm, the advantage of decision tree is that the con...
2018-03-28 22:01:18
208
原创 Spark(day09) -- MLlib(1)
一.Overviewhttps://en.wikipedia.org/wiki/Machine_learning二.Arithmetic (1)Linear Regression(2)K-Nearest NeighborIn essence, KNN algorithm USES distance to measure the similarity between samples.The algo...
2018-03-27 21:57:53
263
原创 Spark(day08) -- SpqrkSql
一.Introductionhttp://spark.apache.org/docs/latest/sql-programming-guide.html二.Syntax1.Create RDD2.Create case class(like table's schema)3.Associate RDD with case class.4.rdd.toDF一.DSL style synt...
2018-03-26 21:11:45
268
原创 Spark(day07) -- SparkStreaming
一.Introductionhttp://spark.apache.org/docs/latest/streaming-programming-guide.html*Important jar:commons-lang3-3.3.2.jarscala-library-2.10.5.jarspark-streaming-flume-sink_2.10-1.6.1.jarspark-streaming...
2018-03-26 20:50:28
174
原创 Spark(day06) -- logstash,kafka
一.The message storage of kafka and Pro-Con Model·Consumer manage to consume offset·Each consumer has their own group·There is queue model in group-Each consumer consume different partiition-In this wa...
2018-03-23 20:55:53
208
原创 Spark(day05) -- Kafka(1)
一.Why Spark need a message queue?• The answer is easy，resolving the coupling problem with other system.• How to gain input data? – spark do not have receiver– and pull data from data source can contro...
2018-03-21 20:51:43
112
转载 spark on yarn和mapreduce on yarn区别
Apache Spark的高性能一定程度上取决于它采用的异步并发模型（这里指server/driver端采用的模型），这与Hadoop 2.0（包括YARN和MapReduce）是一致的。Hadoop 2.0自己实现了类似Actor的异步并发模型，实现方式是epoll+状态机，而Apache Spark则直接采用了开源软件Akka，该软件实现了 Actor模型，性能非常高。尽管二者在ser...
2018-03-21 19:02:56
250
原创 Spark(day04) -- Pratices
1.SparkTA:import org.apache.spark.{SparkConf, SparkContext}object SparkTA { def main(args: Array[String]): Unit = { val conf=new SparkConf().setAppName("SparkTA").setMaster("local[2]") val...
2018-03-17 21:21:15
218
原创 Spark(day03) -- Installation,RDD(WC)
一.Introductionhttps://en.wikipedia.org/wiki/Apache_Spark二.Installation && Test1.tar -xzvf spark-1.6.1-bin-hadoop2.6.tgz2.mv spark-1.6.1-bin-hadoop2.6 spark-1.6.13.vi /etc/profile4, vi slaves5,...
2018-03-14 20:49:14
168
原创 Spark(day02) -- Scala
1.WordCountobject ScalaWC { //定义一个list val lines =List("hello tom hello bob hello jerry","hello tom hello bob hello jj") //压平并切分 val word =lines.flatMap(_.split(" ")) //单词作为key 1作为value v...
2018-03-13 19:28:43
132
原创 Spark(day01) -- Scala
一.Introductionhttps://en.wikipedia.org/wiki/Scala_(programming_language)二.Data Type三.Example1.functional orientationobject HelloScala extends App{ //yield 关键字 生成新数组 val z=for (i <- 1 to 3...
2018-03-10 21:17:29
193
原创 Hadoop总结杂文
*标题后数字对应Hadoop权威指南第4版页数一. Hadoop-Ha规避机制(49)1. 撤销namenode访问共享存储目录的权限2. 通过远程管理命令屏蔽相应的网络端口3. STONITH（shot the other node in the head）:通过一个特定的供电单元对相应主机进行断电操作 二. 通过distcp并行复制(76)用来处理复制大文件默认情况下开启20个map,每个文件...
2018-03-06 20:24:30
230
原创 Hadoop(Day12) -- Hadoop-HA
一.Cluster Design IP Hostname Namenode Datanode ZOOKEEPER HIVE HBASE192.168.16.100 name1 Y N Y N Y192.168.16.101 name2 Y N Y N Y192.1
2018-02-07 10:54:07
375
原创 Hadopp(Day13) -- Sqoop,Flume,Azkaban
一.Sqoop1.Prefacehttps://en.wikipedia.org/wiki/Sqoop2.Installation①transfer tar under path: /usr/local/②tar xzvf xxxx③vi /etc/profilesource /etc/profile3.Practice①Command Exapmlesqoop import --connect ...
2018-02-03 22:11:32
133
原创 BIgData(dayo03) -- MongoDB,MongoCluster
一.Prefacehttps://en.wikipedia.org/wiki/MongoDB二.Installation(1)Win7:1.Adrress: https://www.mongodb.com/download-center#community. mongodb-win32-x86_64-2008plus-ssl-3.4.3-signed.msi
2018-01-31 20:18:36
213
原创 Hadoop(Day11) -- Hbase,Eclipse
一.Preface:https://en.wikipedia.org/wiki/Apache_HBase二.Installation1.Transfer jar under /usr/local2.Tar -xzvf jar3.mv xxx hbase4.vi /etc/profilesource /etc/profi
2018-01-24 21:44:51
238
原创 Hadoop(Day10) -- Eclipse+Hive,DbVisualizer
一.Eclipse + Hive:HIveJdbcClient.java:package com.bsr.hive;import java.sql.Connection;import java.sql.DriverManager;import java.sql.PreparedStatement;import java.sql.ResultSet;import java.s
2018-01-23 21:13:49
175
原创 Hadoop(Day09) -- Hive
一.Preface:https://en.wikipedia.org/wiki/Apache_Hive二.Installation:1.Transfer tar.gz under /usr/local2.tar xzvf 3.vi /etc/profile4.source /etc/profile5.mysql -uroot -prootCREATE USER 'hive' IDENTIFIED ...
2018-01-22 21:56:06
269
原创 Hadoop(Day08) -- MapReduce(Day03), Eclipse
一.MapReduce:1.CombinerDemo.java:package com.bsr.bigdata.mapreduce;import java.net.URI; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.a
2018-01-16 21:59:34
176
原创 Hadoop(Day07) -- MapReduce(Day 02), Eclipse
一.Version：1.MapReduce(v1.0):2.MapReduce(v2.0):二.Code:Example: 1.Counting the data of phone (partition):FlowBean.java:package com.bsr.bigdata.mapreduce.part_pd;import ja
2018-01-15 20:00:21
170
原创 Hadoop(Day06) -- MapReduce, Eclipse
一.IntroductionMapReduce Introduction(Wiki)二.Eclipse + MapReduce1.Setting2.Code:WordcountMapper.java:package com.bsr.bigdata.mapreduce;import java.io.IOException;import org.
2018-01-12 09:38:59
228
原创 Hadoop(Day05) -- Win10, HDFS, Eclipse
一.Installation:1.Unzip hadoop2.Copy one and put it under C:\3.Unzip hadoop.dll and winutils.exe and put them under C:\hadoop\bin4.Copy hadoop.dll under C:\Windows\System32\5.Alter hadoop-env
2018-01-10 21:04:17
202
原创 Hadoop(Day04) -- ZooKeeper
一.IntroductionApache ZooKeeper is a software project of the Apache Software Foundation. It is essentially a distributed hierarchical key-value store, which is used to provide a distributed c
2018-01-09 20:37:11
174
原创 Hadoop(Day03) -- Introduction, Installation
一.Preface1.What is mass data processing:There is a large amount of data, different data complexity, different size of individual data files, different sources of data, and mainly storage intensive
2018-01-08 20:44:46
199
原创 Big Data(day02) -- Architecture, V pattern, Wireless transmission, Design pattern
一.Architecture View1.Logical View -- User2.Development View -- Developer Engineering3.Physical View -- OPS Engineering4.Process View -- Senior member or Architect二.Wireless transmission
2018-01-05 15:30:46
276
原创 Redis(Day02) -- Cluster, Data type, JRedis
一.Setting:1.mkdir -p /redis_c/8000 /redis_c/8001 /redis_c/8002 /redis_c/8003 /redis_c/8004 /redis_c/80052.vi redis-ha.sh#!/bin/bashfor i in `ls /redis_c/`docd /redis_c/$iecho "po
2018-01-05 09:57:48
579
空空如也
django ForeignKey 如何排序？
2019-12-21
关于java多线程的问题，求大神帮忙，很急，感谢
2019-10-08
关于java多线程的问题，求大神帮忙，很急，感谢
2019-10-08
Oracle: 如何将两列数据合并成一列数据并像个set集合查出来?
2019-02-03
TA创建的收藏夹 TA关注的收藏夹
TA关注的人