chinasiyu-CSDN博客

原创 python spark streaming 从 kafka获取nginx日志，秒级统计url pv，并写入mysql

mysql建库建表>create database nginx;>use nginx> create table url_access (id int NOT NULL AUTO_INCREMENT primary key, timestamp varchar(256), url varchar(256), pv long); #!/usr/bin/env ...

2019-01-09 23:44:03 515

原创 python kafka消息传入spark streaming 实时计算打印

实现了从kafka取nginx日志，并在本地打印出来 #!/usr/bin/env python# -*- coding: utf-8 -*-# @Time : 2019/01/09 18:00# @Author : xuanda# @Site : # @File : kafka_to_sparkstreaming.pyfrom pyspark import...

2019-01-09 18:12:34 755

原创 python spark streaming单机测试，streaming源为本地，统计word count

网上有很多pyspark streaming的测试代码，不过大多都是需要结合kafka做消息来源由于懒得搭kafka，所以想本地生成随机数据作为streaming源，测试spark streaming google查了一些文章，其实spark github中就有类似代码，只不过文件名叫 hdfs_wordcount.py，是针对hdfs的examplehttps://github....

2019-01-03 17:08:28 606

原创 python连接standalone版spark

昨天已经完成了standalone的spark搭建，当然今天在实际使用python连接时，发现其实是有问题的。master和slave其实并没有成功连接，原因是启动slave时指定的master ip地址有误，需要通过netstat查看7077在哪个ip上侦听，不可直接写127.0.0.1 今天成功用python连接了昨天搭建的spark集群做简单计算环境：python3...

2019-01-02 16:24:21 479

原创 spark单机模式快速搭建

运维做了8年，带了好几年团队，但是逐渐发现自己技术上飘飘然，前几年还写写代码，做做运维，现在只能探探规划，看看方向。什么spark、docker、k8s、hadoop、open-falcon、devops、aiops都带团队做过，但自己真没具体做过。今天开始在运维、开发领域做个新手，重新从0开始学习，看看自己是不是既能上天又能入地 Spark介绍：Spark程序在集群中独立运行，通过S...

2019-01-01 23:29:11 3700

转载 7.5. Packet Filtering with ipchains

7.5. Packet Filtering with ipchainsthe three builtin chains, input, output, forward policy per chain, see targets jumping from chain to chain, -j $TARGET; wher TARGET=chain the big

2010-01-01 19:31:00 454

转载 7.6. Protecting a Host

7.6. Protecting a HostHost protection in the past was typically performed with application layer checks on the originating IP or hostname. This was (and still is) frequently accomplished

2010-01-01 19:31:00 461

转载 7.8. Further Resources

7.8. Further ResourcesThe use of linux packet filtering features is mature and well-documented in many places throughout the Internet. One of the most thorough introductions to the use o

2010-01-01 19:31:00 754

转载 7.1. Rationale for and Introduction to Packet Filtering

7.1. Rationale for and Introduction to Packet FilteringPacket filtering refers to the technique of conditionally allowing or denying packets entering or exiting a network or host based o

2010-01-01 19:30:00 490

转载 7.2. Limits and Weaknesses of Packet Filtering

7.2. Limits and Weaknesses of Packet FilteringAlthough the functionality offered by linux kernels for protecting network resources with packet filtering allows tremendously specific netw

2010-01-01 19:30:00 467

转载 7.3. General Packet Filter Requirements

7.3. General Packet Filter Requirementsminimum ICMP required to meet the networking needs; xref PMTU discussion source quench parameter problem inbound destination unreachable ou

2010-01-01 19:30:00 467

转载 7.4. The Netfilter Architecture

7.4. The Netfilter Architecturepacket filtering engine in kernel 2.2 (skip history, adequately documented elsewhere) packet filtering engine as part of netfilter in kernel 2.4, backwar

2010-01-01 19:30:00 406

转载 6.1. Concepts of Source NAT

6.1. Concepts of Source NAT 6.1.1. Differences Between SNAT and MasqueradingThough SNAT and masquerading perform the same fundamental function, mapping one address spac

2010-01-01 19:29:00 365

转载 Chapter 7. Packet Filtering

Chapter 7. Packet FilteringTable of Contents7.1. Rationale for and Introduction to Packet Filtering7.1.1. History of Linux Packet Filter Support7.2. Limits and Weaknesses of Packet F

2010-01-01 19:29:00 688

转载 Chapter 6. Masquerading and Source Network Address Translation

Chapter 6. Masquerading and Source Network Address TranslationTable of Contents6.1. Concepts of Source NAT6.1.1. Differences Between SNAT and Masquerading6.1.2. Double SNAT/Masquerad

2010-01-01 19:28:00 492

转载 5.4. Stateless NAT and Packet Filtering

5.4. Stateless NAT and Packet FilteringBecause NAT rewrites the packet as it passes through the IP stack, packet filtering can become complex. With attentiveness to the addressing of the

2009-12-31 18:43:00 796

转载 5.5. Destination NAT with netfilter (DNAT)

5.5. Destination NAT with netfilter (DNAT)Destination NAT with netfilter is commonly used to publish a service from an internal RFC 1918 network to a publicly accessible IP. To enable DN

2009-12-31 18:43:00 367

转载 5.6. Port Address Translation (PAT) from Userspace

5.6. Port Address Translation (PAT) from UserspacePort address translation (hereafter PAT) provides a similar functionality to NAT, but is a more specific tool. PAT forwards requests for

2009-12-31 18:43:00 603

转载 5.2. Application Layer Protocols with Embedded Network Information

5.2. Application Layer Protocols with Embedded Network InformationNetwork address translation is beautifully invisible when it works, but has adverse effects on some protocols. Some netw

2009-12-31 18:42:00 399

转载 5.3. Stateless NAT with iproute2

5.3. Stateless NAT with iproute2Stateless NAT, occasionally maligned as dumb NAT [31], is the simplest form of NAT. It involves rewriting addresses passing through a routing device: inbo

2009-12-31 18:42:00 792

转载 5.1. Rationale for and Introduction to NAT

5.1. Rationale for and Introduction to NATNetwork address translation (NAT) is a technique of transparently mapping an IP address or range to another IP address or range. Any routing dev

2009-12-31 18:41:00 556

转载 Chapter 5. Network Address Translation (NAT)

Chapter 5. Network Address Translation (NAT)Table of Contents5.1. Rationale for and Introduction to NAT5.2. Application Layer Protocols with Embedded Network Information5.3. Stateles

2009-12-31 18:40:00 638

转载 4.9. Routing Policy Database (RPDB)

4.9. Routing Policy Database (RPDB) The routing policy database (RPDB) controls the order in which the kernel searches through the routing tables. Each rule has a priority, and rules a

2009-12-30 20:53:00 885

转载 4.10. ICMP and Routing

4.10. ICMP and RoutingICMP is a very important part of the communication between hosts on IP networks. Used by routers and endpoints (clients and servers) ICMP communicates error conditi

2009-12-30 20:53:00 504

转载 4.8. Routing Tables

4.8. Routing TablesLinux kernel 2.2 and 2.4 support multiple routing tables [22]. Beyond the two commonly used routing tables (the local and main routing tables), the kernel supports u

2009-12-30 20:52:00 1209

转载 4.6. Source Address Selection

4.6. Source Address Selection The selection of the correct source address is key to correct communication between hosts with multiple IP addresses. If a host chooses an address from a

2009-12-30 20:51:00 715

转载 4.7. Routing Cache

4.7. Routing CacheThe routing cache is also known as the forwarding information base (FIB). This term may be familiar to users of other routing systems. The routing cache stores rece

2009-12-30 20:51:00 451

转载 4.3. Sending Packets Through a Gateway

4.3. Sending Packets Through a Gateway By comparison to the total number of publicly accessible hosts on the Internet there is an almost insignificant number of hosts inside any locall

2009-12-30 20:50:00 375

转载 4.4. Operating as a Router

4.4. Operating as a RouterOperating as a router allows a linux machine to accept packets on one interface and transmit them on another. This is the nature of a router. The process of a

2009-12-30 20:50:00 302

转载 4.5. Route Selection

4.5. Route Selection Crucial to the proper ability of hosts to exchange IP packets is the correct selection of a route to the destination. The rules for the selection of route path are

2009-12-30 20:50:00 598

转载 4.2. Routing to Locally Connected Networks

4.2. Routing to Locally Connected Networks Any IP network is defined by two sets of numbers: network address and netmask. By convention, there are two ways to represent these two numbe

2009-12-30 20:49:00 507

转载 4.1. Introduction to Linux Routing

4.1. Introduction to Linux RoutingThe design of IP routing allows for very simple route definitions for small networks, while not hindering the flexibility of routing in complex environm

2009-12-30 20:47:00 462

转载 Chapter 4. IP Routing

Chapter 4. IP RoutingTable of Contents4.1. Introduction to Linux Routing4.2. Routing to Locally Connected Networks4.3. Sending Packets Through a Gateway4.4. Operating as a Router4.5.

2009-12-30 20:46:00 362

转载 Chapter 3. Bridging

Chapter 3. BridgingTable of Contents3.1. Concepts of Bridging3.2. Bridging and Spanning Tree Protocol3.3. Bridging and Packet Filtering3.4. Traffic Control with a Bridge3.5. ebtables

2009-12-30 20:45:00 364

转载 2.3. ARP filtering

2.3. ARP filteringThis section should be part of the "ghetto" which will include documentation on ip arp. Theres nothing more to add here at the moment (low priority). #

2009-12-30 20:44:00 416

转载 2.4. Connecting to an Ethernet 802.1q VLAN

2.4. Connecting to an Ethernet 802.1q VLAN Virtual LANs are a way to take a single switch and subdivide it into logical media segments. A single switch port in a VLAN-capable switch ca

2009-12-30 20:44:00 418

转载 2.5. Link Aggregation and High Availability with Bonding

2.5. Link Aggregation and High Availability with Bonding Networking vendors have long offered a functionality for aggregating bandwidth across multiple physical links to a switch. This

2009-12-30 20:44:00 548

转载 2.2. Proxy ARP

2.2. Proxy ARPOccasionally, an IP network must be split into separate segments. Proxy ARP can be used for increased control over packets exchanged between two hosts or to limit exposur

2009-12-30 20:43:00 395

转载 2.1. Address Resolution Protocol (ARP)

2.1. Address Resolution Protocol (ARP)Address Resolution Protocol (ARP) hovers in the shadows of most networks. Because of its simplicity, by comparison to higher layer protocols, ARP ra

2009-12-30 19:35:00 702

转载 Chapter 2. Ethernet Table of Contents

Chapter 2. EthernetTable of Contents2.1. Address Resolution Protocol (ARP)2.1.1. Overview of Address Resolution Protocol2.1.2. The ARP cache2.1.3. ARP Suppression2.1.4. The ARP Flux Prob

2009-12-30 19:33:00 410

空空如也

空空如也