weixin_41792059的博客

安全启蒙运动 然而在当今的互联网行业中， 对安全的重视程度普遍不高。 有统计显 示， 互联网公司对安全的投入不足收入的百分之一。 在2011年岁末之际， 中国互联网突然卷入了一场有史以来最大的安全危 机。 12月21日， 国内最大的开发者社区CSDN被黑客在互联网上公布了 600万注册用户的数据。 更糟糕的是， CSDN在数据库中明文保存了用户 的密码。 接下来如同一场盛大的交响乐， 黑客随后陆续公布了网易、 人人、 天涯、 猫扑、 多玩等多家大型网站的数据库， 一时间风声鹤唳， 草 木皆兵。 这些数据其实在黑客的地下世界中已经辗转流传了多年， 牵扯到了一条 巨大的黑色产业链。 这次的偶然事件使之浮出水面， 公之于众， 也让用 户清醒地认识到中国互联网的安全现状有多么糟糕。 以往类似的事件我都会在博客上说点什么， 但这次我保持了沉默。 因为 一来知道此种状况已经多年， 网站只是在为以前的不作为而买单； 二来 要解决“拖库”的问题， 其实是要解决整个互联网公司的安全问题， 远非 保证一个数据库的安全这么简单。 这不是通过一段文字、 一篇文章就能 够讲清楚的。 但我想最好的答案， 可以在本书中找到。 经历这场危机之后， 希望整个中国互联网， 在安全问题的认识上， 能够 有一个新的高度。 那这场危机也就物有所值， 或许还能借此契机成就中 国互联网的一场安全启蒙运动。

What’s in This Book? This book contains a mix of theoretical and practical chapters. For the practical chapters, I’ve developed and made available a networking library called Canape Core, which you can use to build your own tools for protocol analysis and exploitation. I’ve also provided an example networked application called SuperFunkyChat, which implements a userto-user chat protocol. By following the discussions in the chapters, you can use the example application to learn the skills of protocol analysis and attack the sample network protocols. Here is a brief breakdown of each chapter: Chapter 1: The Basics of Networking This chapter describes the basics of computer networking with a particular focus on TCP/IP, which forms the basis of application-level network protocols. Subsequent chapters assume that you have a good grasp of the network basics. This chapter also introduces the approach I use to model application protocols. The model breaks down the application protocol into flexible layers and abstracts complex technical detail, allowing you to focus on the bespoke parts of the protocol you’re analyzing. Chapter 2: Capturing Application Traffic This chapter introduces the concepts of passive and active capture of network traffic, and it’s the first chapter to use the Canape Core network libraries for practical tasks. Chapter 3: Network Protocol Structures Download from finelybook www.finelybook.com 24This chapter contains details of the internal structures that are common across network protocols, such as the representation of numbers or human-readable text. When you’re analyzing captured network traffic, you can use this knowledge to quickly identify common structures, speeding up your analysis. Chapter 4: Advanced Application Traffic Capture This chapter explores a number of more advanced capture techniques that complement the examples in Chapter 2. The advanced capture techniques include configuring Network Address Translation to redirect traffic of interest and spoofing the address resolution protocol. Chapter 5: Analysis from the Wire This chapter introduces methods for analyzing captured network traffic using the passive and active techniques described in Chapter 2. In this chapter, we begin using the SuperFunkyChat application to generate example traffic. Chapter 6: Application Reverse Engineering This chapter describes techniques for reverse engineering networkconnected programs. Reverse engineering allows you to analyze a protocol without needing to capture example traffic. These methods also help to identify how custom encryption or obfuscation is implemented so you can better analyze traffic you’ve captured. Chapter 7: Network Protocol Security This chapter provides background information on techniques and cryptographic algorithms used to secure network protocols. Protecting the contents of network traffic from disclosure or tampering as it travels over public networks is of the utmost importance for network protocol security. Chapter 8: Implementing the Network Protocol This chapter explains techniques for implementing the application Download from finelybook www.finelybook.com 25network protocol in your own code so you can test the protocol’s behavior to find security weaknesses. Chapter 9: The Root Causes of Vulnerabilities This chapter describes common security vulnerabilities you’ll encounter in a network protocol. When you understand the root causes of vulnerabilities, you can more easily identify them during analysis. Chapter 10: Finding and Exploiting Security Vulnerabilities This chapter describes processes for finding security vulnerabilities based on the root causes in Chapter 9 and demonstrates a number of ways of exploiting them, including developing your own shell code and bypassing exploit mitigations through return-oriented programming. Appendix: Network Protocol Analysis Toolkit In the appendix, you’ll find descriptions of some of the tools I commonly use when performing network protocol analysis. Many of the tools are described briefly in the main body of the text as well. How to Use This Book If you want to start with a refresher on the basics of networking, read Chapter 1 first. When you’re familiar with the basics, proceed to Chapters 2, 3, and 5 for practical experience in capturing network traffic and learning the network protocol analysis process. With the knowledge of the principles of network traffic capture and analysis, you can then move on to Chapters 7 through 10 for practical information on how to find and exploit security vulnerabilities in these protocols. Chapters 4 and 6 contain more advanced information about additional capture techniques and application reverse engineering, so you can read them after you’ve read the other chapters if you prefer. For the practical examples, you’ll need to install .NET Core Download from finelybook www.finelybook.com 26(https://www.microsoft.com/net/core/), which is a cross-platform version of the .NET runtime from Microsoft that works on Windows, Linux, and macOS. You can then download releases for Canape Core from https://github.com/tyranid/CANAPE.Core/releases/ and SuperFunkyChat from https://github.com/tyranid/ExampleChatApplication/releases/; both use .NET Core as the runtime. Links to each site are available with the book’s resources at https://www.nostarch.com/networkprotocols/. To execute the example Canape Core scripts, you’ll need to use the CANAPE.Cli application, which will be in the release package downloaded from the Canape Core Github repository. Execute the script with the following command line, replacing script.csx with the name of the script you want to execute.

本课题的主要目的是设计面向定向网站的网络爬虫程序，同时需要满足不同的性能要求，详细涉及到定向网络爬虫的各个细节与应用环节。 搜索引擎作为一个辅助人们检索信息的工具。但是，这些通用性搜索引擎也存在着一定的局限性。不同领域、不同背景的用户往往具有不同的检索目的和需求，通用搜索引擎所返回的结果包含大量用户不关心的网页。为了解决这个问题，一个灵活的爬虫有着无可替代的重要意义。 网络爬虫应用智能自构造技术，随着不同主题的网站，可以自动分析构造URL，去重。网络爬虫使用多线程技术，让爬虫具备更强大的抓取能力。对网络爬虫的连接网络设置连接及读取时间，避免无限制的等待。为了适应不同需求，使网络爬虫可以根据预先设定的主题实现对特定主题的爬取。研究网络爬虫的原理并实现爬虫的相关功能,并将爬去的数据清洗之后存入数据库，后期可视化显示。