codoseo-crawler 0.0.1

The CodoSEO crawler: fetching, robots.txt, sitemaps and streaming field extraction.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<!doctype html>
<html lang="en"><head>
<meta charset="utf-8">
<title>Tents &amp; Shelters | Northwind</title>
<meta name="description" content="Backpacking tents &amp; tarps.">
<link rel="canonical" href="https://northwind.test/tents">
<link rel="alternate" hreflang="de" href="/de/zelte">
</head><body>
<h1>Tents  &amp;
 Shelters</h1>
<h2>Two person</h2><h2>Family</h2>
<p>Our tents are light and strong.</p>
<a href="/tents/ridgeline-2p">Ridgeline <b>2P</b></a>
<a href="https://other.test/x">Partner</a>
<a href="mailto:hi@northwind.test">Mail</a>
<script>var hidden = "do not count these words";</script>
<style>.x{color:red}</style>
</body></html>