author | Oleksandr Gavenko <gavenkoa@gmail.com> |
Mon, 19 Oct 2015 17:04:52 +0300 | |
changeset 1769 | cadd23f8127f |
parent 1346 | a2fbf50a43f4 |
child 1905 | fba288d59662 |
permissions | -rw-r--r-- |
1334
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
1 |
.. -*- coding: utf-8; -*- |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
2 |
.. include:: HEADER.rst |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
3 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
4 |
======= |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
5 |
HTTP. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
6 |
======= |
1346
a2fbf50a43f4
Fix: Has no 'contents::' directive.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
1334
diff
changeset
|
7 |
.. contents:: |
1334
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
8 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
9 |
Command line tools for HTTP protocol. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
10 |
===================================== |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
11 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
12 |
http://daniel.haxx.se/docs/curl-vs-wget.html |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
13 |
Comparing curl and wget. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
14 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
15 |
Recursive site download. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
16 |
======================== |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
17 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
18 |
``curl`` does not allow recursive site downloading, only ``wget`` can do this:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
19 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
20 |
$ wget -r -np -nc -e robots=off -p -k http://DOMAIN/PATH |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
21 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
22 |
Get server response header. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
23 |
=========================== |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
24 |
:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
25 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
26 |
$ wget --server-response http://example.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
27 |
$ wget -S http://example.com # short variant |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
28 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
29 |
$ curl --dump-header - http://example.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
30 |
$ curl -D - http://example.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
31 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
32 |
View cookies from site. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
33 |
======================= |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
34 |
:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
35 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
36 |
$ wget --save-cookies FILE -O - http://example.com >/dev/null |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
37 |
$ curl --cookie-jar FILE -o - http://example.com >/dev/null |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
38 |
$ curl -c FILE -o - http://example.com >/dev/null |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
39 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
40 |
Send cookies to site. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
41 |
===================== |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
42 |
:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
43 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
44 |
$ wget --load-cookies FILE http://example.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
45 |
$ curl --cookie $name=$data http://example.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
46 |
$ curl -b $name=$data http://example.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
47 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
48 |
Send specific header line. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
49 |
========================== |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
50 |
:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
51 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
52 |
$ wget --header='Accept-Charset: iso-8859-2' --header='Accept-Language: ru' http://example.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
53 |
$ curl --header 'Accept-Charset: iso-8859-2' http://example.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
54 |
$ curl -H 'Accept-Charset: iso-8859-2' http://example.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
55 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
56 |
Send POST request. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
57 |
================== |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
58 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
59 |
Log in to the server. This can be done only once:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
60 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
61 |
$ wget --save-cookies cookies.txt --post-data 'user=foo&password=bar' http://server.com/auth.php |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
62 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
63 |
Now grab the page or pages we care about:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
64 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
65 |
$ wget --load-cookies cookies.txt -p http://server.com/interesting/article.php |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
66 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
67 |
Web server in Cygwin. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
68 |
===================== |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
69 |
:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
70 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
71 |
$ setup -p apache2,lighttpd,dhttp |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
72 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
73 |
Compressing HTTP data. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
74 |
====================== |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
75 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
76 |
Starting with HTTP/1.1, web clients can indicate support for compression:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
77 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
78 |
Accept-Encoding: gzip, deflate |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
79 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
80 |
Web server notifies the web client of this via the Content-Encoding header in |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
81 |
the response:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
82 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
83 |
Content-Encoding: gzip |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
84 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
85 |
ETags. |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
86 |
====== |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
87 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
88 |
Server respond:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
89 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
90 |
HTTP/1.1 200 OK |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
91 |
Last-Modified: Tue, 12 Dec 2006 03:03:59 GMT |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
92 |
ETag: "10c24bc-4ab-457e1c1f" |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
93 |
Content-Length: 12195 |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
94 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
95 |
Lately client send:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
96 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
97 |
GET /i/yahoo.gif HTTP/1.1 |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
98 |
Host: us.yimg.com |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
99 |
If-Modified-Since: Tue, 12 Dec 2006 03:03:59 GMT |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
100 |
If-None-Match: "10c24bc-4ab-457e1c1f" |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
101 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
102 |
and get respond:: |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
103 |
|
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
104 |
HTTP/1.1 304 Not Modified |
9bf0d5a1f0cf
Include common header with quick links.
Oleksandr Gavenko <gavenkoa@gmail.com>
parents:
diff
changeset
|
105 |