XyloSiteMonitor

Check-in [98b51cc783]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Added script and example definitions
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 98b51cc7832509f6ef78d3ff7ac2007cd08cc25c4d2c6edf2b332871af16e597
User & Date: xylon 2018-11-04 13:30:16
Context
2018-11-04
15:35
added a flag for email only on fail and a flag for giving the run a name (annotation) check-in: 7f4091db1d user: xylon tags: trunk
13:30
Added script and example definitions check-in: 98b51cc783 user: xylon tags: trunk
2018-11-03
18:24
initial empty check-in check-in: e2b6476d58 user: xylon tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Added definitions_example.yml.





































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
- name: My bank does it so well
  expected string: Welcome to Online Banking
  canonical address: https://www.halifax-online.co.uk/personal/logon/login.jsp
  urls:
  - url: www.halifax-online.co.uk/personal/logon/login.jsp
    tests:
    - action: return string
      protocols:
        - TLS
    - action: redirect
      protocols:
        - no-TLS
  - url: halifax-online.co.uk
    tests:
    - action: redirect
      protocols:
        - no-TLS
        - TLS
- name: This site passes its tests
  expected string: Freed Computer
  canonical address: http://www.freedcomputer.net/
  urls:
  - url: www.freedcomputer.net
    tests:
    - action: return string
      protocols:
        - TLS
        - no-TLS
  - url: freedcomputer.net
    tests:
    - action: redirect
      protocols:
        - TLS
        - no-TLS

Added xylositemonitor.py.



































































































































































































































































































































































































































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
#! /usr/bin/env python3

import sys
import os
import re
from io import BytesIO
import argparse

import yaml
import pycurl

# Command line arguments.
parser = argparse.ArgumentParser(description='Tests websites.')
parser.add_argument('--sites-file', dest='sitesfile')
parser.add_argument('--mailto', dest='mailaddress')
args = parser.parse_args()
mailto = args.mailaddress
sitesfile = args.sitesfile

# We need to test both that it's not None and that it's not empty
try:
    re.search('[a-zA-Z0-9]+', mailto).group(0)  # this will error on
                                                # both blank string
                                                # and non-string
except (TypeError, AttributeError):
    mailto = False

# We need to test both that it's not None and that it's not empty
try:
    re.search('[a-zA-Z0-9]+', sitesfile).group(0)  # this will error
                                                   # on both blank
                                                   # string and
                                                   # non-string
except (TypeError, AttributeError):
    sitesfile = "/etc/xylosites.yml"

# don't even try to open sitesfile unless it's there
if not os.path.isfile(sitesfile):
    print('Initialisation Error! Cannot find sitesfile at "' + sitesfile +
          '"\nPlease place it here or specify location with --sites-file=')
    sys.exit()

with open(sitesfile, 'r') as stream:
    sites = yaml.load(stream)

def send_mail(subject):
    import smtplib
    from email.mime.text import MIMEText

    msg = MIMEText(mail_body)
    msg['Subject'] = 'XyloSiteMonitor: ' + subject
    msg['To'] = mailto
    msg['From'] = "xylositemonitor"

    smtpcon = smtplib.SMTP('localhost')  # use host as mail relay
    smtpcon.send_message(msg)
    smtpcon.quit()  # close connection


def header_function(header_line):
    """We have to parse http headers manually becasue libcurl doesn't do it for us."""

    # HTTP standard specifies that headers are encoded in iso-8859-1.
    header_line = header_line.decode('iso-8859-1')

    # Header lines include the first status line (HTTP/1.x ...).
    if header_line[:6] == "HTTP/1":
        # get status code
        status = re.search(r'[0123456789]{3}', header_line).group(0)
        headers['status'] = status
        return

    # We are going to ignore all lines that don't have a colon in them.
    # This will botch headers that are split on multiple lines...
    if ':' not in header_line:
        return

    # Break the header line into header name and value.
    hname, value = header_line.split(':', 1)

    # Remove whitespace that may be present.
    # Header lines include the trailing newline, and there may be whitespace
    # around the colon.
    hname = hname.strip()
    value = value.strip()

    # Header names are case insensitive.
    # Lowercase name here.
    hname = hname.lower()

    # Now we can actually record the header name and value.
    # Note: this only works when headers are not duplicated, see below.
    headers[hname] = value

BCOLORS = {
    "HEADER": '\033[95m',
    "OKBLUE": '\033[94m',
    "OKGREEN": '\033[92m',
    "WARNING": '\033[93m',
    "FAIL": '\033[91m',
    "ENDC": '\033[0m',
    "BOLD": '\033[1m',
    "UNDERLINE": '\033[4m',
    }

def config_fail(message):
    if mailto:
        global mail_body
        mail_body += '  Config Error! ' + message + "\n"
        send_mail('config error!')
    else:
        print(BCOLORS["WARNING"] + '  Config Error! ' + message + BCOLORS["ENDC"])

    sys.exit()

def test_fail(message):
    global fail_count

    if mailto:
        global mail_body
        mail_body += "  Test Fail! " + message + "\n"
    else:
        print(BCOLORS["FAIL"] + "  Test Fail! " + message + BCOLORS["ENDC"])

    fail_count += 1

def test_success():
    global success_count

    if mailto:
        global mail_body
        mail_body += " Test Success!" + "\n"
    else:
        print(BCOLORS["OKGREEN"] + " Test Success!" + BCOLORS["ENDC"])

    success_count += 1

success_count = 0
fail_count = 0
if mailto:
    mail_body = ""

for site in sites:
    name = site["name"]
    try:
        ex_string = site["expected string"]
    except KeyError:
        ex_string = ""  # if this var is needed it will be validated and fail later :)

    try:
        can_address = site["canonical address"]
    except KeyError:
        can_address = ""  # if this var is needed it will be validated and fail later :)

    for urldef in site["urls"]:
        url = urldef["url"]
        if url[:7] == "http://" or url[:8] == "https://":
            config_fail('Do not specify protocol in url.')

        for test in urldef["tests"]:
            action = test["action"]

            for protocol in test["protocols"]:
                if protocol == "TLS":
                    prefix = "https://"
                elif protocol == "no-TLS":
                    prefix = "http://"
                else:
                    config_fail('Supported protocols are "TLS" and "no-TLS".')

                for ipver in ("IPv4", "IPv6",):
                    if mailto:
                        mail_body += ipver + ', does "' + url + '" ' + \
                        action + ' over "' + protocol + '"?' + "\n"
                    else:
                        print(ipver + ', does "' + url + '" ' +
                              action + ' over "' + protocol + '"?')

                    if ipver == "IPv4":
                        curliptype = pycurl.IPRESOLVE_V4
                    elif ipver == "IPv6":
                        curliptype = pycurl.IPRESOLVE_V6

                    buffer = BytesIO()
                    c = pycurl.Curl()
                    c.setopt(c.URL, prefix + url)
                    c.setopt(c.FOLLOWLOCATION, False)
                    c.setopt(c.TIMEOUT, 8)
                    c.setopt(c.ACCEPT_ENCODING, "")
                    c.setopt(c.USERAGENT, "xylositemonitor")
                    c.setopt(c.IPRESOLVE, curliptype)
                    c.setopt(c.WRITEFUNCTION, buffer.write)
                    c.setopt(c.HEADERFUNCTION, header_function)

                    headers = {}
                    try:
                        c.perform()
                    except pycurl.error as e:
                        test_fail(str(e))
                        continue

                    c.close()

                    # Figure out what encoding was sent with the response, if any.
                    # Check against lowercased header name.
                    encoding = None
                    if 'content-type' in headers:
                        content_type = headers['content-type'].lower()
                        match = re.search(r'charset=(\S+)', content_type)
                        if match:
                            encoding = match.group(1)
                    if encoding is None:
                        # Default encoding for HTML is iso-8859-1
                        encoding = 'iso-8859-1'

                    body = buffer.getvalue()
                    responsebody = body.decode(encoding)

                    if 'status' not in headers:
                        test_fail("Can't get HTTP response code!")
                        continue

                    # The test hasn't failed yet!
                    # Now we just need to test that "action" has been
                    # met

                    # There are three supported actions
                    # http success
                    #     this just tests for 200 status
                    # return string
                    #     this checks for 200 and the contents of the page for an expected string
                    # redirect
                    #     this checks that the status is a redirect code to the specified URL
                    if action == "http success":
                        if headers['status'] != "200":
                            test_fail("HTTP status is: " + headers['status'])
                            continue
                        else:
                            test_success()
                            continue

                    if action == "return string":
                        # just check at least the status is 200 before even checking string
                        if headers['status'] != "200":
                            test_fail("HTTP status is: " + headers['status'])
                            continue

                        # we need ex_string var for this test
                        try:
                            re.search('[a-zA-Z0-9]+', ex_string).group(0)  # this
                                                                           # will
                                                                           # error
                                                                           # on
                                                                           # both
                                                                           # blank
                                                                           # string
                                                                           # and
                                                                           # non-string
                        except (TypeError, AttributeError):
                            config_fail('"return string" check specified but ' +
                                        '"ex_string" is not defined!')

                        # now we grep for the expected string in the response body
                        if not ex_string in responsebody:
                            test_fail("Don't find expected string!")
                            continue
                        else:
                            test_success()
                            continue

                    if action == "redirect":
                        if headers['status'][:1] != "3":
                            test_fail("Response code is not a redirect: " +headers['status'])
                            continue

                        if 'location' not in headers:
                            test_fail("Response code is a redirect but no Location header!")
                            continue

                        # we need can_address var for this test
                        try:
                            re.search('[a-zA-Z0-9]+', can_address).group(0)  # this
                                                                             # will
                                                                             # error
                                                                             # on
                                                                             # both
                                                                             # blank
                                                                             # string
                                                                             # and
                                                                             # non-string
                        except (TypeError, AttributeError):
                            config_fail('"redirect" check specified but ' +
                                        '"can_address" is not defined!')

                        # now we check redirect location
                        if not headers['location'] == can_address:
                            test_fail("Redirect location is wrong: " + headers['location'])
                            continue
                        else:
                            test_success()
                            continue

                    # if we got here it means we didn't recognise the action
                    config_fail('action not recognised!')

if mailto:
    mail_body += "\n"
    mail_body += "Summary:\n"
    mail_body += str(success_count) + " tests passed\n"
    mail_body += str(fail_count) + " tests failed\n"

    # OK so we've got our mail body, now we just need to work out what our subject is
    if fail_count > 0:
        send_mail(str(fail_count) + ' failing tests!')
    else:
        send_mail("all " + str(success_count) + "tests passed")

print("")
print("Summary:")
print(str(success_count) + " tests passed")
print(str(fail_count) + " tests failed")