Site outgoing traffic is often limited, especially if it is not a big portal or mall. Thus it wouldn't be wise to select 'heavy', big HTML page to check, it could, under certain circumstances, drain the site of its traffic quota sooner. An ideal solution would be to have specially crafted pages, light in size and causing as little site load as possible (e.g. with few or none database requests and the like).

Another point is sending appropriate headers. In modern Web applications a number of checks may be made to prevent automated access to the system - a common means to ward off such scripts as email harvesters, automated reply bots and the like. Thus, not only sensible information should be passed in headers of the request, but also one should make sure that the POST request is done, if necessary, after the GET request is sent first, and pass whatever cookies are set.

Finally, I would be careful not to send any type of monitoring requests too often. That could not only load site too much, but could also trigger a spam or DoS alert on the site. So, idea is to verify the site presence with lighter tools, such as PING, wherever those are available, and only occasionally send the GET or POST requests.

A polling interval of 60 seconds could be enough for any real-life task, and if necessary, pinging could be performed several times a minute, and only after that a real HTTP request would be sent.

The resume is: to check the site state, one should choose lightweight page(s), perform occasional polling/requesting pages, setting headers as expected and testing the page contents, without an assumption that the error code would be really returned if an error occurs.

Source:

Nowadays many people run a site, and often there are many sites a person has - such as blogs, personal cards, portals and so on. Site downtime is th type of event that could cost too much to its owner, especially if that's well-known site, e-shop or any popular network service. The ability to receive early warning about all site's problem can save much time and other resources to the webmaster.

I should add also, that every site nowadays is not just a server sending static pages; all the Web is now highly dynamic. It means that in fact every Web site consists of one or more Web applications.

There are no standards on the Web applications efficiency, required response time and so on; however, it's a common practice to consider the site response within 1 second a very good time, and the site response within 5-10 seconds an appropriate response time. Thus, there shiuld be means to set the necesary response time to monitor the site, and in every given case to adjust the timing accordingly.

Web application is the one that uses HTTP to communicate. I have used a number of miscellaneous Web applications and I should stress that one should be able to monitor not only GET, but also POST and, perhaps, other type of requests as well. GET types are most common; they are enough to monitor the site main page and Web application default entry. However, if I am to make sure that a feedback form, or any similar kind of facility is working as expected, I should be able to send POST request as well and test the results.

Of course one should be well aware of the logic of any given Web application, since GET requests are often used to retrieve data, and POST requests are used to apply changes. I.e., if a form is known to send, say, an email, precautions should be taken to prevent automated email sending by testing a POST request on a page.

I should also add that checking error codes returned isn't enough, since many Web applications from, say, CMS group may send send custom response along with error code (404 or 503 or anything else) and/or redirect to a special page handling missing pages, unauthorized access attempts and so on. To detect that a page wasn't found one should check the returned page as well, possibly looking for a string (pattern) to detect abnormal situations.

Source: Monitoring Websites: optimal strategy of monitoring

IPHost Network Monitor is able to make use of all the mentioned tools and techniques to detect performance and/or accessibility degradation of a Web site.

To illustrate how HTTP monitoring is done, let's create a test monitor within IP Host Monitor. To start, click on 'New Monitor' button, select 'HTTP(S)' and click 'Next':

Enter URL of the site to watch, say http://amazon.com/:>

Now the newly created monitor appears in the left pane. Click on it and let's do some tuning

You have just created your first monitor using IPHost Network Monitor software.

Like many others, I run a site. Well, several of them, including mostly static ones, like business card sites, to highly dynamic in nature (blogs and portals). There may be more or less dynamic sites (directories, shops, news feeds and so on), but what I need in every case is to know when the site is running smoothly, and when it's in trouble.

To make sure a site is up, in many cases it is sufficient to make sure a single page is loaded and returns what is expected. It may be a site state page, an index page, whatever that is expected to always load with known HTTP response and, possibly, with known content as well.

I can run handmade scripts and make sure the site responds and the data is returned as expected, but when it comes to more sites, the manual operation becomes tedious and hardly acceptable. Of course, I need automated means to watch the state of my presence on the Net.

It is important to check not only the mere existence of page and data returned, but also to make sure the response time is within expected range. Nowadays on the Net, if a site doesn't load quickly enough, possible visitors can as well lose interest to using the site. It can result in many drawbacks and losses, thus it is equally important to check not only the fact the page exists, but also to verify it is loaded within reasonable time.

So, basically, I need not only to know what time the page should load within, but also to tell a heavy traffic condition from site failure or overload.

To handle possible downtime situations, the site administrator should be notified as soon as possible, if that happens. However, it is hardly reasonable to load the site too frequently apart from creating unnecessary load on the server and wasting traffic, it can also result in severe server load, and other unexpected consequences such as search engines penalties applied, since they can notice the pattern of periodic loading of the page. To check the page once in a minute can be quite acceptable even for complex, database-driven heavily used site. Perhaps the best solution would be to create a lightweight monitor, such as PING monitor, and relate the HTTP(S) monitor to it, raising alerts primarily on PING failures.

I certainly do not wish to attack the sites of mine with too frequent requests. It's much better to just ask for a ping response, and much more rarely ask for actual pages. Otherwise, I can cause site overload myself, this is hardly what I want.

Also, it is not always acceptable just to make sure the page is loaded quickly enough and returns proper HTTP response code. There may be need to send not only GET request, perhaps with a query string, but also to send POST request with certain data (to test a Web form functionality). The page may be protected against public access and HTTP authentication may be required. Also, we could also be expecting a definite kind of content returned, i.e. We can expect the page loaded from the site to contain a known string. A simple example: many modern CMSes (content management systems) often intercept 404, 403 and other HTTP error code processing and return a page instead, explaining what kind of error has occurred and why. Thus, unless we analyze the actual page content returned, we could be unable to tell the page we tried to load doesn't exist.

Indeed, I need not only to be sure that the main page of sites opens fine, but also U could be interested in accessibility of contact forms on my site. The feedback may be very important, and I must be sure the visitors can both access the pages and express their opinion, order my products and so on.

Besides, it is not a rare case when page may also be loaded from specific locations (i.e., IP ranges) and it is required to make use of a proxy server to access the site. A typical proxy may be of HTTP/HTTPS or of Socks type. Thus the monitoring program must be able to make use of a proxy to access the site in question (there are further complexities such as possible latency of the proxy, donating to the possible raising response time; proxy stale cache condition, but these can't be detected by the monitor alone).

There may be many resources beyond public ones I need to watch. For example, if I work at a company, it might be required to test whether all its intranet services behave well. Sites are used widely as common way to access data, so there may be intranet sites to control and watch. If this should be done externally, proxy and other gateway services may be of use.

Concluding, a monitoring tool chosen to watch a Web site page(s) should be able to

• access one or more of pages at programmable time periods; perhaps randomly chosen in order not to create a pattern of intentional page loading to raise Web stats
• make use of lighter monitor such as PING to serve as early-warning alarm before trying to actually loading the page
• handle both GET and POST request types, with possible data supplied to either
• investigate response time as well as the actual data returned
• be able to look for HTTP response and specific data in the server output
• be able to make use of proxy servers to access the site being monitored