Page 1 of 1

Bots and Spiders?

Posted: Mon Nov 07, 2016 9:10 am
by Pete
Someone with more knowledge? - Am online and signed in - noted at the bottom the "users" statistics, as per:
Who is online
In total there are 23 users online :: 5 registered, 0 hidden and 18 guests (based on users active over the past 5 minutes)
Most users ever online was 152 on Sat May 23, 2009 6:32 am

Registered users: Baidu [Spider], Bing [Bot], Google Adsense [Bot], Pete, Yahoo [Bot]
Legend: Administrators, Global moderators
Spiders and Bots (not sure the difference?) - as "registered users"??? Hope not a threat.

Re: Bots and Spiders?

Posted: Mon Nov 07, 2016 2:37 pm
by AngelGirl
Pete, as I understand it ... and I'm WAAAY not the professional here (so, probably BJ will be able to answer more succinctly than I) but maybe to assuage your concern ... these are like "search engines" for various stuff like "google", etc., for gathering the stats we see attributed to them or other search engines. Anyway, they are there all the time but I only just noticed them about 18 months ago.

LOL

Re: Bots and Spiders?

Posted: Mon Nov 07, 2016 3:17 pm
by Kenny Wilder
Pete wrote:Someone with more knowledge? - Am online and signed in - noted at the bottom the "users" statistics, as per:
Who is online
In total there are 23 users online :: 5 registered, 0 hidden and 18 guests (based on users active over the past 5 minutes)
Most users ever online was 152 on Sat May 23, 2009 6:32 am

Registered users: Baidu [Spider], Bing [Bot], Google Adsense [Bot], Pete, Yahoo [Bot]
Legend: Administrators, Global moderators
Spiders and Bots (not sure the difference?) - as "registered users"??? Hope not a threat.
Just normal search engines indexing my website 24/7.

Re: Bots and Spiders?

Posted: Mon Nov 07, 2016 9:29 pm
by bjrichus
Kenny Wilder wrote:
Pete wrote:Someone with more knowledge? - Am online and signed in - noted at the bottom the "users" statistics, as per:
Who is online
In total there are 23 users online :: 5 registered, 0 hidden and 18 guests (based on users active over the past 5 minutes)
Most users ever online was 152 on Sat May 23, 2009 6:32 am

Registered users: Baidu [Spider], Bing [Bot], Google Adsense [Bot], Pete, Yahoo [Bot]
Legend: Administrators, Global moderators
Spiders and Bots (not sure the difference?) - as "registered users"??? Hope not a threat.
Just normal search engines indexing my website 24/7.
To quote: http://whatis.techtarget.com/definition/spider

"A spider is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "crawler" or a "bot." Spiders are typically programmed to visit sites that have been submitted by their owners as new or updated. Entire sites or specific pages can be selectively visited and indexed. Spiders are called spiders because they usually visit many sites in parallel at the same time, their "legs" spanning a large area of the "web." Spiders can crawl through a site's pages in several ways. One way is to follow all the hypertext links in each page until all the pages have been read.

The spider for the AltaVista search engine and its Web site is called Scooter . Scooter adheres to the rules of politeness for Web spiders that are specified in the Standard for Robot Exclusion (SRE). It asks each server which files should be excluded from being indexed. It does not (or can not) go through firewalls. And it uses a special algorithm for waiting between successive server requests so that it doesn't affect response time for other users."

If you ever read about something called "Search Engine Optimization" (or SEO), it is all about making the spider programs think your particular web page or site is the best thing possible in a particular subject area (adding meaningful and relevant keywords for example), so that it rises in the listings and appears higher up, so you the user sees it first and is more likely to click on the links to it. Of course, if you pay, you can get the much desired top few spots on the first page of results that the likes of Google display... They are supposed to put them up with a grey background to them. Its all a big business.

Re: Bots and Spiders?

Posted: Tue Nov 08, 2016 7:04 am
by Pete
Thanks - all!