When the search engine bot visits a website, it reads all the visible text on the web page, the content of the various tags in the source code (title tag, meta tags, Dublin Core Tags, comments tags, alt tags, attribute tags, content, etc.), as well as the text within the hyperlinks on the web page. From the content that it extracts, the search engine decides what the website, and web page is about. There are many factors used to figure out what is of value and what matters. Each search engine has its own set of rules, standards and algorithms in order to evaluate and process the information. Depending on how the bot was set up by the search engine, different pieces of information are gathered, weighted, indexed and then added to the search engine’s database. Manipulation of the keywords within these webpage elements form part of what is known as search engine optimization.
After it is added, the information then becomes part of the search engine and directory ranking process. When the search engine visitor submits their query, the search engine digs through its database to give the final listing that is displayed on the results page.
Sometimes bots are unable to access the website they are visiting. If a website is down, the bot may not be able to access the website. When this happens, the website may not be re-indexed, and if it happens repeatedly, the website may drop in the rankings.
