How to Understand Search Engine Mechanism – Insider Secrets to SEO Success

For those working in the SEO realm or those who aspire to, search engines are nothing short of a mystery. It is an everyday adventure to understand the working mechanism and algorithm of this sophisticated online phenomenon. Our days are filled with exciting routine that involves fiddling with search engines. From 301 redirects to server architecture, sky is the limit here.

Let our experts guide small and medium businesses about search engine’s intricate working.

How Wrong Today’s SEO Experts Actually Are

No search engine company, Google or Yahoo, trains people officially in search engine optimization or marketing techniques. They do not reveal their closely guarded secrets. So, most of the SEO experts working in the industry were never trained and know nothing about complex algorithms. Their experience has bestowed them with bits and pieces of knowledge which they use to maximum potential. No wonder these people tremble when Google updates its policies. And by the time they understand, the damage has been done already.

Indexing Vs. Crawling

In SEO realm, ‘Indexing’ refers when we are able to see a site during a search on Google. The pages added in the Google’s database are displayed during search results. Google has to discover the URL before it can proceed to crawl it. Also, these pages have to be crawled before indexing them. Confusing? Yes it is. To put it in plain words – Google relates words or phrase in the webpage with a reference from all the documents related to the webpage’s phrase or words.

So, when we say the page has been indexed, it actually means that some phrases or words related to the webpage now point towards the webpage. The raw documents are archived elsewhere. Google collects info about URLs and proceeds to include these URLs in its crawl scheduler. The schedule list is de-duped and rearranged further so that they can be crawled as per their order of priority. In the next step, Google applies its algorithm to decide whether or not to store the webpage in their index. In the end, every page crawled by Google is not indexed and neither is every webpage crawled.

Other Important Search Engine terms:

  • Robots.txt – This blocks a webpage from crawling. Google can associate certain words with the concerned page but it does not actually crawl the entire content of page.
  • NoIndex Commands are not definitive on the page level.
  • Google may associate the page with words in the index but it may or may not include the page in the search results.
  • Noindex commands are respected well by Yahoo and Bing. Thus, they will not include such pages in its search results.
  • Google also processes some other important elements such as parameter exclusion and canonicals as and when it learns about the webpage.
  • Page rank is in no way linked to the words on the webpage. It is a tool used to measure the quality and quantity of links.
  • PageRank works separately from crawling. So, directives such as Disallow and Noindex etc do not stop PageRAnk flow from one page to the other.
  • Referrer-based tracking does not give you control over PageRank. So, a referrer can not be blocked using .htaccess. And the same cannot be expected to work like a nofollow on Googlebot.
  • A ‘302 redirect’ will definitely pass PageRank.

Finally, This is How Search Engines Work

The inner workings of a search engine will always be as clear mud. However, the important things are:

  1. Indexing is not equal to crawling.
  2. Crawling and PageRank is not same.
  3. Disavow is the only option to blocking PageRank flow.

Surely, you were working with wrong fundamentals. Don’t worry, many SEO specialists are. The key to success is to understand the direction of search engine optimization. Any SEO strategies worth his salt keeps track of the latest developments and moves accordingly.


eBasic is a renowned name in Canadian online marketing sphere. Our SEO and web development stems from years of experience in Web 2.0 solutions. Decoding and reverse engineering search engine algorithms is our specialty. With thousands of satisfied clients to prove our worth, we deliver results that catapult your online business to new heights. Contact 416-907-4996 to discover how SEO can create long queues in front of your office.