In short, because it’s all up to the very unpredictable Google spiders sent out to crawl and index web pages. You might have a website that Google crawls infrequently so it will take longer for them to get all your pages indexed. Google treats websites differently based on your history with them, or lack thereof.
First, let’s look at some terms used. Crawling is what the Googlebots (spiders) do to gather information about your website. Indexing is the final outcome after running your pages through their rather complicated algorithm to determine the relevancy and authority of your site and each page for any number of search terms.
Google Likes Active Sites
Google will crawl an active site more frequently than an inactive site so if you haven’t made any changes to your website in a year it might get crawled on a monthly or semi-annual schedule. If you make frequent additions and updates to your site they might crawl it daily, and if you’re an authority site like CNN.com you’re probably getting crawled and indexed constantly and any changes to content will show up in your Search Engine Results Page (SERP) within minutes, if not seconds.
Since I make substantive changes to my blog articles on average about twice a week any new articles I post will be crawled and indexed within a few days. Most websites that I’m going to be working on for SEO haven’t had much done to them if anything since they were first published so I like to do something that supposedly has the effect of ringing their doorbell to let them know I’ve just delivered some content to their front porch.
Speeding Up Those Google Spiders (Potentially)
I’ll set up websites I’m working on with Google Analytics and Google Webmaster accounts and when I’ve posted a few pages I’ll go into their Google Webmaster account and go to Webmaster Tools > Crawl > Fetch as Google, fill in the site I want indexed and hit Fetch, wait a few seconds, and then hit Submit to index when that button appears. Google says they take this as a “suggestion” and don’t have to do it but I’ve had pretty good success with it.
Once a Googlebot come around to crawl a site and sees that you’ve made some recent changes it will say to itself, “Hey look, they’ve made some recent changes to their site, I better come back sooner next time because they are more likely to make some more changes.” Not only will frequent changes increase your crawl frequency but Google likes to see active sites and will boost your rankings. In my experience it seems if a website is left to sit idle for more than 3 months its rankings drop significantly.
If you’re like me you’ll get inspired to write several articles at one time and publish them on your blog and then not do anything for several weeks. That several week spell of inactivity could be telling Google you have become an inactive site and might not only crawl your site less frequently but also decrease your rankings. To avoid this I publish one post every Friday so if I write 3 articles I’ll schedule them to be published over the next 3 Fridays. Since I use WordPress it’s very easy to publish at a later date by changing the publish date for that post in the page editor.