In October 2021, Microsoft Bing and Yandex announced a new protocol which changes the way Search Engine’s crawl and index new or updated website content. The protocol is called ‘IndexNow’ and it makes it faster and more efficient for Search Engines to find, index and serve new or updated content to users across the world. There’s no denying that SEO’s (including myself!) will be utilising this new protocol to get their hard work acknowledged by Search Engine’s faster, particularly since Google announced they are testing the use of this protocol in November 2021.
What is IndexNow?
IndexNow is a protocol that website owners can use to ‘ping’ and notify participating Search Engines whenever a new page is uploaded to their website or an existing page is changed or deleted. Without IndexNow, Search Engine Spiders have to crawl websites by following internal links or sitemaps, sometimes crawling hundreds or thousands of URLs before finding a newly updated page to add to their index. IndexNow creates a much more efficient process as Search Engines can instantly find new pages without having to recrawl the same URLs over and over again.
One of the beautiful things about IndexNow is that it means competing Search Engines have to work together to provide the most up-to-date version of the web. Bing has made it a requirement that all URLs sent to a participating Search Engine will then be shared to all other participating Search Engines, so website owners only need to ping one Search Engine to have their content updated across all participating Search Engine’s indexes! This will likely work in Bing’s favour, as I would imagine the majority of website owners will choose to ping market leading Search Engine, Google, if Google decides to fully adopt this protocol.
How does IndexNow work?
IndexNow would change the process of crawling and indexing from ‘pull’ (Search Engines exploring websites to find URLs) to ‘push’ (Search Engines being given a list of URLs by the website owner to crawl). There are a few ways in which website owners can implement IndexNow to ensure Search Engines can prioritise the crawling and indexing of new content. Rather than reinventing the wheel, I have taken the below ’developer’ process directly from Bing’s article on IndexNow (ooooh, duplicate content, I’m in trouble now!):
For developers
If you are a developer, good news, IndexNow is very easy to adopt.
- Generate a key supported by the protocol using our online key generation tool.
- Host the key in text file named with the value of the key at the root of your website.
- Start submitting URLs when your URLs are added, updated, or deleted. You can submit one URL or a set of URLs per API call. Submitting one URL is easy as sending a simple HTTP request containing the URL changed and your key: https://www.bing.com/indexnow?url=url-changed&key=your-key
See detailed instructions at the Microsoft Bing IndexNow site or the IndexNow protocol website.
For non-developers
If you’re not a developer, don’t worry, as many platforms are going to be adopting this protocol in the near future, particularly if Google decides on a full roll out of IndexNow. Microsoft Bing has provided open-source code for platforms to support IndexNow, making it easy for websites like ebay, content management systems like WordPress, content delivery networks like Cloudflare or SEO tools like Botify to integrate IndexNow into their own systems.
What does IndexNow mean for SEO?
IndexNow is a game changer for SEO’s because it means we have a new way to communicate with Search Engine’s regarding the work we have been doing on our / our client’s websites. This could have a number of benefits:
Measuring the instant impact of SEO
SEO is often considered the ‘slow burner’ digital marketing strategy, particularly when compared to PPC. While it is true that it does take time for SEO to have a big impact, some gains can be almost instantaneous. When optimising a landing page for a particular keyword, once those optimisations have been put live and the changes have been crawled and indexed, ranking improvements are often noticeable straight away (along with any traffic and conversion increases this results in). With IndexNow, SEO’s could more reliably measure the instant impact of our work, making it much easier to demonstrate the correlation between SEO output and organic performance improvements.
Relieving concerns over crawl budgets for larger websites
For larger websites, particularly ecommerce sites with thousands of products, ‘crawl budget’ can be an issue. ‘Crawl budget’ is a term used to describe how many URLs Googlebot and other spiders can/do crawl on a particular website within a particular time frame. For some larger websites, spiders can get stuck in ‘spider traps’ where they continuously crawl low priority URLs (e.g. if certain URL parameters are not blocked in robots.txt). While this should be avoided with the right SEO setup, sometimes this is an issue, meaning important pages don’t get crawled because the crawl budget is used up on these low priority URLs.
For other sites, new or important pages might just get missed because there are so many other URLs to crawl through in order to find them. With IndexNow, website owners can submit up to 10,000 URLs per day to be crawled, which would make the crawling and indexing of new content on large websites so much more efficient and gives the website owner a lot more control.
More efficient website migrations
Probably my main motivation for wanting all Search Engines to adopt IndexNow is that website migrations may become a much smoother process. In many website migrations, URLs will change, and in some cases the domain will change too. Search Engines can take a while to find and crawl all of the 301 redirects, so the index takes a while to update while website owners sit nervously awaiting the final results of their migration. With IndexNow, hopefully it will be much easier to get 301 redirects processed by Search Engines, allowing website owners and SEO’s to measure the impact of their migration much quicker and seeing less damage to their traffic and conversions in the process.
Reactive digital PR
Link building is still a weapon in most SEO’s arsenal, and a great way to build links is through reactive digital PR. By creating content about recent events (such as Google’s testing of the IndexNow protocol…..), website owners can quickly gain traffic on trending topics, and being the first to reach the top of Google for that subject may result in an early injection of traffic and backlinks. With the IndexNow protocol, Google’s index could update in real time, benefitting the quicker content creators.
SEO’s (like me) won’t have to panic every time Search Console’s ‘Request Indexing’ tool is taken down
One of my favourite tools in Search Console is the ‘Request Indexing’ tool. It’s come to my aid when Google has decided to randomly de-index an important page on my clients website, it’s allowed me to see faster ranking improvements when optimising landing pages and it generally gives me some control over the indexing of my content. There are times, however, when Google takes this tool away and I am left with no way to communicate with Google, which leaves me quite paranoid and disgruntled, as it did in October 2020:
Let me get this straight. There has recently been a huge @Google indexing bug, & now they remove the only tool that gives website owners the power to actually do something about it? SUSPICIOUS. The empire is crumbling. SEOs grab your keywords & backlinks & head to @DuckDuckGo https://t.co/yyoTv05P19
— Tom Crewe (@TomCrewe) October 15, 2020
But with IndexNow, I would likely not need to use the ‘Request Indexing’ tool any more, because indexing would be instantly requested every time I made a change to a website!
Overall, ‘IndexNow’ would make for a more efficient and harmonious web, and is something I would love to see Google adopt permanently. As a Freelance SEO Consultant, it would be incredibly reassuring to have a bit more control over crawling and indexing for my own site and my client’s sites too!