.Google.com's John Mueller addressed a question regarding why Google.com marks web pages that are disallowed from crawling through robots.txt and why the it is actually risk-free to overlook the relevant Search Console files about those creeps.Crawler Website Traffic To Question Guideline URLs.The person inquiring the inquiry recorded that robots were actually creating links to non-existent query specification URLs (? q= xyz) to webpages with noindex meta tags that are also obstructed in robots.txt. What cued the concern is actually that Google is creeping the web links to those web pages, getting obstructed by robots.txt (without seeing a noindex robotics meta tag) after that acquiring shown up in Google Look Console as "Indexed, though blocked out by robots.txt.".The person inquired the adhering to inquiry:." Yet listed below is actually the major question: why would certainly Google.com index pages when they can't also view the web content? What's the advantage because?".Google.com's John Mueller affirmed that if they can not creep the webpage they can not see the noindex meta tag. He likewise creates an exciting mention of the internet site: search driver, suggesting to dismiss the outcomes considering that the "average" users won't view those results.He wrote:." Yes, you're appropriate: if our experts can't crawl the page, we can't see the noindex. That said, if we can't crawl the web pages, at that point there's certainly not a lot for us to mark. Thus while you might find some of those web pages along with a targeted internet site:- question, the normal customer will not find them, so I would not fuss over it. Noindex is actually additionally great (without robots.txt disallow), it only means the URLs will end up being actually crawled (and wind up in the Look Console file for crawled/not listed-- neither of these conditions lead to concerns to the rest of the site). The essential part is that you don't produce them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the limitations being used the Web site: hunt progressed hunt operator for analysis reasons. Some of those reasons is actually because it is actually certainly not attached to the routine hunt index, it is actually a distinct factor completely.Google.com's John Mueller talked about the site search driver in 2021:." The short answer is actually that a website: question is certainly not implied to become complete, neither utilized for diagnostics purposes.An internet site question is a particular sort of hunt that limits the results to a specific web site. It is actually generally simply words site, a colon, and then the internet site's domain.This inquiry limits the results to a certain internet site. It's certainly not suggested to become a comprehensive assortment of all the pages coming from that site.".2. Noindex tag without using a robots.txt is actually alright for these kinds of conditions where a robot is connecting to non-existent web pages that are obtaining uncovered by Googlebot.3. Links along with the noindex tag will create a "crawled/not recorded" item in Search Console and also those will not have a damaging impact on the remainder of the web site.Read through the question and also respond to on LinkedIn:.Why would Google mark pages when they can not even find the web content?Featured Graphic by Shutterstock/Krakenimages. com.