{"id":411,"date":"2014-09-18T10:00:00","date_gmt":"2014-09-18T09:00:00","guid":{"rendered":"http:\/\/www.lcn.com\/blog\/?p=411"},"modified":"2021-07-12T16:54:52","modified_gmt":"2021-07-12T15:54:52","slug":"robots-txt-meta-robots-explained","status":"publish","type":"post","link":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/","title":{"rendered":"A Beginners Guide to Robots.txt and Meta Robots"},"content":{"rendered":"<p>As you begin to optimise your website for search engines, it won&#8217;t be long before you come across the <strong>robots.txt<\/strong> file. It may only be a tiny text file, but the data in there can have a huge effect on your website and its visibility in the <a href=\"https:\/\/www.lcn.com\/learn\/how-search-engines-work\">search engine<\/a> result pages.<\/p>\n<p>Terms like &#8216;<strong>Robots Exclusion Protocol<\/strong>&#8216; may sound like sound something out of a fiction movie, but all will become clear once you&#8217;ve read this article.<\/p>\n<p>&nbsp;<\/p>\n<h2>What is Robots.txt?<\/h2>\n<p>Search engines use web crawlers\u00a0to scan the Internet to find content. These crawlers\u00a0are also known as <strong>spiders<\/strong> and bots. The Robots.txt (Robots exclusion standard) file tells these crawlers\u00a0(and other <a href=\"http:\/\/en.wikipedia.org\/wiki\/User_agent\" target=\"_blank\" rel=\"noopener\">user agents<\/a>) which pages and elements of a site to crawl and index.<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-418 size-full\" src=\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/typical.png\" alt=\"disallowed url path\" width=\"602\" height=\"175\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>In simple terms robots.txt files can be used to help crawlers\u00a0identify and index\u00a0pages on your website. The\u00a0<a href=\"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/\">Robots.txt<\/a>\u00a0file can\u00a0also be used for\u00a0managing security and server performance.<\/p>\n<p>&nbsp;<\/p>\n<h2>Creating your Robots.txt<\/h2>\n<p>As the name suggests, a Robots.txt is simply a .TXT file that can be created using popular text editors such as Notepad on Windows, or TextEdit on Mac.<\/p>\n<p>Your Robots.txt file always should be placed in your websites root directory, as this is the only place crawlers\u00a0will look for it. For example, <em>http:\/\/yoursite.com\/robots.txt<\/em> is a valid location.<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-425 size-full\" src=\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/texteditor.png\" alt=\"text editor\" width=\"518\" height=\"281\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>If your site is built using WordPress, a robots.txt file is created when you start publishing content. You can then control what crawlers will look at under the <strong>Privacy Settings<\/strong> section in your WordPress Admin. There are also a number of WordPress plugin such as <a href=\"https:\/\/wordpress.org\/plugins\/wordpress-seo\/\" target=\"_blank\" rel=\"noopener\">Yoast SEO<\/a>, which offers similar functions and controls to robots.txt files.<\/p>\n<p>The first element of a robots.txt file gives instructions as to which web crawlers the command refers to. In some cases, you may want to restrict access to some crawlers\u00a0but allow others to crawl and index your content.<\/p>\n<p>The second element of the file uses &#8220;<strong>Disallow<\/strong>&#8221; statements to tell a crawler\u00a0where it can and can&#8217;t look for new pages to index.<\/p>\n<p>&nbsp;<\/p>\n<h2>Disallow command<\/h2>\n<p>The following command allows crawlers\u00a0from all search engines full access to every page of your site.\u00a0<\/p>\n<p>User-agent: *<\/p>\n<p>Disallow:<\/p>\n<p>If you want to block the entire site from all crawlers, the command becomes<\/p>\n<pre>User-agent: *\r\nDisallow: \/<\/pre>\n<p>The forward slash indicates that everything within the root directory of your website should be blocked from the crawlers.<\/p>\n<p>This command becomes very useful when you want to only block certain areas of your site.<\/p>\n<p>For example, you may not want the crawlers to index an admin login page or\u00a0a checkout page (as these are not desired <a href=\"https:\/\/www.lcn.com\/blog\/essential-guide-landing-page-optimisation\/\">landing pages<\/a> to your website via search engines). In which case, the command\u00a0would be\u00a0like so&#8230;.<\/p>\n<pre>User-agent: *\r\nDisallow: \/log-in\/\r\nDisallow: \/checkout\/<\/pre>\n<p>All you need to do is just add the relevant path to that page on a new line after &#8220;<em>Disallow:<span style=\"color: #ff0000;\"><span style=\"color: #000000;\">&#8220;.<\/span><\/span><\/em><\/p>\n<p>It&#8217;s important to be aware that search engines could still access your site in other ways if you use the command above. For example, an anchor text link to your site could mean crawlers will still find it.<\/p>\n<p>Confidential content should always be saved in a password-protected file to prevent unwanted visitors from finding it.<\/p>\n<p>&nbsp;<\/p>\n<h2>Restricting specific crawlers\/user agents<\/h2>\n<p>The robots.txt file can also be used for blocking specific crawlers and user agents. You may wish to block Google from indexing your site; for example, if your website is under construction.\u00a0<\/p>\n<p>The following command can be used to restrict specific web crawlers from accessing your content &#8211;<\/p>\n<pre>User-agent:\u00a0BadCrawler\r\nDisallow: \/<\/pre>\n<p>Replacing &#8216;BadCrawler&#8217; with the name of the crawlers\u00a0you want to exclude allows you to block them. You can also restrict crawlers\u00a0from accessing specific directories on your site, like this &#8211;<\/p>\n<pre>User-agent:\u00a0BadCrawler\r\nDisallow: \/private\/<\/pre>\n<p style=\"text-align: left;\">\u00a0<\/p>\n<h2>Meta robots<\/h2>\n<p>Another way to prevent web crawlers from indexing your content is to use the meta robot tag. Unlike the <strong>Disallow <\/strong>tag\u00a0(which goes in your robots.txt file), the meta robots tag is placed into the section of your web page.<\/p>\n<p>The following line of code means that content should not\u00a0be indexed by any crawlers.<\/p>\n<pre>\u00a0<span class=\"token tag\"><span class=\"token punctuation\">&lt;<\/span>meta<\/span> <span class=\"token attr-name\">name<\/span><span class=\"token attr-value\"><span class=\"token punctuation\">=<\/span><span class=\"token punctuation\">\"<\/span>robots<span class=\"token punctuation\">\"<\/span><\/span> <span class=\"token attr-name\">content<\/span><span class=\"token attr-value\"><span class=\"token punctuation\">=<\/span><span class=\"token punctuation\">\"<\/span>noindex<span class=\"token punctuation\">\"<\/span><\/span><span class=\"token punctuation\">&gt;<\/span><\/pre>\n<p>To reiterate, this tag <strong>must be <\/strong>added alongside your other meta tags in the header of your code\u00a0(the tag).<\/p>\n<p>&nbsp;<\/p>\n<h2>No guarantees&#8230;\u00a0<\/h2>\n<p>The robots.txt is a protocol all crawlers\u00a0(other\u00a0user agents)\u00a0<em>should<\/em> follow &#8211; should being the operative word here.<\/p>\n<p>There are many malicious spam crawlers out there crawling content daily, and these crawlers are likely to completely ignore your robots.txt file and it&#8217;s commands. So keep in mind that the robots.txt is not a complete failsafe when looking to hide your website&#8217;s content.<\/p>\n<p>Similarly, remember your robots.txt file is publicly available, which means anyone can see what sections and paths your website you wish to block.<\/p>\n<p>&nbsp;<\/p>\n<h2>Utilise Google Webmaster Tools<\/h2>\n<p>Google&#8217;s <a href=\"https:\/\/www.google.com\/webmasters\/tools\" target=\"_blank\" rel=\"noopener\">Webmaster Tools<\/a> offers a quick and easy way to check if your robots.txt file and its commands working as you intended.<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-447 size-full\" src=\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/blocked-urls.png\" alt=\"blocked urls\" width=\"632\" height=\"224\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>The <strong>Test Robots.txt tool\u00a0<\/strong>tests that the commands you have submitted are correctly formatted and working.<\/p>\n<p>It&#8217;s also worth using the tool to check if content you do want to appear isn&#8217;t being blocked unintentionally.<\/p>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/robots-test.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-449 size-full\" src=\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/robots-test.png\" alt=\"robots test\" width=\"631\" height=\"515\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<p>It&#8217;s important to understand that the <strong>Test Robots.txt tool<\/strong> only provides results for Google user-agents, and other crawlers may respond differently to your robots.txt commands.<\/p>\n<p>&nbsp;<\/p>\n<p><strong>Get the right web hosting for your website! Check out our\u00a0<a href=\"https:\/\/www.lcn.com\/web-hosting\">hosting plans<\/a> and get online today. Have a question? <a href=\"https:\/\/www.lcn.com\/info\/contact\">Contact our friendly UK-based support team<\/a>.<\/strong><\/p>\n<p>&nbsp;<\/p>\n<h2>Your say<\/h2>\n<p>If you have any questions on how to use the robots.txt, or when to use a meta robots command, please let us know in the comments below \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Your websites robot.txt file can be a handy tool to help optimise your site for search engines. However, get it wrong and you&#8217;ll do more harm than good!<\/p>\n","protected":false},"author":31,"featured_media":416,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[35,21],"tags":[4],"class_list":["post-411","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-search-engines","category-technical-seo","tag-beginner"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>A Beginners Guide to Robots.txt and Meta Robots - LCN.com<\/title>\n<meta name=\"description\" content=\"Your websites robot.txt file can be a handy tool to help optimise your site for search engines. However, get it wrong and you&#039;ll do more harm than good!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"A Beginners Guide to Robots.txt and Meta Robots - LCN.com\" \/>\n<meta property=\"og:description\" content=\"Your websites robot.txt file can be a handy tool to help optimise your site for search engines. However, get it wrong and you&#039;ll do more harm than good!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/\" \/>\n<meta property=\"og:site_name\" content=\"Grow a successful website with the LCN Business Hub\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/lcndotcom\" \/>\n<meta property=\"article:published_time\" content=\"2014-09-18T09:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-07-12T15:54:52+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1000\" \/>\n\t<meta property=\"og:image:height\" content=\"667\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Nathan Preedy\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@lcndotcom\" \/>\n<meta name=\"twitter:site\" content=\"@lcndotcom\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nathan Preedy\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/\"},\"author\":{\"name\":\"Nathan Preedy\",\"@id\":\"https:\/\/www.lcn.com\/blog\/#\/schema\/person\/e6493782266f4474f2c90dc96682d069\"},\"headline\":\"A Beginners Guide to Robots.txt and Meta Robots\",\"datePublished\":\"2014-09-18T09:00:00+00:00\",\"dateModified\":\"2021-07-12T15:54:52+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/\"},\"wordCount\":973,\"image\":{\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg\",\"keywords\":[\"Beginner\"],\"articleSection\":[\"Search Engines\",\"Technical SEO\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/\",\"url\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/\",\"name\":\"A Beginners Guide to Robots.txt and Meta Robots - LCN.com\",\"isPartOf\":{\"@id\":\"https:\/\/www.lcn.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg\",\"datePublished\":\"2014-09-18T09:00:00+00:00\",\"dateModified\":\"2021-07-12T15:54:52+00:00\",\"author\":{\"@id\":\"https:\/\/www.lcn.com\/blog\/#\/schema\/person\/e6493782266f4474f2c90dc96682d069\"},\"description\":\"Your websites robot.txt file can be a handy tool to help optimise your site for search engines. However, get it wrong and you'll do more harm than good!\",\"breadcrumb\":{\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#primaryimage\",\"url\":\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg\",\"contentUrl\":\"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg\",\"width\":1000,\"height\":667,\"caption\":\"robot\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"LCN.com\",\"item\":\"\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Blog\",\"item\":\"https:\/\/www.lcn.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Technical SEO\",\"item\":\"https:\/\/www.lcn.com\/blog\/category\/technical-seo\/\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"A Beginners Guide to Robots.txt and Meta Robots\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.lcn.com\/blog\/#website\",\"url\":\"https:\/\/www.lcn.com\/blog\/\",\"name\":\"Grow a successful website with the LCN Business Hub\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.lcn.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.lcn.com\/blog\/#\/schema\/person\/e6493782266f4474f2c90dc96682d069\",\"name\":\"Nathan Preedy\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.lcn.com\/blog\/wp-content\/plugins\/wp-user-avatars\/wp-user-avatars\/assets\/images\/mystery.jpg\",\"url\":\"https:\/\/www.lcn.com\/blog\/wp-content\/plugins\/wp-user-avatars\/wp-user-avatars\/assets\/images\/mystery.jpg\",\"contentUrl\":\"https:\/\/www.lcn.com\/blog\/wp-content\/plugins\/wp-user-avatars\/wp-user-avatars\/assets\/images\/mystery.jpg\",\"caption\":\"Nathan Preedy\"},\"description\":\"Nathan has been with team.blue since 2005 and has a background in Technical Support. He is passionate about helping customers find the best product for them and use it to its full potential.\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"A Beginners Guide to Robots.txt and Meta Robots - LCN.com","description":"Your websites robot.txt file can be a handy tool to help optimise your site for search engines. However, get it wrong and you'll do more harm than good!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/","og_locale":"en_GB","og_type":"article","og_title":"A Beginners Guide to Robots.txt and Meta Robots - LCN.com","og_description":"Your websites robot.txt file can be a handy tool to help optimise your site for search engines. However, get it wrong and you'll do more harm than good!","og_url":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/","og_site_name":"Grow a successful website with the LCN Business Hub","article_publisher":"http:\/\/www.facebook.com\/lcndotcom","article_published_time":"2014-09-18T09:00:00+00:00","article_modified_time":"2021-07-12T15:54:52+00:00","og_image":[{"width":1000,"height":667,"url":"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg","type":"image\/jpeg"}],"author":"Nathan Preedy","twitter_card":"summary_large_image","twitter_creator":"@lcndotcom","twitter_site":"@lcndotcom","twitter_misc":{"Written by":"Nathan Preedy","Estimated reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#article","isPartOf":{"@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/"},"author":{"name":"Nathan Preedy","@id":"https:\/\/www.lcn.com\/blog\/#\/schema\/person\/e6493782266f4474f2c90dc96682d069"},"headline":"A Beginners Guide to Robots.txt and Meta Robots","datePublished":"2014-09-18T09:00:00+00:00","dateModified":"2021-07-12T15:54:52+00:00","mainEntityOfPage":{"@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/"},"wordCount":973,"image":{"@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#primaryimage"},"thumbnailUrl":"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg","keywords":["Beginner"],"articleSection":["Search Engines","Technical SEO"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/","url":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/","name":"A Beginners Guide to Robots.txt and Meta Robots - LCN.com","isPartOf":{"@id":"https:\/\/www.lcn.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#primaryimage"},"image":{"@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#primaryimage"},"thumbnailUrl":"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg","datePublished":"2014-09-18T09:00:00+00:00","dateModified":"2021-07-12T15:54:52+00:00","author":{"@id":"https:\/\/www.lcn.com\/blog\/#\/schema\/person\/e6493782266f4474f2c90dc96682d069"},"description":"Your websites robot.txt file can be a handy tool to help optimise your site for search engines. However, get it wrong and you'll do more harm than good!","breadcrumb":{"@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#primaryimage","url":"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg","contentUrl":"https:\/\/www.lcn.com\/blog\/wp-content\/uploads\/2014\/07\/shutterstock_169002578.jpg","width":1000,"height":667,"caption":"robot"},{"@type":"BreadcrumbList","@id":"https:\/\/www.lcn.com\/blog\/robots-txt-meta-robots-explained\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"LCN.com","item":"\/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https:\/\/www.lcn.com\/blog\/"},{"@type":"ListItem","position":3,"name":"Technical SEO","item":"https:\/\/www.lcn.com\/blog\/category\/technical-seo\/"},{"@type":"ListItem","position":4,"name":"A Beginners Guide to Robots.txt and Meta Robots"}]},{"@type":"WebSite","@id":"https:\/\/www.lcn.com\/blog\/#website","url":"https:\/\/www.lcn.com\/blog\/","name":"Grow a successful website with the LCN Business Hub","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.lcn.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/www.lcn.com\/blog\/#\/schema\/person\/e6493782266f4474f2c90dc96682d069","name":"Nathan Preedy","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.lcn.com\/blog\/wp-content\/plugins\/wp-user-avatars\/wp-user-avatars\/assets\/images\/mystery.jpg","url":"https:\/\/www.lcn.com\/blog\/wp-content\/plugins\/wp-user-avatars\/wp-user-avatars\/assets\/images\/mystery.jpg","contentUrl":"https:\/\/www.lcn.com\/blog\/wp-content\/plugins\/wp-user-avatars\/wp-user-avatars\/assets\/images\/mystery.jpg","caption":"Nathan Preedy"},"description":"Nathan has been with team.blue since 2005 and has a background in Technical Support. He is passionate about helping customers find the best product for them and use it to its full potential."}]}},"_links":{"self":[{"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/posts\/411","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/users\/31"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/comments?post=411"}],"version-history":[{"count":52,"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/posts\/411\/revisions"}],"predecessor-version":[{"id":5438,"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/posts\/411\/revisions\/5438"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/media\/416"}],"wp:attachment":[{"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/media?parent=411"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/categories?post=411"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lcn.com\/blog\/wp-json\/wp\/v2\/tags?post=411"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}