{"id":127,"date":"2020-02-12T10:36:39","date_gmt":"2020-02-12T10:36:39","guid":{"rendered":"https:\/\/www.lcn.com\/support\/knowledge-base\/what-is-robots-dot-txt\/"},"modified":"2020-08-11T16:03:06","modified_gmt":"2020-08-11T16:03:06","slug":"what-is-robots-dot-txt","status":"publish","type":"ht_kb","link":"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/","title":{"rendered":"What is Robots.txt"},"content":{"rendered":"\n<p>A robots.txt file is a protocol used to prevent web crawlers and robots from accessing all or part of a website. This allows you to control which\u202f<strong>web crawlers<\/strong>\u202fand robots can access your site and how often they do so.\u00a0<\/p>\n\n\n\n<p>It\u2019s important to understand that robots.txt is not a way of blocking robots from scanning your\u00a0site. It\u00a0is simply a request to not visit it. Also, because this file is publicly available, anyone can see what sections of the site you\u00a0don\u2019t\u00a0want the robots to access.\u00a0<\/p>\n\n\n\n<p>A Robots.txt file can be created in something like\u202f<strong>notepad<\/strong>\u202for\u202f<strong>textedit<\/strong>. You can find a couple of examples below on how best to use this.\u00a0<\/p>\n\n\n\n<p>This will allow all robots complete access\u00a0<\/p>\n\n\n\n<pre class=\"prettyprint linenums\">User-agent: *Disallow:<\/pre>\n\n\n\n<p>To exclude all robots from the entire server&nbsp;<\/p>\n\n\n\n<pre class=\"prettyprint linenums\">User-agent: *Disallow: \/<\/pre>\n\n\n\n<p>Lastly to allow a single robot\u00a0<\/p>\n\n\n\n<pre class=\"prettyprint linenums\">User-agent: GoogleDisallow:User-agent: *Disallow: \/<\/pre>\n\n\n\n<p>Once you have created your file, you would need to upload it to your root directory via FTP\u00a0or your File Manager. Details of how to do this can be found at the links below:\u00a0<\/p>\n\n\n\n<p><a href=\"https:\/\/www.lcn.com\/support\/articles\/how-to-upload-files-to-your-website-with-ftp\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to upload files to your website with FTP<\/a>&nbsp;<\/p>\n\n\n\n<p><a href=\"https:\/\/www.lcn.com\/support\/articles\/how-to-use-the-lcn-file-manager\/#Upload\" target=\"_blank\" rel=\"noreferrer noopener\">How to upload files using File Manager<\/a><\/p>\n\n\n\n<p>That\u2019s&nbsp;it!&nbsp;&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A robots.txt file is a protocol used to prevent web crawlers and robots from accessing all or part of a website. This allows you to control which\u202fweb crawlers\u202fand robots can access your site and how often they do so.\u00a0 It\u2019s important to understand that robots.txt is not a way of blocking robots from scanning your\u00a0site. [&hellip;]<\/p>\n","protected":false},"author":3,"comment_status":"closed","ping_status":"open","template":"","format":"standard","meta":{"footnotes":""},"ht-kb-category":[15],"ht-kb-tag":[5],"class_list":["post-127","ht_kb","type-ht_kb","status-publish","format-standard","hentry","ht_kb_category-creating-a-new-website","ht_kb_tag-web-hosting"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>What is Robots.txt - LCN.com Support Guide<\/title>\n<meta name=\"description\" content=\"What is Robots.txt? A guide on what is and how to use a robots.txt file on LCN shared web and WordPress hosting service.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is Robots.txt - LCN.com Support Guide\" \/>\n<meta property=\"og:description\" content=\"What is Robots.txt? A guide on what is and how to use a robots.txt file on LCN shared web and WordPress hosting service.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"Customer Support Guides - LCN\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/lcndotcom\" \/>\n<meta property=\"article:modified_time\" content=\"2020-08-11T16:03:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.lcn.com\/support\/wp-content\/uploads\/support-facebook-feed.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@lcndotcom\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/\",\"url\":\"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/\",\"name\":\"What is Robots.txt - LCN.com Support Guide\",\"isPartOf\":{\"@id\":\"https:\/\/www.lcn.com\/support\/#website\"},\"datePublished\":\"2020-02-12T10:36:39+00:00\",\"dateModified\":\"2020-08-11T16:03:06+00:00\",\"description\":\"What is Robots.txt? A guide on what is and how to use a robots.txt file on LCN shared web and WordPress hosting service.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.lcn.com\/support\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is Robots.txt\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.lcn.com\/support\/#website\",\"url\":\"https:\/\/www.lcn.com\/support\/\",\"name\":\"Customer Support Guides - LCN\",\"description\":\"- LCN\",\"publisher\":{\"@id\":\"https:\/\/www.lcn.com\/support\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.lcn.com\/support\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.lcn.com\/support\/#organization\",\"name\":\"LCN.com\",\"url\":\"https:\/\/www.lcn.com\/support\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.lcn.com\/support\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.lcn.com\/support\/wp-content\/uploads\/2020\/02\/ZF_-hcc3.jpg\",\"contentUrl\":\"https:\/\/www.lcn.com\/support\/wp-content\/uploads\/2020\/02\/ZF_-hcc3.jpg\",\"width\":461,\"height\":461,\"caption\":\"LCN.com\"},\"image\":{\"@id\":\"https:\/\/www.lcn.com\/support\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/lcndotcom\",\"https:\/\/x.com\/lcndotcom\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is Robots.txt - LCN.com Support Guide","description":"What is Robots.txt? A guide on what is and how to use a robots.txt file on LCN shared web and WordPress hosting service.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/","og_locale":"en_US","og_type":"article","og_title":"What is Robots.txt - LCN.com Support Guide","og_description":"What is Robots.txt? A guide on what is and how to use a robots.txt file on LCN shared web and WordPress hosting service.","og_url":"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/","og_site_name":"Customer Support Guides - LCN","article_publisher":"https:\/\/www.facebook.com\/lcndotcom","article_modified_time":"2020-08-11T16:03:06+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/www.lcn.com\/support\/wp-content\/uploads\/support-facebook-feed.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_site":"@lcndotcom","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/","url":"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/","name":"What is Robots.txt - LCN.com Support Guide","isPartOf":{"@id":"https:\/\/www.lcn.com\/support\/#website"},"datePublished":"2020-02-12T10:36:39+00:00","dateModified":"2020-08-11T16:03:06+00:00","description":"What is Robots.txt? A guide on what is and how to use a robots.txt file on LCN shared web and WordPress hosting service.","breadcrumb":{"@id":"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.lcn.com\/support\/articles\/what-is-robots-dot-txt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.lcn.com\/support\/"},{"@type":"ListItem","position":2,"name":"What is Robots.txt"}]},{"@type":"WebSite","@id":"https:\/\/www.lcn.com\/support\/#website","url":"https:\/\/www.lcn.com\/support\/","name":"Customer Support Guides - LCN","description":"- LCN","publisher":{"@id":"https:\/\/www.lcn.com\/support\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.lcn.com\/support\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.lcn.com\/support\/#organization","name":"LCN.com","url":"https:\/\/www.lcn.com\/support\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.lcn.com\/support\/#\/schema\/logo\/image\/","url":"https:\/\/www.lcn.com\/support\/wp-content\/uploads\/2020\/02\/ZF_-hcc3.jpg","contentUrl":"https:\/\/www.lcn.com\/support\/wp-content\/uploads\/2020\/02\/ZF_-hcc3.jpg","width":461,"height":461,"caption":"LCN.com"},"image":{"@id":"https:\/\/www.lcn.com\/support\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/lcndotcom","https:\/\/x.com\/lcndotcom"]}]}},"_links":{"self":[{"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/ht-kb\/127","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/ht-kb"}],"about":[{"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/types\/ht_kb"}],"author":[{"embeddable":true,"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/comments?post=127"}],"version-history":[{"count":1,"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/ht-kb\/127\/revisions"}],"predecessor-version":[{"id":1119,"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/ht-kb\/127\/revisions\/1119"}],"wp:attachment":[{"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/media?parent=127"}],"wp:term":[{"taxonomy":"ht_kb_category","embeddable":true,"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/ht-kb-category?post=127"},{"taxonomy":"ht_kb_tag","embeddable":true,"href":"https:\/\/www.lcn.com\/support\/wp-json\/wp\/v2\/ht-kb-tag?post=127"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}