How can we know which URLs can be crawled as robots.txt tells if we don't know to which folder a URL belong...
I'm going to code a web crawler but before I want to know what is going to be possible to crawl.
Tell me if I'm wrong, but in robots.txt websites indicate folders not URLs that can and can't be crawled, so how can we know to which folder a URL belong to ?
web-crawler robots.txt
asked Jan 21 at 13:53
Yung StunaYung Stuna
82
add a comment |
I'm going to code a web crawler but before I want to know what is going to be possible to crawl.
Tell me if I'm wrong, but in robots.txt websites indicate folders not URLs that can and can't be crawled, so how can we know to which folder a URL belong to ?
web-crawler robots.txt
asked Jan 21 at 13:53
Yung StunaYung Stuna
82
add a comment |
I'm going to code a web crawler but before I want to know what is going to be possible to crawl.
Tell me if I'm wrong, but in robots.txt websites indicate folders not URLs that can and can't be crawled, so how can we know to which folder a URL belong to ?
web-crawler robots.txt
asked Jan 21 at 13:53
Yung StunaYung Stuna
82
I'm going to code a web crawler but before I want to know what is going to be possible to crawl.
Tell me if I'm wrong, but in robots.txt websites indicate folders not URLs that can and can't be crawled, so how can we know to which folder a URL belong to ?
web-crawler robots.txt
web-crawler robots.txt
asked Jan 21 at 13:53
Yung StunaYung Stuna
82
asked Jan 21 at 13:53
Yung StunaYung Stuna
82
asked Jan 21 at 13:53
Yung StunaYung Stuna
82
asked Jan 21 at 13:53
Yung StunaYung Stuna
82
asked Jan 21 at 13:53
Yung StunaYung Stuna
82
82
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The robots.txt file excludes directory prefixes. For example, if you have a robots.txt excluding a directory /foo, then /foo/bar.html must not be crawled.
For any URL you want to crawl, you have to check whether its path matches one of the directives in the robots file.
See the Google documentation for more info and examples:
The path value is used as a basis to determine whether or not a rule applies to a specific URL on a site. With the exception of wildcards, the path is used to match the beginning of a URL (and any valid URLs that start with the same path).
Note that URLs do not have to indicate actual directories on a server. /download.php?what=thestuff could be functionally equivalent to /download/thestuff and point to the same resource.
answered Jan 21 at 14:01
slhckslhck
161k47447470
1
What makes me confuse is the term "directory" because it makes me think that robots.txt exclude some directories path on the server instead of excluding URLs paths. Can you confirm that if the robots.txt contain :User-agent: *Disallow: /download/thestuffI can still crawl tohttp://dl/something.htmleven ifsomething.html's path on the server is/download/thestuff?
– Yung Stuna
Jan 21 at 16:45
Yes, you can still crawl that URL. The physical/actual directories on the server do not matter. It's only about matching the URL against the directive in therobots.txtfile.
– slhck
Jan 21 at 17:07
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1396630%2fhow-can-we-know-which-urls-can-be-crawled-as-robots-txt-tells-if-we-dont-know-t%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The robots.txt file excludes directory prefixes. For example, if you have a robots.txt excluding a directory /foo, then /foo/bar.html must not be crawled.
For any URL you want to crawl, you have to check whether its path matches one of the directives in the robots file.
See the Google documentation for more info and examples:
The path value is used as a basis to determine whether or not a rule applies to a specific URL on a site. With the exception of wildcards, the path is used to match the beginning of a URL (and any valid URLs that start with the same path).
Note that URLs do not have to indicate actual directories on a server. /download.php?what=thestuff could be functionally equivalent to /download/thestuff and point to the same resource.
answered Jan 21 at 14:01
slhckslhck
161k47447470
1
What makes me confuse is the term "directory" because it makes me think that robots.txt exclude some directories path on the server instead of excluding URLs paths. Can you confirm that if the robots.txt contain :User-agent: *Disallow: /download/thestuffI can still crawl tohttp://dl/something.htmleven ifsomething.html's path on the server is/download/thestuff?
– Yung Stuna
Jan 21 at 16:45
Yes, you can still crawl that URL. The physical/actual directories on the server do not matter. It's only about matching the URL against the directive in therobots.txtfile.
– slhck
Jan 21 at 17:07
add a comment |
The robots.txt file excludes directory prefixes. For example, if you have a robots.txt excluding a directory /foo, then /foo/bar.html must not be crawled.
For any URL you want to crawl, you have to check whether its path matches one of the directives in the robots file.
See the Google documentation for more info and examples:
The path value is used as a basis to determine whether or not a rule applies to a specific URL on a site. With the exception of wildcards, the path is used to match the beginning of a URL (and any valid URLs that start with the same path).
Note that URLs do not have to indicate actual directories on a server. /download.php?what=thestuff could be functionally equivalent to /download/thestuff and point to the same resource.
answered Jan 21 at 14:01
slhckslhck
161k47447470
1
What makes me confuse is the term "directory" because it makes me think that robots.txt exclude some directories path on the server instead of excluding URLs paths. Can you confirm that if the robots.txt contain :User-agent: *Disallow: /download/thestuffI can still crawl tohttp://dl/something.htmleven ifsomething.html's path on the server is/download/thestuff?
– Yung Stuna
Jan 21 at 16:45
Yes, you can still crawl that URL. The physical/actual directories on the server do not matter. It's only about matching the URL against the directive in therobots.txtfile.
– slhck
Jan 21 at 17:07
add a comment |
The robots.txt file excludes directory prefixes. For example, if you have a robots.txt excluding a directory /foo, then /foo/bar.html must not be crawled.
For any URL you want to crawl, you have to check whether its path matches one of the directives in the robots file.
See the Google documentation for more info and examples:
The path value is used as a basis to determine whether or not a rule applies to a specific URL on a site. With the exception of wildcards, the path is used to match the beginning of a URL (and any valid URLs that start with the same path).
Note that URLs do not have to indicate actual directories on a server. /download.php?what=thestuff could be functionally equivalent to /download/thestuff and point to the same resource.
answered Jan 21 at 14:01
slhckslhck
161k47447470
The robots.txt file excludes directory prefixes. For example, if you have a robots.txt excluding a directory /foo, then /foo/bar.html must not be crawled.
For any URL you want to crawl, you have to check whether its path matches one of the directives in the robots file.
See the Google documentation for more info and examples:
The path value is used as a basis to determine whether or not a rule applies to a specific URL on a site. With the exception of wildcards, the path is used to match the beginning of a URL (and any valid URLs that start with the same path).
Note that URLs do not have to indicate actual directories on a server. /download.php?what=thestuff could be functionally equivalent to /download/thestuff and point to the same resource.
answered Jan 21 at 14:01
slhckslhck
161k47447470
answered Jan 21 at 14:01
slhckslhck
161k47447470
answered Jan 21 at 14:01
slhckslhck
161k47447470
answered Jan 21 at 14:01
slhckslhck
161k47447470
161k47447470
1
What makes me confuse is the term "directory" because it makes me think that robots.txt exclude some directories path on the server instead of excluding URLs paths. Can you confirm that if the robots.txt contain :User-agent: *Disallow: /download/thestuffI can still crawl tohttp://dl/something.htmleven ifsomething.html's path on the server is/download/thestuff?
– Yung Stuna
Jan 21 at 16:45
Yes, you can still crawl that URL. The physical/actual directories on the server do not matter. It's only about matching the URL against the directive in therobots.txtfile.
– slhck
Jan 21 at 17:07
add a comment |
1
What makes me confuse is the term "directory" because it makes me think that robots.txt exclude some directories path on the server instead of excluding URLs paths. Can you confirm that if the robots.txt contain :User-agent: *Disallow: /download/thestuffI can still crawl tohttp://dl/something.htmleven ifsomething.html's path on the server is/download/thestuff?
– Yung Stuna
Jan 21 at 16:45
Yes, you can still crawl that URL. The physical/actual directories on the server do not matter. It's only about matching the URL against the directive in therobots.txtfile.
– slhck
Jan 21 at 17:07
1
1
What makes me confuse is the term "directory" because it makes me think that robots.txt exclude some directories path on the server instead of excluding URLs paths. Can you confirm that if the robots.txt contain :
User-agent: * Disallow: /download/thestuff I can still crawl to http://dl/something.html even if something.html 's path on the server is /download/thestuff ?– Yung Stuna
Jan 21 at 16:45
What makes me confuse is the term "directory" because it makes me think that robots.txt exclude some directories path on the server instead of excluding URLs paths. Can you confirm that if the robots.txt contain :
User-agent: * Disallow: /download/thestuff I can still crawl to http://dl/something.html even if something.html 's path on the server is /download/thestuff ?– Yung Stuna
Jan 21 at 16:45
Yes, you can still crawl that URL. The physical/actual directories on the server do not matter. It's only about matching the URL against the directive in the
robots.txt file.– slhck
Jan 21 at 17:07
Yes, you can still crawl that URL. The physical/actual directories on the server do not matter. It's only about matching the URL against the directive in the
robots.txt file.– slhck
Jan 21 at 17:07
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1396630%2fhow-can-we-know-which-urls-can-be-crawled-as-robots-txt-tells-if-we-dont-know-t%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
