varleysheen
Level 1
Write the reason you're deleting this FAQ
Hi,
on majestic, it only crawls 19 of my posts and really wants it to crawl all as I believe my TF will be higher. anybody know what is going on
Are you sure you want to delete this post?
Are you sure you want to delete this post?
Cristian
Another good way to figure out what is going on is to write the following command in Google Search: "
This command will show you the exact number of pages of your domain indexed by Google. You can then compare that number with the number crawled by Majestic.
Also, I won't rely on web crawlers to much, if you really want to proper crawl a website I suggest you try Screaming Frog (the paid version) and I assure you if you configure it correctly it will crawl everything to the last link, image, and URL.
Another problem you may have that can stuck Majestic from crawling your whole website are robots.txt restriction you may have as well as robots meta tags with no-index or no-follow implemented on some of your pages.
If you don't have the money for a paid version of screaming frog, let me know your domain address and I'll do a full crawl and tell you where the problems are and overall result.
Cheers. How many posts do you have? Because if you only have 19 posts Majestic will only crawl those 19 posts. Another good way to figure out what is going on is to write the following command in Google Search: "[code]site:http://www.yourwebsitename.com[/code]Another good way to figure out what is going on is to write the following command in Google Search: "site:http://www.yourwebsitename.com" This command will show you the exact number of pages of your domain indexed by Google. You can then compare that number with the number crawled by Majestic. Also, I won't rely on web crawlers to much, if you really want to proper crawl a website I suggest you try Screaming Frog (the paid version) and I assure you if you configure it correctly it will crawl everything to the last link, image, and URL. Another problem you may have that can stuck Majestic from crawling your whole website are robots.txt restriction you may have as well as robots meta tags with no-index or no-follow implemented on some of your pages. If you don't have the money for a paid version of screaming frog, let me know your domain address and I'll do a full crawl and tell you where the problems are and overall result. Cheers.
Are you sure you want to delete this post?