Crawlers are largely a separate product category. There is some overlap with the self-service keyword tools (Ahrefs, for instance, does both), but crawling is another important piece of the puzzle. We tested several tools with these capabilities either as their express purpose or as features within a larger platform. Ahrefs, DeepCrawl, Majestic, and LinkResearchTools are all primarily focused on crawling and backlink tracking, the inbound links coming to your site from another website. Moz Pro, SpyFu, SEMrush, and AWR Cloud all include domain crawling or backlink tracking features as part of their SEO arsenals.
Another excellent guide is Google’s “Search Engine Optimization Starter Guide.” This is a free PDF download that covers basic tips that Google provides to its own employees on how to get listed. You’ll find it here. Also well worth checking out is Moz’s “Beginner’s Guide To SEO,” which you’ll find here, and the SEO Success Pyramid from Small Business Search Marketing.

QUOTE: “So there’s three things that you really want to do well if you want to be the world’s best search engine you want to crawl the web comprehensively and deeply you want to index those pages and then you want to rank or serve those pages and return the most relevant ones first….. we basically take PageRank as the primary determinant and the more PageRank you have that is the more people who link to you and the more reputable those people are the more likely it is we’re going to discover your page…. we use page rank as well as over 200 other factors in our rankings to try to say okay maybe this document is really authoritative it has a lot of reputation because it has a lot of PageRank … and that’s kind of the secret sauce trying to figure out a way to combine those 200 different ranking signals in order to find the most relevant document.” Matt Cutts, Google
Users will occasionally come to a page that doesn't exist on your site, either by following a broken link or typing in the wrong URL. Having a custom 404 page30 that kindly guides users back to a working page on your site can greatly improve a user's experience. Your 404 page should probably have a link back to your root page and could also provide links to popular or related content on your site. You can use Google Search Console to find the sources of URLs causing "not found" errors31.
The existing content may speak to core audiences, but it isn’t producing many strong organic results. For example, the content header Capitalizing on the Right Skills at the Right Time With Business Agility may seem OK, but it doesn’t include a keyword phrase within striking distance. The lengthy URL doesn’t help matters. Extraneous words prevent any focus and the URL is bogged down by “business” and “agility” duplication:
I like the competition analysis tools, it provides paid and organic data, which gives me an idea on how to catch up and outrank the immediate competition for my clients. It also provides data for the potential traffic, which helps show clients the potential gains of the campaign. And with the marketing plan, I know what needs to be improved in order to get results for my clients.
These cloud-based, self-service tools have plenty of other unique optimization features, too. Some, such as AWR Cloud and Searchmetrics, also do search position monitoring—which means tracking how your page is doing against popular search queries. Others, such as SpyFu and LinkResearchTools, have more interactive data visualizations, granular and customizable reports, and return on investment (ROI) metrics geared toward online marketing and sales goals. The more powerful platforms can sport deeper analytics on paid advertising and pay-per-click (PPC) SEO as well. Though, at their core, the tools are all rooted in their ability to perform on-demand keyword queries.
In 1998, two graduate students at Stanford University, Larry Page and Sergey Brin, developed "Backrub", a search engine that relied on a mathematical algorithm to rate the prominence of web pages. The number calculated by the algorithm, PageRank, is a function of the quantity and strength of inbound links.[22] PageRank estimates the likelihood that a given page will be reached by a web user who randomly surfs the web, and follows links from one page to another. In effect, this means that some links are stronger than others, as a higher PageRank page is more likely to be reached by the random web surfer.
One common scam is the creation of "shadow" domains that funnel users to a site by using deceptive redirects. These shadow domains often will be owned by the SEO who claims to be working on a client's behalf. However, if the relationship sours, the SEO may point the domain to a different site, or even to a competitor's domain. If that happens, the client has paid to develop a competing site owned entirely by the SEO.

One of its content headers and SEO page titles is: Why Understanding the Four Major Learning Styles Will Help You Reach More Employees This Open Enrollment. The highest ranking for that page is 23rd for the phrase “benefits of learning styles” (30 monthly searches), which appears on the third page of Google. Maybe it’s good content – just not an effective string of words for SEO.
Hi Claire, you’re welcome. It depends. If the keyword seems like a Featured Snippet would make sense (for example, it’s a term that could use a definition or there’s a list of steps or tips), I’d still try snippetbait. One other thing I’d keep in mind is that Featured Snippets tend to float in and out. For example, the keyword “how to get more subscribers on YouTube”. That featured snippet tends to appear (with us ranking in it) and disappear on a weekly basis. Just Google testing stuff out.
The self-service keyword research tools we tested all handle pricing relatively similarly, pricing by month with discounts for annual billing with most SMB-focused plans ranging in the $50-$200 per month range. Depending on how your business plans to use the tools, the way particular products delineate pricing might make more sense. KWFinder.com is the cheapest of the bunch, but it's focused squarely on ad hoc keyword and Google SERP queries, which is why the product sets quotas for keyword lookups per 24 hours at different tiers. Moz and Ahrefs price by campaigns or projects, meaning the number of websites you're tracking in the dashboard. Most of the tools also cap the number of keyword reports you can run per day. SpyFu prices a bit differently, providing unlimited data access and results but capping the number of sales leads and domain contacts.
Mike Murray has shaped online marketing strategies for hundreds of businesses since 1997, including Fortune 500 companies. A former journalist, he has led SEO studies and spoken at regional and national Internet conferences. Founder of Online Marketing Coach, Mike is passionate about helping clients identify their best opportunities for online marketing success based on their strengths, his advice and industry trends. You can find him at his blog, Online Marketing Matters or on Twitter @mikeonlinecoach.
Use common sense – Google is a search engine – it is looking for pages to give searchers results, 90% of its users are looking for information. Google itself WANTS the organic results full of information. Almost all websites will link to relevant information content so content-rich websites get a lot of links – especially quality links. Google ranks websites with a lot of links (especially quality links) at the top of its search engines so the obvious thing you need to do is ADD A LOT of INFORMATIVE CONTENT TO YOUR WEBSITE.
Enterprise SEO platforms put all of this together—high-volume keyword monitoring with premium features like landing page alignments and optimization recommendations, plus on-demand crawling and ongoing position monitoring—but they're priced by custom quote. While the top-tier platforms give you features like in-depth keyword expansion and list management, and bells and whistles like SEO recommendations in the form of automated to-do lists, SMBs can't afford to drop thousands of dollars per month.
QUOTE: “We are a health services comparison website…… so you can imagine that for the majority of those pages the content that will be presented in terms of the clinics that will be listed looking fairly similar right and the same I think holds true if you look at it from the location …… we’re conscious that this causes some kind of content duplication so the question is is this type … to worry about? “

There are other parts of SEO which you should pay attention to after your audit to make sure you stay competitive. After all, the technical foundation isn't the end of the road for SEO success. It's important to pay attention to your competition's SEO activity, keep an eye on the newest search engine best practices, and maintain local SEO best practices if your business depends on customers visiting a physical address. All of these are elements of a successful SEO strategy and should be corollary to your audit and ongoing SEO maintenance.


Place strategic search phrases on pages. Integrate selected keywords into your website source code and existing content on designated pages. Make sure to apply a sug­gested guideline of one to three keywords/phrases per content page and add more pages to complete the list. Ensure that related words are used as a natural inclu­sion of your keywords. It helps the search engines quickly determine what the page is about. A natural approach to this works best. In the past, 100 to 300 words on a page was recommended. Many tests show that pages with 800 to 2,000 words can outperform shorter ones. In the end, the users, the marketplace, content and links will determine the popularity and ranking numbers.
If you link out to irrelevant sites, Google may ignore the page, too – but again, it depends on the site in question. Who you link to, or HOW you link to, REALLY DOES MATTER – I expect Google to use your linking practices as a potential means by which to classify your site. Affiliate sites, for example, don’t do well in Google these days without some good quality backlinks and higher quality pages.
The caveat in all of this is that, in one way or another, most of the data and the rules governing what ranks and what doesn't (often on a week-to-week basis) comes from Google. If you know where to find and how to use the free and freemium tools Google provides under the surface—AdWords, Google Analytics, and Google Search Console being the big three—you can do all of this manually. Much of the data that the ongoing position monitoring, keyword research, and crawler tools provide is extracted in one form or another from Google itself. Doing it yourself is a disjointed, meticulous process, but you can piece together all the SEO data you need to come up with an optimization strategy should you be so inclined.
Forget whether or not to put your keyword in it, make it relevant to a searcher and write it for humans, not search engines. If you want to have this 20-word meta description which accurately describes the page you have optimised (for one or two keyword phrases) when people use Google to search, make sure the keyword is in there. There is more likelihood it will be used by Google as your snippet description in SERPs.
You should build a website to benefit your users, and any optimization should be geared toward making the user experience better. One of those users is a search engine, which helps other users discover your content. Search Engine Optimization is about helping search engines understand and present content. Your site may be smaller or larger than our example site and offer vastly different content, but the optimization topics we discuss below should apply to sites of all sizes and types. We hope our guide gives you some fresh ideas on how to improve your website, and we'd love to hear your questions, feedback, and success stories in the Google Webmaster Help Forum1.
The last piece of the complicated SEO tool ecosystem is the enterprise tier. This roundup is geared toward SEO for small to midsize businesses (SMBs), for which these platforms are likely priced out of reach. But there's a handful of enterprise SEO software providers out there that essentially roll all of the self-service tools into one comprehensive platform. These platforms combine ongoing position monitoring, deep keyword research, and crawling with customizable reports andanalytics.

The basics of GOOD SEO hasn’t changed for years – though effectiveness of particular elements has certainly narrowed or changed in type of usefulness – you should still be focusing on building a simple site using VERY simple SEO best practices – don’t sweat the small stuff, while all-the-time paying attention to the important stuff  – add plenty of unique PAGE TITLES and plenty of new ORIGINAL CONTENT. Understand how Google SEES your website. CRAWL it, like Google does, with (for example) Screaming Frog SEO spider, and fix malformed links or things that result in server errors (500), broken links (400+) and unnecessary redirects (300+). Each page you want in Google should serve a 200 OK header message.


Some search engines have also reached out to the SEO industry, and are frequent sponsors and guests at SEO conferences, webchats, and seminars. Major search engines provide information and guidelines to help with website optimization.[18][19] Google has a Sitemaps program to help webmasters learn if Google is having any problems indexing their website and also provides data on Google traffic to the website.[20] Bing Webmaster Tools provides a way for webmasters to submit a sitemap and web feeds, allows users to determine the "crawl rate", and track the web pages index status.
A poor 404 page and user interaction with it, can only lead to a ‘poor user experience’ signal at Google’s end, for a number of reasons. I will highlight a poor 404 page in my audits and actually programmatically look for signs of this issue when I scan a site. I don’t know if Google looks at your site that way to rate it e.g. algorithmically determines if you have a good 404 page – or if it is a UX factor, something to be taken into consideration further down the line – or purely to get you thinking about 404 pages (in general) to help prevent Google wasting resources indexing crud pages and presenting poor results to searchers. I think rather that any rating would be a second order scoring including data from user activity on the SERPs – stuff we as SEO can’t see.
The SEO starter guide describes much of what your SEO will do for you. Although you don't need to know this guide well yourself if you're hiring a professional to do the work for you, it is useful to be familiar with these techniques, so that you can be aware if an SEO wants to use a technique that is not recommended or, worse, strongly discouraged.
Some search engines have also reached out to the SEO industry, and are frequent sponsors and guests at SEO conferences, webchats, and seminars. Major search engines provide information and guidelines to help with website optimization.[18][19] Google has a Sitemaps program to help webmasters learn if Google is having any problems indexing their website and also provides data on Google traffic to the website.[20] Bing Webmaster Tools provides a way for webmasters to submit a sitemap and web feeds, allows users to determine the "crawl rate", and track the web pages index status.
The ranking of your website is partly decided by on-page factors. On-page SEO factors are all those things you can influence from within your actual website. These factors include technical aspects (e.g. the quality of your code and site speed) and content-related aspects, like the structure of your website or the quality of the copy on your website. These are all crucial on-page SEO factors.

While the title tag is effectively your search listing’s headline, the meta description (another meta HTML element that can be updated in your site’s code, but isn’t seen on your actual page) is effectively your site’s additional ad copy. Google takes some liberties with what they display in search results, so your meta description may not always show, but if you have a compelling description of your page that would make folks searching likely to click, you can greatly increase traffic. (Remember: showing up in search results is just the first step! You still need to get searchers to come to your site, and then actually take the action you want.)


QUOTE: “The average duration metric for the particular group of resources can be a statistical measure computed from a data set of measurements of a length of time that elapses between a time that a given user clicks on a search result included in a search results web page that identifies a resource in the particular group of resources and a time that the given user navigates back to the search results web page. …Thus, the user experience can be improved because search results higher in the presentation order will better match the user’s informational needs.” High Quality Search Results based on Repeat Clicks and Visit Duration


Your article reaches me at just the perfect time. I’ve been working on getting back to blogging and have been at it for almost a month now. I’ve been fixing SEO related stuff on my blog and after reading this article (by the way is way too long for one sitting) I’m kind of confused. I’m looking at bloggers like Darren Rowse, Brian Clark, and so many other bloggers who use blogging or their blogs as a platform to educate their readers more than thinking about search rankings (but I’m sure they do).
Ego and assumptions led me to choose the wrong keywords for my own site. How did I spend three years optimizing my site and building links to finally crack the top three for six critical keywords, only to find out that I wasted all that time? However, in spite of targeting the wrong words, Seer grew the business. In this presentation, Will shows you the mistakes made and share with you the approaches that can help you build content that gets you thanked.
×