For example, let's say the keyword difficulty of a particular term is in the 80s and 90s in the top five spots on a particular search results page. Then, in positions 6-9, the difficulty scores drop down into the 50s and 60s. Using that difficulty score, a business can begin targeting that range of spots and running competitive analysis on the pages to see who your website could knock out of their spot.
Website-specific crawlers, or software that crawls one particular website at a time, are great for analyzing your own website's SEO strengths and weaknesses; they're arguably even more useful for scoping out the competition's. Website crawlers analyze a website's URL, link structure, images, CSS scripting, associated apps, and third-party services to evaluate SEO. Not unlike how a website monitoring tool scans for a webpage's overall "health," website crawlers can identify factors such as broken links and errors, website lag, and content or metadata with low keyword density and SEO value, while mapping a website's architecture. Website crawlers can help your business improve website user experience (UX) while identifying key areas of improvement to help pages rank better. DeepCrawl is, by far, the most granular and detailed website crawler in this roundup, although Ahrefs and Majestic also provide comprehensive domain crawling and website optimization recommendations. Another major crawler we didn't test is Screaming Frog, which we'll soon discuss in the section called "The Enterprise Tier."
The caveat in all of this is that, in one way or another, most of the data and the rules governing what ranks and what doesn't (often on a week-to-week basis) comes from Google. If you know where to find and how to use the free and freemium tools Google provides under the surface—AdWords, Google Analytics, and Google Search Console being the big three—you can do all of this manually. Much of the data that the ongoing position monitoring, keyword research, and crawler tools provide is extracted in one form or another from Google itself. Doing it yourself is a disjointed, meticulous process, but you can piece together all the SEO data you need to come up with an optimization strategy should you be so inclined.
Does this article have an excessive amount of ads that distract from or interfere with the main content? (OPTIMISE FOR SATISFACTION FIRST – CONVERSION SECOND – do not let the conversion get in the way of satisfying the INTENT of the page. For example – if you rank with INFORMATIONAL CONTENT with a purpose to SERVE those visitors – the visitor should land on your destination page and not be deviated from the PURPOSE of the page – and that was informational, in this example – to educate. SO – educate first – beg for social shares on those articles – and leave the conversion on Merit and slightly more subtle influences rather than massive banners or whatever that annoy users). We KNOW ads (OR DISTRACTING CALL TO ACTIONS) convert well at the top of articles – but Google says it is sometimes a bad user experience. You run the risk of Google screwing with your rankings as you optimise for conversion so be careful and keep everything simple and obvious.
While Google is on record as stating these quality raters do not directly influence where you rank (without more senior analysts making a call on the quality of your website, I presume?) – there are some things in this document, mostly of a user experience nature (UX) that all search engine optimisers and Webmasters of any kind should note going forward.
QUOTE: “Over time, we’ve seen sites try to maximize their “search footprint” without adding clear, unique value. These doorway campaigns manifest themselves as pages on a site, as a number of domains, or a combination thereof. To improve the quality of search results for our users, we’ll soon launch a ranking adjustment to better address these types of pages. Sites with large and well-established doorway campaigns might see a broad impact from this change.” Google 2015
I added one keyword to the page in plain text because adding the actual ‘keyword phrase’ itself would have made my text read a bit keyword stuffed for other variations of the main term. It gets interesting if you do that to a lot of pages, and a lot of keyword phrases. The important thing is keyword research – and knowing which unique keywords to add.
Mike Levin is the Senior SEO Director at Ziff Davis, PCMag's parent company. His career goes back 25 years to the halls of Commodore Computers, as an original Amiga fanboy to be beamed up by the mothership just as it imploded. Over his past 10 years in NYC, Mike's highlights have included leading the Apple Store, Kraft and JCPenney SEO accounts whi... See Full Bio
I prefer simple SEO techniques and ones that can be measured in some way. I have never just wanted to rank for competitive terms; I have always wanted to understand at least some of the reasons why a page ranked for these key phrases. I try to create a good user experience for humans AND search engines. If you make high-quality text content relevant and suitable for both these audiences, you’ll more than likely find success in organic listings and you might not ever need to get into the technical side of things, like redirects and search engine friendly URLs.