Website-specific crawlers, or software that crawls one particular website at a time, are great for analyzing your own website's SEO strengths and weaknesses; they're arguably even more useful for scoping out the competition's. Website crawlers analyze a website's URL, link structure, images, CSS scripting, associated apps, and third-party services to evaluate SEO. Not unlike how a website monitoring tool scans for a webpage's overall "health," website crawlers can identify factors such as broken links and errors, website lag, and content or metadata with low keyword density and SEO value, while mapping a website's architecture. Website crawlers can help your business improve website user experience (UX) while identifying key areas of improvement to help pages rank better. DeepCrawl is, by far, the most granular and detailed website crawler in this roundup, although Ahrefs and Majestic also provide comprehensive domain crawling and website optimization recommendations. Another major crawler we didn't test is Screaming Frog, which we'll soon discuss in the section called "The Enterprise Tier."
The ranking of your website is partly decided by on-page factors. On-page SEO factors are all those things you can influence from within your actual website. These factors include technical aspects (e.g. the quality of your code and site speed) and content-related aspects, like the structure of your website or the quality of the copy on your website. These are all crucial on-page SEO factors.
Crawlers are largely a separate product category. There is some overlap with the self-service keyword tools (Ahrefs, for instance, does both), but crawling is another important piece of the puzzle. We tested several tools with these capabilities either as their express purpose or as features within a larger platform. Ahrefs, DeepCrawl, Majestic, and LinkResearchTools are all primarily focused on crawling and backlink tracking, the inbound links coming to your site from another website. Moz Pro, SpyFu, SEMrush, and AWR Cloud all include domain crawling or backlink tracking features as part of their SEO arsenals.
Think, that one day, your website will have to pass a manual review by ‘Google’ – the better rankings you get, or the more traffic you get, the more likely you are to be reviewed. Know that Google, at least classes even useful sites as spammy, according to leaked documents. If you want a site to rank high in Google – it better ‘do’ something other than exist only link to another site because of a paid commission. Know that to succeed, your website needs to be USEFUL, to a visitor that Google will send you – and a useful website is not just a website, with a sole commercial intent, of sending a visitor from Google to another site – or a ‘thin affiliate’ as Google CLASSIFIES it.
What does it mean for a site to be SEO friendly? It goes beyond just posting quality content (though that’s a very important part!). There are all kinds of ways big and small that can prevent your site from being seen by search engines and thus by users. Our free audit tool begins by looking at some of the most important facets of your website you might not even be aware of.
This can be broken down into three primary categories: ad hoc keyword research, ongoing search position monitoring, and crawling, which is when Google bots search through sites to determine which pages to index. In this roundup, we'll explain what each of those categories means for your business, the types of platforms and tools you can use to cover all of your SEO bases, and what to look for when investing in those tools.
What would be the purpose of/reason for moving back to a different url? If its been a few years, I’d leave it alone unless you watched everything decline since moving to the main url. Moving the forum to a new url now would probably be a bit chaotic, not only for your main url but for the forum itself…. Only reason I could imagine myself moving the forum in this scenario would be if all those links were really awful and unrelated to the url it currently sits on…
I’ve got by, by thinking external links to other sites should probably be on single pages deeper in your site architecture, with the pages receiving all your Google Juice once it’s been “soaked up” by the higher pages in your site structure (the home page, your category pages). This tactic is old school but I still follow it. I don’t need to think you need to worry about that, too much, in 2019.
A poor 404 page and user interaction with it, can only lead to a ‘poor user experience’ signal at Google’s end, for a number of reasons. I will highlight a poor 404 page in my audits and actually programmatically look for signs of this issue when I scan a site. I don’t know if Google looks at your site that way to rate it e.g. algorithmically determines if you have a good 404 page – or if it is a UX factor, something to be taken into consideration further down the line – or purely to get you thinking about 404 pages (in general) to help prevent Google wasting resources indexing crud pages and presenting poor results to searchers. I think rather that any rating would be a second order scoring including data from user activity on the SERPs – stuff we as SEO can’t see.
Consider the current status of your website. When you create a marketing strategy, look for the “low-hanging fruit”, or the opportunities that will make the biggest impact with the least amount of work Click & Tweet! . So before you launch a search marketing campaign, research your website to see where you may have the potential to grow an organic SEO strategy that is already working before putting money into an SEM campaign.
In a few cases, see what happens if you make more risky changes. I’m working with a website that wasn’t even in the top 100 positions for many of its 20 strategic keywords. Based on some data, it looked like the client’s sweet spot for keywords may be in the 10 to 30 range for average search value. We targeted one phrase with 700 searches a month. It’s now ranking No. 12 on Google after making two sets of SEO changes on one page. Ultimately, the client may need a new page to grab a spot among the top 10 positions.
I think ranking in organic listings is a lot about trusted links making trusted pages rank, making trusted links making trusted pages rank ad nauseam for various keywords. Some pages can pass trust to another site; some pages cannot. Some links can. Some cannot. Some links are trusted enough to pass ranking signals to another page. Some are not. YOU NEED LINKS FROM TRUSTED PAGES IF YOU WANT TO RANK AND AVOID PENALTIES & FILTERS.
Google expects pages to “be edited, reviewed, and updated on a regular basis” especially if they are for important issues like medical information, and states not all pages are held to such standards, but one can expect that Google wants information updated in a reasonable timescale. How reasonable this is, is dependant on the TOPIC and the PURPOSE of the web page RELATIVE to competing pages on the web.
Make it as easy as possible for users to go from general content to the more specific content they want on your site. Add navigation pages when it makes sense and effectively work these into your internal link structure. Make sure all of the pages on your site are reachable through links, and that they don't require an internal "search" functionality to be found. Link to related pages, where appropriate, to allow users to discover similar content.
QUOTE: “Medium pages achieve their purpose and have neither high nor low expertise, authoritativeness, and trustworthiness. However, Medium pages lack the characteristics that would support a higher quality rating. Occasionally, you will find a page with a mix of high and low quality characteristics. In those cases, the best page quality rating may be Medium.” Google Quality Evaluator Guidelines, 2017
If you are just starting out, don’t think you can fool Google about everything all the time. Google has VERY probably seen your tactics before. So, it’s best to keep your plan simple. GET RELEVANT. GET REPUTABLE. Aim for a healthy, satisfying visitor experience. If you are just starting out – you may as well learn how to do it within Google’s Webmaster Guidelines first. Make a decision, early, if you are going to follow Google’s guidelines, or not, and stick to it. Don’t be caught in the middle with an important project. Do not always follow the herd.
Rob Marvin served as PCMag's Associate Features Editor from December 2017 to December 2019. He wrote features, news, and trend stories on all manner of emerging technologies. Beats included: big tech coverage, startups, business and venture capital, blockchain and cryptocurrencies, artificial intelligence, augmented and virtual reality, IoT and aut... See Full Bio
Having a different description meta tag for each page helps both users and Google, especially in searches where users may bring up multiple pages on your domain (for example, searches using the site: operator). If your site has thousands or even millions of pages, hand-crafting description meta tags probably isn't feasible. In this case, you could automatically generate description meta tags based on each page's content.
If you are using Responsive Web Design, use meta name="viewport" tag to tell the browser how to adjust the content. If you use Dynamic Serving, use the Vary HTTP header to signal your changes depending on the user-agent. If you are using separate URLs, signal the relationship between two URLs by tag with rel="canonical" and rel="alternate" elements.
I had the same issue. I invested time to go to the website of each of these tools, had to read through the specs of what they offer in their free account etc etc. Some of them did not even allow you to use one single feature until you gave them details for a credit card (even thouhg they wouldn’t charge it for 10-15 days or so). I did not enjoy this approch at all. Free is free. “Free version” should only talk about what you can do in the free version. Same goes for trial version.
Brian, I’m going through Step 3, which is referring to the one version of the website. I found a very good free tool (https://varvy.com/tools/redirects/) to recommend. It checks on the redirect and gives you a visual number of hops. More hops mean more delay. For example, if I use your manual method to check on https://uprenew.com, all looks good. However, if I use the tool and check, I realize there is an unnecessary 1 hop/delay, whereby I can fix it. Hope this helps. : )