If you are just starting out in the world of digital marketing and SEO, Screaming Frog is about to become your best friend.
The only weird and funny thing about Screaming Frog is its name. In reality, it’s a highly useful tool, which many SEO specialists can’t work without. Learning how to use this software can help you achieve top-notch results with your marketing campaign. Once you start relying on Screaming Frog, you won’t understand how you managed to work without it.
This detailed guide will help you learn the ins and outs of the tool to start taking full advantage of it immediately.
What Is Screaming Frog?
Screaming Frog is an SEO spider tool. Spider software crawls around on the website, collects data and presents it to the user. For example, Google uses several such tools to check websites in order to rank them. Thanks to these programs, users get to see high-quality websites on the first page of search results. Other search engines use spider software as well.
The Screaming Frog software is a desktop application that you can download to your computer.
The spider checks your website and collects SEO data, such as URL, metadata, schema categories, and more. One of the top benefits of using such a tool when you are just starting out is that it discovers SEO issues which you may have missed. Meaning, it catches such mistakes as broken links, improper redirects, missing metadata, etc. The first thing you can use the spider for is to tweak your website for SEO.
The tool collects a variety of useful data, including the status of internal and external links, their security, issues with meta titles, page loading time, word counts, image information, alt text…the list goes on and on.
While collecting data and auditing your website and content, Screaming Frog follows AJAX standards for web crawling. It follows the same directives as the Google bot, including no-follow and non-index tags in robots.txt.
Overall, Screaming Frog is a highly useful website crawler that can help identify SEO issues on your website and make decisions needed to rank higher on Google search.
How Much is Screaming Frog Website Crawler?
The program comes with a free version, which offers limited options. However, it’s good enough for newbies to learn how to use the tool and figure out its efficiency. The paid version has about twice as many features and costs around $200 per year.
How to Use Screaming Frog SEO Crawler
Below are a few things you should do for the crawling process to be as efficient as possible.
Setting Up Your Computer
Before you learn how to use Screaming Frog, you need to set up your computer. When you start crawling sites, you’ll understand that some of them require much more of your system memory than others. All the data Screaming Frog finds needs to be stored somewhere.
So before you start crawling, give Screaming Frog as much of your system’s RAM as possible to ensure speed and efficiency. This is necessary for websites with over 150,000 URLs to be checked.
Click the “configuration” tab within the program and go to system>memory. For 32-bit computers, 1GB of RAM (2 GB for 64 bit) is a default setting. Consider allowing Screaming Frog to use 8GB so it can crawl up to 5M URLs. However, make sure you use at least 2GB less than your total RAM.
Restart the tool for the changes to apply.
Adjust Tool Configurations
Once you set up the computer, you need to adjust configurations according to your needs. Go to Configuration > Spider > Basic. Here, you can see default settings for each crawling attempt. Make sure you change the settings for each crawl to get the results you need.
In case you are planning to crawl a JavaScript website and need to check internal navigation, you have to execute several crawls with different configurations for a specific page instead of the entire domain. Use the “rendering” tab (Configuration > Spider > Basic > Rendering) to make sure the crawler can find such instances.
To catch specific things on the website, you need to set custom tool filters (Configuration > Custom > Search). You can take advantage of these filters to point the crawlers’ attention to the parameters you need the most. You can save settings for the future crawls so you won’t have to go through the configuration every time (File > Configuration > Save As). Next time you need a crawl, you can load the configuration (File > Configuration > Load).
Crawling the First Website
Once everything is set up, you can proceed to crawl your first website. Use the default “Spider” mode for the initial crawl. It will check the website and collect links based on the configurations and filters you set up. (Mode > Spider > Enter URL > Click Start).
Remember, you can stop and resume the crawling process at any time. In case you turn the system off or exit the tool before the crawling is completed, you will lose all data collected during the current crawl. If you need to leave, you can stop the crawl, save it, and resume your work later.
If the tool crashes during a crawl, then you may have memory issues. Check how much RAM you have available.
How to Read Screaming Frog Results
Let’s talk about how to work with the results you get from Screaming Frog’s crawling efforts.
Link Auditing
Since proper links are extremely important for your rankings, link auditing can help you figure out what works and what doesn’t. You can also use the tool to check what the competitive websites are doing with their linking strategy. Knowing how to read Screaming Frog results can help you with crawling efforts in the future.
Go to the Spider’s configuration menu and check off all subdomains while unchecking CSS, images, Flash, JavaScript and all other options you don’t need for the link auditing process. Figure out if you want to crawl nofollow links and check the box accordingly. Run the crawl.
Once the crawling is finished, go to the Advanced Report tab and choose “All Links”. This action will generate and export a CSV of all the crawled links with their location, destination, directives, anchor texts, and other information.
As you export the file, you can start working with the data. Here are a few things you can do:
- Sort links by outlinks by clicking the internal tab. These results will show you pages that have the most links to your sites. Pages that have over 100 links can be marked as suspicious by Google so you may want to check why a page has such a huge link number.
- Click status code (internal tab). It will show you broken links (404 status). It’s important to fix broken links because they hurt your rankings. The 301 or other redirect status pages may be redirecting to the home page or harmful websites. Check each link to figure out if it needs to be removed. Advanced Report tab also allows you to generate reports according to the status codes.
Content Auditing
The majority of ranking factors are based on content. That’s why content auditing with the Screaming Frog can bring useful results. To perform a content audit, run a full crawl with CSS, scripts, images, etc. For such an audit, you need as much data as the SEO crawler can collect.
Use the internal tab to filter by HTML and sort the word count column from low to high. Pages with less than 500 words on them usually have thin content. Go over these pages to figure out if you can remove them entirely or add high-quality content to beef them up. Sometimes, thin content pages are product pages with a small description. Then deleting them is a mistake. However, adding some related content is a possibility.
Then go to the images tab and filter results by “missing alt text”. This can help you identify images without alt tabs. Such pictures are hindering your SEO efforts. You can go further to filter “alt text over 100 characters”. Similar to missing alt text, excessive descriptions won’t help your search rankings or improve user experience.
Filter the title results for “titles over 70 characters”. Big titles will get cut by Google, thus making your website appear hard-to-read in the search results. Change titles to fit into the 60-character count.
In the same filter tab, check for duplicate pages and pages with duplicate meta titles. If you see duplicate meta titles, it means that you have similar content. This can lead to a Panda penalty from Google. In case only meta titles are similar while pages are unique, you need to change the title for one of them. If the content is duplicated, remove one page and redirect its URL to the other one.
Use the URL tab to find any duplicate URLs to canonicalize or fix. You can also filter by various options to figure out if you have hard-to-read of non-standards URLs, which require changes. Pay special attention to pages with excessive underscores of non-ASCII characters.
Check the meta description tag to filter by duplicate. Any duplicated meta description on unique or duplicate pages requires your attention. These duplicates are hurting your rankings and reputation.
Generate an XML Sitemap
Since XML sitemaps can be highly useful to Google because they allow it to know where your pages are and when you updated them last, creating such a map can help your rankings. Screaming Frog allows generating such a map.
- Crawl your website completely (with subdomains)
- Go to “Advanced Export” and click the XML Sitemap feature (it saves your sitemap as an Excel table so you can edit it later)
- Open the table and select “read online” and “open as SML table”
- Edit your sitemap
- Save it back as an XML file and upload it to Google.
Check an Insecure Report
When learning how to read Screaming Frog results, you need to pay special attention to the Insecure Content Report. You can use to check how safe your website is from cyberattacks.
Go to the “Reports” menu and choose “Insecure Content”. The tool will generate a report that lists all JavaScript, CSS, images, and other elements that can be insecure on your website. In case you have a long list of such insecure content, it’s time to beef up the security.
Final Thoughts
If you master all of the above options, you are ready to improve your website. Screaming Frog is a highly robust tool with numerous features. The more you use it, the more interesting opportunities you will discover. Good luck!