Table of Contents
- Process Methodology
- Starting Your Research
- Time to Get Your Keyword File Started
- If Using Google Keyword Planner
- Developing Your Keyword File
- Time To Dig In
- A Look at Search Intent
- Keyword Research for Existing Websites
- Your Master Seed List
- What About Getting Accurate Search Volume Data?
- Using The SEMRush API
- Mapping Search Intent
- Why is this process so valuable?
I already know what you’re thinking; it probably goes something like:
Seriously? Why would anyone write another post on keyword research?
The reason is simple; times change.
The sheer amount of data and tools now available, not to mention the increased need to understand contributing factors such as latent semantic indexing (LSI) and the potential effects of RankBrain, change the landscape of information.
Not to mention the fact that Google has now taken away more of our data, rendering many of the tools on the market (whom relied on scraping Google’s Keyword Planner) useless.
Then on top of that, with Google’s various signaling algorithms and all this data in near constant flux – I thought it was a reasonable time to revisit what I still believe to be, the most important aspect of SEO.
UPDATE: This post has been updated to reflect Google’s change to free use of keyword planner and makes use of alternative keyword research tools to fetch search volume data and fill in holes.
You can find
a ton millions, literally ~4.6 million+ pages on the internet about keyword research;
95% of those pages will say the same thing; use keyword planner, sort by volume, use some other tool to get a competitiveness or difficulty score, blah blah.
Most of the processes are near identical, and leave a lot of stones unturned.
Not to mention one near-constant, gaping hole; a process to select and prioritize keywords based on a realistic timeline.
This post is going to run through my current process.
Which has bits and pieces from many others, but will also have you mine a considerable amount of additional competitive metrics to make real, informed decisions in terms of which keywords you should be targeting – and more so, how to create a priority list you can actually attack.
My methodology for the keyword research process is relatively straight-forward:
- Start with common sense, and then use data to validate your ideas.
- Use websites ranking well with large relevant footprints to expand your keyword list.
- Find patterns commonly used as head terms, to allow you to abstract your common modifiers and define the “long tail.”
- Grab existing footprint data (*only valid for existing websites).
- Expand your list of seed terms to include tangential and related topics and concepts.
- Retrieve search volume and commercial value data.
- Sort by search volume and use negative keywords to filter down your list to core terms.
- Leverage existing SERP’s to get a list of all your competitors at the keyword-level.
- Mine directional competitive metrics for all SERP competitors.
- Build multi-layer filters to identify easy wins, high value targets, and create timelines for ranking.
Starting Your Research
So first things first, you need to turn on your brain and think for a few minutes.
Start by thinking of just one general phrase that describes the topic or concept you’re doing keyword research for; and make it a phrase, not a word.
It can be 2, 3, or 4 words… as long as it’s not just one word.
The term “keyword” is actually a bit misleading.. cause phrases are keywords.. as least the most qualified and most valuable ones.
Got your top-level term?
Now it’s time to get some data to validate your idea.
I prefer to use SEMRush, but in a recent update they removed ALL export functionality from their free tier.. kind of a bust in my opinion in terms of driving user adoption, but I’m not their product manager so no use in complaining… moving along;
*IF* you have an SEMRush account (like I do – and you probably should if you’re serious about SEO), then you can save yourself some serious time by starting there – so if you do – type your seed keyword into the SEMRush box below and hit enter.
*IF NOT* – Click here to get started with Keyword Planner (which now requires a paying account)
If you used the box above – you are now staring at a results screen inside SEMRush that looks like this:
Now let’s run through what you’re looking at and how to make use of all this glorious data.
A Look at Phrase Match Keywords
So drawing your attention to the bottom left hand side of this dashboard, click the blue button to “view full report:”
Which brings you to this screen:
^you’ll quickly notice that the free version only gives you access to 10 results, but for the purposes of my process right now – that’s OK.
Time to Get Your Keyword File Started
Go ahead and click export in the top right corner of that report to grab the data on those first 10 terms:
For the purposes of this process go ahead and choose the Excel option, and save that file:
Now use the left-hand navigation and go to the “Related Keywords” view:
and grab and export from there too.
Combine those files into one sheet via ye olde copy and paste, which should leave you with a file looking something like this:
Before I move forward I need to take a quick detour into the land of formerly free-use keyword research via keyword planner, so if you’ve completed the SEMRush steps above, jump ahead here ›
If Using Google Keyword Planner
The logic for using keyword planner, and getting the most out of it, is the same as SEMRush – you just get
a bit way less data – especially now since Google requires you to be running active bids on AdWords to get the data you’re after.
So unless you’re planning on running an AdWords campaign – you can skip the section below…
Jump into keyword planner’s default interface (below) and have that seed keyword you picked out earlier handy:
Select the option to “Search for new keywords using a phrase…” and enter your seed term;
TIP: Click into the “keyword options” panel and check the box for “Only show ideas closely related to my search terms” and click Save.
Then click the “Get ideas” button, and you’ll land here:
and here is where it starts to get interesting…
Before diving straight into the keyword ideas, there’s a little known filtering option that’s worth checking out; click on the drop down toward the top of the page with the default label of “Search volume trends:”
using these filters can provide all sorts of incredibly important directional data both for SEO but also for paid search.
For instance, let’s say I wanted to test a new product or service offering by sending some PPC traffic at it prior to kicking off a content or link building campaign. Using the “Breakdown by location” option I can see which geographies I should be targeting both for local pitches but also for potential geo-gating:
moving right along – grab all the keyword ideas in here by clicking the download button:
You will then be presented with a modal window with some options, for the purposes of this exercise I recommend NOT segmenting by month and downloading to an Excel CSV, which will leave you here:
Developing Your Keyword File
First things first, let’s clean up this mess.
Regardless of which route you took; free with keyword planner, paid with SEMRush (or term, volume, and competitiveness data out of pretty much any other keyword tool), your file has a bunch of fluff data that is not going to help at this phase of the process – so let’s go ahead and clear that out.
Cleaning Up SEMRush Data
First delete the 2 columns we don’t need; Number of Results and Trends.
There may be some instances where you would want to look at these but for my process at this point you won’t need them.
Now it’s time to stack a couple filters to lean on data to do more of the work for us.
So select Row 1, then click the “Data” tab, and then the “Filter” tab.
First sort Column B (Search Volume) from largest to smallest, and then filter column D (CPC) and uncheck the boxes next to anything under $1.00.
Why filter out keywords with less than $1 CPC?
I would consider keywords related to keyword research to be what I would consider a mature SERP, in that the demand is stable and there’s clearly terms in this vertical that have commercial intent – we can cut out some of the junk by removing terms that don’t have enough commercial intent to warrant at least a $1 per click price.
Yes, it’s possible these terms are great top of funnel terms where visitors are gathering information to start their buyer journey – but today we’re focusing on the money.
Cleaning Up Keyword Planner Data
Unlike SEMRush, keyword planner exports a bunch of junk you can cut right out;
Then just like with SEMRush, select row 1 > go to Data > Filter.
Sort by your Column B (Avg. Monthly Search Volume) and then filter out the “Suggested bid” column for any bids under $1.00.
You should be left with file that resembles this:
Time To Dig In
What you’re left with is a handful of keyword phrases, monthly search volume, AdWords competition, and Google’s suggested cost per click bid price.
IF you’re fortunate enough to have this data from running an AdWords campaign, or you did your research recently enough that it’s still relevant (i.e. within the past 30 days if your business isn’t hyper-seasonal) we can pick up the process of making sense of these numbers.
First you’re going to want to slice cull this list down to be a representative list of seed terms.
I prefer to do this by abstracting out the root term, and building a list of modifiers.
What’s a Modifier?
So as a quick aside, a modifier is a word or phrase that is appended or prepended to your root seed keyword phrase that changes or enhances it’s meaning.
So for example, in the list above the root term is “keyword research,” and the modifiers shown in that snapshot are:
- how to do
Even though there are 15 keywords in the list shown in the snapshot, those 8 terms are the only modifiers – making sense?
Using Modifiers to Expand Your Seed List
What you can do from here (if it makes sense for your website and content vertical – which I’ll come back to in a moment), is to use a tool like mergewords.com to quickly blow out your list of seed permutations. (H/T to Ollie Hayman for the find).
Using multipliers is especially useful for working up long-tail phrases to be included in body copy for things like recipes, business processes, and other topics where the words tend to be the same but the order can vary significantly depending on the writer.
Using Scraping Tools to Further Expand Your List
Tools like ubersuggest still afford great uses for expanding on initial lists of seed terms.
and BONUS now also includes monthly search volume, though if I’m being honest I’m not sure where they’re getting it from – I sincerely doubt they’re maintaining their own index (no hard feelings ubersuggest).
There’s also a free tool or two that will scrape many of the sources that people use to search everyday (Google, YouTube, Ebay, Amazon, the App Store, etc.) and provide you with large lists of keyword ideas relevant to your core seed term.
From there though it’s still up to you to run through the painstaking process of getting search volume and competitive metrics for all those words.
A Look at Search Intent
Gone are the days of just picking keywords because of high search volume or low competition.
The modern customer journey requires a deep understanding of what searchers are expecting to see when they query google seeking answers.
This means classifying modifiers into buckets that allow you to infer the intent behind the words.
Inferring intent varies greatly between search verticals, but still all follow a general pattern of mirroring the stages of the conversion funnel:
Let me explain what you’re looking at in the above diagram.
The stages of the traditional conversion funnel are AIDA, and as I mentioned these align rather nicely with the 4 top-level buckets of search intent.
As searchers modify their keywords to include intent modifiers, the desire of the searcher moves further down the conversion funnel.
The practical application of this is that as keywords get bigger, i.e. include more words, they become more “qualified” in terms of indicating where the searcher is at in the above conversion funnel.
So if you look at how the keyword universe as a whole is comprised:
Common patterns and modifiers found in search intent buckets include:
- Single word keywords, i.e. shoes, shorts, pants, CRM, etc.
- Brand terms
- Names of products or services
- Colors: black, blue, red, green
- Sizes: small, medium, large, 32, 11
- Sex or age: mens, womens, kids, toddler
- keywords containing “vs”
- Location: Philadelphia, Seattle, NYC
Understanding searchers implied intent by tagging your keywords for which stage of the buying funnel they represent, will allow you to develop better content to serve their expectations.
This means you stand a better chance at providing the information they’re looking for when they’re seeking it and have the opportunity to move them further down the funnel towards a transaction.
We’ll come back to this a bit later in the post.
Keyword Research for Existing Websites
Existing websites inherently have a leg up in the keyword research process – unless you’ve been blocking all robots from crawling and indexing the pages on your site, you will be able to get some data on how your site is currently categorized from a contextual relevance stand-point.
My favorite way to do this is to grab your homepage URL and head back over to SEMRush (yes again).
They have a cool feature that will show you not only the keywords your site is currently ranking for, but the other sites and URL’s that are ranking, the number of keywords you share, and give you the opportunity to dive into those overlap sets to get more insights.
Leveraging Google Analytics (and Search Console)
So you have an existing website, hopefully if you’re reading this since you’re interested in SEO – you understand how critical it is to have as much data as possible for your website.
So naturally it makes sense to start with keywords you already have visibility for. The screenshot above is taken directly out of Google Analytics (GA) with Google Search Console (GSC) hooked up.
Once you’ve verified your site with GSC and authenticated your account within GA you get all this lovely impression, CTR, and average position data flowing in.
GA is wonderful for a lot of reasons, but perhaps one of the greatest is that it’s still one place where greedy Google doesn’t handcuff all your data – so you can export all of this lovely data showing your current term relevancy out and into a useable format; a spreadsheet.
You can export for Excel, CSV, TSV, or even direct to Google Sheets.
So for the purposes of this post – let’s do exactly that, sort descending by Clicks > set the results to the largest number you can (5000) > and click the old Export > Google Sheets;
Your Master Seed List
Now that you have all this great data from all these sources – you’re ready to use an intelligence tool to fetch all of your competitive data.
My go to for this is still TermExplorer, though there are some caveats now thanks to Google’s recent Keyword Planner changes.
We’ll need to grab search volume data from elsewhere and merge this in, however the data points that are still very important that TE does afford us are:
- Keyword use in page title, domain, URL, and on page
- Word count of page
- # of links to the domain
- # of links to the page
- Age of domain
- And provides a solid score for:
- Link strength
So with that let’s run a bulk keyword job in TermExplorer (this can be done with a free account) – just know you’ll need to get the search volume out of something like SEMRush (or one of the other *very* few data sources that maintains their own keyword index).
Let that run in the background (it’s going to take a little while) and in the meantime take that seed list and let’s go get search volume data for these keywords – so I’m going to head back to SEMRush.
Once TE finishes up running it’s discovery process expanding on your seed list, scroll to the bottom and make sure you set “Display ____ Records” to All – otherwise you will only export what’s currently shown on your screen:
For Those Who Value Data
If you are someone who still refuses to pay for data, you can stop here – good luck.
This next part is where TermExplorer actually becomes worth paying for.
Now “Select All” > “Download CSV” and then “Send checked keywords to Keyword Analyser”
We’ll come back to the keyword analyzer later (and you’re going to need some credits to get all the data you’re going to want):
- YES these will cost money
- NO it’s not much (you can spend $34 and you’ll have all the data you need for 6 months for this project – with credits left for other projects should you want to use them)
- YES I promise this is extremely valuable, and I’m going to show you exactly why.
Once your Keyword Analyser report finishes, you should see a screen that looks like this:
This is all well and good but the BULK of the data we’re after is not displayed here – so click the button to download CSV:
Now got to your google drive and upload the CSV – you should see something like this:
Notice anything strange? Yeah… all that keyword search volume is wrong.
However, all the rest of this data is not only accurate, but extremely helpful. What TermExplorer’s analyzer does is not only goes out and scrapes all the ranking URL’s on the first page SERP, but then it goes and scrapes each of those URL’s and pulls all their link data.
So even though at first it seems like there are loads of duplicate results – LOOK CLOSER – instead it’s displaying all the URL’s ranking on the first page for each keyword.
What About Getting Accurate Search Volume Data?
Well, we’ll have to do something about that.
Good news is that now that we’re in Google Sheets, we can use something like SEMRush’s API to fill in the blank that we will soon have.
So first step – delete all of the data out of Column C (Average Monthly Searches).
Next add in a new column to the left of Column C, and name this Column “API Call” (just for reference).
Next it’s time to go fetch the data we’ll need from SEMRush.
Using The SEMRush API
I promise this sounds scarier than it is.
The first thing you’ll need is the syntax for importing data into your Google Sheet using SEMRush’s API, but lucky you – I’m going to give that to you:
[snippet slug=semrush-api-google-sheets lang=vbscript]
In order to use that script you’ll need to grab your SEMRush API Key and then replace the section where I have “YourSEMRushAPIKeyGoesHere” with your key (kind of obvious right?).
Now once you have your script ready to go, you’re going to want to paste that into Cell C2 (right under your header titled “API Call”)
You should see it render “Loading…” for a few seconds and then pull back the search volume and populate the previously empty cell in Column D.
Now comes the easy (and fun) part – simply grab the bottom right-hand corner of that cell and drag it down to the bottom of your file (or if you prefer copy the cell, scroll all the way to the bottom of the Column, hold shift and paste).
You should now be the proud owner of a file that looks something like this:
Now to do a bit of house cleaning…
Once all of you data has finished populating via the API (and depending on how many rows of data you’re requesting this could take several minutes) you will now want to lock all that data into your file so it’s no longer dynamic.
So again, create a new Column, this time to the left of the Average CPC column, and again name this “Monthly Searches.”
Now Copy all the data from Column D and then Paste Special > Values.
Mapping Search Intent
Remember that time we talked about search intent and how important it is?
Now it’s time to put it to put the concept to work to help you prioritize your keywords and come up with an initial target list.
To do this you will want to create another column in your keyword file and title it “Search Intent.”
As for mapping search intent there are a few ways to do it:
- Manually (bleh!)
- Using a Macro (like the one I created for Excel several years ago)
- Using Sentinel
Sentinel is a keyword intelligence tool me and my team created to help automatically take all the data you just gathered, automatically tag for intent based on your specific keyword vertical and the modifiers therein, and then automatically prioritize which keywords you should be going after based on rank potential.
Better yet – it’s free. ALL the functionality is Free. There are paid tiers but only for massive amounts of data where there’s a cost to host all of it, otherwise – completely free for all the automatic keyword opportunity goodness.
Regardless of how you get your search intent mapped to your keywords, once you have it baked into your keyword file there’s a few filters you can now use to find your opportunities.
First turn on Column Filtering by selecting row 1 and going to Data > Filter.
Now filter Page Links from lowest to highest and then using the filter on the search intent column remove the checkboxes for Informational and Navigational.
You’re left with an ordered list of bottom of the funnel keywords with the least amount of links needed to crack rankings.
To take this a step further you can additionally sort the words on page from Z to A (least to most) to then find even easier ranking opportunities as the amount of content needed will be the least among your keyword population.
Pay attention to the number in Column A (“Position”) as this will show the rank potential you can realistically target for a given keyword based on hitting or exceeding the ranking heuristics in that row, i.e. words on page, keyword use in meta attributes, and link count.
Can you see how you can start to glean immediately actionable information from this data?
Another quick win is looking for pages ranking for keywords where the keyword is not being used in any of the meta attributes (title, domain, URL, or on-page content) – these can usually be cracked simply by standing up a basic page and targeting the keyword with the page title.
Why is this process so valuable?
One simple reason; it takes into considering rank potential – and to date, it’s the only keyword research process that accounts for the fact that you cannot and will not rank for all the keywords you want to.
Now you have all the data you need to make directional decisions in terms of prioritizing your keyword lists and then mapping intent.
If you still prefer to run through the entire process of building out what I call a keyword matrix using all the data I’ve discussed in this post, you may want to check out my course called Master Keyword Research which will take you through the exact steps to do this, or again – you could just use Sentinel.