TL;DR
- Development resources can be thin, but that doesn’t mean visibility optimizations need to stop.
- Ensure your URLs are always visible to the search engine crawlers.
- Utilize this template to create static HTML & XML sitemaps.
What Is An HTML & XML Sitemap?
An HTML Sitemap or ‘frontend sitemap’ is a web page on a website that lists all the important pages on the site in a hierarchical structure. It is designed to help both users and search engines navigate a website and find content quickly and easily.
HTML sitemaps are typically created for larger websites with many pages, but I typically create them for all websites I work on to eliminate any doubt. By providing a clear overview of the site’s structure, an HTML sitemap allows crawlers & visitors to find the information they need more easily. They are typically linked footer or main navigation menu, and they can be organized in a number of ways, such as by category, by topic, or by page type. See the New York Times HTML Sitemap example.
An XML Sitemap is a file that lists all the ‘indexable’ URLs of a website and can help search engines like Google and Bing, to crawl, render and index a website more effectively. These are typically structured like /sitemap.xml and should be referenced in the robots.txt file.
By submitting an XML sitemap to search engines, you can ensure that all the pages on your website are discovered and ideally indexed, even if they are not linked to other pages on your site (which they should be, fix the orphan pages). This can help improve your website’s visibility in search engine results and make it easier for users to find the information they are looking for.
Video Walkthrough – Manually Create HTML & XML Sitemap
Manual HTML & XML Sitemaps
Ideally, you would have the frontend and XML sitemaps created dynamically so you never need to add/remove URLs manually. Unfortunately, that is not always the case and SEOs either need to get crafty or accept that the issue will remain unoptimized until development can tackle the issue. I’m pretty impatient regarding visibility optimizations so I wanted to create a workaround to help speed it up. This is a bandaid fix to ensure we have crawler visibility until the development team can complete a more sustainable process. Even though it’s a gritty temporary fix, it works!
How To Create An HTML Sitemap
Most SEOs or digital marketers will have access to create new pages so this is typically an easy optimization. There is a lot of debate about frontend sitemaps and whether or not they are worth the time investment. My methodology with SEO is to eliminate all doubt. I want to ensure there is no issue with search engine crawlers accessing our most important links through the HTML. If the navigation and other sections have a lot of JS in front of the links, this eliminates any doubt that the crawlers will find the page.
Static HTML Sitemap Process
- Visit the Frontend Sitemap / XML Sitemap Template (Frontend Sitemap Sheet).
- Make a copy under File in the menu.
- If you have Screaming Frog, crawl your site and copy all URLs into Column A and all H1s into Column B.
- If you do not have Screaming Frog I recommend getting it, very handy for all people who work on the web. If you do not have access to a crawler you will need to manually grab your most important URLs and paste them into the sheet.
- Copy The Formula Results in Column F.
- Paste Formula Results (Values Only) in Column H.
- Create a page on your website and paste the final results onto the page. The HTML links are already formatted.
- I recommend doing some additional formatting for easier viewing, but the technical setup is complete regardless.
Now that you have a list of your URLs formatted for HTML links you can create a frontend sitemap page. I like to break out the sections with headers for easier viewing. While this is mainly for the search engine crawlers to find the important URLs without any issue, it can also be helpful to users who are familiar with sitemaps. Making it more user-friendly can always help.
The best practice for the HTML/frontend sitemap is to link it in the global footer similar to what I have done on this website. As new pages get created or old ones get removed you will need to update accordingly. This template should be saved as a master copy and updated over time with the relevant pages.
How To Create An XML Sitemap
The XML sitemap is probably the most important asset for search engine crawler visibility. This is something that should be heavily invested in to ensure it’s error-free and houses all URLs that should be discovered by search engines. Most modern content management systems are built to handle this with no problem, but errors can always come up. If development resources are thin and you need pages added quickly you can always create a static XML sitemap. All you need to do is create a file with the ‘.xml’ extension and upload it to your website. You will then be able to add it to your robots.txt file and submit it to Google Search Console.
You may also see in Google Search Console that there are URLs indexed, but missing from the XML sitemap. If these are URLs you want to be discovered in the search engine, you should add them during this process.
Static XML Sitemap Process
- Visit the Frontend Sitemap / XML Sitemap Template (XML Sitemap Sheet).
- Make a copy under File in the menu.
- Similar to above if you have Screaming Frog, crawl your site and copy all URLs into Column E.
- Copy the Formula Results in Column F.
- Paste Formula Results (Values Only) in Column G.
- Copy the Final Results into a text editor (Notepad++ or Sublime Text).
- Make sure to add the start and end lines in Column B for the XML sitemap format.
- Save the file as a .xml.
- Upload the file to your website through the media library or FTP.
What Comes After The HTML & XML Sitemaps?
SEO gets broken down into two main areas, visibility and content. Without visibility, the best content in the world will never rank. These two manual processes will help improve your organic visibility. Even though you should always strive for dynamic setups, this will accomplish the job just as well.
You will need to stay on top of both of these since they are static. If new pages are added you will need to go through the process again to add them. If pages are removed you will need to go through the template and remove them, then republish. I recommend creating a master version and having that be the source of truth. Do not change the XML file without changing the spreadsheet because you may end up messing up URLs if you switch back and forth on how you update.
How I Help
I am mainly using this site as a portfolio to create content that can help others in the industry. I have learned so much from people in this field and I want to do my part to help as well. I will continue to create processes and free templates that can hopefully help anyone in SEO and Digital Marketing. If there is anything you are looking for or have questions about please reach out to me on my Contact page or on Twitter!
Sitemap Resources
- How to Create an XML Sitemap: Tips & Best Practices – SEMrush
- How to Optimize XML Sitemaps: 13 SEO Best Practices – Search Engine Journal
- 7 Reasons Why An HTML Sitemap Is Still A Must-Have – Search Engine Journal
