A primer on JavaScript SEO for WordPress

The technologies which power the web, and which power parts of WordPress, are changing. It’s increasingly common for developers to use JavaScript to make our websites behave more and more like apps. Done well, these approaches create rich, interactive experiences. When built poorly, they can be catastrophic for SEO.

For most WordPress website owners and managers, you shouldn’t need to worry about JavaScript SEO. Most WordPress plugins and themes work fine with JavaScript, and won’t cause any special SEO problems. But if you’re developing bespoke themes or plugins, then you should make sure that you follow best practice, and avoid the common pitfalls.

If you’d like to learn more about JavaScript SEO for WordPress, this article provides a starting point, and signposts resources which you can use to learn more.

Front-end vs back-end technologies

Traditionally, most websites have a distinct ‘back-end’ (or ‘server-side’) and ‘front-end’ (or ‘client-side’). Note that this isn’t the same as “the public or user-facing parts of my website” or “my WordPress admin area“; we’re talking about the technologies which power the website, not the different parts of a WordPress website.

In WordPress, the back-end is usually a combination of PHP (a scripting language) and MySQL (a database technology). These store and process your data, and run the logic which constructs your front-end.

HTML, CSS & JavaScript

The front-end is a combination of HTML, CSS and JavaScript. HTML builds the structure and content (words and images). CSS manages presentation (colours, layouts and styles). JavaScript adds behaviour (interactivity, movement, and responsiveness).

When your browser requests a URL, the back-end determines what the server should respond with. That’s typically a package (or a series of packages) of HTML, CSS and JavaScript files. Your browser downloads and reads those files, and uses them to construct the front-end.

As you navigate through a site, this process repeats each time you visit a different page. Whenever you click a link, the page you’re on unloads, the new page is sent from the server to your browser.

If you’d like to learn more about the difference between back-end and front-end development, we recommend this guide by Chris Castiglione on One Month.

JavaScript in WordPress

Many WordPress sites use JavaScript in plugins and themes to power specific parts of their site. Interactive components like sliders, calendars, popups and similar functionality often rely on JavaScript.

In WordPress, it’s best practice to split JavaScript into individual files, and to ‘enqueue’ them via WordPress’ wp_enqueue_script() function. That allows plugins and themes to share resources, to declare dependencies, and to manage versioning (or even to dequeue other resources).

When it comes to shared resources, it’s common for the front-end of WordPress themes and plugins to have been built using jQuery; a JavaScript ‘library’ which acts as a development framework (and which is included by default, along with many addons and extensions). jQuery isn’t the only option, though, and more modern (or bespoke) themes might use libraries like React. We’ll discuss React, and more modern JavaScript frameworks, later in this article.

You can learn more about including JavaScript in WordPress themes and plugins from this guide on wordpress.org.

Performance considerations

From an SEO perspective, it’s important to consider performance when you’re using JavaScript.

If you’re running lots of plugins, you might be loading a lot of JavaScript, or, running lots of complex scripts. That can slow down your website.

It’s important to follow best practice for JavaScript optimization, like minifying files, using caching, only loading what you need for specific pages, and loading non-critical resources asynchronously.

Many of the popular caching plugins should handle most of this for you, but it’s always good practice to check your site’s Core Web Vitals metrics using Google’s PageSpeed Insights reports.

You can learn more about best practice for optimizing JavaScript in this KeyCDN article by Cody Arseault.

Introducing Ajax

For more complex websites, some user interactions (or other scenarios) might mean that you want to update parts of a page without having to reload it.

For example, we might want to let a user submit a form and see a message based on the outcome, without reloading the page.

In these cases, we can use Ajax (“Asynchronous JavaScript and XML”) to send our data from the front-end to the back-end, and then listen for the response. We can then use JavaScript to update the contents of the page and show our message, based on the data we get back from the server.

This kind of functionality powers many of the interactive tools and processes we’ve all become familiar with as the web has evolved; like editing content, exploring maps, submitting forms, and using live chat.

From an SEO perspective, it’s important to remember that any content which is only loaded after user interaction might be undiscoverable by search engines and other systems. In general, you shouldn’t load critical page content via Ajax.

If you’d like to learn more about Ajax, we recommend starting with this W3Schools tutorial.

A schematic of how Ajax works by Daniel Haischt, via Wikimedia Commons

Ajax in WordPress

In WordPress, many plugins and themes use Ajax to dynamically get, set, and update the contents of pages based on user interactions. They typically do this in one of two ways:

1. Requesting a WP REST API endpoint
2. Requesting /wp-admin/admin-ajax.php (with an action)

If you recognize the admin-ajax.php filename, that might be because you’ve seen it show up in your robots.txt file.

WordPress’ default robots.txt file contains an instruction to explicitly allow crawling of this URL, even though its parent folder (/admin-ajax/) is blocked.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

This is because many themes and plugins relied so heavily on (sending, and) retrieving data and content from this endpoint that much of it wasn’t discoverable. By permitting crawlers to follow links to admin-ajax.php endpoints (like admin-ajax.php?action=get-some-critical-content), search engines could crawl and index otherwise ‘invisible’ content.

Well-developed plugins and websites shouldn’t need to rely on exposing content in this way. Instead, they should consider how their content is discovered, crawled and indexed regardless of which of the Ajax approaches they use.

This article won’t go into depth around comparing the technical differences between these approaches, other than to highlight that the WP REST API is far superior to the (legacy) admin-ajax.php approach. It’s easier to develop with, provides a better framework for managing permissions, and is significantly faster to respond.

If you’d like to learn more about the differences between the approaches, you can read more in this article, or see the performance differences here.

The admin area

In 2015, Matt Mullenweg (creator of WordPress) implored the developer community to “Learn JavaScript deeply“. That’s because WordPress’ admin tools and areas are being gradually upgraded; moving away from pure PHP, to use JavaScript-powered editing interfaces.

In the future, more parts of WordPress will utilise JavaScript to enable fluid, flexible editing and management. Your WordPress website, starting with the admin area, will behave more and more like a Single page application.

Single page applications

Over the past few years, a new type of website has emerged. As users expect websites to behave more and more like apps, single page applications (or ‘SPAs’) have become increasingly common.

SPAs are ‘apps’, powered by JavaScript, running inside the browser. When you request a URL, instead of the server returning the HTML, CSS and JavaScript for the page, the server returns the code for the application. Once it’s loaded, the app takes over all of the logic around loading, presenting, and interacting with content.y

All of the logic for how the app work is stored and executed client-side. The app only communicates with the back-end (via Ajax) when it needs to retrieve new content or data.

This makes it much easier to build responsive, fast experiences, where the page doesn’t need to reload when there are updates to content, layouts and components.

Websites which have lots of moving, changing parts, which update as you interact with them are well-suited for being built as SPAs. In fact, many of the ‘app-like’ websites which you use in daily life – like Gmail, Facebook, Twitter and even PayPal – use SPAs in their main web services. You’ll notice that as you browse through their views, and interact with their tools, the page doesn’t ‘reload’.

Developers build most SPAs with open source JavaScript frameworks, like Angular, React, and Vue.

Whilst faster, sleeker websites can definitely be a good thing for users, they can introduce a whole range of SEO challenges.

SEO challenges

Historically, search engines and other systems (like social media crawlers) have struggled with SPAs. Because the initial response from the server doesn’t contain the page’s content (it only contains the JavaScript required to load the app), systems which don’t run the JavaScript application never see the content.

Even today, few search engines and social networks support JavaScript completely or effectively. That means that, unless you go to special lengths to expose your content, everything in your SPA may be ‘invisible’ to anything other than human users.

Even systems which do consume JavaScript do so at a cost. It’s vastly more expensive (in time, processing power, and cost) to download and run a JavaScript app, and to make sense of what it does, than it is to read simple HTML.

Conceptual challenges

One of the key concepts in SEO, and in how Google works, is that a webpage should be about a thing. A given page should have a clear topic, and a clear focus. That page resides on a URL, which identifies it uniquely, and which shouldn’t change or be removed.

This is how most conventional websites, and certainly most WordPress websites, work. You write pages about topics.

But for websites which behave more like apps, it isn’t always easy to answer the question, “what is a page?”. As parts of pages change and transition as the user interacts with them (with or without changing the URL), it becomes difficult to know which state of each ‘page’ is about which topic.

Even if search engines can successfully parse and ‘explore’ a JavaScript app, it can be difficult for them to associate topics and relevance to a specific page or URL. These concepts don’t fit neatly with the nature of the platform.

Performance challenges

There are performance problems, too. Because your browser needs to download a (potentially large, complex) application before it can construct the webpage, that can take some time.

Even if the app is split into smaller pieces, and only the necessary ‘route’ is downloaded for the requested page, that can still take much longer to load than a conventional ‘static’ website might (though clever techniques like using WPGraphQL can make passing back-end data to the front-end much simpler and faster; and even more-so when you integrate it with Yoast SEO via this plugin).

Not only can the initial payload be slow, but it can also be the case that the more a page behaves like an app (and less like a website), the slower it might become. That’s because web browsers like Chrome make smart decisions on things like resource prioritisation, lazy-loading, and processing. If the HTML is bottlenecked or obfuscated, as is often the case with complex JavaScript apps, these processes might struggle or fail.

Rendering challenges

When Google (and others) can’t reliably access and assess your content, that can lead to serious SEO problems.

To avoid some these problems, you can use different approaches to rendering your content, and ensure that search engines and social networks can see and digest your content.

Pre-rendering

One possible solution to these problems is to “pre-render” pages. In this context, rendering is the process by which the app uses JavaScript to construct HTML when a page is requested. So, “pre-rendering” is when you do that before a page is requested.

If your back-end is smart, it can run a process which loads the website application, and stores the rendered HTML output. We can build up a cache for our pages, and serve that “pre-rendered” content users or search engines. Pre-rendering is often done on demand (when a page is first loaded). but more sophisticated approaches might run on a schedule, or when content is changed or updated.

But using pre-rendering alone doesn’t solve our problem. In fact,
it puts us back where we started. We have a server which, instead of responding to the browser with a JavaScript application, it responds by delivering pre-rendered HTML, CSS and JavaScript files. Many of the advantages we had from responding with an application – the fluid, dynamic content and experiences – are gone. We need a middle ground.

Hydration

One possible solution is to use a technique called hydration. In this approach, the back-end still responds with a pre-rendered, static version of the page. But it also responds with a version of the application code. Once this code loads, the static page (or, parts of it) is seamlessly transformed into the app.

This kind of approach can be extremely effective, but can also be more complex to configure and maintain. It’s also tricky to avoid performance pitfalls, where the app (re)loads content which was already downloaded in the initial response.

Dynamic rendering

To address these problems, Google makes special allowance for a concept they refer to as ‘dynamic rendering’. In this approach, a website uses pre-rendering, but only serves pre-rendered responses to Google (and other search engines / bots, etc). Normal users get the full JavaScript app experience. Search engines and crawlers get a pre-rendered, static HTML version of the site which is easier for them to understand.

Whilst this could technically be considered a form of cloaking by Google’s definitions, they go to lengths in their documentation to explain that dynamic rendering is not the same as cloaking, and is their preferred solution.

Dynamic rendering isn’t a perfect solution, though. It still means that you’re essentially managing two versions of the website (which must be ‘kept in sync’). You’re also micromanaging which types of users and systems see which version. That’s potentially a lot of micromanagement and a lot of things which could go wrong.

Isomorphic / Universal JavaScript

In an ideal world, you wouldn’t need to maintain two different versions of your website, or to rely on complex hydration techniques. You’d just have one set of code and logic, and the site would deliver server-side generated content, but also allow app-like interactions.

We can achieve all of these goals by using ‘isomorphic’ or ‘universal’ JavaScript (commonly referred to as ‘server-side rendering’). In this approach, our back-end and front-end can both run the same JavaScript code. That means that the back-end can respond with the rendered HTML and then hydrate it, but without needing to define and manage additional code to define how that should work.

This approach provides all of the advantages of a JavaScript app and can bypass all of the drawbacks of pre-rendering, hydration, or dynamic rendering. It can be complex to configure, though, and is only supported with certain JavaScript frameworks and approaches.

Headless WordPress

Surprisingly, relatively few WordPress websites today are deployed as Single Page Applications. We can speculate that this may be due to a combination of factors – including differing developer skillsets (WordPress is heavily biased towards PHP & MySQL), and a lack of broad theme/plugin support.

That’s not to say that it’s not possible, however. In fact, WordPress’ REST API (and/or solutions like WP GraphQL) can be used to provide the data and back-end connectivity for a SPA. This Medium article by Brijesh Dhanani shows how easy it is to create a simple SPA using WordPress as the back-end and React for the front-end.

We call this kind of approach ‘headless’. Headless sites remove all of the parts of (the front end of) the CMS – in our case WordPress – which a normal user would see and interact with. The front-end is entirely powered by a JavaScript application, and content (and data) is fetched ‘on-demand’ from the back-end. Most SPAs use headless content management systems, and might even draw from multiple back-ends and systems, using JavaScript to communicate with APIs.

If you’d like to learn more about headless WordPress sites, we recommend looking into Frontity; a leading framework with excellent documentation.

Challenges with headless SEO

One of the drawbacks to headless sites is that it’s hard for the front end to know how the ‘SEO stuff’ on the page should work. On even the simplest of sites, complex logic is often required to determine how a page should behave from an SEO perspective. Crawling controls, indexing directives, meta tags and structured data must all interact correctly, without error.

For simple posts and pages, this logic isn’t overly complex. But when you need to start considering post archives, pagination, indexing controls and other edge-cases, it becomes much more complex. When there’s no ‘back end’, the SPA must define and manage all of these rules from scratch. Even the logic for ‘simple’ components like canonical URL tags is surprisingly complex when defined and built from scratch. Many headless sites and SPAs fall afoul of this complexity and fail to achieve basic SEO standards.

This has meant that, despite being on the cutting-edge of performance and user experience, many headless sites have performed poorly when it comes to SEO.

Headless SEO in WordPress, with Yoast SEO

At Yoast, we’re convinced that JavaScript websites are only going to become more popular, more powerful, and preferred by users. But the current standards and tools let these websites down.

That’s why, in Yoast SEO 14.0 (in April 2020), we added a headless SEO API. Since then, headless WordPress sites running our plugin automatically get all of the appropriate <head> content, for whatever type of request/page, appended to their standard WordPress API requests. Developers who want even customization can request the <head> output for any URL by using our special REST API syntax. E.g., this URL returns all of the meta tags we output for our Yoast SEO plugin page.

You can learn more about the Yoast REST API in our developer docs.

Static HTML sites

One of the advantages of headless websites is that they can be completely de-coupled from the back-end. Because the front-end only communicates with the back-end via APIs to fetch content and data, it doesn’t matter where those individual pieces of the site live.

This has given rise to a trend for people to develop static websites.

Frameworks like Gatsby (who talk specifically about headless WordPress here) generate the HTML and content as part of a ‘build’ process (which runs initially, then whenever a post is updated or site setting is changed). That HTML can be hosted and accessed somewhere separate from any back-end components like databases and admin interfaces. That can provide significant security and performance benefits.

Increasingly, there’s an overlap between isomorphic SPAs, static HTML sites, and headless sites. These technologies are converging, and some of the most flexible, powerful and performant websites use a blend of each approach.

You can learn more about static WordPress sites from our friends
at Strattic.

JAMstack

We know this approach to web and app development as ‘JAMstack’. That stands for JavaScript, APIs, and Markup, which are the technologies which these sites use to construct pages and interfaces.

This is a distinctly different technical and conceptual approach to how a conventional WordPress website might operate; which is often on a LAMPstack (Linux, Apache, MySQL and PHP) – though a JAMstack website might rely on a WordPress site’s REST API for content and ‘back-end’ requests, which may use a LAMPstack.

You can learn more about the principles behind JAMstack here.

In conclusion

As we mentioned at the start, most WordPress website owners and managers shouldn’t need to worry about JavaScript SEO. As long as the theme and plugins that you’re using follow best practice, then you won’t need to take any additional action.

But theme and plugin developers should be aware that the web is changing, and that there’s a strong trend for next-generation web experiences to be powered by JavaScript, and by Single Page Apps.

JavaScript websites are going to become more commonplace. Parts of WordPress (particularly themes, and ‘the front-end’) will increasingly use, and be powered by, JavaScript. If we’re to be successful in optimizing and managing these sites, there’s a lot to learn.

Building and managing any flavour of headless, static WordPress SPA from scratch is a huge amount of work, and getting it wrong can be disastrous for SEO. Thankfully, there are a range of popular frameworks and tools which make it much easier, and we have some suggestions on what best practice looks like.

What does ‘best practice’ look like?

For now, best practice for JavaScript SEO in WordPress is to make sites which look, feel and behave as much like a ‘traditional’ sites as possible.

That means that the initial request to the server should return the complete, rendered source code. The initial page should be functionally identical, whether JavaScript is used or not. Once the page is loaded, JavaScript can be used to enhance and add behaviour or functionality; but navigating the site, and accessing content shouldn’t rely on this.

This can be achieved via any of the rendering solutions we’d identified, but is most reliable when using an isomorphic approach.

Of the many tools, libraries and frameworks available, we recommend using more mature and fully-featured support for SEO and rendering management. In practice, depending on your technical environment, that tends to mean React (or its lightweight sibling, Preact), Next, Nuxt (which uses Vue.js), or Angular (using Angular Universal).

Further reading

JavaScript SEO is a huge, complex, and technical topic – and exploring its relationship with WordPress adds even more moving parts.

We hope that this guide is a helpful starting point. If you’d like to learn more, we recommend these excellent guides and articles:

• Google’s guide to JavaScript SEO basics
• Tomek Rudzki‘s “Ultimate Guide to JavaScript SEO”, on the Onely blog
• Patrick Stox‘s “JavaScript SEO: What You Need to Know” guide, on the Ahrefs blog
• Alexis Sanders “Making Your Bot Experience As Good As Your User Experience” guide, on the Moz blog

Let us know if you have any questions, or if we’ve missed anything!

The post A primer on JavaScript SEO for WordPress appeared first on Yoast.