Skip to main content

Prevent Google from indexing Angular components

The problem

Our web page is multilanguage, server-side render Angular application, and creating a JavaScript file that will dynamically change content (like SPA does) will be a hard thing to do. Some of the problems:

  • We need to copy existing text from JSON files that we use for translation. And we need to keep in mind to update both places when changing the content

The solution

We need a solution for hiding content or whole components wherever we want in Angular Web application. This way we could keep using our Angular components and Angular features like before and hide contents from Google bots.

TLTR;

  • Create folder for JavaScript files

Create folder for JavaScript files

Create folder hidefrombot in /src where we will add our dummy JavaScript file that we will use to determine if the agent is a robot or not. You can add a hidefrombot folder somewhere else, but you will need to adjust angular-cli.json different also.

Hide folder from Google bot

To disallow a Google bot to see out the script we need to hide the whole folder. We can do that by adding restriction in robot.txt file:

Disallow: /hidefrombot

If we don’t have that file, create it.

Create dummy isRobot.js file

This dummy script will be used just to see if the browser agent can access it. If the browser agent can’t execute isRobot.js then the agent is a robot. This file can be empty, or add some comment, or whatever. We do not care what is in a file, we just want to know can it be executed or not.

Add folder in angular-cli.json

To make sure that our isRobot.js is present after the application is built and deployed we need to add it like an asset. We can do that by adjusting the angular-cli.json file. There we need to add our folder in the asset part. Something like this:

{
  "project": {
    "name": "hide.from.google"
  },
  "apps": [{
    "root": "src",
    "outDir": "dist",
    "assets": [
      "assets",
      "favicon.ico",
      "robots.txt",
      {
        "glob": "*.xml",
        "input": "config/sitemaps",
        "output": "./"
      },
      "denyrobots"
    ]
  }]
}


If you created your folder somewhere else, you need to adjust your angular-cli.json file.
isRobot method and how to use it in component
We should create a service for this method or add it to our existing service. The best way is to do a one-time check-in our app, and use that information through our whole app. That way, we will try to get isRobot.js file only once and our app will stay optimized.
Assume that we have a service called SeoOptimisationService in seo-optimisation.service.ts file. Create a new method checkForCrawlingRobot and try to execute our script. Our service should also contain a flag variable that will show if the agent is Google bot or not.
isNotCrawlingRobot: boolean = false;isRobot () { const contentScriptElement = this.document.createElement(‘script’); contentScriptElement.src = ‘/ hidefrombot / isRobot.js’; contentScriptElement.async = true; contentScriptElement.onload = () => {  this.isNotCrawlingRobot = true; }; this.document.head.appendChild(contentScriptElement);}
This method will add isRobot.js in our head and try to execute it. If this happens onload event will fire and there we can change our variable.
To keep our app optimized we should call this method in our app.compoment.ts file in ngAfterViewInit() life circle. This way, calling this script won’t block our app to render and we won’t get negative points from Google. Calling our service:
// check if browser agent is a crawling robotthis.seoOptimisationService.isRobot();
The last thing we need to do is to use this service in our component. Simple as that.
Just add seoOptimisationService to our component and use isNotCrawlingRobotflag in our template.
<div *ngIf=”seoOptimisationService.isNotCrawlingRobot” class=”info-content-box”><p> Our customer service is open, the store will be open again from the end of April due to the coronavirus. </p></div>
You may wonder: what if the component is rendered before we get onload event, because we do not know when will get a response from that call. Well, we do not have to worry. When variable isNotCrawlingRobot is changed in service, it will be changed also in our component and everything will be fine.
Now, we can hide whatever content or component in our Angular application using only one service! Nice, right?




Comments

Post a Comment

Popular posts from this blog

4 Ways to Communicate Across Browser Tabs in Realtime

1. Local Storage Events You might have already used LocalStorage, which is accessible across Tabs within the same application origin. But do you know that it also supports events? You can use this feature to communicate across Browser Tabs, where other Tabs will receive the event once the storage is updated. For example, let’s say in one Tab, we execute the following JavaScript code. window.localStorage.setItem("loggedIn", "true"); The other Tabs which listen to the event will receive it, as shown below. window.addEventListener('storage', (event) => { if (event.storageArea != localStorage) return; if (event.key === 'loggedIn') { // Do something with event.newValue } }); 2. Broadcast Channel API The Broadcast Channel API allows communication between Tabs, Windows, Frames, Iframes, and  Web Workers . One Tab can create and post to a channel as follows. const channel = new BroadcastChannel('app-data'); channel.postMessage(data); And oth...
Post a Comment
Read more

Certbot SSL configuration in ubuntu

  Introduction Let’s Encrypt is a Certificate Authority (CA) that provides an easy way to obtain and install free  TLS/SSL certificates , thereby enabling encrypted HTTPS on web servers. It simplifies the process by providing a software client, Certbot, that attempts to automate most (if not all) of the required steps. Currently, the entire process of obtaining and installing a certificate is fully automated on both Apache and Nginx. In this tutorial, you will use Certbot to obtain a free SSL certificate for Apache on Ubuntu 18.04 and set up your certificate to renew automatically. This tutorial will use a separate Apache virtual host file instead of the default configuration file.  We recommend  creating new Apache virtual host files for each domain because it helps to avoid common mistakes and maintains the default files as a fallback configuration. Prerequisites To follow this tutorial, you will need: One Ubuntu 18.04 server set up by following this  initial ...
Post a Comment
Read more

Working with Node.js streams

  Introduction Streams are one of the major features that most Node.js applications rely on, especially when handling HTTP requests, reading/writing files, and making socket communications. Streams are very predictable since we can always expect data, error, and end events when using streams. This article will teach Node developers how to use streams to efficiently handle large amounts of data. This is a typical real-world challenge faced by Node developers when they have to deal with a large data source, and it may not be feasible to process this data all at once. This article will cover the following topics: Types of streams When to adopt Node.js streams Batching Composing streams in Node.js Transforming data with transform streams Piping streams Error handling Node.js streams Types of streams The following are four main types of streams in Node.js: Readable streams: The readable stream is responsible for reading data from a source file Writable streams: The writable stream is re...
Post a Comment
Read more