Disallow Oembed Media Links

As of  Drupal 9.5.0 stable and Varbase 9.0.10​
The Drupal core issue #3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO was committed. The old setup could be changed to follow the new Drupal core standard robots.txt 
​Dropped the use of the varbase-robots-additions.txt as its content were committed to drupal-scaffold asset templates in Drupal 9.5.0 #158​
Remove the following from the root composer.json file for the project.
        "[web-root]/robots.txt": {
          "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
        }
Old Information About The Issue
Search engines are indexing the media/oembed links in websites. And the search results for the Oembed remote media can be viewed in search results. Then a landing page will open with only the content of the remote media in it.
Current Status In Every Project 
Developers add the following Oembed media link in the robots.txt to be disallowed
# Oembed media
Disallow: /media/oembed
Disallow: /en/media/oembed
Disallow: /fr/media/oembed
Disallow: /es/media/oembed
Disallow: /ar/media/oembed
​
This is a repetitive issue that the support team faces when starting the handover process to push a site to production before launching a project.
Drupal core robots.txt file does not have the Disallow: /media/oembed
and only following with http://www.robotstxt.org/robotstxt.html​
Waiting on the Drupal core issue #3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO to be committed.
Adding Disallow Media Oembed in Old Projects
As of Varbase 9.0.6 and later do not need to do any work on this part.
If the project was built from Varbase 9.0.5 and older versions need to do the following steps:
Add line “Disallow: /media/oembed“ to be configured from Varbase to be always included in robots.txt
Regarding the other languages will be dealt with by the delivery team to add the variation according to the available languages on the website
Add the following in src/assets/varbase-robots-additions.txt file
# Varbase Robots additions to Robots.txt
# --------------------------------------
# Oembed media
Disallow: /media/oembed
Disallow: /*/media/oembed
Add the following append for for the file-mapping in Varbase Project template. To make sure it will be added, even if the robots.txt was changed or updated.
      "file-mapping": {
        "[web-root]/robots.txt": {
          "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
        }
      }
Projects may change the "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt" to their custom robots-additions.txt file.
Updated the default Varbase Project with the Include Disallow Oembed media link in the robots.txt for Varbase Project template #136 issue.
Actions After the Fix in Drupal 9.5.0
The Drupal core issue #3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO was committed. The old setup could be changed to follow the new Drupal core standard robots.txt 
​Dropped the use of the varbase-robots-additions.txt as its content were committed to drupal-scaffold asset templates in Drupal 9.5.0 #158​
Remove the following from the root composer.json file.
        "[web-root]/robots.txt": {
          "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
        }
Configuring SEO Features
Configuring Security Features
Last modified 4d ago