Disallow Oembed Media Links
The Drupal core issue #3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO was committed. The old setup could be changed to follow the new Drupal core standard
robots.txt
Remove the following from the root
composer.json
file for the project. "[web-root]/robots.txt": {
"append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
}
Search engines are indexing the
media/oembed
links in websites. And the search results for the Oembed remote media can be viewed in search results. Then a landing page will open with only the content of the remote media in it.
Current Status In Every ProjectDevelopers add the following Oembed media link in therobots.txt
to be disallowed# Oembed mediaDisallow: /media/oembedDisallow: /en/media/oembedDisallow: /fr/media/oembedDisallow: /es/media/oembedDisallow: /ar/media/oembed
This is a repetitive issue that the support team faces when starting the handover process to push a site to production before launching a project.
Drupal core robots.txt file does not have the
Disallow: /media/oembed
and only following with http://www.robotstxt.org/robotstxt.htmlWaiting on the Drupal core issue #3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO to be committed.
As of Varbase 9.0.6 and later do not need to do any work on this part.
If the project was built from Varbase 9.0.5 and older versions need to do the following steps:
Add line “Disallow: /media/oembed
“ to be configured from Varbase to be always included inrobots.txt
Regarding the other languages will be dealt with by the delivery team to add the variation according to the available languages on the website
- Add the following in
src/assets/varbase-robots-additions.txt
file
# Varbase Robots additions to Robots.txt
# --------------------------------------
# Oembed media
Disallow: /media/oembed
Disallow: /*/media/oembed
- Add the following
append
for for thefile-mapping
in Varbase Project template. To make sure it will be added, even if the robots.txt was changed or updated.
"file-mapping": {
"[web-root]/robots.txt": {
"append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
}
}
Projects may change the "append":
"docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
to their custom robots-additions.txt file.Updated the default Varbase Project with the Include Disallow Oembed media link in the robots.txt for Varbase Project template #136 issue.
The Drupal core issue #3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO was committed. The old setup could be changed to follow the new Drupal core standard
robots.txt
Remove the following from the root
composer.json
file. "[web-root]/robots.txt": {
"append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
}
Last modified 4d ago