Disallow Oembed Media Links
Last updated
9.0.x
Last updated
As of stable and
Remove the following from the root composer.json file for the project.
Search engines are indexing the media/oembed links in websites. And the search results for the Oembed remote media can be viewed in search results. Then a landing page will open with only the content of the remote media in it.
Current Status In Every ProjectDevelopers add the following Oembed media link in the
robots.txtto be disallowed
This is a repetitive issue that the support team faces when starting the handover process to push a site to production before launching a project.
As of Varbase 9.0.6 and later do not need to do any work on this part.
If the project was built from Varbase 9.0.5 and older versions need to do the following steps:
Add line “
Disallow: /media/oembed“ to be configured from Varbase to be always included inrobots.txtRegarding the other languages will be dealt with by the delivery team to add the variation according to the available languages on the website
Add the following in src/assets/varbase-robots-additions.txt file
Projects may change the "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt" to their custom robots-additions.txt file.
Remove the following from the root composer.json file.
Drupal core file does not have the Disallow: /media/oembed
and only following with
Waiting on the Drupal core issue to be committed.
Add the following append for for the file-mapping in template. To make sure it will be added, even if the robots.txt was changed or updated.
Updated the default Varbase Project with the issue.
The Drupal core issue was committed. The old setup could be changed to follow the new Drupal core standard robots.txt