Disallow Oembed Media Links
Last updated
Last updated
As of stable and
Remove the following from the root composer.json
file for the project.
Search engines are indexing the media/oembed
links in websites. And the search results for the Oembed remote media can be viewed in search results. Then a landing page will open with only the content of the remote media in it.
Current Status In Every ProjectDevelopers add the following Oembed media link in the
robots.txt
to be disallowed
This is a repetitive issue that the support team faces when starting the handover process to push a site to production before launching a project.
As of Varbase 9.0.6 and later do not need to do any work on this part.
If the project was built from Varbase 9.0.5 and older versions need to do the following steps:
Add line “
Disallow: /media/oembed
“ to be configured from Varbase to be always included inrobots.txt
Regarding the other languages will be dealt with by the delivery team to add the variation according to the available languages on the website
Add the following in src/assets/varbase-robots-additions.txt
file
Projects may change the "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
to their custom robots-additions.txt file.
Remove the following from the root composer.json
file.
Drupal core file does not have the Disallow: /media/oembed
and only following with
Waiting on the Drupal core issue to be committed.
Add the following append
for for the file-mapping
in template. To make sure it will be added, even if the robots.txt was changed or updated.
Updated the default Varbase Project with the issue.
The Drupal core issue was committed. The old setup could be changed to follow the new Drupal core standard robots.txt