LogoLogo
10.0.x
10.0.x
  • Overview
    • About Varbase
    • Roadmap
    • Release Cycle and Supported Versions
    • How to Get Support
  • Developers
    • Getting Started
    • Installing Varbase
      • Requirements
      • Installing Varbase locally with DDEV
      • Installing Varbase locally with Lando
    • Understanding Varbase
      • Basic Concepts
      • Core Components
        • Varbase Core
          • Varbase Admin
          • Varbase Page
          • Varbase Security
          • Varbase Webform
          • Varbase Default Content
          • Varbase Tour
        • Varbase Media
        • Varbase Editor
        • Varbase Email
        • Varbase SEO
        • Varbase Workflow
        • Varbase Dashboards
        • Varbase Layout Builder
          • Varbase Landing page (Layout Builder)
          • Varbase Heading Block
          • Varbase Rich Text Block
          • Varbase HTML Code Block
          • Varbase Media (Image/Video) Block
          • Varbase Gallery Block
        • Varbase Components
        • Vartheme Claro
        • Vartheme BS5
      • Optional Components
        • Varbase AI
          • Varbase AI Default recipe
          • Varbase AI Editor Assistant recipe
          • Varbase AI Image Alt recipe
          • Varbase AI Taxonomy Tagging recipe
          • Varbase AI Agents recipe
        • Varbase API
        • Varbase Internationalization
        • Varbase Hero Slider
        • Varbase Carousels
        • Varbase Blog
        • Varbase Search
        • Varbase Social Single Sign-On
        • Varbase Media Twitter
        • Varbase Media Instagram
        • Varbase Content Planner
        • Varbase Bootstrap Paragraphs
          • Varbase Bootstrap Paragraphs Text and Image
        • Varbase Landing Page (Paragraphs)
      • Additional Components
        • Varbase Media Header
        • Varbase FAQs
        • Varbase Reports
        • Varbase Commerce
        • Varbase YouTube Import
      • Development Components
        • Varbase Development
        • Reroute Email
        • Varbase Updates Helper
      • Deprecated Components
        • Varbase Media Hero Slider
        • Varbase Style Guide
        • Vartheme BS4
        • Varbase Total Control Dashboard
    • Configuring a Varbase Site
      • Navigation
        • Defining Active Menu Trail Using Menu Position
      • Using View Modes Inventory
      • Content Moderation and Workflows
      • URL Aliases in Varbase/Drupal
        • Configuring URL Aliases
        • Managing URL Aliases
      • Managing URL Patterns for Multilingual Websites
      • Managing Date and Time Formats to Display Across the Site
      • Using Varbase Layout Builder
        • Creating a Landing Page With Varbase Layout Builder
        • Activating Varbase Layout Builder for Content Types
      • Configuring Pathologic When Going Live
      • Configuring SEO Features
        • Disallow Oembed Media Links
      • Configuring Security Features
        • Spam Protection
        • Password Policies
        • Other Security Kits
        • Flood Contorl
        • Enable HTTP Password Authentication
      • Configuring JSON:API Features
      • Configure Varbase Media Hero Slider
      • Configuring Varbase Media Header
      • Configuring Varbase Mailer Settings
        • Configure Symfony Mailer
      • Configure Level of Logging and Errors
      • Entity Definition Update Manager Class
      • Module Installer Factory Class
      • Configure Ckeditor 5 Media Embed
    • Theme Development with Varbase
      • Understanding The Vartheme Starterkit Theme
      • Creating Your Own Theme
      • Integration of Varbase with Storybook 1.0 - CL Server
      • Integration of Varbase with Storybook 2.0
      • Customize a Varbase Single Directory Components (SDC) In a Custom Theme
      • Local Tasks and Moderation Sidebar
      • Add Preloaded Fonts in Vartheme
      • Command to Install Needed Theming Tools
      • Manually Install Needed Theming Tools
    • Launching a Varbase Site to Production
    • Updating a Varbase Site
      • Understanding Varbase Updater Package
      • Handling Patches When Updating
      • Handling Configuration Updates
      • Updating Varbase to work with Composer 2.0
      • Updating Drush to the Latest Stable Version
      • Version Update Guides
        • Updating Varbase ~9.0 to Drupal 10
        • Updating from Varbase 8.x to 9.x
    • Varbase Patches
    • Frequently Asked Questions (FAQs)
    • Extending Varbase
      • Overriding Varbase
      • Creating Your Own Feature
      • Install Needed YARN and Webpack Tools
      • Compiling Provided Component Styles
      • Check Standards/Practice Coding And Linting
  • Content Designers
    • Accessing the Administration Area
      • Website Environments
    • Dashboard
      • Reports
    • Content Management
      • Create Content
        • Rich-Text Editor “WYSIWYG”
      • Modify Content
      • Delete Content
      • Preview Content
      • Bulk Edit Content
      • Publish Content
      • Authoring Information
    • Content Structure
      • EntityQueues
        • Add a New Term
      • Menu Management
        • Add Menu Items
        • Update Menu Items
        • Adding a Page to the Menu
      • Taxonomies
        • List Terms
        • Add Terms
        • Reorder and Edit Terms
    • Content Translation
      • Translating Content
      • Language Switcher
      • Deleting a Translation
      • Editing a Translation
    • Content Workflow
    • Layout Builder
      • Section
        • Section Styles
      • Block Management
        • Block Styles
      • Reordering Blocks
    • Webforms
      • Create New Webform View
      • Setting up Emails Handler Conditions
      • Setting up Confirmations Message Settings Types
    • Media Management
      • Accessing the Media Library
      • Uploading Media Files
        • Image
        • Remote Video
        • Video
      • Applying Metadata and Descriptions
      • Resizing Images
    • User Management
      • Login & Registration
      • Add User
      • Edit User
      • Delete User
      • Simulating Another User
      • Default Roles and Site Personas
        • Edit Roles' Permissions
        • Managing Roles
    • URL Management
      • URL Aliases
      • Generating URL Alias
      • URL Redirects
      • Domain Redirects
    • SEO Management
      • XML Sitemap
      • Meta Tags in Varbase
      • Disallow Indexing for Content Items
    • Site Configuration
      • System
        • Clearing Varbase Cache
        • Google Analytics
        • Mail Templates
      • User Interface
        • Coffee
        • External Links
        • Gin Moderation Sidebar
        • Layout Builder Modal
        • Moderation Sidebar
        • Shortcuts
        • Taxonomy Manager
        • Tours
        • CL Components
        • Antibot
        • Responsive Preview
        • Navigation Settings
      • Development
      • Varbase Settings
      • Web Services
        • OpenAPI
    • Additional Tips
      • Content Uploading Standards
      • Common Error Handling Tips
      • Glossary
  • Contributing
    • Contributing to Documentation
Powered by GitBook
On this page
  • Old Information About The Issue
  • Adding Disallow Media Oembed in Old Projects
  • Actions After the Fix in Drupal 9.5.0
  1. Developers
  2. Configuring a Varbase Site
  3. Configuring SEO Features

Disallow Oembed Media Links

PreviousConfiguring SEO FeaturesNextConfiguring Security Features

Last updated 1 year ago

As of stable and

The Drupal core issue was committed. The old setup could be changed to follow the new Drupal core standard robots.txt

Remove the following from the root composer.json file for the project.

        "[web-root]/robots.txt": {
          "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
        }

Old Information About The Issue

Search engines are indexing the media/oembed links in websites. And the search results for the Oembed remote media can be viewed in search results. Then a landing page will open with only the content of the remote media in it.

Current Status In Every Project

Developers add the following Oembed media link in the robots.txt to be disallowed

# Oembed media
Disallow: /media/oembed
Disallow: /en/media/oembed
Disallow: /fr/media/oembed
Disallow: /es/media/oembed
Disallow: /ar/media/oembed

This is a repetitive issue that the support team faces when starting the handover process to push a site to production before launching a project.

Adding Disallow Media Oembed in Old Projects

As of Varbase 9.0.6 and later do not need to do any work on this part.

If the project was built from Varbase 9.0.5 and older versions need to do the following steps:

Add line “Disallow: /media/oembed“ to be configured from Varbase to be always included in robots.txt Regarding the other languages will be dealt with by the delivery team to add the variation according to the available languages on the website

  • Add the following in src/assets/varbase-robots-additions.txt file

# Varbase Robots additions to Robots.txt
# --------------------------------------
# Oembed media
Disallow: /media/oembed
Disallow: /*/media/oembed
      "file-mapping": {
        "[web-root]/robots.txt": {
          "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
        }
      }

Projects may change the "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt" to their custom robots-additions.txt file.

Actions After the Fix in Drupal 9.5.0

Remove the following from the root composer.json file.

        "[web-root]/robots.txt": {
          "append": "docroot/profiles/varbase/src/assets/varbase-robots-additions.txt"
        }

Drupal core file does not have the Disallow: /media/oembed and only following with

Waiting on the Drupal core issue to be committed.

Add the following append for for the file-mapping in template. To make sure it will be added, even if the robots.txt was changed or updated.

Updated the default Varbase Project with the issue.

The Drupal core issue was committed. The old setup could be changed to follow the new Drupal core standard robots.txt

Drupal 9.5.0
Varbase 9.0.10
#3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO
Dropped the use of the varbase-robots-additions.txt as its content were committed to drupal-scaffold asset templates in Drupal 9.5.0 #158
robots.txt
http://www.robotstxt.org/robotstxt.html
#3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO
Varbase Project
Include Disallow Oembed media link in the robots.txt for Varbase Project template #136
#3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO
Dropped the use of the varbase-robots-additions.txt as its content were committed to drupal-scaffold asset templates in Drupal 9.5.0 #158