RobotsTxtBundle

This bundle provides robots.txt generator for Symfony2

Installation

$ composer require evheniy/robots-txt-bundle "1.*"

Or add to composer.json

"evheniy/robots-txt-bundle": "1.*"

AppKernel:

public function registerBundles()
    {
        $bundles = array(
            ...
            new Evheniy\RobotsTxtBundle\RobotsTxtBundle(),
            ...
        );
        ...
...

config.yml:

robots_txt:
  - 'User-agent: *'

Or with sitemap:

robots_txt:
  - 'User-agent: *'
  - 'Sitemap: http://test.com/sitemap.xml'

The last step

app/console robots.txt:dump

Using different environments

The main idea is to use robots.txt for different environments, for example: for dev hiding project from indexing robots like Google

config_prod.yml:

robots_txt:
  - 'User-agent: *'
  - 'Sitemap: http://test.com/sitemap.xml'

config_dev.yml:

robots_txt:
  - 'User-agent: *'
  - 'Disallow: /'

And run it for prod:

app/console robots.txt:dump --env=prod

And run it for dev:

app/console robots.txt:dump --env=dev

Or just

app/console robots.txt:dump

License

This bundle is under the MIT license.

Документация на русском языке

Demo

Build a robots.txt