Getting Started

Minimal Setup for First Use

Check the PHP constraint first. 2.0.0 aligns the PHP version configuration with the other atoolo projects, so the composer.json platform requirement may have moved (upwards). Read composer.json in the released tag before upgrading a project on an older PHP.
Install 2.0
```
composer require atoolo/crawler-teaser-indexer:^2.0
```
2.0 is a major: the indexing side is no longer self-contained — it now builds on atoolo/index-bundle, which arrives as a dependency. Confirm it resolved:
```
composer show atoolo/index-bundle
composer why atoolo/index-bundle
```
Register both bundles in config/bundles.php (Symfony still requires explicit registration of each bundle, dependency or not):
```
Atoolo\Index\AtooloIndexBundle::class => ['all' => true],
Atoolo\Crawler\AtooloCrawlerTeaserIndexerBundle::class => ['all' => true],
```
If you previously hand-wired index services, Solr clients, or index commands yourself, delete those definitions — that is what the index-bundle now provides, and duplicates will fight each other in the container.
Re-verify the scheduling configuration. 2.0 refactored the scheduling command. The atoolo.crawler.schedule parameter you used in 1.x may still exist, may have been renamed, or may now be consumed by a different command. Inspect rather than assume:
```
php bin/console list atoolo
php bin/console debug:config atoolo_crawler_teaser_indexer
```
Then keep (or adapt) your cron list:
```
# config/packages/atoolo_crawler_master.yaml
parameters:
  atoolo.crawler.schedule:
    - "0 3 * * *"
  atoolo.crawler.retry_status_codes:
    - 408
    - 500
```

Define the crawler configuration in base_dir/indexer/atooloTeaserCrawler.php:

return [
  "name" => "My External Teaser Indexer",
  "data" => [
    "sp_crawling_sites" => [
      [
        "sp_id" => "my_site",
        "sp_url" => "https://example.com",
        "sp_title_css" => ["h1", ".title"],
        "sp_start_urls" => [["sp_url" => "https://example.com/blog", "sp_extraction_depth" => 1]],
      ]
    ]
  ]
];

Run it
```
php bin/console crawler:index -vvv
```

Where to Look First

vendor/atoolo/index-bundle/ — the index command, index definitions, and Solr wiring now live here. This is the first place to look when something that used to be "in the crawler" is now missing.
vendor/atoolo/crawler-teaser-indexer/config/ — the bundle's own service and parameter definitions; confirms whether your 1.x parameters survived the refactor.
bin/console list atoolo — cheapest way to see what the refactored scheduling command actually exposes.

First Use Case: Indexing a Blog

Point sp_start_urls at the blog index page, target post titles with sp_title_css (e.g. ".post-title"), run the crawler, and confirm the teasers land in the Solr core defined by the index-bundle. Do this on a throwaway index first — 2.0 changes which component owns the write path, so a misconfigured index definition shows up as "crawl succeeded, nothing indexed".

Implementation Patterns

Laravel integration (this is a Symfony bundle)

There is no first-class Laravel provider. Three viable shapes, in order of preference:

Separate Symfony kernel/process (recommended). Keep the crawler as its own deployable (or a micro-kernel in the same repo) and invoke it from Laravel.

// app/Console/Commands/RunCrawler.php
use Illuminate\Support\Facades\Process;

class RunCrawler extends Command
{
    protected $signature = 'crawler:index {--env=prod}';

    public function handle(): int
    {
        $result = Process::timeout(3600)
            ->path(config('crawler.symfony_path'))
            ->run("php bin/console crawler:index --env={$this->option('env')} -vvv");

        $this->output->write($result->output());
        return $result->exitCode();
    }
}

Then schedule it with Laravel's scheduler:

Schedule::command('crawler:index')->dailyAt('03:00')->withoutOverlapping();

This keeps the 2.0 atoolo/index-bundle container isolated from your Laravel container, which is what you want.

Bridge the bundle into Laravel via a Symfony-bundle loader (e.g. spatie/laravel-symfony-bundle). Workable, but the index-bundle's DI, config loading, and console wiring are Symfony-shaped; expect to debug container compilation rather than crawler behaviour. Only take this if you need the crawler to share models/services with the Laravel app.
Console command proxy — instantiate the Symfony console application inside an Artisan command and register IndexCommand manually. Note that after 2.0 the index command comes from atoolo/index-bundle, so you must register that bundle's command class, not assume it lives in the crawler namespace. Verify the FQCN against vendor/ before copying old snippets.

Workflow Integration

Scheduled execution: prefer your host scheduler (Laravel's Schedule, cron, systemd timer) over relying solely on the in-bundle scheduler parameter — the 2.0 "scheduling command" refactor means the internal entry point may change again; an external crawler:index invocation is the stable contract.
Dynamic configuration: keep atooloTeaserCrawler.php returning a plain array and generate it (or load it) per site:
```
$config = include __DIR__.'/indexer/atooloTeaserCrawler.php';
$crawler = new Crawler($config['data']);
```
Solr indexing: the write path is now the index-bundle's. Define the index/target there, and treat the crawler config as producer-side only. Run with --env=prod for realistic container caching.
Retries: keep transient-failure handling in the crawler config rather than in your wrapper:
```
"sp_max_retry" => 3,
"sp_delay_ms" => 1000,
```

Common Patterns

CSS selector fallbacks: always provide alternatives — ["h1", ".title", "#main-title"].
Scope control: sp_allow_prefixes / sp_deny_endings to keep the crawl on-blog and off binaries.
Content scoring: sp_content_scoring_active + sp_content_scoring_min_score to drop stub pages.
De-duplication: sp_strip_query_params_active to collapse ?page=2 variants into one teaser.
Queue the wrapper, not the crawl: dispatch your Artisan command as a job with a generous $timeout and a unique lock so a slow crawl doesn't overlap the next scheduled run.

Gotchas and Tips

Upgrade Pitfalls (1.x → 2.0)

Duplicate index services. The two use atoolo/index-bundle commits mean anything you wired by hand in 1.x (Solr client, index command, document factory) is now also provided by a bundle. Duplicate definitions surface as container-compile errors or, worse, silently wrong service ordering. Grep your config/services.yaml for Atoolo\Index before upgrading.
Scheduling command refactor. The scheduler entry point was reworked. A deploy that only runs cache:clear may keep serving a stale compiled container that still expects the old command signature. Rebuild the container (and warm the prod cache) as part of the release:
```
php bin/console cache:clear --env=prod
php bin/console cache:warmup --env=prod
```
PHP version alignment. Platform requirements moved to match the other atoolo projects. If CI pins an older PHP image, composer will resolve to 1.x or fail outright — check the matrix before assuming the upgrade "just didn't happen".
Tooling churn is dev-only, but it affects forks. The phpcs ruleset was dropped in favour of php-cs-fixer, phplint moved from phive to composer, and phive tools were updated. If you maintain a fork or vendor patch, your phpcs.xml pipeline will silently stop enforcing the atoolo ruleset — port the config to .php-cs-fixer.php.

Behavioural Pitfalls

Missing required selectors — URLs are skipped when mandatory fields (title, intro text) have no matching selector. Always ship at least one working selector per mandatory field.
Solr type mismatch — a datetime field receiving a bare date fails indexing; set "sp_datetime_only_date" => true to normalise to YYYY-MM-DD 00:00:00.
Rate limiting — keep sp_parallel_requests modest (start at 3); 429s from the target site look like crawler failures, not politeness failures.
Config caching — parameter changes in atoolo_crawler_master.yaml need a cache clear before they take effect.

Debugging

php bin/console crawler:index -vvv is still the fastest feedback loop.
php bin/console debug:config / debug:container --tag=kernel.event_listener to confirm which listeners the index-bundle integration actually registered — 2.0 may have moved the teaser event out of the crawler bundle.
Dry-run selectors in browser DevTools before touching YAML; a wrong selector looks identical to a network failure at -vvv.
Check Solr logs first when the crawl logs look clean but nothing is searchable.

Extension Points

Custom extractors: implement ExtractorInterface for non-standard HTML structures.
Post-processing: hook the teaser-extracted event and mutate documents before indexing. Re-verify the event name and priority after the index-bundle migration:
```
Atoolo\Crawler\EventListener\TeaserPostProcessor:
  tags:
    - { name: kernel.event_listener, event: crawler.teaser.extracted, method: onTeaserExtracted }
```
Dynamic config loading: fetch per-site config from a database or API instead of static PHP files when sites are tenant-managed.
Index definitions: extend or override the index-bundle's index config rather than re-implementing the write path.

Configuration Quirks

OpenGraph fallback: if sp_title_opengraph yields nothing, the crawler falls back to sp_title_css.
Query params: sp_strip_query_params_active removes tracking/pagination params to avoid duplicate teasers.
User-Agent: always set a descriptive sp_user_agent ("MyBot/1.0 ([email protected])") — default agents get blocked, and the resulting 403s are retried per atoolo.crawler.retry_status_codes rather than reported clearly.
Undocumented option drift: several sp_* keys are convention-driven rather than schema-validated. After a major release, verify the option names actually present in the shipped config before trusting a 1.x snippet.

Crawler Teaser Indexer Laravel Package