Ezmigrationbundle Laravel Package

Fixed: the --admin-login option had been broken for migrate commands since version 6.3.0

• Fixed: exception thrown at end of migration if the migration steps include sql executing transaction commits

• Fixed: correctly abort a migration when it leaves a database transaction pending (nb: this can be detected only for transactions started using Doctrine, not for transactions started using sql begin statements)

• Improved: reporting of errors happening before/during/after migration execution, esp. anything related to transactions

• Improved: when generating migrations, try harder to reset the repository to the originally connected user in case of exceptions being thrown

BC notes (for developers extending the bundle):

• MigrationService::getFullExceptionMessage gained a 2nd parameter: $addLineNumber = false
• AfterMigrationExecutionException produces a different error message when passed 0 for the $step parameter
• service ez_migration_bundle.migration_service requires an added setConnection call in its definition

Fixed php warning in class PHPExecutor due to trait being used twice

• Fixed: php warning when generating Role migrations for roles with policy limitations

• Fixed: in rare circumstances (having two siteaccesses configured with the same repo and root node, but different languages), the TagMatcher could use the wrong language when matching by tag keyword

• BC change (for developers extending the bundle): class TagMatcher changed its constructor signature. the same applies to service ez_migration_bundle.tag_matcher

• New: migration step migration_definition/include. This allows one migration to basically include another, the same way it is possible to do that in php.

  It is useful for scenarios such as fe. creating a library of reusable migrations, which can be run multiple times with different target contents every time. This is often achieved by copy-pasting the same migration logic many times. As an alternative it is now possible to create a "library" migration, driven by references, and store it only once, in a separate folder, then create many "specific execution" migrations which set up values for the required references and include the library migration's definition.

  Please note that migrations which rely on external resources, such as in this case would be the included migration, go against the principle of migrations being immutable for ease of replay and analysis.

  Ex:

    -
        type: migration_definition
        mode: include
        file: a_path
  

• Improved: when executing migrations with the set-reference cli option, the injected references will be saved in the migration status

• BC change (for developers extending the bundle): method MigrateCommand::executeMigrationInProcess changed its signature

• BC change (for developers extending the bundle): Migrationservice methods executeMigration, executeMigrationInner and resumeMigration should now be called using a different signature. They do still work with the previous signature, but that usage is considered deprecated

• Fixed: when setting references to a ContentType sorting attributes, numeric values were used instead of their string representation

• Fixed: when generating contentType migrations, do export the default_always_available, default_sort_order and default_sort_field attributes

• Fixed: migrations created using kaliop:migration:generate would create yml which was not valid for import, for content fields of type eztags

• Improved: it is now possible to set references to the value of content fields which are recursive arrays (only 1-level arrays were supported previously)

• Improved: in step reference/set, when reference resolving for value is enabled, it will be done recursively if value is an array

• Improved: migration step mail/send learned how to deal with multiple attachment files, using an array for element attach

• Improved: added a cookbook recipe about adding tags to an eztags field

• BC change (for developers extending the bundle): class AbstractExecutor gained a few method and properties, and it lost method parseReferenceDefinition, which was moved to trait ReferenceSetterTrait

• New: when matching contents, it is now possible to filter based on empty fields, eg:

  -
      type: content
      mode: load
      match:
          and:
              - content_type_identifier: file
              - attribute: {'file': empty}
  

• New: command k:m:migration learned action --fail. It should be used sparingly, only to set manully to failed status migrations which for any reason got stuck in an incorrect status, f.e. those which are still listed as executing after the corresponding process is terminated

• Fixed: migrations creating/updating contents with an ezmatrix field would result in corrupted data. Also, trying to create a content/create migration for a content with an ezmatrix field would lead to a crash

• Fixed: migrations creating/updating contents with an ezmatrix field used to work with an undocumented yaml format, up to version 5.14.0. We now allow that format to be used as well, besides the preferred format - although such format most likely does not work with the ezsystems/ezplatform-matrix-fieldtype bundle and is to be considered deprecated (see issue #250 for details).

• Fixed: make error messages from subprocesses be echoed to the console when running k:m:migrate -p with eZP 2.0 and later

• Improved: bumped the version of phpunit used to run the tests from 4.x/5.x to 5.x/8.x

• Improved: updated documentation in README and in Cookbooks

• Changed: renamed master branch on Github to main

• BC change (for developers extending the bundle): const Kaliop\eZMigrationBundle\Command\MigrateCommand::VERBOSITY_CHILD has been transformed into static variable Kaliop\eZMigrationBundle\Command\MigrateCommand::$VERBOSITY_CHILD

• BC change (for developers extending the bundle): all \Exception generated by the bundle have been converted into Kaliop\eZMigrationBundle\API\Exception\MigrationBundleException or subclasses. The same applies for all previously existing Migration Bundle exception classes.

Fix: backport fix for issue #232: bad method signature for EmbeddedRegexpReferenceResolverTrait

• New: everywhere a reference was previously resolved, ie. using reference:myref or [reference:myref] syntax it is now possible to use eval:expression or [eval:expression].

  The syntax for "expression" is the one of the Symfony ExpressionLanguage component. See: https://symfony.com/doc/current/components/expression_language/syntax.html

  Ex: to take the value of an existing reference and add 1 to it: [eval: 1 + resolve('reference:myref')]

  Ex: to take the value of an existing reference and concatenate 'a' to it: [eval: resolve('reference:myref') ~ 'a']

  BC BREAK: note that this can be an issue if you have existing migrations which might have the text [eval: in their data. If this is a problem for your environment, you can fix it by overriding the definition of Symfony service ez_migration_bundle.reference_resolver.customreference.flexible and remove from its arguments the service [@ez_migration_bundle](https://github.com/ez_migration_bundle).reference_resolver.expression

• New: migration steps php/call_function and php/call_static_method, to ease one-off calling php code as part of a yaml migration. See the relevant DSL for details.

  Ex: it is possible to add an element to an array-valued reference, with an admittedly cumbersome syntax, given here as a self-contained example:

  -
      type: reference
      mode: set
      identifier: pippo
      value: [a, b]
  -
      type: php
      mode: call_function
      function: array_merge
      arguments: ['reference:pippo', ['c']]
      references:
          pluto: result
  

• New: multiple migration steps url_alias/... and url_wildcard/... are now available to manage urls aliases. Please read their documentation in UrlAliases.yml and UrlWildcards.yml for details

• New: migration steps loop/break and loop/continue

• New: migration step file/load_csv, allows to easily initialize references with long list of values

• New: for migration steps content/create and content/update, when content fields of type eZBinaryFile, eZImage or eZMedia are defined using array syntax (instead of a single string defining the file path), references are now resolved for each element of the array. Eg:

  -
      type: content
      mode: create
      content_type: an_image_type
      attributes:
          image_field:
              path: 'reference:a-reference-name'
              alternativeText: 'looking good'
  

• New: references are now resolved in the following migration step elements: file/load_csv/separator, file/load_csv/enclosure, file/load_csv/escape, file/save/overwrite, file/copy/overwrite, file/move/overwrite, http/call/method, http/call/client, migration/cancel/message, migration/fail/message, migration/sleep/seconds migration/suspend/message migration/suspend/sleep, process/run/timeout, process/run/working_directory, process/run/environment, process/run/fail_on_error,

• New: migration step migration/sleep now supports the if clause

• New: command migrate and mass_migrate can pass down to children processes custom php.ini settings, such as f.e. memory_limit and error_reporting. Useful to run migrations as subprocesses in hostile environments

• Improved: it is now possible to set references to the value of content fields which are of type array

• Improved: content/update and location/update steps will throw an exception if there is nothing to update in their definition. This might happen f.e. if there is a typo in the yaml, and was silently ignored beforehand

• Improved: we now strive to always save paths to migration definition files as relative (to the app's root directory). This should help when copying the eZ database between different environments, such as fe. Prod and QA, which reside in different root directories in their respective servers/VMs, and then running kaliop:migration:status.

  BC BREAK: the paths reported for migration definitions by commands status and info will now most often not be absolute paths, but relative paths instead

  BC BREAK: if you are running the migration commands from a directory which is not the application's root dir, and use the --path option with a relative path, be aware that the path will now resolve to the app's root dir instead of the current dir

• Fixed: when running kaliop:migration:status, migration definition files found in a different location than what is stored in the db were not being reported as such

• Fixed: when running kaliop:migration:status --path ..., the status of skipped or failed migrations might be reported incorrectly as not-executed, due to mixing up relative and absolute paths. NB: in order for this to work properly, please execute the migration found in file vendor/kaliop/ezmigrationbundle/MigrationVersions/20220101000200_FixExecutedMigrationsPaths.php

• Fixed: a warning generated by the mass_migrate command

Resolve references in urlalias creation steps

Changes compared to 6.0.0-rc3:

• new: migration steps loop/break and loop/continue

NB: we are now aiming to release 6.0.0 as next version, instead of 5.16. This because the amount of changes introduced is quite notable.

Changes compared to 5.16.0-rc2:

• added md5 to the eval: resolver
• do check location/update step definitions for having some attribute of the location to update, as we now do for content/update
• unbreak dumping of all defined references
• fix one warning in massmigrate command
• work on migrating the CI test suite from Travis to GitHub

• Improved: it is now possible to set references to the value of content fields which are of type array
• Fixed: made the resolver for eval: something compatible with Symfony 2.8 / eZPublishPlatform 5.4

For the rest of changes, see the release notes of 5.16.0-rc1

• New: command migrate and mass_migrate can pass down to children processes custom php.ini settings, such as f.e. memory_limit and error_reporting. Usefult to run migrations as subprocesses in hostile environments

• New: multiple migration steps url_alias and url_wildcard are now available to manage urls aliases. Please read their documentation in Resources/doc/DSL for details

• New: migration step file/load_csv, allows to easily initialize references long list of values

• New: everywhere a reference was previously resolved, ie. using reference:myref or [reference:myref] syntax it is now possible to use eval:expression or [eval:expression].

  Ex: to take the value of an existing reference and add 1 to it: [eval: 1 + resolve('reference:myref')]

  Ex: it is possible to add an element to an array-valued reference, with an admittedly cumbersome syntax, given here as a self-contained example:

  -
      type: reference
      mode: set
      identifier: pippo
      value: [a, b]
  -
      type: reference
      mode: set
      identifier: pippo
      value: "eval: array_merge(resolve('reference:pippo'), ['c'])"
      resolve_references: true
      overwrite: true
  

  The syntax for "expression" is the one of the Symfony ExpressionLanguage component. See: https://symfony.com/doc/current/components/expression_language/syntax.html

  Note that this can be a BC break if you have existing migrations which might have the text [eval: in their data. If this is a problem for your environment, you can fix it by overrideing the definition of Symfony service ez_migration_bundle.reference_resolver.customreference.flexible and remove from its arguments the service [@ez_migration_bundle](https://github.com/ez_migration_bundle).reference_resolver.expression

Fixed: creating ezmedia fields starting from an array definition did not take into account the path element

• New: it is now possible to dump all of a content's languages when generating content/create and content/update migrations. In order to do so, pass --lang=all on the command line

• Fixed: generating content/create and content/update migrations would fail with eZPlatform 1 and later for any contents with non-null ezbinaryfile and ezmedia fields

• Fixed: allow usage of shorthand notation when setting references in file migration steps

• Improved: reduced the amount of test infrastructure setup code by relying on an external tool: https://github.com/tanoconsulting/euts

• New: support for eZMatrix fieldType (issue #217).

• New: Content and Location matchers, used in load, update and delete steps for Content and Location can now match by QueryType (issue #239)

• New: migration step user/create can now assign roles to the newly created user (besides the roles automatically inherited from the user's groups) (issue #77)

• New: taught the kaliop:migration:status command to display full migration path by using the --show-path option (issue #152)

• Improved: when the kaliop:migration:status command is run with --path, it will now filter out according to the given paths not only the available migrations, but also the registered/executed/failed/suspended ones

• Improved: taught the test-execution command teststack.sh to generate code coverage reports, by running teststack.sh runtests -- --coverage-html=/some-dir. Note that it might take a long time to run

• Improved: allow to run unit tests on a PostgreSQL database instead of MySQL. At the moment this works correctly for testing against eZPublish Platform but not against eZPlatorm 1/2/3

• Deprecated: matching using keys: contenttype_id, contenttypegroup_id, objectstate_id, objectstategroup_id, usergroup_id has been deprecated in favour of content_type_id, content_type_group_id, object_state_id, object_state_group_id, userg_roup_id. The same applies for the equivalent ..._identifier keys. This makes the DSL more consistent.

The number of new features, improvements and bug fixes in this release is probably the biggest this project has seen in a "minor" version so far; there just happens to be no "major" changes that warrant a bump in the major version number. The 'developer experience' has been the primary focus of attention, along with test coverage. Besides suggesting that everybody upgrade to the latest release, I recommend to read carefully the release notes, as long as boring as those might seem...

• Improved: a single value for a content field of type ezcountry can be specified as a string instead of an array (issue #190)

• New: taught the kaliop:migration:status command to sort migrations by execution date using --sort-by (issue #224)

• New: taught the kaliop:migration:migrate command a new option: --set-reference (issue #162). This is allows to inject any desired reference value into the migrations.

• New: taught the kaliop:migration:resume command the same new option: --set-reference

• Improved: references can now be set using a simplified syntax. eg:

      -
          type: content_type
          mode: load
          match:
                  identifier: philosophers_stone
          references:
              my_ref_name: content_type_id
  

  Note that you still need to use the old syntax for reference creation in order to be able specify overwrite: true

• New: taught the reference/set migration step to resolve environment variables besides Symfony parameters (issue #199). Eg:

    -
        type: reference
        mode: set
        identifier: myReference
        value: '%env(PWD)%'
  

• New: : taught the reference/set migration step not to resolve environment variables at all, eg:

    -
        type: reference
        mode: set
        identifier: a_funny_string
        value: 'reference:or_not_to_reference'
        resolve_references: false
  

• New: taught the SQL migration step, when specified in yaml format, to resolve references embedded in the sql statement (issue #199), eg:

    -
        type: sql
        resolve_references: true
        mysql: "UPDATE emp SET job='sailor' WHERE ename='[reference:example_reference]'"
  

• New migration step: sql/query, which can be used to run SELECT queries on the database (issue #199). Unlike the existing sql/exec step (previously known simply as sql), this step allows to set reference values with the selected data. Ex:

      -
          type: sql
          mode: query
          mysql: "SELECT login FROM ezuser WHERE email like '%[@ez](https://github.com/ez).no'"
          expect: any
          references:
              -
                  identifier: users_count
                  attribute: count
              -   identifier: users_login
                  attribute: results.login
  

  For more details, see the complete specification in file SQL.yml

• New migration steps: content_type_group/load and trash/load

• New migration step: migration/fail, which is similar to migration/cancel, but leaves the migration marked as failed instead of executed

• New: migration step proces/run now supports element fail_on_error, which triggers a migration failure if the external process executed returns a non zero exit code (issue #234)

• New: all load/update/delete steps, as well as a couple non-repository-related steps, support the optional expect element. This is used to validate the number of matched items, as well as altering the value of the references created.

  • use expect: one to enforce matching of exactly one element, and set scalar values to references
  • use expect: any to allow steps matching of any number of elements, and set array values to references
  • use expect: many to enforce matching of one or more elements, and set array values to references
  • using expect enforces validation of the number of matched elements regardless of the fact that there are any reference definitions in the step, whereas references_type and references_allow_empty only activated if there was at least one reference defined
  • also, the validation of the number of matched elements, when required, now happens before any item deletion/update action takes place. Up until now, for update steps, only a subset of the validation was enforced before the action, and the rest was validated afterwards

• Improved: using the not element in matching clauses would not work for most types of steps, when the element not-to-be-matched was not present in the repository. Notable exceptions being Content and Locations matches. This case now works. Example of a migration that would fail: find all content types except the one 'philosophers_stone'

      -
          type: content_type
          mode: load
          match:
              not:
                  identifier: philosophers_stone
  

• New: most load/update/delete steps support the optional match_tolerate_misses element (issue #235). When setting it to true, the migration will not abort if there are no items in the repository matching the specified conditions. Example of a migration that would previously always fail: update non-existing content type 'philosophers_stone'

      -
          type: content_type
          mode: update
          match:
                  identifier: philosophers_stone
          # make this step successful in case we have not found the stone yet...
          match_tolerate_misses: true
  

• Fixed: setting references using jmespath syntax in migration steps migration_definition/generate

• BC change: when matching users by email in steps user/update, user/delete, user/load the migration will now be halted if there is no matching user found. This can be worked around by usage of match_tolerate_misses: true

• Improved: step reference/dump will not echo anything to stdout any more in case the migrate command is run with -q

• Improved: when generating migrations for Role creation/update, the bundle now tries harder to sort the Role Policies in a consistent way, which should make it easier to diff two Role definitions and spot changes

• Improved: made console command kaliop:migration:migrate survive the case of migrations registered in the database as 'to do' but without a definition file on disk anymore - a warning message is echoed before other migrations are run in this case

• Improved: made console command kaliop:migration:migrate -vv more verbose than kaliop:migration:migrate -v. Besides printing one message before each step begins execution, it also displays time taken and memory used for each step (issue #200). Also, improved the output of kaliop:migration:migrate -v by printing step numbers

• New: migration steps can now take advantage of $context['output] to echo debug/warning messages (issue #201). When set, it is set to an OutputInterface object.

• Improved: many new and improved test cases

• New: taught the test-execution command teststack.sh two new actions: console and dbconsole, as well as a few new options: -r runtests, cleanup ez-cache and cleanup ez-logs. It also accepts the name of a testcase to be run instead of the whole suite and other phpunti command line options, when executing runtests.

• Fixed: regressions when running the test-execution command teststack.sh with the -u option or resetdb action

• BC change: some options for the test-execution command teststack.sh have been renamed, see teststack.sh -h for the new list

• BC change: the references_type and references_allow_empty step elements have been replaced by a new element: expect. The references_type and references_allow_empty step elements are still handled correctly, but considered deprecated; equivalence matrix: expect: one <==> references_type not set or equal to scalar expect: any <==> references_type: array and references_allow_empty: true expect: many <==> references_type: array and references_allow_empty not set or equal to false For developers: the RepositoryExecutor class and its subclasses have dropped/changed methods that deal with setting references. You will have to adapt your code if you had subclassed any of them

• BC change: the database tables used by the bundle are now created by default with charset utf8mb4 and collation utf8mb4_general_ci (issue #176). They used to default to utf8 and utf8_unicode_ci. This is in general not a big issue, as there are no queries with joins between our tables and the eZP ones, but in case you have custom code that does those queries, those might fail if the charset or collation differ. To fix that, you can set different values to the Symfony parameters ez_migration_bundle.database_charset and ez_migration_bundle.database_collation. Note that this 'change' only applies to new installations of the bundle - if the migration tables already existed in your database before upgrading to the latest Migration Bundle version, they will not be modified.

• BC change: some cases of \InvalidArgumentException being thrown have been replaced with Kaliop\eZMigrationBundle\API\Exception\InvalidStepDefinitionException

• Improved: make the bundle compatible with PHP 7.4

• Improved: made it easier to run the test suite locally using multiple Docker stacks for different php/mysql versions

• New: new constraints isnull and notnull are now supported in 'if' clauses to match references values

• New: the migration_definition/generate step now supports an 'if' clause

• New: the migration_definition/generate step now supports setting a reference to the whole definition

• New migration step: migration_definition/save. Useful in content migrations / syndication scenarios

• BC changes: the migration_definition/generate step now uses a different syntax for setting references. The old one is still accepted but deprecated (key json_path has been replaced by attribute)

• Fixed issue #232: error with EmbeddedRegexpReferenceResolverTrait.php and php 7.4

• Improved: massively reworked travis setup to make it friendlier to ezplatform 3 installations

• Fix issue #216: cannot update a location's parent matching it by remote id

• Improved: when creating/updating content, allow to set references to location_remote_id

• Improved: add plumbing to allow future usage of custom content types for UserGroups

• Fix issue #210: cannot match locations by group

• Fix: matching users by usergroup_id

• Fix: file migration steps would not work when using an if element

• Fix issue #207: java.lang.NegativeArraySizeException error when using SOLR multi core

• Improved the DSL docs for the management of Roles (see issue #211)

• Implemented request #205: allow to generate migrations for tag creation independently of content

• Implemented request #215: better error message when migrations fail because an invalid admin account is used to run them

• Implemented request #211: allow to unassign roles from groups on update

• Allow more flexibility in tag matching:

  • allow to match all tags
  • when specifying a parent-tag id, the remote_id can be used in its place

• Implemented request #204: an event of class MigrationGeneratedEvent is now emitted when a migration definition is generated via the command kaliop:migration:generate, allowing developers to easily customize the generated migrations

• Improved: it is now possible to set a reference to the remote_id of any created/updated/deleted userGroup

• Added a Docker-Compose based stack to ease execution of the test suite locally. See the main README for details on use

Fix issue: usergroup_id matching for users was not working.

Fix issue #202: RoleManager::createLimitation fails when using array reference

(note: all changes since 5.9.0 listed here)

• Fixed: match 'all' languages would raise an exception

• Fixed: the migrate command terminates with non-0 exit code when any migration failed, even if it is given the -i option

• Fixed: the mass_migrate command terminates with non-0 exit code when any migration or subprocess failed

• Improved: when migrations fail, the error message is written to stderr instead of stdout, for both the migrate and mass_migrate commands

• Improved: better error output by the migrate and mass_migrate commands. In particular:

  • they now report the number of non-executed migrations besides the failed and skipped ones.
  • the error output when using the -p option has been made more similar to the one of the standard case

• Improved: better support for -v and -q options for the migrate and mass_migrate commands, esp. when used together with -p

• New: the migrate and mass_migrate commands accept an option survive-disconnected-tty. This helps in cases where you would normally run the migrations using screen or tmux, such as over ssh connections which risk being dropped before the migrations have finished executing

• New: the migrate and mass_migrate commands accept an option force-sigchild-enabled. This is useful when you are running on eg. Debian and Ubuntu linux, and run the migrations using separate subprocesses: in such scenario there are chances that migrations will be reported as having failed executing even though they have not. Using the force-sigchild-handling option should fix that. For reference, see comment 12 in this ticket: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/516061

• BC changes:

  • code which relies on parsing the output and/or exit code of migrate and mass_migrate commands should be adjusted

Use 5.9.3 or later instead

Use 5.9.3 or later instead

• New: the role/create migration step now resolves references for role names. Same for role/update.

• New: new migration steps language\update, language\load, section\load, role\load, object_state\load, object_state_group\load

• New: more flexible matching for migration step language\delete

• New: more reference resolving in section creation and update

• New: the generate command now has a --list-types option that will have it list all migration types available for generation

• Fix: warnings generated when creating array-valued refs using an empty collection of items

• Fix: references would not be resolved for Author and Selection fields, when the field value is given in array form. Ex: this will now be resolved

    ...
    attributes:
        country: # an ezselection field
            - italy
            - reference:mycountry
  

• BC changes:

  • the language\delete step should not be used any more with a lang element, but with match instead