Description

This page describes how to execute long-running console commands, to make sure they don't run out of memory. An example is a custom import command or the indexing command provided by the Solr Bundle.

Solution

Reducing memory usage

To avoid quickly running out of memory while executing such commands you should make sure to:

1. Always run in prod environment using: --env=prod
   1. See Environments for further information on Symfony environments.
   2. See Logging and debug configuration for some of different features enabled in development environments, which by design uses memory.

2. Avoid Stash (Persistence cache) using to much memory in prod:

   1. If your system is running, or you need to use cache, then disable Stash InMemory cache as it does not limit the amount of items in cache and grows exponentially:

      Code Block
      title config_prod.yml (snippet, not a full example for stash config)
      stash: caches: default: inMemory: false

      Also if you use FileSystem driver, make sure memKeyLimit is set to a low number, default should be 200 and can be lowered like this:

      Code Block
      title config_prod.yml
      stash: caches: default: FileSystem: memKeyLimit: 100

   2. If your setup is offline and cache is cold, there is no risk of stale cache and you can actually completely disable Stash cache. This will improve performance of import scripts:

      Code Block
      title config_prod.yml (full example)
      stash: caches: default: drivers: [ BlackholeBlackHole ] inMemory: false

3. For logging using monolog, if you use either the default  fingers_crossed , or buffer handler, make sure to specify buffer_size to limit how large the buffer grows before it gets flushed:

   Code Block
   title config_prod.yml (snippet, not a full example for monolog config)
   monolog: handlers: main: type: fingers_crossed buffer_size: 200

4. Run PHP without memory limits using: php  -d memory_limit=-1 app/console <command>
   
5. Disable xdebug (PHP extension to debug/profile php use) when running the command, this will cause php to use much more memory.

Process forking with Symfony

Anchor
process-forking process-forking

The recommended way to completely avoid "memory leaks" in PHP in the first place is to use processes, and for console scripts this is typically done using process forking which is quite easy to do with Symfony.

The things you will need to do:

1. Change your command so it supports taking slice parameters, like for instance a batch size and a child-offset parameter.
   1. If defined, child-offset parameter denotes if a process is child, this could have been accomplished with two commands as well.
   2. If not defined, it is master process which will execute the processes until nothing is left to process.
2. Change the command so that the master process takes care of forking child processes in slices.
   1. For execution in-order, you may look to our platform installer code used to fork out solr indexing after installation to avoid cache issues.
   2. For parallel execution of the slices, see Symfony doc for further instruction.