Performance

Profiling

Profiling is about analysing the execution of a program and measure aggregated data. These data can be the elapsed time for each function, the executed SQL queries…

While profiling does not improve the performance of a program by itself, it can prove very helpful in finding performance issues and identifying which part of the program is responsible for them.

Odoo provides an integrated profiling tool that allows recording all executed queries and stack traces during execution. It can be used to profile either a set of requests of a user session, or a specific portion of code. Profiling results can be either inspected with the integrated speedscope open source app allowing to visualize a flamegraph view or analyzed with custom tools by first saving them in a JSON file or in the database.

Enable the profiler

The profiler can either be enabled from the user interface, which is the easiest way to do so but allows profiling only web requests, or from Python code, which allows profiling any piece of code including tests.

Analyse the results

To browse the profiling results, make sure that the profiler is enabled globally on the database, then open the developer mode tools and click on the button in the top-right corner of the profiling section. A list view of the ir.profile records grouped by profiling session opens.

Each record has a clickable link that opens the speedscope results in a new tab.

Speedscope falls out of the scope of this documentation but there are a lot of tools to try: search, highlight of similar frames, zoom on frame, timeline, left heavy, sandwich view…

Depending on the profiling options that were activated, Odoo generates different view modes that you can access from the top menu.

• The Combined view shows all the SQL queries and traces merged togethers.

• The Combined no context view shows the same result but ignores the saved execution context <performance/profiling/enable>`.

• The sql (no gap) view shows all the SQL queries as if they were executed one after another, without any Python logic. This is useful for optimizing SQL only.

• The sql (density) view shows only all the SQL queries, leaving gap between them. This can be useful to spot if eiter SQL or Python code is the problem, and to identify zones in where many small queries could be batched.

• The frames view shows the results of only the periodic collector.

Collectors

Whereas the profiler is about the when of profiling, the collectors take care of the what.

Each collector specializes in collecting profiling data in its own format and manner. They can be individually enabled from the user interface through their dedicated toggle button in the developer mode tools, or from Python code through their key or class.

There are currently four collectors available in Odoo:

By default, the profiler enables the SQL and the Periodic collectors. Both when it is enabled from the user interface or Python code.

SQL collector

The SQL collector saves all the SQL queries made to the database in the current thread (for all cursors), as well as the stack trace. The overhead of the collector is added to the analysed thread for each query, which means that using it on a lot of small queries may impact execution time and other profilers.

It is especially useful to debug query counts, or to add information to the Periodic collector in the combined speedscope view.

class odoo.tools.profiler.SQLCollector[source]

Saves all executed queries in the current thread with the call stack.

Periodic collector

This collector runs in a separate thread and saves the stack trace of the analysed thread at every interval. The interval (by default 10 ms) can be defined through the Interval option in the user interface, or the interval parameter in Python code.

It is one of the best way to analyse performance as it should have a very low impact on the execution time thanks to its separate thread.

class odoo.tools.profiler.PeriodicCollector(interval=0.01)[source]

Record execution frames asynchronously at most every interval seconds.

Parameters

(float) (interval) – time to wait in seconds between two samples.

QWeb collector

This collector saves the Python execution time and queries of all directives. As for the SQL collector, the overhead can be important when executing a lot of small directives. The results are different from other collectors in terms of collected data, and can be analysed from the ir.profile form view using a custom widget.

It is mainly useful for optimizing views.

class odoo.tools.profiler.QwebCollector[source]

Record qweb execution with directive trace.

Sync collector

This collector saves the stack for every function’s call and return and runs on the same thread, which greatly impacts performance.

It can be useful to debug and understand complex flows, and follow their execution in the code. It is however not recommended for performance analysis because the overhead is high.

class odoo.tools.profiler.SyncCollector[source]

Record complete execution synchronously. Note that –limit-memory-hard may need to be increased when launching Odoo.

Performance pitfalls

• Be careful with randomness. Multiple executions may lead to different results. E.g., a garbage collector being triggered during execution.

• Be careful with blocking calls. In some cases, external c_call may take some time before releasing the GIL, thus leading to unexpected long frames with the Periodic collector. This should be detected by the profiler and give a warning. It is possible to trigger the profiler manually before such calls if needed.

• Pay attention to the cache. Profiling before that the view/assets/… are in cache can lead to different results.

• Be aware of the profiler’s overhead. The SQL collector’s overhead can be important when a lot of small queries are executed. Profiling is practical to spot a problem but you may want to disable the profiler in order to measure the real impact of a code change.

• Profiling results can be memory intensive. In some cases (e.g., profiling an install or a long request), it is possible that you reach memory limit, especially when rendering the speedscope results, which can lead to an HTTP 500 error. In this case, you may need to start the server with a higher memory limit: --limit-memory-hard $((8*1024**3)).

Database population

Odoo CLI offers a database population feature through the CLI command odoo-bin populate.

Instead of the tedious manual, or programmatic, specification of test data, one can use this feature to fill a database on demand with the desired number of test data. This can be used to detect diverse bugs or performance issues in tested flows.

To populate a given model, the following methods and attributes can be defined.

Model._populate_sizes

Return a dict mapping symbolic sizes ('small', 'medium', 'large') to integers, giving the minimal number of records that _populate() should create.

The default population sizes are:

• small : 10

• medium : 100

• large : 1000

Model._populate_dependencies

Return the list of models which have to be populated before the current one.

Return type

list

Model._populate(size)[source]

Create records to populate this model.

Parameters

size (str) – symbolic size for the number of records: 'small', 'medium' or 'large'

Model._populate_factories()[source]

Generates a factory for the different fields of the model.

factory is a generator of values (dict of field values).

Factory skeleton:

def generator(iterator, field_name, model_name):
    for counter, values in enumerate(iterator):
        # values.update(dict())
        yield values

See odoo.tools.populate for population tools and applications.

Returns

list of pairs(field_name, factory) where factory is a generator function.

Return type

list(tuple(str, generator))

 Note

It is the responsibility of the generator to handle the field_name correctly. The generator could generate values for multiple fields together. In this case, the field_name should be more a “field_group” (should be begin by a “_”), covering the different fields updated by the generator (e.g. “_address” for a generator updating multiple address fields).

from odoo.tools import populate

class CustomModel(models.Model)
    _inherit = "custom.some_model"
    _populate_sizes = {"small": 100, "medium": 2000, "large": 10000}
    _populate_dependencies = ["custom.some_other_model"]

    def _populate_factories(self):
        # Record ids of previously populated models are accessible in the registry
        some_other_ids = self.env.registry.populated_models["custom.some_other_model"]

        def get_some_field(values=None, random=None, **kwargs):
            """ Choose a value for some_field depending on other fields values.

            :param dict values:
            :param random: seeded :class:`random.Random` object
            """
            field_1 = values['field_1']
            if field_1 in [value2, value3]:
                return random.choice(some_field_values)
            return False

        return [
            ("field_1", populate.randomize([value1, value2, value3])),
            ("field_2", populate.randomize([value_a, value_b], [0.5, 0.5])),
            ("some_other_id", populate.randomize(some_other_ids)),
            ("some_field", populate.compute(get_some_field, seed="some_field")),
            ('active', populate.cartesian([True, False])),
        ]

    def _populate(self, size):
        records = super()._populate(size)

        # If you want to update the generated records
        # E.g setting the parent-child relationships
        records.do_something()

        return records

Population tools

Multiple population tools are available to easily create the needed data generators.

odoo.tools.populate.cartesian(vals, weights=None, seed=False, formatter=<function format_str>, then=None)[source]

Return a factory for an iterator of values dicts that combines all vals for the field with the other field values in input.

Parameters

• vals (list) – list in which a value will be chosen, depending on weights

• weights (list) – list of probabilistic weights

• seed – optional initialization of the random number generator

• formatter (function) – (val, counter, values) –> formatted_value

• then (function) – if defined, factory used when vals has been consumed.

Returns

function of the form (iterator, field_name, model_name) -> values

Return type

function (iterator, str, str) -> dict

odoo.tools.populate.compute(function, seed=None)[source]

Return a factory for an iterator of values dicts that computes the field value as function(values, counter, random), where values is the other field values, counter is an integer, and random is a pseudo-random number generator.

Parameters

• function (callable) – (values, counter, random) –> field_values

• seed – optional initialization of the random number generator

Returns

function of the form (iterator, field_name, model_name) -> values

Return type

function (iterator, str, str) -> dict

odoo.tools.populate.constant(val, formatter=<function format_str>)[source]

Return a factory for an iterator of values dicts that sets the field to the given value in each input dict.

Returns

function of the form (iterator, field_name, model_name) -> values

Return type

function (iterator, str, str) -> dict

odoo.tools.populate.iterate(vals, weights=None, seed=False, formatter=<function format_str>, then=None)[source]

Return a factory for an iterator of values dicts that picks a value among vals for each input. Once all vals have been used once, resume as then or as a randomize generator.

Parameters

• vals (list) – list in which a value will be chosen, depending on weights

• weights (list) – list of probabilistic weights

• seed – optional initialization of the random number generator

• formatter (function) – (val, counter, values) –> formatted_value

• then (function) – if defined, factory used when vals has been consumed.

Returns

function of the form (iterator, field_name, model_name) -> values

Return type

function (iterator, str, str) -> dict

odoo.tools.populate.randint(a, b, seed=None)[source]

Return a factory for an iterator of values dicts that sets the field to a random integer between a and b included in each input dict.

Parameters

• a (int) – minimal random value

• b (int) – maximal random value

• seed (int) –

Returns

function of the form (iterator, field_name, model_name) -> values

Return type

function (iterator, str, str) -> dict

odoo.tools.populate.randomize(vals, weights=None, seed=False, formatter=<function format_str>, counter_offset=0)[source]

Return a factory for an iterator of values dicts with pseudo-randomly chosen values (among vals) for a field.

Parameters

• vals (list) – list in which a value will be chosen, depending on weights

• weights (list) – list of probabilistic weights

• seed – optional initialization of the random number generator

• formatter (function) – (val, counter, values) –> formatted_value

• counter_offset (int) –

Returns

function of the form (iterator, field_name, model_name) -> values

Return type

function (iterator, str, str) -> dict

Good practices

Batch operations

When working with recordsets, it is almost always better to batch operations.

def _compute_count(self):
    for record in self:
        domain = [('related_id', '=', record.id)]
        record.count = other_model.search_count(domain)

def _compute_count(self):
    if self.ids:
        domain = [('related_id', 'in', self.ids)]
        counts_data = other_model.read_group(domain, ['related_id'], ['related_id'])
        mapped_data = {
            count['related_id'][0]: count['related_id_count'] for count in counts_data
        }
    else:
        mapped_data = {}
    for record in self:
        record.count = mapped_data.get(record.id, 0)

for name in ['foo', 'bar']:
    model.create({'name': name})

create_values = []
for name in ['foo', 'bar']:
    create_values.append({'name': name})
records = model.create(create_values)

for record_id in record_ids:
    model.browse(record_id)
    record.foo  # One query is executed per record.

records = model.browse(record_ids)
for record in records:
    record.foo  # One query is executed for the entire recordset.

for values in values_list:
    message = self.browse(values['id']).with_prefetch(self.ids)

Reduce the algorithmic complexity

Algorithmic complexity is a measure of how long an algorithm would take to complete in regard to the size n of the input. When the complexity is high, the execution time can grow quickly as the input becomes larger. In some cases, the algorithmic complexity can be reduced by preparing the input’s data correctly.

for record in self:
    for result in results:
        if results['id'] == record.id:
            record.foo = results['foo']
            break

mapped_result = {result['id']: result['foo'] for result in results}
for record in self:
    record.foo = mapped_result.get(record.id)

invalid_ids = self.search(domain).ids
for record in self:
    if record.id in invalid_ids:
        ...

invalid_ids = set(invalid_ids)
for record in self:
    if record.id in invalid_ids:
        ...

invalid_ids = self.search(domain)
for record in self - invalid_ids:
    ...

Use indexes

Database indexes can help fasten search operations, be it from a search in the or through the user interface.

name = fields.Char(string="Name", index=True)