`to_js_literal` method is taking excessively long time

[rdp1414]

Describe the bug

The to_js_literal method is taking excessively long time.

---

Steps Reproduce

1. First, let us create a 4000 points dataset.

import seaborn as sns

original_df = sns.load_dataset('iris')
df = sns.load_dataset('iris')
for i in range(26):
    df = df.append(original_df, ignore_index=True)
df

2. Now, create a Highcharts Python chart.

from highcharts_core.chart import Chart
from highcharts_core.options.series.scatter import ScatterSeries
from highcharts_core.options.plot_options import PlotOptions

ch = Chart.from_pandas(
    df,
    property_map={
        'x': 'petal_width',
        'y': 'sepal_length',
    },
    series_type='scatter',
)

# Disable boost.
ch.options.plot_options = PlotOptions(series=ScatterSeries(turbo_threshold=0))

3. Then, when chart is displayed. the bulk of time goes in to_js_literal method. This particular method takes 21 seconds - See python profiling screenshot.

display(ch)

---

Expected behavior

I observed that:

1. IF the number of rows in the input DataFrame decreases, so does the time taken by to_js_literal method.
2. Why is this method taking so much time? I looked at the code - it seems that this is a recursive method. Due to this method taking 21 seconds, entire rendering time is 28 seconds. This is not a great user experience that you've to wait for half-a-minute to view your chart.

---

Your Environment

• OS: Windows, Jupyter Lab (notebook).
• Python Version: 3.10.4
• Highcharts Version: Installed Highcharts Python using pip install highcharts-core command.

[hcpchris]

Hi @rdp1414 ! This is definitely a known performance issue that particularly affects datasets with large numbers of data points. We have on our roadmap fundamentally improving performance, which is an effort that is currently underway.

In this case, the to_js_literal() method is not recursive, but the issue is that it basically has to be executed on every object within the overall chart configuration tree. And this means that it needs to be run on every data point, which gets serialized to its JS literal object representation. So even if that were to only take 1 millisecond per data point, with 4000 data points you're looking at 4 seconds spent just doing that - which obviously is suboptimal. This is why the serialization to JS literal is a major area of optimization that we are looking at.

While it's not ready for release yet, we're likely to introduce two different types of optimization:

Earlier on:

• we will support serialization of data points to a simple array (i.e. unless a data point has special (non-default) configuration settings applied, there is no need to serialize it to its full JS literal object representation - it can be stored and serialized in a simple array, which is a much simpler / faster process
• we will be pruning the JS literal serialization logic to make sure that there are no unnecessary round-trips being made

Longer term, we are thinking about implementing JS literal serialization through either a C or Rust extension to get maximal performance. But that's a longer-term effort.

Both of these sets of improvements are works-in-progress, and are unlikely to be released in the next few weeks. You can keep track of progress by watching the roadmap item, however.

[abbas-mohd56]

Just wanted to add that I'm facing a similar performance issue but with to_json method instead. I have a Dataframe with multiple columns and around 9k rows. When I try to create a LineSeries object for each column, the to_json method can take up to 30s to generate the json string.

[hcpchris]

So good news on this issue: some of the performance improvements we've implemented (and which are slated for release in v.1.4) are benchmarking at accelerating the .to_js_literal() method by 50 - 90%, and accelerating the .to_json() method by about 30 - 90%.

In both cases, the performance improvement should be very significant, but the absolute impact will depend on how complex your chart configurations are and how many data points you have. Either way, the next release in the coming weeks should see some very significant improvements on this front.

[hcpchris]

So this issue should see a significant improvement now that v.1.4.0 has been released. In our benchmarking tests, the .to_js_literal() method sees a 50 - 90% performance improvement over v.1.3.7, depending on various factors. Given that, I'll close this issue for the time being, but if you still see poor performance, please feel free to either re-open or file a new issue and we'll take a deeper dive to see what may be causing such degraded performance.