Challenges in slowing down

On my first day at my new job, the CTO introduced me to my new colleagues. After sampling some chocolate provided by HR, I found myself yet to set up my workstation. Suddenly, a call came in from a senior colleague who was traveling abroad for business. He inquired about my familiarity with time series analysis, splines, and signal processing. Drawing from my prior experience and graduate studies, I assured him of my expertise in these areas. He mentioned that his team would send the necessary files soon. While expressing my eagerness to assist, I explained the need to first set up my PC.

Slow times have their own rules

Once my workstation was ready, I delved into the data. However, upon graphing it, I noticed minimal activity. The data, being time series, appeared piecewise constant. Initially, I attempted ARIMA, but it didn’t converge in most cases. Next, I explored seasonalities and trends using classical methods, but they failed. Even employing techniques like Spline, Fourier analysis, or Wavelets proved fruitless; in fact, these representations seemed worse than the raw data. Feeling the pressure to produce results within my first weeks on the job, I exhaustively tried every technique in the field.

Doubts crept in, wondering if my initial confidence in my knowledge was too presumptuous. I worried that instead of making a good impression, I’d be perceived as incapable. Nonetheless, the firm executive requested that I present my findings. So, I prepared a slideshow and shared it with other stakeholders in the project, leading to a scheduled conference call where I was to present the slideshow.

Indeed, during the meeting, I emphasized the inefficacy of traditional techniques in our specific case, and I was pleasantly surprised when everyone agreed with my assessment. Following the meeting, the client consulted their chief data scientist, who acknowledged that my findings reinforced the limitations of classical methods. She endorsed the adoption of our firm’s specialized technique tailored for slow time series. With a positive outcome, the meeting concluded, leaving us all satisfied.

A typical graph of a slow series

It’s worth noting that the client, being a large corporation accustomed to classical time series methods, initially hesitated to collaborate with our startup due to our reliance on slow time series analysis. However, this experience also highlighted my own limited familiarity with the concept of slow time series.

Don’t treat snails as rabbits

To grasp this concept, let’s revisit the context: a cybersecurity firm operating in the automotive industry. The data under analysis consisted of time series, notably exhibiting a phenomenon of near-constant values with sporadic changes. Parameters such as climate control settings, air pressure readings, or control activations remained static for prolonged durations despite frequent sampling, sometimes exceeding ten times per second. How does one effectively handle such a series? Let’s delve into the concept of storing them.

It’s evident that storing every data point in such a slow time series is impractical. An initial approach involves aggregating the series, consolidating multiple values within a given time window, typically a minute. However, this assumes prior knowledge of when data changes occur, a rarity in practice, necessitating an alternative strategy.

A crucial insight is that slow time series tend to have a predominantly zero value distribution, making them sparse. By employing a sparse storage approach, only non-zero values and their corresponding timestamps are retained, thereby reducing storage requirements significantly. While this entails storing two values per data point instead of one, omitting zero values results in substantial space savings. A rule of thumb suggests employing sparse storage when more than two-thirds of the series comprises zero values, essentially achieving lossless compression. Additionally, storing the difference series alongside the initial value enables the reconstruction of the original data, further enhancing storage efficiency.

Caution: Slow series

In scenarios where the data demonstrates local linearity, such as sensor readings for climate control or battery usage, employing the second difference could provide advantages. By calculating the second difference, we can capture the rate of change in the data, which may offer more meaningful insights than the raw values themselves. In such instances, it is recommended to retain the first two elements of the series.

Adopting this approach facilitates a comprehensive analysis of the frequency and magnitude of changes within the data. Examining the number of changes and their respective magnitudes enables the discovery of valuable patterns and anomalies present in the dataset.

Analyzing the frequency of changes provides a deeper understanding of the dynamics of the observed system. It aids in identifying periods of stability versus periods of volatility, crucial for decision-making and anomaly detection.

Furthermore, assessing the magnitude of changes offers insights into the significance of alterations within the data. Large deviations from expected behavior may indicate significant events or anomalies requiring further scrutiny. Conversely, periods of minimal change may indicate stable conditions or operational efficiency.

By detecting patterns and abnormalities in the data, analysts can gain valuable insights into underlying processes or systems. This information can be leveraged for various purposes, such as operational optimization, future behavior prediction, or risk and opportunity identification.

It’s crucial to recognize that sparse matrix storage poses a substantial challenge in computer science, affecting storage, performance, and analysis. While there’s potential for numerous blog posts dedicated to this topic, it falls outside the scope of the current discussion. Nonetheless, our primary focus remains on time series analysis within this context.

In this article, I delve into the intricacies of managing time series data characterized by slow dynamics. These datasets present unique challenges that demand specialized analytical approaches. By carefully selecting the appropriate tools and techniques, it becomes possible to enhance the quality of results obtained from both the analytical processes and the management of data storage and computational resources. This focus on optimal tool selection is pivotal for achieving superior outcomes across various aspects of analysis, storage, and computational efficiency.