key: cord-0543164-8si70zlh
authors: Pharasi, Hirdesh K.; Sadhukhan, Suchetana; Majari, Parisa; Chakraborti, Anirban; Seligman, Thomas H.
title: Dynamics of the market states in the space of correlation matrices with applications to financial markets
date: 2021-07-12
journal: nan
DOI: nan
sha: 7b1ee35e64207db61c8e3dc05eb23718e2acc352
doc_id: 543164
cord_uid: 8si70zlh

The concept of states of financial markets based on correlations has gained increasing attention during the last 10 years. We propose to retrace some important steps up to 2018, and then give a more detailed view of recent developments that attempt to make the use of this more practical. Finally, we try to give a glimpse to the future proposing the analysis of trajectories in correlation matrix space directly or in terms of symbolic dynamics as well as attempts to analyze the clusters that make up the states in a random matrix context.

The purpose of this chapter is to give an overview of recent developments in the evolution of the concept of 'states' and 'regions' of financial markets based on correlation analysis [1] [2] [3] . Many refinements have been conceived, some of which try to account for a practical applicability of the concept 1, 2, 4-11 . It opens a new perspective, namely the stochastic dynamics of markets in the space of correlation matrices; the dynamics are discrete in time by the nature of markets governed by individual and countable transactions. The space in which the correlation moves, is of high dimensionality and not a discrete grid! This continuous approach may also be relevant to other applications such as social systems, traffic, economy, ecological systems, neurology and many more [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] . Two points stand out in this context: On the one hand, the correlation matrices of financial markets have been studied over fairly long epochs, mostly with zero or 50% overlapping epoch, which led to a low number of points for the time scale considered. On the other hand, the clusters which define market states are not obviously visible in the reduced spaces, where they have been visualized using multidimensional scaling (MDS) 23 . Smaller epoch shifts (bigger overlapping epochs) in time, to overcome the above mentioned reason as well as for practical purpose, indicate that a trader wants up to date analysis to make his decisions as pointed out in Ref. 3 . Using MDS, one can visualize the similar correlation matrices into 2D and 3D clusters but without a cleanly separated boundaries. The transition matrices, consist of transition counts between the clusters, have shown their underlined relevance [3] [4] [5] . Therefore, following the trajectories on shorter time scales as well as introducing symbolic dynamics seems very attractive. The former is in an advanced stage and will be described in this chapter in some detail, while the latter will be proposed and potential usefulness will be pointed out. In both cases, practically useful results are still outstanding. In the framework of these ideas, we start with a brief description of the origins of the correlation analysis and its evolution. We show how this naturally leads to an evolution in a space, whose dimension is determined by the length of the time series considered, i.e., by the time horizon as well as the number of free matrix elements in the correlation matrix. The analysis, based on the distances in the correlation matrix space between the points touched in time, defines the trajectory and the geometrical structure of the trajectory. Studying the correlation of market sectors following a technique proposed in Ref. 2 yields similar results as we shall see. The high dimensionality of the space in practice makes visualization very difficult.

We could try to represent each state by, say, its average correlation matrix and this would reduce the problem to a discrete space as we would identify all correlation matrices in a cluster with a single label or symbol. At this point it should be noticed that for traffic a three phase theory 24 is used; similarly, a four market states model is possible for the S&P 500 market and a higher number of states is certainly required for the Nikkei 225 market due to higher complexity of the Japanese market 4 . Furthermore in the Japanese market a dominance of the average correlation for the structure of market states, found in the S&P 500 market 1 , clearly does not hold. This opens a perspective for the search of a second relevant parameter, which we shall not pursue here.

In the framework of published and submitted works, which we will briefly describe, we propose two routes to advance the use of correlation matrices and market states. The first is MDS, say, to two or three dimensions, and the second is a symbolic dynamics after a partitioning of the set of observed correlation matrices according to the cluster structure used for defining the states. It, in turn, partitions the space on which the symbolic dynamics evolve in time. We shall show that in the two relevant financial markets (S&P 500 and Nikkei 225) this structure is only marginally affected by MDS, which thus emerges as a very effective tool for visualization and interpretation. We will barely outline the second option, namely the possible usefulness of the symbolic dynamics resulting from the developed cluster structure. In financial markets, we have previously opted epochs shifted by one day 3 , thus omitting the intra day trading data, both because they are more difficult to access and of different nature than daily trading data. We limit our study to a time interval 2006-2019 excluding the year 2020, though we will show scaled trajectory for 2020 that is interesting yet lack of interpretation at present.

We shall develop basics and outline the development that led over the years to the present work. Next we give some technical aspects of data handling and processing. Then we proceed to show results for the correlations of the USA market as reflected by the S&P 500 and the Tokyo Market by the Nikkei 225 using both full correlation matrices and market sector averages. We start by discussing the market states, their transition matrices and then passing to the trajectories in the space of correlation matrices with visualizations including at this point some aspects of the 2020 development of markets during the starting period of COVID-19 pandemic. At this point we shall also mention trajectories in the symbolic dynamics. No conclusions in either case can be presented as it is an ongoing work. Finally we shall try to give an outlook as to where we think the field is heading and what could be done about applications to other fields.

We analyze the daily adjusted closure prices S k (t) of stock k for the S&P 500 and Nikkei 225 indices at different time horizons. The daily trading data is freely available at Yahoo finance website 25 . We consider only those stocks which are continuously traded over the considered time period. We again filtered out a few stocks time series that have more than two consecutive missing trading days entries. The price value for the missing days has chosen to be the same as the one on the previous day entry. We then calculate the logarithmic price returns r k (t) = ln S k (t + ∆t) − ln S k (t) for each stock k at time t, where t = 1, 2, . . . , T tot with T tot is total number of the trading days present in the time horizon considered, and k = 1, . . . , N for N number of stocks and ∆t = 1 day for daily price returns.

To understand the co-movement of the financial market constituents (stocks) and the time evolution of the market conditions, we calculate the Pearson correlation coefficients C i j (τ) = ( r i r j − r i r j )/σ r i σ r j where i, j = 1, . . . , N, τ is the epoch number of size T trading days and σ is the standard deviation. Here ... denotes an averaging over an epoch length of T trading days which is shifted by ∆T days through the data. The choice of epoch is important because market conditions evolute with time and the cross-correlations between any pair of stocks may not be stationary for a long epoch, on the other hand, short epoch size leads to "noise" or "fluctuation" due to the singularity present in the corresponding correlation matrix. Therefore, the distribution of eigenvalue spectrum of the empirical cross-correlation matrix C C C(τ) contains "random" and as well as non-random contribution 26, 27 . That motivates us to compare the eigenvalue statistics of C C C(τ) with a large random correlation ensemble constructed from mutually uncorrelated time series or white noise known as Wishart matrix. We will introduce Wishart Orthogonal Ensembles (WOE) in detail in the next subsection.

We construct a rectangular random data matrix A A A = [A i j ] of N random time series each of length T , i.e., of order N × T with real independent elements drawn from a standard Gaussian distribution with fixed mean and variance. The Wishart matrix is then constructed as W W W = 1 T A A AA A A of size N × N, where A A A denotes the transpose of the matrix A A A. Wishart matrices are random matrix models used to describe universal features of covariance matrices. We here consider the entries as real, known in the literature as Wishart orthogonal ensemble and also, by construction, it is a real symmetric positive semidefinite matrix. To obtain the correlation matrix, we fix the value of mean to zero and variance to σ 2 = 1. In the context of time series, W W W may be interpreted as the correlation matrix, calculated over stochastic time series of time horizon T for N statistically independent variables.

Following the Marcenko-Pastur distribution, the probability density functionρ(λ ) of the eigenvalues of such correlation matrices can be analytically shown for the limit N → ∞ and T → ∞ as

where Q = T /N fixed. The maximum and minimum eigenvalue can be derived from λ max min = σ 2 1 ± 1/ √ Q 2 . For Q ≤ 1, one has to take into account of at least N − T zeros, hence the densityρ(λ ) is normalized to Q and not to unity.

To obtain the stationarity in time series, the consideration of shorter time series by breaking a long time series into epochs, each of size T (such that T tot /T = Fr) are useful. However, if N (stocks)> T , we get a singular correlation matrix with N − T + 1 zero eigenvalues, results in poor eigenvalue statistics. We introduce here the power map technique 28, 29 to lift the degeneracy of zero eigenvalues by providing a non-linear distortion (ε, also called noise suppression parameter) to each element (W i j ) of the Wishart matrix W W W : W i j → (sign W i j )|W i j | 1+ε . This technique helps to get an "emerging spectrum" of eigenvalues by removing the degenerated eigenvalues at zero even for very small distortions, e.g., ε = 0.001 (see e.g., Refs. 4, 30, 31 for recent studies and applications).

In pairwise (dis)similarity analysis, the correlation matrix is first calculated for each epoch (which represents a specific time frame) and we compute the (dis)similarity by measuring the distance between the two correlation matrices. Such analysis is performed for each pair of correlation matrices (frames) and defined as:

..| and ... denote the absolute and average value over all the matrix elements, respectively and τ = 1, . . . , Fr with Fr is the total number of epochs of size T constructed from the overlapping shift of ∆T days. This technique yields a symmetric matrix, with all positive off-diagonal elements Fr(Fr − 1)/2, where Fr is the number of frames. Diagonal elements are zero for no such (dis-)similarities present between the same frames. We use multidimensional scaling 23 , which reveals the structure of (dis-)similarity data by creating a map, it constructs a geometrical representation from the relative distances (ζ ) of correlation frames in such a way that the points corresponding to similar objects are located close together, while those corresponding to dissimilar objects are located far apart.

In this section, we present an overview of recent developments in the evolution of the concept of 'states' of financial stock markets based on correlation analysis. We start with the very first work on the characterization of "market states" based on the similarity of the correlation of market states by Munnix et. al. 1 . Figure 1 shows the temporal evolution of the market states of the S&P 500 market for 19-year observation period 1992-2010. Each point in the figure represents a correlation matrix measured over the previous two months. The market is clustered into eight states (S1 to S8) based on a top-down clustering method using a similarity matrix ζ . In this method, initially all the correlation matrices are considered as a single cluster and then further divided into sub-clusters using k-means clustering algorithm. The division process stops as the average radii of all the points to centroids becomes smaller than a certain threshold. In the Fig. 1 , initial points are clustered in lower states (S1 and S2) with less transitions to other states, hence shows a stable behavior of the market before 1996. On the other hand, The market shows the transitions between eight characterized states (S1 to S8) based on the top-down clustering method. The lower states (i.e., S1 and S2), denoting as normal periods of the market, tend to cluster in time revealing that the market remains in those particular states for a longer time. On the other hand, higher states are scattered and irregular in time showing the rapid to and fro transitions between the states with the higher probability of transitions to the nearby states. Figure one can observe more frequent jumps for higher states with scattered points after 1996 and shows high market volatility. In this method, on one side, the number of clusters is dependent on an arbitrary threshold and on the other side, the correlation matrices constructed by short time series are highly singular and dressed with noise.

The noise dressing has an enormous effect on portfolio optimization and proper risk assessment 32 . It has been found by Laloux et. al. 33 that the correlation matrices are noisy due to the finiteness of the time series. Guhr et. al. 34 has presented the power map method to suppress the noise of the correlation matrices, which effectively increases the length of time series. This method is more suitable for the analysis of short time series. In Fig. 2 , we show the effect of the power map method and the length of time series on the spectral density ρ(λ ) of the Wishart orthogonal ensemble (WOE). We use various epoch lengths T = 1000, 5000, 10000, 40000 and noise suppression parameters ε = 0, 0.265, 0.383, 0.63 and show that the spectral density ρ(λ ) of WOE approaches towards identity matrix for higher value of T (for fixed N and ε) and ε (for fixed N and T ). Fig. 2 (a) shows the spectral density ρ(λ ) of a Wishart matrix ensemble (ensemble = 1000) of size N × N, constructed from N = 500 (fixed) time series of real independent Gaussian variables, for four different epoch lengths T (1000, 5000, 10000, 40000) with unit mean and variance σ 2 . The variance of the distribution decreases with the increment of T and move towards the identity matrix for higher T . The same can be achieved by increasing the value of the noise suppression parameter ε of the power map. Fig. 2 (b) shows the effect of the power map by varying the value of ε but keeping N and T fixed. The comparison of spectral densities ρ(λ ) in Fig. 2 (a) and (b) show that the similar variance can be achieved either by increasing the length of the time series or by increasing the value of noise suppression parameter ε.

To suppress the noise of the correlation matrices and to avoid the arbitrariness of the threshold in the top-bottom clustering technique 1 , Pharasi et. al. 4 developed a new approach to obtain the optimum number of market states for a financial market. Similar approach can be applied to other complex systems. The optimization is achieved using two parameters-intra cluster measure and noise suppression parameter. Intra cluster measure d intra is an averaged euclidean distance of points to cluster centroids of the k-means clustering performed on the MDS map. It is constructed from the similarity matrix ζ using 805 noise-suppressed (ε = 0.6) correlation matrices of USA market and 798 noise-suppressed (ε = 0.6) correlation matrices of Japanese market and overlapping epoch T (d) show the transition counts of paired market states for S&P 500 and Nikkei 225 markets, respectively. The optimum number of cluster for k ≥ 4 is chosen such as the standard deviation of the ensemble is lowest keeping the number of clusters highest. Our analysis depicts that k = 4 and 5 are the optimal number of clusters for S&P 500 and Nikkei 225, respectively. We cluster similar correlation matrices into these optimized k number of "market states". The market evolves through the transitions between these market states. Sometimes the market remains in a particular state for a longer time and sometimes it jumps shortly to another state and bounces back or evolves further. Transitions to nearby states are highly probable. Highest transition counts of market states occurs when market remains in the same state i.e. along the diagonal followed by first off diagonal as transitions to nearby states. The jump from the lowest S1 and S2 state to the critical state (S4 in USA and S5 in JPN) is less probable which acts as a precursor to the critical state and as a best state to hedge against the critical state.

Recently, Pharasi et. al. in Ref. 3 have presented an improved criteria for the classification of "market state", mainly due to the increased attention to the transition matrix of paired states. The optimization of the number of market states is done by using the following two parameters-number of clusters and noise suppression. However, here the preference is given to the transition matrix which avoids large jumps from lower states to the critical state. It brings a better perspective to hedge against the market's critical events without a prohibitive cost. By using the same overlapping epoch of length 20 days but the shifts of 1 day, the results showed significantly improved statistics as compared to the shift of 10 days in Ref 4 . Figure 4 In contrast to Fig. 3 , here we use ensemble of 1000 different initial conditions for different number of clusters k and noise suppression parameter ε. It is found that the minimum of σ d intra for k ≥ 4 at k = 5 and ε = 0.9 for S&P 500 and at k = 7 and ε = 0 for Nikkei 225 markets. This optimum number of cluster is used for the k-means clustering for 3503 correlation frames with noise-suppression (ε = 0.9) for S&P 500 market and 3439 correlation frames without noise-suppression (ε = 0) for Nikkei 225 market. The 2D projection of 3D k-means clustering for S&P market on xz-plane with k = 5 clusters and Nikkei 225 market on xy-plane with k = 7 clusters, respectively, are shown in Fig. 4 (c) and (d). The distribution of clusters is different for two market-for S&P 500 market, k-means clustering divides the distribution of correlation frames in 3D correlation space in the horizontal direction (S1, S2, S3, S4, S5) but for the Nikkei 225, it divides in both vertically stacked (S2 & S3 and S4 & S5) as well as horizontal directions. market is nearly tridiagonal and shows zero transitions from lower states (S1, S2, and S3) to critical state (S5) with twelve transitions from the penultimate state (S4) to critical state (S5). Thus, S4 behaves as a precursor to the market crashes and a good agent to hedge. For (d) Nikkei 225 market, the transition counts of the paired market states between S2 and S3 are smaller than S2 and S4. Here, state S6 behaves as a precursor to the critical state S7. The figure is adapted from Ref. 3 . for S&P 500 market is nearly tridiagonal and shows zero transitions from lower states (S1, S2, and S3) to critical state (S5) with twelve transitions from penultimate state (S4) to critical state (S5). Thus, S4 behaves as a precursor to the market crashes and a good agent to hedge. For Nikkei 225 market, the transition counts of the paired market states between S2 and S3 (S2→ S3=25, vertically stacked) are smaller than S2 and S4 (S2→S4=45, in horizontal direction). The same behavior is noticed during the transitions between states S4 and S5, and S4 and S6. Therefore, state S6 behaves as a precursor to the critical state S7 with one transition, contrary to S&P 500, from S5 to critical state S7.

We consider here the same data of N = 350 stocks of the S&P 500 index (N = 156 stocks of the Nikkei 225) over the period from 2006-2019 with T tot = 3523 (T tot = 3459) trading days 3 . We then calculate a sectorial averaged matrices M by taking intra-and inter-sectorial average of the correlation matrices C constructed from the overlapping epoch of T = 20 days and shifted by ∆ = 1 day. The symmetric matrices M are not correlation matrices any more and the diagonal elements are not trivial. We adopt the similar approach used by Rinn et. al. 2 , where they performed a cluster analysis of sectorially averaged correlation matrices and stochastic process analysis to identify financial market dynamics by a few important variables. The market showed dynamical stability and transitions between the stable quasi-stationary states. The method allows to project the dynamics of high-dimensional system to a low-dimensional space. In this manner, we have the matrices M with dimension N S × N S , where N S is the number of the sectors in the market-N S = 10 for S&P 500 and N S = 6 for Nikkei 225. The number of independent variables in the M matrix are N S (N S + 1)/2. The direct advantage of analyzing these matrices M, in comparison to full correlation matrices C, is their low dimensional space, i.e., 45D for S&P 500 and 15D for Nikkei 225 markets which are much smaller than the original N(N − 1)/2 dimensional space of correlation matrix C. Working in lower dimensional space has an advantage of reduced computational time and provides a useful tool for studying sectoral behavior. We then apply the k-means clustering analysis on these matrices M after applying the appropriate noise suppression 31, 34 , based on the optimization using Fig. 6 . We can achieve the same dimensional reduction by taking the average of log-return time series for each sectors first and then calculate the correlations among these N S time series. But, we have found that this method is not effective as by taking average of time series, we average out a lot of information and unusual land up on high correlations even in the normal calm periods.

In Fig. 6 , we optimize of number of market states based on the minimum standard deviation of intra cluster distances σ d intra . Fig. 6 (a) and (b) show the optimization using intra-and inter-sectorial averaged matrices M for S&P 500 and Nikkei 225 markets, respectively. We apply k-means clustering on MDS map, constructed from noise suppressed sectorial averaged matrices M. We use 1000 different initial conditions for the calculation of σ d intra . According to the minimum of standard deviations of intra cluster distances, the best choice (k > 4) for S&P 500 is k = 5 clusters and ε = 0.2 (see, Fig. 6 (a) ), and for Nikkei 225, k = 5 clusters and ε = 0.3 (see, Fig. 6 (b) ). Figure 7 shows the classification of the sectorial averaged matrices M of S&P 500 and Nikkei 225 markets. The US market is clustered into five market states and corresponding k-means clustering and transition matrix of paired market states are shown in Fig. 7 (a) and (b) , respectively. Here we choose k = 5 clusters based on the robust measure (minimum of σ d intra for k > 4) of k-means clustering using Fig. 6 (a) . We have found that the k-means clustering and transition matrix show similar behavior as shown in the sectorial analysis (see, Fig. 4) . But for the Japanese market, we preferred eight clusters over five due to (i) the transition matrix which avoids large jumps from lower states to the critical state and (ii) the similar distribution of clusters as obtained from the stocks analysis (see, Fig. 4 (d) ). The Japanese market is more complex than USA market and the distribution of k-means clusters (k = 8) is in both vertical and horizontal directions in JPN market which is only horizontal direction for USA market. Transition matrices, consists of transition counts between different states, for S&P 500 and Nikkei 225 represent a valuable tool to analyze the behavior of critical state (S5) are shown in Figs. 7 (b) and (d) , respectively. In this method, we have found that the number of epochs in the crash state of S&P 500 market are higher (false positive) than one obtained from stock analysis in Fig. 5 . Furthermore, by increasing the number of clusters provide us appropriate way in which risk assessment can be studied with further details. The transition matrix, shown in Figs. 7 (d) for Nikkei 225, with eight number of clusters has less number of epochs in critical state which would be helpful for better risk assessment.

The displacement between stocks analysis and sector analysis for S&P 500 with five market states and noise suppressed ε = 0.2 is equal to 465 days. We must note that the maximum of displacement, in this case, is d i = ±1, with respect to the stock analysis of the Ref. 3 Figure 6 . Optimization of number of market states based on the minimum standard deviation of intra cluster distances σ d intra for (a) S&P 500, and (b) Nikkei 225 markets. We use MDS map, constructed from noise suppressed sectorial averaged matrices M, for k-means clustering. We use 1000 different initial conditions for the calculation of σ d intra . According to the minimum of standard deviations of intra cluster distances, the best choice (k > 4) for (a) S&P 500 is k = 5 clusters and ε = 0.2, and for Nikkei 225, k = 5 clusters and ε = 0.3.

In this section, we compare the COVID-19 crash of S&P 500 and Nikkei 225 market with other critical events as well as normal periods of both markets. For this purpose, we considered the daily adjusted closure price data and the observation period is given in Tables 2 and 3 For S&P 500 market, we optimize the number of clusters using σ d intra using Fig 6 (a) . For the classification of the Japanese market into market states, we preferred eight clusters over five due to the transition matrix which avoids large jumps from lower states to the critical state.

observation period of 125 trading days keeping the crash day at the center. For the analysis, we have considered a period of 125 trading days keeping the crash date at the center (hollow black star). The trajectory consists of equidistant points with some intermittent abrupt increments during critical events. The (dis-)similarity measure is the distance among correlation matrices so when the transition occurs between two consecutive correlation matrices in the same or nearby market states then the corresponding distance measure shows small and nearly equidistant increments. When the transitions happen between different market states then the corresponding distance measure shows big increments. We have also compared here three crashes and one normal period.

We use MDS that reproduces the original metric or distances. Classical multidimensional scaling at various dimensions (D=1, 2, 3, 4, size of ζ ) of (dis)similarity matrix produces co-ordinates from a D-dimensional map. We can measure the Euclidean distance between pairs of all co-ordinates. Table 1 Fifth, sixth, and seventh columns show the variances σ x 2 , σ y 2 , σ z 2 , respectively, of the 3D MDS map using the similarity matrices. For the analysis, we take the ratio of variances of y-and x-directions (σ r 2 = σ y 2 /σ x 2 ) in eighth column. We have found that the ratio stays below 0.4 for critical events and goes above 0.4 for normal periods. There is a false positive event, the ratio for the critical event IPO Facebook Debut (Serial No 7) shows the value higher than 0.4, we are still investigating the reason behind it.

The Tables 2 and 3 show the analysis of characterization of critical and normal events based on the variance of MDS map of correlation matrix trajectories of 125 days (6 months approx.) for the S&P 500 and Nikkei 225 markets, respectively. We have considered data from various critical and normal periods, details are given in the second column of the table with the corresponding starting and ending date in third and fourth columns, respectively. We categorized the normal events into two groups: Normal-1 period, where the trajectory is bounded between states S1 and S2, and Normal-2 period, where the trajectory is bounded among three states S1 to S3 based on the characterization shown in Figs. 4 and 5. We use the variances of the 3D MDS map to calculate the ratio of variances of y-and x-directions (σ r 2 = σ y 2 /σ x 2 ). Note that, the ratio stays below 0.4 for critical events and goes above 0.4 for normal periods. There is a false positive event for the S&P 500 market where the value for the critical event IPO Facebook Debut (Serial No 7) goes above 0.4. Also, for Nikkei 225 market the ratio of variances for the critical events in August 2011 fall (Serial No 2) and IPO Facebook Debut (Serial No 8) show the value higher than 0.4. We are still investigating the reason behind it.

We have presented a brief overview of the evolution of the states of financial markets based on the similarity of correlation matrices obtained over T = 40 and 20 days epochs with shifts ranging from the entire length of the epoch using daily shifts. The adjusted closing data of the S&P 500 and the Nikkei 225 indices are appropriately purified. The line of ideas starts with the work by Munnix et. al. 1 in 2012, where they proposed to group similar correlation structures as "market state" and showed the evolution of the financial market through these market states. They used a top-bottom k-means clustering technique, which was later replaced by minimization of intra-cluster size while requiring stability of noise reduction in order to determine the optimal Normal-2(S1 to S4) Aug 15, 2017 Jan 10, 2018 0.01 0.0052 0.0045 0.523 number of clusters. We then describe a more recent development to take advantage of some remaining freedom of choice by searching for desirable properties of the transition matrix between states. It is important at this stage, to mention that noise reduction is carried out at the level of the correlation matrix rather than at the level of individual time series. We briefly discuss the power map introduced and used extensively by the group of Guhr et. al. 32, 34 . The amusing interpretation of this method as an artificial lengthening of the time series 34 is revisited by analyzing its effects in synthetic data obtained from correlated Wishart ensembles 35 . The line of argument leads up to recent work detailing the use for risk assessment 3 . It brings a better perspective to hedge against the market's critical events without a prohibitive cost. By using the same overlapping epoch of length 20 days but the shifts of 1 day, the results showed significantly improved statistics as compared to 10 days shifts. Using this criterion over the period of 2006-2019, the S&P 500 and Nikkei 225 markets is characterized into five market states with ε = 0.7 and seven market states with ε = 0 (no suppression).

Further, we studied the cluster analysis of market states constructed from averaged intra-and inter-sectorial matrices to identify the market dynamics. Using the same data over the period 2016-2019, the results for S&P 500 and Nikkei 225 are statistically similar to the one obtained from the stock analysis. We showed that the better choice of transition matrix over a minimum of σ d intra . In the Japanese case, the preference is given to a transition matrix that avoids large jumps from lower states to the critical state. Using this approach, we have a better transition matrix and similar distribution of clusters as obtained from the analysis of the stocks. Note that in this approach, we do not perform the clustering of a correlation matrix. A simple-minded approach averaging the series corresponding to each sector yields a different result that does not seem satisfactory. Finally, we try to look a little into the future, by inspecting the time dynamics in the dimensionally scaled space of the correlation matrices in a representation dimensionally scaled to three dimensions, which seems quite interesting, though no conclusions emerge. This leads to an alternative proposition using the market states as the domains on which symbolic dynamics can be built and this also seems like a promising lane to expand the research. A more detailed analysis of the structure of each cluster is an another interesting topic of future research.

Identifying states of a financial market

Dynamics of quasi-stationary systems: Finance as an example

Dynamics of market states and risk assessment

Identifying long-term precursors of financial market crashes using correlation patterns

Complex market dynamics in the light of random matrix theory

Zooming into market states

Stability and hierarchy of quasi-stationary states: financial markets as an example

Exact multivariate amplitude distributions for non-stationary gaussian or algebraic fluctuations of covariances or correlations

Uncovering the dynamics of correlation structures relative to the collective market motion

Constructing analytically tractable ensembles of stochastic covariances with an application to financial data

Non-stationarity in financial markets: Dynamics of market states versus generic features

Identifying the global terror hubs and vulnerable motifs using complex network dynamics

Quasi-stationary states in temporal correlations for traffic systems: Cologne orbital motorway as an example

Critical Transitions in Nature and Society

Anticipating critical transitions

Ecology for bankers

Why Stock Markets Crash: Critical Events in Complex Financial Systems

Social and environmental risk factors in the emergence of infectious diseases

Unification of theoretical approaches for epidemic spreading on complex networks

Statistical physics of fracture, friction, and earthquakes

Evolution of genuine cross-correlation strength of focal onset seizures

Physical approach to complex systems

Multidimensional scaling: I. theory and method

Introduction to modern traffic flow theory and control: the long road to three-phase traffic theory

Accessed on 7th January, 2020 for S&P 500 and 14th

A random matrix theory approach to financial cross-correlations

Correlated wishart ensembles and chaotic time series

A new method to estimate the noise in financial correlation matrices

Time series, correlation matrices and random matrix models

Emerging spectra characterization of catastrophic instabilities in complex systems

Emerging spectra of singular correlation matrices under small power-map deformations

Power mapping with dynamical adjustment for improved portfolio optimization

Noise dressing of financial correlation matrices

A new method to estimate the noise in financial correlation matrices

Market states: A new understanding

Normal-2(S1 to S4) Jun 19, 2017