Human-Independent analysis of the NIS800–53 regulations

The robots are taking over rules organization?!

Last time, we showed that humans are pretty bad at grouping regulations into families. So this time, I am going more human-independently. Our definition of a good, true family will stay the same:

“a good true family is one that the family members are more related/connected to other members of their own family than to members of other families!“

In this continuation of my villain arc, I reject the work humans had done on saying the connection between regulations and letting the machines take over 😈.

First step — let the machines make the connections

To do that, I started by using NLP analysis to discover the connections between the regulations in three different ways:

• BM25 — A classical NLP method that emphasizes rare word similarity will help ensure that regulations that discuss the same uncommon concepts/systems are related.
• HuggingFace🤗 Sentence Transformers — An open-source NLP embedding model, taken from the MTEB Leaderboard. I chose the best one (for the time) that was small enough to run locally (under 1B parameters) and could take a long text paragraph (at least 2048 tokens). This will give an open-source semantic similarity.
• SaaS NLP embedding service — A SaaS NLP embedding service that can use a match bigger model than what I can run locally. For this, we can also look at the MTEB Leaderboard and choose the provider with the best proprietary model for us. This will give a SaaS semantic similarity.

The full code with explanations is in NIS800–53 analysis NLP and NIS800–53 analysis 2

BM25

For the BM25 I used the bm25s Python library,

# Create your corpus here
corpus = Active_NIS_regulations.loc[:,"Full Text"].reset_index(drop=True)

# optional: create a stemmer
stemmer = Stemmer.Stemmer("english")

# Tokenize the corpus and only keep the ids (faster and saves memory)
corpus_tokens = bm25s.tokenize(corpus, stopwords="en", stemmer=stemmer)

# Create the BM25 model and index the corpus
retriever = bm25s.BM25()
retriever.index(corpus_tokens)

# Query the corpusquery = corpus
query_tokens = bm25s.tokenize(query, stemmer=stemmer)

# Get top-k results as a tuple of (doc ids, scores). Both are arrays of shape (n_queries, k)
results, scores = retriever.retrieve(query_tokens, k=len(query), sorted=True)

Because normal BM25 scores are unbonded, I decided to normalize the pair scores by dividing them by the BM25 score of the regulation with itself. Since each regulation must be connected to itself, I want the self-score to be normalized to 1.

# get the bm25 score of the regulation with themselves
self_scores = np.zeros(len(scores))

for i in range(len(scores)):
    # self_scores[i] = scores[i,i]
    self_scores[i] = scores[i, results[i,:] == i][0]

# normalize the bm25 scores matrix
normalized_scores = scores[:,:] / self_scores[:,None]

Looking at the distribution of the normalized BM25 scores

count     1.014049e+06
mean      5.929372e-02
std       7.646549e-02
min       0.000000e+00
10%       0.000000e+00
25%       1.213569e-02
50%       3.832134e-02
75%       8.084672e-02
90%       1.388958e-01
95%       1.878110e-01
99%       3.350398e-01
99.5%     4.363332e-01
99.75%    5.733947e-01
99.9%     9.444073e-01
max       1.019123e+00
dtype: float64

distribution of the normalized BM25 scores

The scores are mostly around 0, and the distribution looks to be of type power distribution and not a normal distribution.

HuggingFace🤗 Sentence Transformers

For HuggingFace🤗 Sentence Transformers I used stella_en_400M_v5

# This model supports two prompts: "s2p_query" and "s2s_query" for sentence-to-passage and sentence-to-sentence tasks, respectively.
# They are defined in `config_sentence_transformers.json`
query_prompt_name = "s2s_query"
queries = Active_NIS_regulations.loc[:,"Full Text"].to_list()

# ！The default dimension is 1024, if you need other dimensions, please clone the model and modify `modules.json` to replace `2_Dense_1024` with another dimension, e.g. `2_Dense_256` or `2_Dense_8192` !
# on gpu
# model = SentenceTransformer("dunzhang/stella_en_400M_v5", trust_remote_code=True).cuda()
# you can also use this model without the features of `use_memory_efficient_attention` and `unpad_inputs`. It can be worked in CPU.
model = SentenceTransformer(
    "dunzhang/stella_en_400M_v5",
    trust_remote_code=True,
    device="cpu",
    config_kwargs={"use_memory_efficient_attention": False, "unpad_inputs": False}
)
query_embeddings = model.encode(queries, prompt_name=query_prompt_name)
print(query_embeddings.shape)

SentenceTransformer_similarities = model.similarity(query_embeddings, query_embeddings)
print(SentenceTransformer_similarities)

Looking at the distribution of the Sentence Transformers scores

count     1.014049e+06
mean      5.936755e-01
std       6.154253e-02
min       4.002191e-01
10%       5.197783e-01
25%       5.514804e-01
50%       5.897334e-01
75%       6.302466e-01
90%       6.697699e-01
95%       6.961381e-01
99%       7.612001e-01
99.5%     7.999417e-01
99.75%    8.654743e-01
99.9%     9.759435e-01
max       1.000001e+00

distribution of the Sentence Transformers scores

Unlike the BM25, the distribution of the Sentence Transformers scores looks to be a normal distribution.

SaaS NLP embedding service

For the SaaS NLP embedding service I used the free Google GanAI embedding service(*text-embedding-004*)

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

for model in genai.list_models():
  if 'embedContent' in model.supported_generation_methods:
    print(model.name)

texts = Active_NIS_regulations.loc[:,"Full Text"].to_list()

response = genai.embed_content(model='models/text-embedding-004',
                               content=texts,
                               task_type='semantic_similarity')

Looking at the distribution of the SaaS scores

count     1.014049e+06
mean      6.327341e-01
std       6.400480e-02
min       3.768515e-01
10%       5.529555e-01
25%       5.896739e-01
50%       6.312641e-01
75%       6.730189e-01
90%       7.116534e-01
95%       7.363882e-01
99%       7.964514e-01
99.5%     8.298782e-01
99.75%    8.755882e-01
99.9%     9.751884e-01
max       9.999989e-01
dtype: float64

the distribution of the SaaS scores

Extracting Related Controls

After extracting all the pairs scores from each of the three methods, the next step in the process was to make the new Related Controls based on each one.

# what is the minimal score for a control to be considered related
# # a hardcoded threshold
# minimal_related_SaaS_score: float = 0.5
# # a threshold as a quantile of the scores distribution
# minimal_related_SaaS_score: float = SaaS_scores.quantile(.99)
# a threshold as the number of expected relations
n = 7 # number of expected relations
minimal_related_SaaS_score: float = SaaS_scores.quantile(1-n/SaaS_similarities.shape[1])

# set a minimal number of relations wanted per regulation (that are different from the main)
min_relations_per_regulation: int = 1

# add a column for the SaaS Related Controls
Active_NIS_regulations.loc[:,"SaaS Related Controls"] = None

# go over every regulation
for i in range(SaaS_similarities.shape[0]):
    # go over all the other regulations to check if they are related
    regulations_scores:dict = {}
    related_regulations:dict = {}
    for j in range(SaaS_similarities.shape[1]):
        if i == j:
            continue
        else:
            key = (Active_NIS_regulations.iloc[j,:]["Main Control Name"],Active_NIS_regulations.iloc[j,:]["Control Identifier"])
            # if key[0] != Active_NIS_regulations.iloc[i,:]["Main Control Name"]:
            if key[1] != Active_NIS_regulations.iloc[i,:]["Control Identifier"]:
                if key not in regulations_scores:
                    regulations_scores[key] = SaaS_similarities[i,j].item()
                elif SaaS_similarities[i,j].item() > regulations_scores[key]:
                    regulations_scores[key] = SaaS_similarities[i,j].item()

    regulations_scores = dict(sorted(regulations_scores.items(), key=lambda item: item[1], reverse=True))

    out_regulations:int = 0
    for k, v in regulations_scores.items():
        if out_regulations >= min_relations_per_regulation and v <  minimal_related_SaaS_score:
            break
        else:
            related_regulations[k[1]] = v
            if k[0] != Active_NIS_regulations.iloc[i,:]["Main Control Name"]:
                out_regulations += 1

    related_regulations = dict(sorted(related_regulations.items(), key=lambda item: item[1], reverse=True))
    Active_NIS_regulations.at[Active_NIS_regulations.iloc[i,:].name, "SaaS Related Controls"] = related_regulations
    print(f"Regulation {i} ({Active_NIS_regulations.iloc[i,0]}) has related regulations: {related_regulations}")

For that, I needed to determine a threshold for the scores. I checked three main ways for this:

1. A hardcoded threshold — a hardcoded number above which we will say the two regulations are related. The problem with it is that it is highly dependent on the scoring method and is mostly hard to know or estimate.
2. A threshold as a quantile of the scores distribution — taking all the relations at the top P% score. The problem with it is that it makes the number of relations dependent on the regulations (with more regulations, we will get more passing relations per regulation).
3. A threshold as the number of expected relations — calculating a threshold in a way to make the average number of relations per regulation the number we desire. The problem with it is that it is dependent on someone to determine what the desired number of relations per regulation is.

In the end, I chose option 3 because it is easiest to tune to get a similar connections distribution, in order to be as similar to the spirit of the original way when it was humanly done.

In addition, I added one more rule: each regulation/regulation part must have at least a minimum number of relations (default to 1) relating to different regulations/regulations parts that are out of its trivial relations. This rule was added to make the connections graph have fewer connected components, preferably just one connected component of the full connections graph.

Making an automatic pipeline for regulations families detection

In the least time, I made a Jupiter notebook for the analysis process. This time, I want to do it in a more programmatic and reusable way and build a modular pipeline instead.

The pipeline has three main parts:

1. Making new relations columns — this is what I discussed in the previous part.
2. Building the relations graph — this time I have done it in a function and added support for multi-relations sources.
3. Analysing the relations graph for communities detection —similar to the previous analysis, running communities detection on the relations graph and analysing the results, just now written as modular functions.

The first step is what the previous section talked about, making new human-independent relations using NLP.

The second step of building the relations graph was upgraded from last time and now supports multi-relations sources. The multi-relations sources support helps to mitigate biases from single sources and brings multiple angles of view on how to relate regulations.

The third step of analyzing the relations graph for communities detection is now running as a function, and the analysis of the results has been put into a function and upgraded to include balance metrics and better statistics. Another upgrade was implemented here, adding 4 unsupervised clustering measures: **Silhouette Score, [Calinski Harabasz Score](https://en.wikipedia.org/wiki/Calinski%E2%80%93Harabasz_index), [Davies Bouldin Score](https://en.wikipedia.org/wiki/Davies%E2%80%93Bouldin_index), and [Dunn Score](https://en.wikipedia.org/wiki/Dunn_index)**, all of which were slightly modified to work on graphs.

In the end, the full pipeline is as follows:

• For each new connection method, calculate the related regulations for every regulation.
• Choose a combination of connection methods and construct the regulations connection graph accordingly.
• Inspect the regulations connection graph for its connected components
• For the largest connected component (and any other desired large connected components), run all the wanted clustering algorithms.
• Submit each of the new clustering results, along with the default families, to the grading function to obtain all the parameters we want to compare across all the clustering results.
• Return all of the clustering results and their comparison parameters to the user.

diagram of the automatic pipeline for regulations families detection

For testing purposes, I checked the pipeline with the original Related Controls column to see if it gives similar results to the original analysis. The results are similar (to the level of two different runs of the original analysis).

pipeline run on original NIS800–53 connection data, greener is better redder is worst

I also need to emphasize that I have built the pipeline in a way that it can be easily adapted to any other regulation or sets of regulations other than NIS800–53, so similar tests for families/groups correctness/quality can also be done without major changes.

Running the automatic regulations families detection pipeline on the new NLP-discovered related controls

The final step for this time was to run the pipeline I made on the new NLP-discovered related controls. I decided to run it on the combination of all three NLP methods: “normalized bm25 Related Controls”, ”SentenceTransformer Related Controls”, ”SaaS Related Controls”. I chose this to try and maximize the benefits from each of the methods and decrease the biases that any single method can have.

results of the pipeline on the new NLP made related controls, greener is better redder is worst

The result I got where somewhat surprising.

• Similar to the original case with the human-made “Related Controls” column, “Divisive Communities”, “Label propagation”, and “Centrality Communities” give us very unbalanced new families that we don’t want.
• ”Modularity based communities” gives us better balance scores with the new NLP-based relations, but now its statistical connection probabilities scores are lower than the default families, but all of its new unsupervised clustering measures are still better than the default families.
• ”Louvain Communities” statistical connection probabilities scores dropped below that of the default families. But similar to the analysis with the original relations, it still has a better Silhouette Score, Calinski Harabasz Score, and Dunn Score than the default families.
• ”Fluid Communities” is the best performer here, giving better balance scores than the default families, and both its empiric connection probabilities scores and statistical connection probabilities scores are better than the default families and similar to the analysis with the original relations it still has better Silhouette Score, Calinski Harabasz Score, Davies Bouldin Score than the default families.
• Still, none of the methods have recreated the human-made family splits in any significant way.

An Important Notice

I also need to note that in all the cases I have seen I saw that the Silhouette Score was close to 0 and the Davies Bouldin Score was greater than 1. Both are strong indications that the groups are overlapping and that there is no clear separation. This is consistent with the images we get of the connection graphs of a total mess with no clear groups.

Why graph analysis over normal n-dimensional clustering?

• The original analysis used graph analysis, and I wanted to keep the same line with the new NLP-based related controls analysis.
• For some methods, like BM25, it is easier to obtain closely related entities (which are needed for graph analysis) than to obtain a good n-dimensional representation of the entities (which is needed for normal n-dimensional clustering).
• Graph-based methods make it easier to combine multiple different inputs into the clustering analysis (making it easier to combine any number of NLP-based related controls discovery methods we want).
• Graph space can be a non-metric space, and I see it as an advantage because we can say that regulation R1 can be close to regulation R2 because of reason A, and regulation R1 can be close to regulation R3 because of reason B, but regulations R2 and R3 don’t need to be close to each other.

Final Conclusion

To my deep sorrow, my current villain arc has somewhat failed.

In a surprising twist, the default NIS800–53 families performed better when provided with NLP-discovered controls compared to those provided by humans. Is this the beginning of a machine rebellion?

However, this is not the end. The machine uprising has still made its mark, and the process has been mostly automated and upgraded compared to previous attempts, with humans being almost completely pushed out. None of the graph community methods have significantly recreated the human-made family splits, indicating that machines are still distinct from humans.

Despite this, I conclude that the NIS800–53 families are more a product of human thought than a reflection of true connections among all the regulations, and the automated, machine-discovered families should take their place.

Future Work

In this study, I utilized existing clustering algorithms, which performed well and provided better results than the default options. However, these algorithms currently group each regulation into a single family. For future work, we should consider relaxing this restriction, allowing each regulation to be associated with multiple families. This approach would be more logical, as it recognizes that a single regulation can contribute to various topics and, therefore, belong to more than one family.