Semantic Text Similarity for Cybersecurity Maturity Assessment

As part of Cybersecurity Maturity Assessment assignment, I attempted to perform it using Semantic text similarity across multiple documents. The results were quite promising and provided some interesting key lessons.

Objective: Conduct cyber security maturity assessment(IKAS) BSSN 2025

Challenge: It requires to map hundreds of clause with relevant existing system policy as evidence of framework assessment

Task: Run Semantic Analysis between IKAS framework with various system policy documents (one-to-many approach). Collect relevant evidence of system implementation.

Input Data: 180 clauses of cyber security maturity framework

Sources: PDF documents containing company policies regarding information security systems and other relevant materials

Pre-processing: PDF Miner to chunk sentences based on topic policy.

Model: Indobert-large-p2. The pretrained BERT model uses indonesian based language. The model is trained using a masked language modeling (MLM) objective and next sentence prediction (NSP) objective

Cyber ​​Security Maturity Assessment Instrument (IKAS) 2025

In 2025, Badan Siber & Sandi Negara (BSSN) Indonesia has released a new Cybersecurity Maturity framework called Cyber ​​Security Maturity Assessment Instrument (IKAS).

IKAS framework with four assessment domains

Cyber ​​Security Maturity Assessment Instrument (IKAS) consist of 4 domains. The following is an explanation of each domain and the weighting distributions that can be used:

1. Identification (20–25%): Identification includes asset inventory, risk assessment, and threat analysis and is a critical initial step in identifying what needs to be protected and understanding potential risks. Identification is used for strategic planning.
2. Protection (30–35%): Protection focuses on implementing controls to protect systems and data from threats, including access controls, encryption, and security policies. Investing in robust protection can reduce the frequency and impact of incidents.
3. Detection (20–25%): Detection involves monitoring, log analysis, and the use of tools to identify suspicious activity or breaches. Detection aims to ensure that threats are identified as soon as they emerge and before they cause significant damage.
4. Mitigation and Recovery (20–25%): Incident response encompasses responding to and mitigating the impact of detected incidents, including isolating affected systems, repairing, and recovering. The goal is to minimize the impact of incidents that have already occurred and to ensure a rapid and effective recovery.

Semantic Analysis

Textual similarity is an NLP task which provides a quantitative value to the similarity between two text embeddings. As the name suggests, semantic textual similarity (STS) is a quantitative comparison between the semantic meanings of two text embeddings and can be calculated using cosine similarity.

Methodology

An overview of the overall algorithm framework

Split PDF documents into datafarme with PDFMiner

I used PDFMiner to automate document chunking based on topic on each complay system policies. Here an example of document split based on topic policy.

Example of document separation based on discussion topics

Embedding

In this section, the document is tokenized into chunks. Then, the maximum chunk length is determined. Each group of chunks is aggregated using average aggregation. This means the average score of each word vector in the chunk is calculated.

Pairwise Cosine similarity matrix

Next, the program calculates similarity scores between documents. The calculation is performed pairwise, comparing one document group with another. The longer the documents, the larger the resulting matrix.

Post-processing results

I converted the resulting matrix into a dataframe. I used the reference ID as the dataframe index, sorted it descending, and collected the top-3 documents with the highest similarity. At last, I used human justification to determine which documents were most relevant.

Example of results of 3 documents with the highest similarity

Note: Semantic text similarity works based on context similarity, if the input contains too much unnecessary information, it still adds value and contributes to a higher score.

Model evaluation

To evaluate the model perfomance, I compare the mapping result with the actual human expert assessment result. The result show fairly high similarity with accuracy on 83%.

Model evaluation results

Results

After collecting all the top n documents with the highest similarity scores, we proceed with a justification decision to determine which documents are most relevant to the regulatory clause. This approach helps us navigate the selection of relevant documents through a semantic similarity. Finally, we need to use human justification to determine the level scores within the IKAS framework and represent the result as spider chart.

Example of Cyber ​​Security Maturity Level assessment results using a spider chart

Limitation

During this process, I realized there were still limitations to this approach. The model might fail to distinguish documents effectively, resulting in similar scores across multiple documents. At this point, the top n documents might not represent the most relevant documents we were looking for. I flagged this situation and manually searched for relevant documents.

Another point is that the model will still generate similarity scores even if there is no relevant information within the documents. This occurs when the topic lacks relevant documents for comparison. This situation also needs to be addressed and flagged.

Further works

• Apply cascade IR: Before implementing transformer model, I could use relatively simpler model like DPR, TILDE v2, etc.
• Apply Re-ranker with cross directional model
• Apply metadata for filtering or add flag information
• Store embedding dataset into vector DB, like Chroma, Postgre (pgvector)

Closing Remark

This publication is produced for educational or information only, If there are any mistake in data, judgement, or methodology that I used to produce this publication.

• * Please consider to contact the writer using contact information at Profile. I would like to discuss and sharing more about the topic. Thank you.

Best Regards,

Andriyan Saputra