Contents

ATT&CK DS Event Mappings Notebook

Importing Python Libraries

# Importing library to manipulate data
import pandas as pd

# Importing library to manipulate yaml data
import yaml
import requests

Importing Data Sources Mapping Yaml File

yamlUrl = 'https://raw.githubusercontent.com/OTRF/OSSEM-DM/main/attack_event_mapping/_all_attack_ds_mappings.yml'
yamlContent = requests.get(yamlUrl)
yamlMapping = yaml.safe_load(yamlContent.text)
mapping = pd.json_normalize(yamlMapping)
mapping.head()
name definition collection_layers platforms contributors data_components references
0 User Account Security principal or entity that represents a... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [{'name': 'user account creation', 'type': 'ac... [https://docs.microsoft.com/en-us/windows/secu...
1 Schedule Task Information about scheduled work that the Task... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [{'name': 'schedule task creation', 'type': 'a... [https://docs.microsoft.com/en-us/windows/win3...
2 WMI object Information about objects from the system clas... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [{'name': 'wmi object context', 'type': 'infor... [https://docs.microsoft.com/en-us/windows/win3...
3 Process Information about instances of computer progra... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [{'name': 'process context', 'type': 'informat... [https://docs.microsoft.com/en-us/windows/win3...
4 Module Information about portable executable files, s... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [{'name': 'module load', 'type': 'activity', '... [https://docs.microsoft.com/en-us/windows/win3...

Preparing Data Sources Mapping Dataframe

# Splitting rows for data_components list
dcListExp = mapping.explode('data_components').reset_index(drop=True).rename(columns={'name':'data_sources'})

# Splitting columns for data_components dict
dcDictExp = dcListExp['data_components'].apply(pd.Series).merge(dcListExp,left_index=True,right_index=True)\
.reset_index(drop=True).rename(columns={'name':'components'}).drop(['data_components'], axis = 1)

# Splitting rows for relationships list
drListExp = dcDictExp.explode('relationships').reset_index(drop=True)

# Splitting columns for relationships dict
drDictExp = drListExp['relationships'].apply(pd.Series).merge(drListExp,left_index=True,right_index=True)\
.reset_index(drop=True).rename(columns={'name':'data_relationships'}).drop(['relationships'], axis = 1)

# Splitting rows for telemetry list
tListExp = drDictExp.explode('telemetry').reset_index(drop=True)

# Splitting columnes for telemetry dict
tDictExp = tListExp['telemetry'].apply(pd.Series).merge(tListExp,left_index=True,right_index=True)\
.reset_index(drop=True).drop(['telemetry'], axis = 1)

# Splitting rows for event_id list
dataSourcesMapping = tDictExp.explode('event_id').reset_index(drop=True)

dataSourcesMapping.head()
log_provider log_channel audit_category audit_sub_category event_id event_name data_relationships id source_data_element relationship target_data_element components type data_sources definition collection_layers platforms contributors references
0 Microsoft-Windows-Security-Auditing Security Account Management User Account Management 4720 A user account was created. User created User 0C55D5FB-1FCC-454B-9C4A-0E72116D4BE3 user created user user account creation activity User Account Security principal or entity that represents a... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [https://docs.microsoft.com/en-us/windows/secu...
1 Microsoft-Windows-Security-Auditing Security Account Management User Account Management 4726 A user account was deleted. User deleted User B215E53A-D35F-4ED2-BBFE-9516C30B9F96 user deleted user user account deletion activity User Account Security principal or entity that represents a... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [https://docs.microsoft.com/en-us/windows/secu...
2 Microsoft-Windows-Security-Auditing Security Account Management User Account Management 4722 A user account was enabled. User enabled User B3F266ED-866B-4263-8551-C0FBCE341805 user enabled user user account enable activity User Account Security principal or entity that represents a... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [https://docs.microsoft.com/en-us/windows/secu...
3 Microsoft-Windows-Security-Auditing Security Account Management User Account Management 4725 A user account was disabled. User disabled User 2B984BA0-4788-4FAA-8E10-F6C1DC07C35E user disabled user user account disable activity User Account Security principal or entity that represents a... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [https://docs.microsoft.com/en-us/windows/secu...
4 Microsoft-Windows-Security-Auditing Security Account Management User Account Management 4740 A user account was locked out. User locked User 8F03BCE0-7EAB-41BA-976F-FBA406412A16 user locked user user account lock activity User Account Security principal or entity that represents a... [host] [Windows] [Jose Rodriguez @Cyb3rPandaH, Roberto Rodrigue... [https://docs.microsoft.com/en-us/windows/secu...

Visualizing Relationships Among Data Elements: Network Graph

  • Getting dataframe of relationships among data elements

relationships = dataSourcesMapping[['source_data_element','target_data_element','relationship']]\
      .drop_duplicates()\
      .dropna().reset_index(drop = True)\
      .rename(columns={'source_data_element':'source','target_data_element':'target'})\
      .replace(['windows registry key','windows registry key value'],['registry key','registry key value'])
relationships.head()
source target relationship
0 user user created
1 user user deleted
2 user user enabled
3 user user disabled
4 user user locked

  • Importing libraries

import networkx as nx

from bokeh.io import output_notebook, show
from bokeh.models import (BoxSelectTool,LassoSelectTool,PanTool,BoxZoomTool,ResetTool,TapTool,
                          WheelZoomTool,HoverTool,Circle, MultiLine,Plot, Range1d,ColumnDataSource,
                          PointDrawTool,LabelSet,EdgesAndLinkedNodes, NodesAndLinkedEdges,HoverTool)
from bokeh.palettes import Spectral5
from bokeh.plotting import from_networkx

output_notebook()
Loading BokehJS ...

  • Visualizing network graph

# Creating networkx graph object
G = nx.from_pandas_edgelist(relationships, source = 'source', target = 'target',edge_attr = 'relationship')

# Creating plot object
plot = Plot(plot_width=800, plot_height=500,
            x_range=Range1d(-1.1, 1.1), y_range=Range1d(-1.1, 1.1))
plot.title.text = "Relationships among Data Elements"
plot.title.align = "center"
plot.title.text_font_size = "30px"

# Creating renderer objects
graph_renderer = from_networkx(G, nx.spring_layout, scale=1, center=(0, 0))

graph_renderer.node_renderer.glyph = Circle(size=60, fill_color=Spectral5[0])
graph_renderer.node_renderer.selection_glyph = Circle(size=15, fill_color=Spectral5[3])
graph_renderer.node_renderer.hover_glyph = Circle(size=15, fill_color=Spectral5[1])

graph_renderer.edge_renderer.glyph = MultiLine(line_color="gray", line_alpha=0.9, line_width=1)
graph_renderer.edge_renderer.selection_glyph = MultiLine(line_color=Spectral5[3], line_width=4)
graph_renderer.edge_renderer.hover_glyph = MultiLine(line_color=Spectral5[1], line_width=4)

graph_renderer.selection_policy = NodesAndLinkedEdges()
graph_renderer.inspection_policy = NodesAndLinkedEdges()
plot.renderers.append(graph_renderer)

# Adding nodes labels
x,y = zip(*graph_renderer.layout_provider.graph_layout.values())

names = []
for i in nx.nodes(G): names.append(i) 
    
nodes_table = ColumnDataSource({'x': x, 'y': y,'name': names})
labels = LabelSet(x='x',y='y',text='name',render_mode='canvas',y_offset=-10,text_align='center',
                  text_font_size='12px',source = nodes_table, background_fill_color=None,text_color = 'black')

plot.renderers.append(labels)

# Adding tools
plot.add_tools(HoverTool(tooltips=None),WheelZoomTool(),TapTool(),BoxSelectTool(),BoxZoomTool(),
               PanTool(),LassoSelectTool(),ResetTool())

# Showing graph
show(plot)

Mapping (sub)techniques to data components & relationships & event logs

Gathering ATT&CK content

  • Importing libraries

# Importing library to interact with up to date ATT&CK content available in STIX format via public TAXII server
from attackcti import attack_client

  • Gathering (sub)techniques

# Instantiating attack_client class
lift = attack_client()

# Collecting all techniques (Revoked and not revoked) for windows platform within the enterprise matrix
attck = lift.get_techniques_by_platform(name = 'Windows', stix_format = False)

# Removing revoked techniques
attck = lift.remove_revoked(attck)

# Generating a dataframe with information collected
attck = pd.json_normalize(attck)

# Selecting columns
attck = attck[['tactic','technique_id','technique','data_sources']]

# Showing information collected
attck.head()
tactic technique_id technique data_sources
0 [credential-access, collection] T1557.002 ARP Cache Poisoning [Packet capture, Netflow/Enclave netflow]
1 [persistence, privilege-escalation] T1547.012 Print Processors [Process monitoring, Windows Registry, File mo...
2 [defense-evasion] T1564.007 VBA Stomping [Process monitoring, File monitoring]
3 [credential-access] T1558.004 AS-REP Roasting [Windows event logs, Authentication logs]
4 [defense-evasion] T1218.012 Verclsid [Process use of network, Process command-line ...

  • Splitting data_sources field

attck = attck.explode('data_sources').reset_index(drop=True)
attck.head()
tactic technique_id technique data_sources
0 [credential-access, collection] T1557.002 ARP Cache Poisoning Packet capture
1 [credential-access, collection] T1557.002 ARP Cache Poisoning Netflow/Enclave netflow
2 [persistence, privilege-escalation] T1547.012 Print Processors Process monitoring
3 [persistence, privilege-escalation] T1547.012 Print Processors Windows Registry
4 [persistence, privilege-escalation] T1547.012 Print Processors File monitoring

  • Updating data_sources names

yamlNamesUrl = 'https://raw.githubusercontent.com/OTRF/OSSEM-DM/main/docs/mitre_attack/_new_data_sources_names.yaml'
yamlNamesContent = requests.get(yamlNamesUrl)
names = yaml.safe_load(yamlNamesContent.text)
names = pd.DataFrame(list(names[0].items()), columns=['data_sources', 'new_names'])

attck = pd.merge(attck, names, on = 'data_sources', how = 'left')
namesUpdate = attck['new_names'].fillna(attck['data_sources'])
attck = attck.drop(columns = ['new_names']).assign(data_sources = namesUpdate)
attck = attck.drop_duplicates(subset = ['technique_id','data_sources'])\
        .sort_values(by = ['technique'])\
        .reset_index(drop = True)
attck.head()
tactic technique_id technique data_sources
0 [credential-access, collection] T1557.002 ARP Cache Poisoning Packet capture
1 [credential-access, collection] T1557.002 ARP Cache Poisoning Netflow/Enclave netflow
2 [credential-access] T1558.004 AS-REP Roasting Logon session
3 [credential-access] T1558.004 AS-REP Roasting Windows event logs
4 [privilege-escalation, defense-evasion] T1548 Abuse Elevation Control Mechanism Windows registry

  • Mapping techniques to data components & relationships * event logs

techniques = pd.merge(attck, dataSourcesMapping, on = 'data_sources', how = 'left')\
             [['tactic','technique_id','technique','data_sources','components','data_relationships',\
               'log_provider','event_id']]

techniques.head()
tactic technique_id technique data_sources components data_relationships log_provider event_id
0 [credential-access, collection] T1557.002 ARP Cache Poisoning Packet capture NaN NaN NaN NaN
1 [credential-access, collection] T1557.002 ARP Cache Poisoning Netflow/Enclave netflow NaN NaN NaN NaN
2 [credential-access] T1558.004 AS-REP Roasting Logon session logon session context Logon session modified Microsoft-Windows-Security-Auditing 4672.0
3 [credential-access] T1558.004 AS-REP Roasting Logon session logon session creation User requested creation Logon session Microsoft-Windows-Security-Auditing 4648.0
4 [credential-access] T1558.004 AS-REP Roasting Logon session logon session creation User requested logon session creation from Ip Microsoft-Windows-Security-Auditing 4648.0

Use Case: SMB/Windows Admin Shares (T1021.002)

  • What are the recommended data sources?

attck[attck['technique_id']=='T1021.002']
tactic technique_id technique data_sources
1091 [lateral-movement] T1021.002 SMB/Windows Admin Shares Logon session
1092 [lateral-movement] T1021.002 SMB/Windows Admin Shares Process

  • What are the potential event logs that we could consider in our analysis?

techniques[techniques['technique_id']=='T1021.002']
tactic technique_id technique data_sources components data_relationships log_provider event_id
17642 [lateral-movement] T1021.002 SMB/Windows Admin Shares Logon session logon session context Logon session modified Microsoft-Windows-Security-Auditing 4672.0
17643 [lateral-movement] T1021.002 SMB/Windows Admin Shares Logon session logon session creation User requested creation Logon session Microsoft-Windows-Security-Auditing 4648.0
17644 [lateral-movement] T1021.002 SMB/Windows Admin Shares Logon session logon session creation User requested logon session creation from Ip Microsoft-Windows-Security-Auditing 4648.0
17645 [lateral-movement] T1021.002 SMB/Windows Admin Shares Logon session logon session creation User requested logon session creation from Port Microsoft-Windows-Security-Auditing 4648.0
17646 [lateral-movement] T1021.002 SMB/Windows Admin Shares Logon session logon session creation User created Logon session Microsoft-Windows-Security-Auditing 4624.0
... ... ... ... ... ... ... ... ...
17708 [lateral-movement] T1021.002 SMB/Windows Admin Shares Process process network connection Process bound to Port Microsoft-Windows-Security-Auditing 5158.0
17709 [lateral-movement] T1021.002 SMB/Windows Admin Shares Process process network connection Host blocked local port bind on Process Microsoft-Windows-Security-Auditing 5159.0
17710 [lateral-movement] T1021.002 SMB/Windows Admin Shares Process process network connection Host blocked local port bind on Ip Microsoft-Windows-Security-Auditing 5159.0
17711 [lateral-movement] T1021.002 SMB/Windows Admin Shares Process process network connection Host blocked local port bind on Port Microsoft-Windows-Security-Auditing 5159.0
17712 [lateral-movement] T1021.002 SMB/Windows Admin Shares Process process network connection Process attempted to bind on Port Microsoft-Windows-Security-Auditing 5159.0

71 rows × 8 columns

  • Contribution opportunity identified after testing the framework!!

The Mordor project contains a dataset that represents a variation of the SMB/Windows Admin Shares (T1021.002) sub-technique.

Offensive tradecraft: Adversaries leverage SMB to copy files over the network to either execute code remotely or exfiltrate data.

Initial detection strategy: Using the data sources recommended by the ATT&CK team, we were able to get some context related to the adversary behavior from a Procees and User Authentication perspective. However, more context around the manipulation or creation of files might be helpful to define a better detection strategy.

Contribution opportunity: Security event logs such as 5140 (A network share object was accessed) and 5145 (A network share object was checked) from Windows Security Auditing and 11 (File created) from Sysmon can help us with the conext required to define a detection strategy for this specific sub-technqiue variation. Therefore, adding the File data source object (file share access & file access data components) to the recommended data sources for this sub-technique might be a good idea.

dataSourcesMapping[['data_sources','components','data_relationships','log_provider','event_id']][(dataSourcesMapping['data_sources'] == 'File') & \
                   ((dataSourcesMapping['event_id'] == 5140) |\
                    (dataSourcesMapping['event_id'] == 5145) |\
                    (dataSourcesMapping['event_id'] == 11))]
data_sources components data_relationships log_provider event_id
87 File file creation Process created File Microsoft-Windows-Sysmon 11
92 File file access User accessed File Microsoft-Windows-Security-Auditing 5145
103 File file modification Process modified File Microsoft-Windows-Sysmon 11
108 File file share access User accessed File Share Microsoft-Windows-Security-Auditing 5140

I hope you find this notebook helpful!! :D

Data Model Relationships