Skip to content

Commit 9d4baa8

Browse files
Birger SchachtBirger Schacht
authored andcommitted
Shadowserver feed updates
Microsoft Sinkhole Report has been replaced by two reports: Microsoft Sinkhole Events Report and Microsoft Sinkhole HTTP Events Report. Added the config, added the tests, csv files, license files and mentioned the changes in the CHANGELOG. Fixes: #2019
1 parent 103a584 commit 9d4baa8

File tree

9 files changed

+343
-0
lines changed

9 files changed

+343
-0
lines changed

CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,8 @@ CHANGELOG
6262
#### Parsers
6363
- `intelmq.bots.parsers.microsoft.parser_ctip`: Map `Payload.domain` to `destination.fqdn` instead of `extra.payload.domain` as it matches to `destination.ip` from `DestinationIp` (PR#2023 by Sebastian Wagner).
6464
- Removed `intelmq.bots.parsers.malwaredomains` because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
65+
- `intelmq.bots.parsers.shadowserver._config`:
66+
- Add support for the new feeds *Microsoft Sinkhole Events Report*, *Microsoft Sinkhole HTTP Events Report* (PR#2036 by Birger Schacht).
6567

6668
#### Experts
6769
- `intelmq.bots.experts.truncate_by_delimiter.expert`: Cut string if its length is higher than a maximum length (PR#1967 by Marius Karotkis).

docs/user/bots.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1627,6 +1627,8 @@ These are the supported feed name and their corresponding file name for automati
16271627
HTTP-Scanners `hp_http_scan`
16281628
ICS-Scanners `hp_ics_scan`
16291629
IP-Spoofer-Events `event4_ip_spoofer`
1630+
Microsoft-Sinkhole-Events IPv4 `event4_microsoft_sinkhole`
1631+
Microsoft-Sinkhole-Events-HTTP IPv4 `event4_microsoft_sinkhole_http`
16301632
NTP-Monitor `scan_ntpmonitor`
16311633
NTP-Version `scan_ntp`
16321634
Open-Chargen `scan_chargen`

intelmq/bots/parsers/shadowserver/_config.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2849,6 +2849,8 @@ def scan_exchange_identifier(field):
28492849
('IPv6-Sinkhole-HTTP-Drone', 'sinkhole6_http', ipv6_sinkhole_http_drone), # legacy (replaced by event46_sinkhole_http)
28502850
('IP-Spoofer-Events', 'event4_ip_spoofer', event4_ip_spoofer),
28512851
('Microsoft-Sinkhole', 'microsoft_sinkhole', microsoft_sinkhole), # legacy (replaced by event46_sinkhole_http)
2852+
('Microsoft-Sinkhole-Events IPv4', 'event4_microsoft_sinkhole', event46_sinkhole),
2853+
('Microsoft-Sinkhole-Events-HTTP IPv4', 'event4_microsoft_sinkhole_http', event46_sinkhole_http),
28522854
('NTP-Monitor', 'scan_ntpmonitor', ntp_monitor),
28532855
('NTP-Version', 'scan_ntp', ntp_version),
28542856
('Open-Chargen', 'scan_chargen', open_chargen),
Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# SPDX-FileCopyrightText: 2021 Birger Schacht
2+
#
3+
# SPDX-License-Identifier: AGPL-3.0-or-later
4+
5+
# -*- coding: utf-8 -*-
6+
7+
import os
8+
import unittest
9+
10+
import intelmq.lib.test as test
11+
import intelmq.lib.utils as utils
12+
from intelmq.bots.parsers.shadowserver.parser import ShadowserverParserBot
13+
14+
with open(os.path.join(os.path.dirname(__file__),
15+
'testdata/event4_microsoft_sinkhole.csv')) as handle:
16+
EXAMPLE_FILE = handle.read()
17+
EXAMPLE_LINES = EXAMPLE_FILE.splitlines()
18+
19+
EXAMPLE_REPORT = {"feed.name": "ShadowServer Microsoft Sinkhole",
20+
"raw": utils.base64_encode(EXAMPLE_FILE),
21+
"__type": "Report",
22+
"time.observation": "2015-01-01T00:00:00+00:00",
23+
"extra.file_name": "2019-01-01-event4_microsoft_sinkhole.csv",
24+
}
25+
EVENTS = [{'__type': 'Event',
26+
'classification.identifier': 'b68-zeroaccess-2-32bit',
27+
'classification.taxonomy': 'malicious-code',
28+
'classification.type': 'infected-system',
29+
'destination.asn': 8075,
30+
'destination.geolocation.cc': 'HK',
31+
'destination.geolocation.city': 'HONG KONG',
32+
'destination.geolocation.region': 'HONG KONG',
33+
'destination.ip': '168.63.134.179',
34+
'destination.port': 16464,
35+
'extra.destination.naics': 334111,
36+
'extra.destination.sector': 'Information',
37+
'extra.public_source': 'MSDCU',
38+
'extra.source.naics': 517311,
39+
'feed.name': 'ShadowServer Microsoft Sinkhole',
40+
'malware.name': 'b68-zeroaccess-2-32bit',
41+
'protocol.transport': 'tcp',
42+
'source.asn': 7303,
43+
'source.geolocation.cc': 'AR',
44+
'source.geolocation.city': 'CASEROS',
45+
'source.geolocation.region': 'BUENOS AIRES',
46+
'source.ip': '190.229.1.2',
47+
'source.port': 52955,
48+
'time.source': '2021-06-07T00:00:00+00:00',
49+
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
50+
EXAMPLE_LINES[1]])),
51+
},
52+
{'__type': 'Event',
53+
'classification.identifier': 'b68-zeroaccess-2-32bit',
54+
'classification.taxonomy': 'malicious-code',
55+
'classification.type': 'infected-system',
56+
'destination.asn': 8075,
57+
'destination.geolocation.cc': 'IE',
58+
'destination.geolocation.city': 'DUBLIN',
59+
'destination.geolocation.region': 'DUBLIN',
60+
'destination.ip': '52.169.3.4',
61+
'destination.port': 16464,
62+
'extra.destination.naics': 334111,
63+
'extra.destination.sector': 'Information',
64+
'extra.public_source': 'MSDCU',
65+
'extra.source.naics': 517311,
66+
'extra.source.sector': 'Communications, Service Provider, and Hosting Service',
67+
'feed.name': 'ShadowServer Microsoft Sinkhole',
68+
'malware.name': 'b68-zeroaccess-2-32bit',
69+
'protocol.transport': 'tcp',
70+
'source.asn': 5769,
71+
'source.geolocation.cc': 'CA',
72+
'source.geolocation.city': 'LAVAL',
73+
'source.geolocation.region': 'QUEBEC',
74+
'source.ip': '96.20.3.4',
75+
'source.port': 16464,
76+
'time.source': '2021-06-07T00:00:00+00:00',
77+
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
78+
EXAMPLE_LINES[2]])),
79+
},
80+
{'__type': 'Event',
81+
'classification.identifier': 'b68-zeroaccess-2-32bit',
82+
'classification.taxonomy': 'malicious-code',
83+
'classification.type': 'infected-system',
84+
'destination.asn': 8075,
85+
'destination.geolocation.cc': 'HK',
86+
'destination.geolocation.city': 'HONG KONG',
87+
'destination.geolocation.region': 'HONG KONG',
88+
'destination.ip': '168.63.134.179',
89+
'destination.port': 16464,
90+
'extra.destination.naics': 334111,
91+
'extra.destination.sector': 'Information',
92+
'extra.public_source': 'MSDCU',
93+
'extra.source.naics': 517311,
94+
'feed.name': 'ShadowServer Microsoft Sinkhole',
95+
'malware.name': 'b68-zeroaccess-2-32bit',
96+
'protocol.transport': 'tcp',
97+
'source.asn': 8151,
98+
'source.geolocation.cc': 'MX',
99+
'source.geolocation.city': 'MEXICO CITY',
100+
'source.geolocation.region': "CIUDAD DE MEXICO",
101+
'source.ip': '187.222.5.6',
102+
'source.port': 55049,
103+
'time.source': '2021-06-07T00:00:00+00:00',
104+
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
105+
EXAMPLE_LINES[3]])),
106+
},
107+
]
108+
109+
110+
class TestShadowserverParserBot(test.BotTestCase, unittest.TestCase):
111+
"""
112+
A TestCase for a ShadowserverParserBot.
113+
"""
114+
115+
@classmethod
116+
def set_bot(cls):
117+
cls.bot_reference = ShadowserverParserBot
118+
cls.default_input_message = EXAMPLE_REPORT
119+
120+
def test_event(self):
121+
""" Test if correct Event has been produced. """
122+
self.run_bot()
123+
for i, EVENT in enumerate(EVENTS):
124+
self.assertMessageEqual(i, EVENT)
125+
126+
127+
if __name__ == '__main__': # pragma: no cover
128+
unittest.main()
Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
# SPDX-FileCopyrightText: 2021 Birger Schacht
2+
#
3+
# SPDX-License-Identifier: AGPL-3.0-or-later
4+
5+
# -*- coding: utf-8 -*-
6+
7+
import os
8+
import unittest
9+
10+
import intelmq.lib.test as test
11+
import intelmq.lib.utils as utils
12+
from intelmq.bots.parsers.shadowserver.parser import ShadowserverParserBot
13+
14+
with open(os.path.join(os.path.dirname(__file__),
15+
'testdata/event4_microsoft_sinkhole_http.csv')) as handle:
16+
EXAMPLE_FILE = handle.read()
17+
EXAMPLE_LINES = EXAMPLE_FILE.splitlines()
18+
19+
EXAMPLE_REPORT = {'feed.name': 'HTTP Microsoft Sinkhole IPv4',
20+
"raw": utils.base64_encode(EXAMPLE_FILE),
21+
"__type": "Report",
22+
"time.observation": "2015-01-01T00:00:00+00:00",
23+
"extra.file_name": "2019-01-01-event4_microsoft_sinkhole_http.csv",
24+
}
25+
EVENTS = [{'__type': 'Event',
26+
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
27+
'classification.identifier': 'necurs',
28+
'classification.taxonomy': 'malicious-code',
29+
'classification.type': 'infected-system',
30+
'destination.asn': 8075,
31+
'destination.geolocation.cc': 'US',
32+
'destination.geolocation.city': 'ASHBURN',
33+
'destination.geolocation.region': 'VIRGINIA',
34+
'destination.ip': '40.121.206.97',
35+
'destination.port': 80,
36+
'destination.url': 'http://40.121.206.97/locator.php',
37+
'extra.destination.naics': 334111,
38+
'extra.destination.sector': 'Information',
39+
'extra.public_source': 'MSDCU',
40+
'malware.name': 'necurs',
41+
'protocol.application': 'http',
42+
'protocol.transport': 'tcp',
43+
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
44+
EXAMPLE_LINES[1]])),
45+
'source.asn': 8386,
46+
'source.geolocation.cc': 'TR',
47+
'source.geolocation.city': 'KEPEZ',
48+
'source.geolocation.region': 'ANTALYA',
49+
'source.ip': '31.206.1.2',
50+
'source.port': 49245,
51+
'time.observation': '2015-01-01T00:00:00+00:00',
52+
'time.source': '2021-06-07T00:00:00+00:00'},
53+
{'__type': 'Event',
54+
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
55+
'classification.identifier': 'caphaw',
56+
'classification.taxonomy': 'malicious-code',
57+
'classification.type': 'infected-system',
58+
'destination.asn': 8075,
59+
'destination.fqdn': '3fo8jrthz3y.rgk.cc',
60+
'destination.geolocation.cc': 'US',
61+
'destination.geolocation.city': 'REDMOND',
62+
'destination.geolocation.region': 'WASHINGTON',
63+
'destination.ip': '204.95.99.204',
64+
'destination.port': 443,
65+
'destination.url': 'http://3fo8jrthz3y.rgk.cc/index.php',
66+
'extra.destination.naics': 334111,
67+
'extra.destination.sector': 'Information',
68+
'extra.http_agent': 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.6103)',
69+
'extra.http_referer': 'null',
70+
'extra.public_source': 'MSDCU',
71+
'extra.source.naics': 517312,
72+
'malware.name': 'caphaw',
73+
'protocol.application': 'http',
74+
'protocol.transport': 'tcp',
75+
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
76+
EXAMPLE_LINES[2]])),
77+
'source.asn': 28573,
78+
'source.geolocation.cc': 'BR',
79+
'source.geolocation.city': 'SAO PAULO',
80+
'source.geolocation.region': 'SAO PAULO',
81+
'source.ip': '177.140.3.4',
82+
'source.port': 35919,
83+
'time.observation': '2015-01-01T00:00:00+00:00',
84+
'time.source': '2021-06-07T00:00:00+00:00'},
85+
{'__type': 'Event',
86+
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
87+
'classification.identifier': 'necurs',
88+
'classification.taxonomy': 'malicious-code',
89+
'classification.type': 'infected-system',
90+
'destination.asn': 8075,
91+
'destination.geolocation.cc': 'US',
92+
'destination.geolocation.city': 'ASHBURN',
93+
'destination.geolocation.region': 'VIRGINIA',
94+
'destination.ip': '40.121.206.97',
95+
'destination.port': 80,
96+
'destination.url': 'http://40.121.206.97/locator.php',
97+
'extra.destination.naics': 334111,
98+
'extra.destination.sector': 'Information',
99+
'extra.public_source': 'MSDCU',
100+
'extra.source.naics': 517311,
101+
'malware.name': 'necurs',
102+
'protocol.application': 'http',
103+
'protocol.transport': 'tcp',
104+
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
105+
EXAMPLE_LINES[3]])),
106+
'source.asn': 132199,
107+
'source.geolocation.cc': 'PH',
108+
'source.geolocation.city': 'MANDAUE',
109+
'source.geolocation.region': 'CEBU',
110+
'source.ip': '180.190.5.6',
111+
'source.port': 49264,
112+
'time.observation': '2015-01-01T00:00:00+00:00',
113+
'time.source': '2021-06-07T00:00:01+00:00'},
114+
{'__type': 'Event',
115+
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
116+
'classification.identifier': 'necurs',
117+
'classification.taxonomy': 'malicious-code',
118+
'classification.type': 'infected-system',
119+
'destination.asn': 8075,
120+
'destination.geolocation.cc': 'US',
121+
'destination.ip': '40.121.206.97',
122+
'destination.geolocation.city': 'ASHBURN',
123+
'destination.geolocation.region': 'VIRGINIA',
124+
'destination.port': 80,
125+
'destination.url': 'http://40.121.206.97/news/stream.php',
126+
'extra.destination.naics': 334111,
127+
'extra.destination.sector': 'Information',
128+
'extra.public_source': 'MSDCU',
129+
'malware.name': 'necurs',
130+
'protocol.application': 'http',
131+
'protocol.transport': 'tcp',
132+
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
133+
EXAMPLE_LINES[4]])),
134+
'source.asn': 37129,
135+
'source.geolocation.cc': 'KE',
136+
'source.geolocation.city': 'NAIROBI',
137+
'source.geolocation.region': 'NAIROBI CITY',
138+
'source.ip': '197.157.7.8',
139+
'source.port': 55307,
140+
'time.observation': '2015-01-01T00:00:00+00:00',
141+
'time.source': '2021-06-07T00:00:01+00:00'},
142+
{'__type': 'Event',
143+
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
144+
'classification.identifier': 'necurs',
145+
'classification.taxonomy': 'malicious-code',
146+
'classification.type': 'infected-system',
147+
'destination.asn': 8075,
148+
'destination.geolocation.cc': 'US',
149+
'destination.geolocation.city': 'ASHBURN',
150+
'destination.geolocation.region': 'VIRGINIA',
151+
'destination.ip': '40.121.206.97',
152+
'destination.port': 80,
153+
'destination.url': 'http://40.121.206.97/locator.php',
154+
'extra.destination.naics': 334111,
155+
'extra.destination.sector': 'Information',
156+
'extra.public_source': 'MSDCU',
157+
'extra.source.naics': 517311,
158+
'extra.source.sector': 'Communications, Service Provider, and Hosting Service',
159+
'malware.name': 'necurs',
160+
'protocol.application': 'http',
161+
'protocol.transport': 'tcp',
162+
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
163+
EXAMPLE_LINES[5]])),
164+
'source.asn': 812,
165+
'source.geolocation.cc': 'CA',
166+
'source.geolocation.city': 'OTTAWA',
167+
'source.geolocation.region': 'ONTARIO',
168+
'source.ip': '174.114.9.10',
169+
'source.port': 59000,
170+
'time.observation': '2015-01-01T00:00:00+00:00',
171+
'time.source': '2021-06-07T00:00:01+00:00'}]
172+
173+
174+
class TestShadowserverParserBot(test.BotTestCase, unittest.TestCase):
175+
"""
176+
A TestCase for a ShadowserverParserBot.
177+
"""
178+
179+
@classmethod
180+
def set_bot(cls):
181+
cls.bot_reference = ShadowserverParserBot
182+
cls.default_input_message = EXAMPLE_REPORT
183+
184+
def test_event(self):
185+
""" Test if correct Event has been produced. """
186+
self.run_bot()
187+
for i, EVENT in enumerate(EVENTS):
188+
self.assertMessageEqual(i, EVENT)
189+
190+
191+
if __name__ == '__main__': # pragma: no cover
192+
unittest.main()
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
"timestamp","protocol","src_ip","src_port","src_asn","src_geo","src_region","src_city","src_hostname","src_naics","src_sector","device_vendor","device_type","device_model","dst_ip","dst_port","dst_asn","dst_geo","dst_region","dst_city","dst_hostname","dst_naics","dst_sector","public_source","infection","family","tag","application","version","event_id"
2+
"2021-06-07 00:00:00","tcp","190.229.1.2",52955,7303,"AR","BUENOS AIRES","CASEROS",,517311,,,,,"168.63.134.179",16464,8075,"HK","HONG KONG","HONG KONG",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
3+
"2021-06-07 00:00:00","tcp","96.20.3.4",16464,5769,"CA","QUEBEC","LAVAL",,517311,"Communications, Service Provider, and Hosting Service",,,,"52.169.3.4",16464,8075,"IE","DUBLIN","DUBLIN",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
4+
"2021-06-07 00:00:00","tcp","187.222.5.6",55049,8151,"MX","CIUDAD DE MEXICO","MEXICO CITY",,517311,,,,,"168.63.134.179",16464,8075,"HK","HONG KONG","HONG KONG",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
5+
"2021-06-07 00:00:00","tcp","75.84.7.8",64190,20001,"US","CALIFORNIA","NORTH HOLLYWOOD",,517311,"Communications, Service Provider, and Hosting Service",,,,"52.169.7.8",16464,8075,"IE","DUBLIN","DUBLIN",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
6+
"2021-06-07 00:00:00","tcp","24.15.9.10",60373,7922,"US","ILLINOIS","HOMER GLEN",,517311,"Communications, Service Provider, and Hosting Service",,,,"104.40.6.5",16464,8075,"US","CALIFORNIA","SAN FRANCISCO",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
7+
"2021-06-07 00:00:00","tcp","124.101.11.12",50386,4713,"JP","FUKUOKA","FUKUOKA",,517311,"Communications, Service Provider, and Hosting Service",,,,"23.99.101.165",16465,8075,"HK","HONG KONG","HONG KONG",,334111,"Information","MSDCU","b68-zeroaccess-2-64bit",,"b68-zeroaccess-2-64bit",,
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
SPDX-FileCopyrightText: 2021 Birger Schacht
2+
SPDX-License-Identifier: AGPL-3.0-or-later
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"timestamp","protocol","src_ip","src_port","src_asn","src_geo","src_region","src_city","src_hostname","src_naics","src_sector","device_vendor","device_type","device_model","dst_ip","dst_port","dst_asn","dst_geo","dst_region","dst_city","dst_hostname","dst_naics","dst_sector","public_source","infection","family","tag","application","version","event_id","http_url","http_host","http_agent","forwarded_by","ssl_cipher","http_referer"
2+
"2021-06-07 00:00:00","tcp","31.206.1.2",49245,8386,"TR","ANTALYA","KEPEZ",,,,,,,"40.121.206.97",80,8075,"US","VIRGINIA","ASHBURN",,334111,"Information","MSDCU","necurs",,"necurs",,,,"/locator.php","40.121.206.97",,,,
3+
"2021-06-07 00:00:00","tcp","177.140.3.4",35919,28573,"BR","SAO PAULO","SAO PAULO",,517312,,,,,"204.95.99.204",443,8075,"US","WASHINGTON","REDMOND",,334111,"Information","MSDCU","caphaw",,"caphaw",,,,"/index.php","3fo8jrthz3y.rgk.cc","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.6103)",,,"null"
4+
"2021-06-07 00:00:01","tcp","180.190.5.6",49264,132199,"PH","CEBU","MANDAUE",,517311,,,,,"40.121.206.97",80,8075,"US","VIRGINIA","ASHBURN",,334111,"Information","MSDCU","necurs",,"necurs",,,,"/locator.php","40.121.206.97",,,,
5+
"2021-06-07 00:00:01","tcp","197.157.7.8",55307,37129,"KE","NAIROBI CITY","NAIROBI",,,,,,,"40.121.206.97",80,8075,"US","VIRGINIA","ASHBURN",,334111,"Information","MSDCU","necurs",,"necurs",,,,"/news/stream.php","40.121.206.97",,,,
6+
"2021-06-07 00:00:01","tcp","174.114.9.10",59000,812,"CA","ONTARIO","OTTAWA",,517311,"Communications, Service Provider, and Hosting Service",,,,"40.121.206.97",80,8075,"US","VIRGINIA","ASHBURN",,334111,"Information","MSDCU","necurs",,"necurs",,,,"/locator.php","40.121.206.97",,,,
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
SPDX-FileCopyrightText: 2021 Birger Schacht
2+
SPDX-License-Identifier: AGPL-3.0-or-later

0 commit comments

Comments
 (0)