mirror of
https://git.FreeBSD.org/ports.git
synced 2024-11-25 00:51:21 +00:00
9ff6a0ecc0
Despite the name, the crawler gets info for more services than just DNS:
- DNS:
- all A/AAAA records (for the 2nd level domain and www.subdomain), annotated
with GeoIP
- TXT records (with SPF and DMARC parsed for easier filtering)
- TLSA (for the 2nd level domain and www.subdomain)
- MX
- DNSSEC validation
- nameservers:
- each server IP annotated with GeoIP
- HOSTNAME.BIND, VERSION.BIND, AUTHORS.BIND and fortune (also for all IPs)
- users can add custom additional RRs in the config file
- E-mail (for every server from MX):
- SMTP server banners (optional, ports are configurable)
- TLSA records
- Web:
- HTTP status & headers (inc. parsed cookies) for ports 80 & 443 on each IP
from A/AAAA records
- certificate info for HTTPS (optionally with an entire cert chain)
- webpage content (optional)
- everything of the above is saved for each step in the redirect history --
the crawler follows redirects until it gets a non-redirecting status or hits
a configurable limit
WWW: https://gitlab.nic.cz/adam/dns-crawler
26 lines
1.0 KiB
Plaintext
26 lines
1.0 KiB
Plaintext
Despite the name, the crawler gets info for more services than just DNS:
|
|
- DNS:
|
|
- all A/AAAA records (for the 2nd level domain and www.subdomain), annotated
|
|
with GeoIP
|
|
- TXT records (with SPF and DMARC parsed for easier filtering)
|
|
- TLSA (for the 2nd level domain and www.subdomain)
|
|
- MX
|
|
- DNSSEC validation
|
|
- nameservers:
|
|
- each server IP annotated with GeoIP
|
|
- HOSTNAME.BIND, VERSION.BIND, AUTHORS.BIND and fortune (also for all IPs)
|
|
- users can add custom additional RRs in the config file
|
|
- E-mail (for every server from MX):
|
|
- SMTP server banners (optional, ports are configurable)
|
|
- TLSA records
|
|
- Web:
|
|
- HTTP status & headers (inc. parsed cookies) for ports 80 & 443 on each IP
|
|
from A/AAAA records
|
|
- certificate info for HTTPS (optionally with an entire cert chain)
|
|
- webpage content (optional)
|
|
- everything of the above is saved for each step in the redirect history --
|
|
the crawler follows redirects until it gets a non-redirecting status or hits
|
|
a configurable limit
|
|
|
|
WWW: https://gitlab.nic.cz/adam/dns-crawler
|