©2019 Security Unleashed | New Delhi

  • Animesh Gupta

Photon - Incredibly Fast Crawler

Updated: Aug 1, 2018



Photon is a lightning fast web crawler which extracts URLs, files, intel & endpoints from a target.

Yep, you can use 100 threads and Photon won't complain about it because its in Ninja Mode.


Why Photon?


Not Your Regular Crawler


Crawlers are supposed to recursively extract links right? Well that's kind of boring so Photon goes beyond that. It extracts the following information:

URLs (in-scope & out-of-scope)

URLs with parameters (example.com/gallery.php?id=2)

Intel (emails, social media accounts, amazon buckets etc.)

Files (pdf, png, xml etc.)

JavaScript files & Endpoints present in them

Strings based on custom regex pattern


The extracted information is saved in an organized manner.



Intelligent Multithreading


Here's a secret, most of the tools floating on the internet aren't properly multi-threaded even if they are supposed to. They either supply a list of items to threads which results in multiple threads accessing the same item or they simply put a thread lock and end up rendering multi-threading useless.

Ninja Mode

In Ninja Mode, 3 online services are used to make requests to the target on your behalf. So basically, now you have 4 clients making requests to the same server simultaneously which gives you a speed boost if you have a slow connection, minimizes the risk of connection reset as well as delays requests from a single client.



Usage

-u --url


Run Photon against a single website. photon -u http://cybertechops.com


Specifying a URL with it's schema i.e. http(s):// is optional but you must add www. if the website has it.


-l --level


Depth of crawling.

photon -u http://cybertechops.com -l 3


-d --delay


You can keep a delay between requests made to the target by specifying the time in seconds.

photon -u http://cybertechops.com -d 1


-t --threads


Number of threads to use.

photon -u http://cybertechops.com -t 10


-c --cookie


Cookie to send

photon -u http://cybertechops.com -c "PHPSESSID=931b431d2"


-n --ninja


Toggles Ninja Mode on/off.

photon -u http://cybertechops.com --ninja


--dns


Create an image displaying target domain's DNS data.

photon -u http://cybertechops.com --dns


-s --seeds

Lets you add custom seeds, seperated by commas.

photon -u http://cybertechops.com -s "https://www.cybertechops.com/blog"


-r --regex

Specify custom regex pattern to extract strings.

photon -u http://cybertechops.com -r "\d{10}"


PluginsPhoton's capabilites can be further extended by using plugins.

Available plugins:

  • dnsdumpster: Generates an image containing the DNS data of the target domain.

  • Exporter: Plugin to export results in JSON, support for more formats is being worked on.

Plugins in active development:

  • Quark: A plugin to plot a graph making it easier to inspect relationships between different webpages using Quark.

  • XSStrike: Modular & targeted version of XSStrike to be used with Photon.

  • dnsdumpster: A new version of the plugin is in development which will save the DNS data in a nicely formatted HTML file.


Download Photon