selecrawler

Simple utility kit for Java like web crawler using selenium and chrome.

About robots rule

You will need to learn robots rule if you want to create web crawler.

You can find robots rule link below:

• https://en.wikipedia.org/wiki/Robots_exclusion_standard

Note Maven

mvn archetype:generate -DgroupId=jp.igapyon.selecrawler -DartifactId=selecrawler -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false