Name		Name	Last commit message	Last commit date
.gitignore		.gitignore
README.md		README.md
configure.go		configure.go
crawl_page.go		crawl_page.go
get_html.go		get_html.go
get_urls_from_html.go		get_urls_from_html.go
get_urls_from_html_test.go		get_urls_from_html_test.go
go.mod		go.mod
go.sum		go.sum
main.go		main.go
normalize_url.go		normalize_url.go
normalize_url_test.go		normalize_url_test.go
print_report.go		print_report.go
print_report_test.go		print_report_test.go

Repository files navigation

Go Webcrawler

Usage

make sure that you have Go installed
run go build -o crawler && ./crawler <URL> <maxConcurrency> <maxPages>

Description

This webcrawler fetches all internal links on a webpage. It uses concurrency to be faster and can be limited to a max number of pages.

About

Webcrawler written in Go

0 stars

1 watching

Report repository

Releases

Packages

Used by

Contributors

Languages