samuel
/
scraper
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2 Commits 1 Branch 0 Tags 29 MiB
Go 100%
Go to file
Samuel Campos 98d3cbe7e2 Improve updateNews function 
Start developping updateSchedule function
Use of go modules
 		2020-10-10 19:08:06 +02:00
match Improve updateNews function 2020-10-10 19:08:06 +02:00
news Improve updateNews function 2020-10-10 19:08:06 +02:00
postgres Improve updateNews function 2020-10-10 19:08:06 +02:00
requests Improve updateNews function 2020-10-10 19:08:06 +02:00
utils Improve updateNews function 2020-10-10 19:08:06 +02:00
.gitignore Initial Commit 2020-10-05 10:24:33 +02:00
go.mod Improve updateNews function 2020-10-10 19:08:06 +02:00
go.sum Improve updateNews function 2020-10-10 19:08:06 +02:00
main.go Improve updateNews function 2020-10-10 19:08:06 +02:00
readme.md Initial Commit 2020-10-05 10:24:33 +02:00
scraper Improve updateNews function 2020-10-10 19:08:06 +02:00

readme.md

1- INTRODUCTION

Scraper is a project made in Golang to scrap different types of data about sports, such as :

  • all news about sports
  • schedules and scores [coming soon]
  • team staff [coming soon]
  • details about players [coming soon]
  • tv schedule [coming soon]

For that, different technologies have been used :

  • Golang (with go routines, goquery, gofeed, net/proxy)
  • PostgreSQL (used to store data)
  • InfluxDB (used to store details about each program execution) [coming soon]

Several websites are stored in order to gather different types of data :

  • eurosport.fr
  • rugbyrama.fr
  • fftt.com
  • footao.tv [coming soon]
  • football.fr [coming soon]
  • football365.fr [coming soon]
  • football-direct.com [coming soon]
  • footmercato.net
  • lequipe.fr
  • matchendirect.fr [coming soon]
  • programme-television.org [coming soon]
  • transfermarkt.fr [coming soon]

All these data are collected in nonprofit purpose for 1bet.fr, a website made for free sports predictions between friends.

I decline any responsibility about your eventual usages of this project.

## 2- DEPLOYMENT

The deployment is very simple as the binary scraper can be used directly.

A PostgresSQL database is needed for this program, as well as some environment variables, all clearly listed in postgres.go.