Fetch
Documentation for Fetch.
Quick start
The package can be installed with the Julia package manager. From the Julia REPL, type ] to enter the Pkg REPL mode and run:
pkg> add Fetch
Download file from Google drive
Download file or Google Sheet from Google drive via the share link:
using Fetch
link = "https://drive.google.com/file/d/1OiX6gEWRm57kb1H8L0K_HWN_pzc-sk8y/view?usp=sharing"
gdownload(link, pwd())
Download dataset from Kaggle
Download dataset from Kaggle via the name:
using Fetch
dataset = "ningjingyu/fetchtest"
kdownload(dataset, pwd())
Or via the url of the home page of the dataset:
using Fetch
url = "https://www.kaggle.com/ningjingyu/fetchtest"
kdownload(url, pwd())
Intergrate with DataDeps.jl
According to DataDeps.jl, DataDep
can be construct as following:
DataDep(
name::String,
message::String,
remote_path::Union{String,Vector{String}...},
[checksum::Union{String,Vector{String}...},];
fetch_method=fetch_default
post_fetch_method=identity
)
By using Fetch.jl
, one can upload their dataset to Google drive, and construct DataDep
by setting fetch_method=gdownload
.
using DataDeps
using Fetch
register(DataDep(
"FetchTest",
"""Test dataset""",
"https://drive.google.com/file/d/1OiX6gEWRm57kb1H8L0K_HWN_pzc-sk8y/view?usp=sharing",
"b083597a25bec4c82c2060651be40c0bb71075b472d3b0fabd85af92cc4a7076",
fetch_method=gdownload,
post_fetch_method=unpack
))
datadep"FetchTest"
Or to Kaggle
using DataDeps
using Fetch
register(DataDep(
"FetchTest",
"""Test dataset""",
"ningjingyu/fetchtest",
"65492e1f4c6affb7955125e5e4cece2bb547e482627f3af9812c06448dae40a9",
fetch_method=kdownload,
post_fetch_method=unpack
))
datadep"FetchTest"
According to the document of Kaggle-api one needs to set their environment variables KAGGLE_USERNAME
and KAGGLE_KEY
, or simply download the api token from Kaggle, and place this file in the location ~/.kaggle/kaggle.json
(on Windows in the location C:\Users\<Windows-username>\.kaggle\kaggle.json
).
Index
Fetch.gdownload
โ Methodgdownload(url, localdir)
Download file or Google Sheet from Google drive.
Fetch.unshortlink
โ Methodunshortlink(url)
return unshorten url or the url if it is not a short link