crslab package

Subpackages

• crslab.config package
• crslab.data package
• crslab.evaluator package
• crslab.model package
• crslab.quick_start package
• crslab.system package

Submodules

class crslab.download.DownloadableFile(url, file_name, hashcode, zipped=True, from_google=False)

Bases: object

A class used to abstract any file that has to be downloaded online.

Any task that needs to download a file needs to have a list RESOURCES that have objects of this class as elements.

This class provides the following functionality:

• Download a file from a URL

• Untar the file if zipped

• Checksum for the downloaded file

An object of this class needs to be created with:

• url <string> : URL or Google Drive id to download from

• file_name <string> : File name that the file should be named

• hashcode <string> : SHA256 hashcode of the downloaded file

• zipped <boolean> : False if the file is not compressed

• from_google <boolean> : True if the file is from Google Drive

checksum(dpath)

Checksum on a given file.

Parameters

dpath – path to the downloaded file.

download_file(dpath)

crslab.download.build(dpath, dfile, version=None)

crslab.download.check_build(path, version_string=None)

Check if ‘.built’ flag has been set for that task.

If a version_string is provided, this has to match, or the version is regarded as not built.

crslab.download.download(url, path, fname, redownload=False, num_retries=5)

Download file using requests. If redownload is set to false, then will not download tar file again if it is present (default False).

crslab.download.download_from_google_drive(gd_id, destination)

Use the requests package to download a file from Google Drive.

crslab.download.make_dir(path)

Make the directory and any nonexistent parent directories (mkdir -p).

crslab.download.mark_done(path, version_string=None)

Mark this path as prebuilt.

Marks the path as done by adding a ‘.built’ file with the current timestamp plus a version description string if specified.

Parameters

• path (str) – The file path to mark as built.

• version_string (str) – The version of this dataset.

crslab.download.move(path1, path2)

Rename the given file.

crslab.download.remove_dir(path)

Remove the given directory, if it exists.

crslab.download.untar(path, fname, deleteTar=True)

Unpack the given archive file to the same directory.

Parameters

• path (str) – The folder containing the archive. Will contain the contents.

• fname (str) – The filename of the archive file.

• deleteTar (bool) – If true, the archive will be deleted after extraction.

Module contents