Utils methods¶
-
scrapper_helpers.utils.
caching
(key_func=<function default_key_func>)¶ A decorator that creates local dumps of the decorated function’s return values for given parameters. It can take a key_func argument that determines the name of the output file.
-
scrapper_helpers.utils.
default_key_func
(*args)¶ This method creates the default string representation of the input parameters, used for caching filename
-
scrapper_helpers.utils.
finder
(many=True, *finder_args, **finder_kwargs)¶ - Search for an element(or elements depending on variable ‘many’) with certain key(tag, class, id, ...)
- in a web page markup
Parameters: - many – decide whether searching for one or more elements
- finder_args – key to find
Returns: element or list of elements
-
scrapper_helpers.utils.
flatten
(container)¶ Flattens a list :param container: list with nested lists :type container: list :return: list with elements that were nested in container :rtype: list
-
scrapper_helpers.utils.
get_number_from_string
(s, number_type, default)¶ Returns a numeric value of number_type created from the given string or default if the cast is not possible.
-
scrapper_helpers.utils.
get_random_user_agent
()¶ Randoms user agent to prevent “python” user agent :return: Random user agent from USER_AGENTS :rtype: str
-
scrapper_helpers.utils.
html_decode
(s)¶ Returns the ASCII decoded version of the given HTML string. This does NOT remove normal HTML tags like <p>.
-
scrapper_helpers.utils.
key_md5
(*args)¶ This method creates an MD5 from the input parameters, used for caching filename
-
scrapper_helpers.utils.
key_sha1
(*args)¶ This method creates an SHA-1 from the input parameters, used for caching filename
-
scrapper_helpers.utils.
normalize_text
(text, lower=True, replace_spaces='_')¶ This method returns the input string, but normalizes is it for use in the url. :param text: input string :rtype: string :return: Normalized string. lowercase, no diacritics, ‘-‘ instead of ‘ ‘
-
scrapper_helpers.utils.
replace_all
(text, dic)¶ This method returns the input string, but replaces its characters according to the input dictionary. :param text: input string :param dic: dictionary containing the changes. key is the character that’s supposed to be changed and value is
the desired valueReturn type: string Returns: String with the according characters replaced