Abstract class containing the network-specific configuration for threads. More...

#include <Config.hpp>

Inheritance diagram for crawlservpp::Network::Config:

Collaboration diagram for crawlservpp::Network::Config:

Classes
struct	Entries
	Configuration entries for analyzer threads. More...

Configuration Loader
void	loadConfig (const std::string &configJson, LogQueue &warningsTo)
	Loads a configuration. More...

Parsing Options
enum	StringParsingOption { Default = 0, SQL, SubURL, URL, Trim }
	Options for parsing strings. More...

enum	CharParsingOption { FromNumber = 0, FromString }
	Options for parsing `char's`. More...

Configuration Parsing
void	category (const std::string &category)
	Sets the category of the subsequent configuration items to be checked for. More...

void	option (const std::string &name, bool &target)
	Checks for a configuration option of type `bool`. More...

void	option (const std::string &name, std::vector< bool > &target)
	Checks for a configuration option of type array of `bool's`. More...

void	option (const std::string &name, char &target, CharParsingOption opt)
	Checks for a configuration option of type `char`. More...

void	option (const std::string &name, std::vector< char > &target, CharParsingOption opt)
	Checks for a configuration option of type array of `char's`. More...

void	option (const std::string &name, std::int16_t &target)
	Checks for a configuration option of type 16-bit integer. More...

void	option (const std::string &name, std::vector< std::int16_t > &target)
	Checks for a configuration option of type array of 16-bit integers. More...

void	option (const std::string &name, std::int32_t &target)
	Checks for a configuration option of type 32-bit integer. More...

void	option (const std::string &name, std::vector< std::int32_t > &target)
	Checks for a configuration option of type array of 32-bit integers. More...

void	option (const std::string &name, std::int64_t &target)
	Checks for a configuration option of type 64-bit integer. More...

void	option (const std::string &name, std::vector< std::int64_t > &target)
	Checks for a configuration option of type array of 64-bit integers. More...

void	option (const std::string &name, std::uint8_t &target)
	Checks for a configuration option of type unsigned 8-bit integer. More...

void	option (const std::string &name, std::vector< std::uint8_t > &target)
	Checks for a configuration option of type array of unsigned 8-bit integers. More...

void	option (const std::string &name, std::uint16_t &target)
	Checks for a configuration option of type unsigned 16-bit integer. More...

void	option (const std::string &name, std::vector< std::uint16_t > &target)
	Checks for a configuration option of type array of unsigned 16-bit integers. More...

void	option (const std::string &name, std::uint32_t &target)
	Checks for a configuration option of type unsigned 32-bit integer. More...

void	option (const std::string &name, std::vector< std::uint32_t > &target)
	Checks for a configuration option of type array of unsigned 32-bit integers. More...

void	option (const std::string &name, std::uint64_t &target)
	Checks for a configuration option of type unsigned 64-bit integer. More...

void	option (const std::string &name, std::vector< std::uint64_t > &target)
	Checks for a configuration option of type array of unsigned 64-bit integers. More...

void	option (const std::string &name, float &target)
	Checks for a configuration option of type floating-point number. More...

void	option (const std::string &name, std::vector< float > &target)
	Checks for a configuration option of type array of floating-point numbers. More...

void	option (const std::string &name, std::string &target, StringParsingOption opt=Default)
	Checks for a configuration option of type string. More...

void	option (const std::string &name, std::vector< std::string > &target, StringParsingOption opt=Default)
	Checks for a configuration option of type array of strings. More...

void	warning (const std::string &warning)
	Adds a warning to the logging queue. More...

Module-specific Configuration Parsing
virtual void	checkOptions ()=0
	Checks the module-specific configuration after parsing. More...

Configuration
struct crawlservpp::Network::Config::Entries	networkConfig
	Configuration for networking. More...

Parsing (Network Configuration)
void	parseBasicOption () override
	Parses basic network configuration options. More...

void	resetBase () override
	Resets basic network configuration options. More...

void	parseOption () override=0
	Parses additional configuration options. More...

void	reset () override=0
	Resets additional configuration options. More...

Helper (Network Configuration)
const std::string &	getProtocol () const
	Gets the protocol to be used for networking. More...

Detailed Description

Abstract class containing the network-specific configuration for threads.

See also the documentation of libcurl's curl_easy_setopt function for more information about specific networking options.

Member Enumeration Documentation

◆ CharParsingOption

enum crawlservpp::Module::Config::CharParsingOption

protectedinherited

Options for parsing char's.

Enumerator

FromNumber

Get char by its numeric value.

FromString

Get char from the beginning of a string.

Also supports certain escaped characters.

See also: Helper::Strings::getFirstOrEscapeChar

◆ StringParsingOption

enum crawlservpp::Module::Config::StringParsingOption

protectedinherited

Options for parsing strings.

Enumerator
Default	Uses a string as it is.
SQL	Requires a SQL-compatible string.
SubURL	Converts a string to a sub-URL.
URL	Converts a string to a URL (without the protocol).
Trim	Trims a string.

Member Function Documentation

◆ category()

void crawlservpp::Module::Config::category ( const std::string & category )

inlineprotectedinherited

Sets the category of the subsequent configuration items to be checked for.

Parameters

category Constant reference to a string containing the name of the category.

References crawlservpp::Struct::ConfigItem::category.

◆ checkOptions()

virtual void crawlservpp::Module::Config::checkOptions ( )

protectedpure virtualinherited

Checks the module-specific configuration after parsing.

Needs to be overridden by module-specific configuration child classes.

Implemented in crawlservpp::Module::Extractor::Config, crawlservpp::Module::Crawler::Config, crawlservpp::Module::Analyzer::Config, and crawlservpp::Module::Parser::Config.

Referenced by crawlservpp::Module::Config::loadConfig().

◆ getProtocol()

const std::string & crawlservpp::Network::Config::getProtocol ( ) const

inline

Gets the protocol to be used for networking.

Returns: A const reference to the string containing the URI component of the protocol to be used for networking, either "https://" or "http://".

References networkConfig, and crawlservpp::Network::Config::Entries::protocol.

Referenced by crawlservpp::Module::Extractor::Thread::onReset(), and crawlservpp::Module::Crawler::Thread::onReset().

◆ loadConfig()

void crawlservpp::Module::Config::loadConfig	(	const std::string &	configJson,
		LogQueue &	warningsTo
	)

inlineinherited

Loads a configuration.

Parameters

configJson	Constant reference to a string containing the configuration as JSON.
warningsTo	Reference to a queue to which warnings will be added that occur during the parsing of the configuration, also known as the "logging queue".

Exceptions

Module::Config::Exception if the configuration JSON cannot be parsed.

References crawlservpp::Struct::ConfigItem::category, crawlservpp::Module::Config::checkOptions(), crawlservpp::Struct::ConfigItem::name, crawlservpp::Module::Config::parseBasicOption(), crawlservpp::Helper::Json::parseRapid(), crawlservpp::Struct::ConfigItem::str(), crawlservpp::Struct::ConfigItem::value, and crawlservpp::Main::Exception::view().

Referenced by crawlservpp::Module::Analyzer::Thread::cleanUpQueries(), crawlservpp::Module::Parser::Thread::onReset(), crawlservpp::Module::Extractor::Thread::onReset(), and crawlservpp::Module::Crawler::Thread::onReset().

◆ option() [1/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		bool &	target
	)

inlineprotectedinherited

Checks for a configuration option of type bool.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a boolean variable into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [2/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< bool > &	target
	)

inlineprotectedinherited

Checks for a configuration option of type array of bool's.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector of `bool's` into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [3/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		char &	target,
		CharParsingOption	opt
	)

inlineprotectedinherited

Checks for a configuration option of type char.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a variable of the type `char` into which the value of the configuration entry will be written if it is encountered.
opt	Parsing options used for the configuration option.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Module::Config::FromNumber, crawlservpp::Module::Config::FromString, crawlservpp::Helper::Strings::getFirstOrEscapeChar(), crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [4/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< char > &	target,
		CharParsingOption	opt
	)

inlineprotectedinherited

Checks for a configuration option of type array of char's.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector of `char's` into which the value of the configuration entry will be written if it is encountered.
opt	Parsing options used for the configuration option.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Module::Config::FromNumber, crawlservpp::Module::Config::FromString, crawlservpp::Helper::Strings::getFirstOrEscapeChar(), crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [5/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::int16_t &	target
	)

inlineprotectedinherited

Checks for a configuration option of type 16-bit integer.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a variable into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [6/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< std::int16_t > &	target
	)

inlineprotectedinherited

Checks for a configuration option of type array of 16-bit integers.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector into which the value of the configuration entry will be stored if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [7/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::int32_t &	target
	)

inlineprotectedinherited

Checks for a configuration option of type 32-bit integer.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a variable into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [8/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< std::int32_t > &	target
	)

inlineprotectedinherited

Checks for a configuration option of type array of 32-bit integers.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector into which the value of the configuration entry will be stored if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [9/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::int64_t &	target
	)

inlineprotectedinherited

Checks for a configuration option of type 64-bit integer.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a variable into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [10/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< std::int64_t > &	target
	)

inlineprotectedinherited

Checks for a configuration option of type array of 64-bit integers.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector into which the value of the configuration entry will be stored if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [11/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::uint8_t &	target
	)

inlineprotectedinherited

Checks for a configuration option of type unsigned 8-bit integer.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a variable into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [12/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< std::uint8_t > &	target
	)

inlineprotectedinherited

Checks for a configuration option of type array of unsigned 8-bit integers.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector into which the value of the configuration entry will be stored if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [13/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::uint16_t &	target
	)

inlineprotectedinherited

Checks for a configuration option of type unsigned 16-bit integer.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a variable into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [14/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< std::uint16_t > &	target
	)

inlineprotectedinherited

Checks for a configuration option of type array of unsigned 16-bit integers.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector into which the value of the configuration entry will be stored if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [15/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::uint32_t &	target
	)

inlineprotectedinherited

Checks for a configuration option of type unsigned 32-bit integer.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a variable into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [16/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< std::uint32_t > &	target
	)

inlineprotectedinherited

Checks for a configuration option of type array of unsigned 32-bit integers.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector into which the value of the configuration entry will be stored if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [17/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::uint64_t &	target
	)

inlineprotectedinherited

Checks for a configuration option of type unsigned 64-bit integer.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a variable into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [18/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< std::uint64_t > &	target
	)

inlineprotectedinherited

Checks for a configuration option of type array of unsigned 64-bit integers.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector into which the value of the configuration entry will be stored if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [19/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		float &	target
	)

inlineprotectedinherited

Checks for a configuration option of type floating-point number.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a variable into which the value of the configuration entry will be written if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [20/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< float > &	target
	)

inlineprotectedinherited

Checks for a configuration option of type array of floating-point numbers.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector into which the value of the configuration entry will be stored if it is encountered.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Struct::ConfigItem::name, crawlservpp::Struct::ConfigItem::str(), and crawlservpp::Struct::ConfigItem::value.

◆ option() [21/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::string &	target,
		StringParsingOption	opt = `Default`
	)

inlineprotectedinherited

Checks for a configuration option of type string.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a string into which the value of the configuration entry will be stored if it is encountered.
opt	Parsing option for the configuration entry.

Exceptions

Module::Config::Exception if no category has been set.

See also: category

References crawlservpp::Helper::Strings::checkSQLName(), crawlservpp::Module::Config::Default, crawlservpp::Struct::ConfigItem::name, crawlservpp::Module::Config::SQL, crawlservpp::Struct::ConfigItem::str(), crawlservpp::Module::Config::SubURL, crawlservpp::Helper::Strings::trim(), crawlservpp::Module::Config::Trim, crawlservpp::Module::Config::URL, and crawlservpp::Struct::ConfigItem::value.

◆ option() [22/22]

void crawlservpp::Module::Config::option	(	const std::string &	name,
		std::vector< std::string > &	target,
		StringParsingOption	opt = `Default`
	)

inlineprotectedinherited

Checks for a configuration option of type array of strings.

Ignores the option and adds a warning to the warnings queue, if the requested type does not match the data type in the configuration JSON.

Parameters

name	Constant reference to a string containing the name of the option to check for.
target	Reference to a vector into which the value of the configuration entry will be stored if it is encountered.
opt	Parsing option for the configuration entry.

Exceptions

Module::Config::Exception if no category has been set.