A domain is a group of processing entities such as cores in a specific NUMA domain or a Intel MIC entity. More...

#include <dart_types.h>

Public Attributes
char	host [((int)(256))]
	Hostname of the domain's node or 0 if unspecified. More...

char	domain_tag [((int)(32))]
	Hierarchical domain identifier, represented as dot-separated list of relative indices on every level in the locality hierarchy. More...

struct dart_domain_locality_s **	aliases

int	num_aliases

dart_locality_scope_t	scope
	Locality scope of the domain. More...

int	level
	Level in the domain locality hierarchy. More...

int	global_index
	The domain's global index within its scope. More...

int	relative_index
	The domain's index within its parent domain. More...

struct dart_domain_locality_s *	parent
	Pointer to descriptor of parent domain or 0 if no parent domain is specified. More...

int	num_domains
	Number of subordinate domains. More...

struct dart_domain_locality_s **	children
	Array of subordinate domains of size `num_domains` or 0 if no subdomains are specified. More...

int	is_symmetric
	Whether sub-domains have identical hardware configuration. More...

dart_team_t	team
	Team associated with the domain. More...

int	num_units
	Number of units in the domain. More...

dart_global_unit_t *	unit_ids
	Global IDs of units in the domain. More...

int	num_nodes

int	node_id

int	num_cores

int	shared_mem_bytes

Detailed Description

A domain is a group of processing entities such as cores in a specific NUMA domain or a Intel MIC entity.

Domains are organized in a hierarchy. In this, a domain may consist of heterogenous child domains. Processing entities in domains on the lowest locality level are homogenous.

Domains represent the actual hardware topology but also can represent grouping from user-defined team specifications.

Use cases:

To determine whether units in a domain have access to common shared memory, test if domain descriptor field
- num_nodes is set to 1, or
- scope is set to DART_LOCALITY_SCOPE_NODE or greater.
The maximum number of threads for a single unit, e.g. for MKL routines, can be calculated as:

(dloc.num_cores x dloc.num_threads)

from a domain descriptor dloc with scope DART_LOCALITY_SCOPE_UNIT.
A simple metric of processing power of components in a homogenous domain (minimum number of instructions per second) can be calculated as:

dmhz(dloc) = (dloc.num_cores x dloc.min_threads x dloc.min_cpu_mhz)

This metric then can be used to balance workload between homogenous domains with different processing components. A simple balance factor wb can be calculated as:

wb = dmhz(dloc_a) / dmhz(dloc_b)

from domain descriptors dloc_a and dloc_b.

Illustrating example:

domain (top level, heterogenous)
domain_tag:  "."
host:        "number-crunch-9000"
scope:       DART_LOCALITY_SCOPE_GLOBAL
level:         0
num_nodes:     4
num_cores:   544 (4 nodes x 136 cores per node)
min_threads:   2
max_threads:   4
num_domains:   4 (4 nodes)
domains:
:
|-- domain (compute node, heterogenous)
:   domain_tag:  ".0"
:   scope:       DART_LOCALITY_SCOPE_NODE
:   level:         1
:   num_nodes:     1
:   num_cores:   136 (16 host cores + 2x60 MIC cores)
:   min_threads:   2
:   max_threads:   4
:   num_domains:   3 (1 host + 2 MICs)
:   domains:
:   :
:   |-- domain (host, homogenous)
:   :   domain_tag:  ".0.0"
:   :   scope:       DART_LOCALITY_SCOPE_PROC_GROUP
:   :   level:         2
:   :   num_nodes:     1
:   :   num_numa:      2
:   :   num_cores:    16
:   :   min_threads:   2
:   :   max_threads:   2
:   :   num_domains:   2
:   :   :
:   :   |-- domain (NUMA domain at host)
:   :   :   domain_tag:  ".0.0.1"
:   :   :   scope:       DART_LOCALITY_SCOPE_UNIT
:   :   :   level:        3
:   :   :   num_nodes:    1
:   :   :   num_numa:     1
:   :   :   num_cores:    8
:   :   :   num_domains:  8
:   :   :   :
:   :   :   '
:   :   :   ...
:   :   :
:   :   '-- domain (NUMA domain at host)
:   :       domain_tag:  ".0.0.1"
:   :       scope:       DART_LOCALITY_SCOPE_UNIT
:   :       level:        3
:   :       num_nodes:    1
:   :       num_numa:     1
:   :       num_cores:    8
:   :       num_domains:  8
:   :       :
:   :       '
:   :       ...
:   :
:   |-- domain (MIC, homogenous)
:   :   domain_tag:  ".0.1"
:   :   scope:       DART_LOCALITY_SCOPE_PROC_GROUP
:   :   level:         2
:   :   num_nodes:     1
:   :   num_cores:    60
:   :   min_threads:   4
:   :   max_threads:   4
:   :   num_domains:   0
:   :
:   '-- domain (MIC, homogenous)
:       domain_tag:  ".0.2"
:       scope:       DART_LOCALITY_SCOPE_PROC_GROUP
:       level:         2
:       num_nodes:     1
:       num_cores:    60
:       min_threads:   4
:       max_threads:   4
:       domains:
:       num_domains:   2
:       :
:       |-- domain (unit of MIC cores, homogenous)
:       :   domain_tag:  ".0.2.0"
:       :   scope:       DART_LOCALITY_SCOPE_UNIT
:       :   level:        3
:       :   num_nodes:    1
:       :   num_cores:   30
:       :   num_domains:  0
:       :
:       '-- domain (unit of MIC cores, homogenous)
:           domain_tag:  ".0.2.1"
:           scope:       DART_LOCALITY_SCOPE_UNIT
:           level:        3
:           num_nodes:    1
:           num_cores:   30
:           num_domains:  0
:
|-- domain (compute node, heterogenous)
:   domain_tag:  ".1"
:   scope:       DART_LOCALITY_SCOPE_NODE
:   level:         1
:   num_cores:   136
:   num_domains:   3
:   domains:
:   :
:   '
:   ...
'
...

Definition at line 597 of file dart_types.h.

Member Data Documentation

◆ children

struct dart_domain_locality_s** dart_domain_locality_s::children

Array of subordinate domains of size num_domains or 0 if no subdomains are specified.

Definition at line 630 of file dart_types.h.

◆ domain_tag

char dart_domain_locality_s::domain_tag[((int)(32))]

Hierarchical domain identifier, represented as dot-separated list of relative indices on every level in the locality hierarchy.

Definition at line 606 of file dart_types.h.

◆ global_index

int dart_domain_locality_s::global_index

The domain's global index within its scope.

Definition at line 618 of file dart_types.h.

Referenced by dash::util::LocalityDomain::leader_unit().

◆ host

char dart_domain_locality_s::host[((int)(256))]

Hostname of the domain's node or 0 if unspecified.

Definition at line 600 of file dart_types.h.

◆ is_symmetric

int dart_domain_locality_s::is_symmetric

Whether sub-domains have identical hardware configuration.

Definition at line 633 of file dart_types.h.

◆ level

int dart_domain_locality_s::level

Level in the domain locality hierarchy.

Definition at line 615 of file dart_types.h.

Referenced by dash::util::LocalityDomain::leader_unit().

◆ num_domains

int dart_domain_locality_s::num_domains

Number of subordinate domains.

Definition at line 627 of file dart_types.h.

◆ num_units

int dart_domain_locality_s::num_units

Number of units in the domain.

Definition at line 638 of file dart_types.h.

◆ parent

struct dart_domain_locality_s* dart_domain_locality_s::parent

Pointer to descriptor of parent domain or 0 if no parent domain is specified.

Definition at line 624 of file dart_types.h.

◆ relative_index

int dart_domain_locality_s::relative_index

The domain's index within its parent domain.

Definition at line 620 of file dart_types.h.

◆ scope

dart_locality_scope_t dart_domain_locality_s::scope

Locality scope of the domain.

Definition at line 613 of file dart_types.h.

Referenced by dash::util::LocalityDomain::leader_unit().

◆ team

dart_team_t dart_domain_locality_s::team

Team associated with the domain.

Definition at line 636 of file dart_types.h.

◆ unit_ids

dart_global_unit_t* dart_domain_locality_s::unit_ids

Global IDs of units in the domain.

Definition at line 640 of file dart_types.h.

The documentation for this struct was generated from the following file:

/tmp/tmporruphar/dart-if/include/dash/dart/if/dart_types.h

Public Attributes

Detailed Description

Member Data Documentation

◆ children

◆ domain_tag

◆ global_index

◆ host

◆ is_symmetric

◆ level

◆ num_domains

◆ num_units

◆ parent

◆ relative_index

◆ scope

◆ team

◆ unit_ids