JASS::parser::token Class Reference

A token as returned by the parser. More...

#include <parser.h>

Collaboration diagram for JASS::parser::token:

Public Types
enum	token_type {   alpha, numeric, xml_start_tag, xml_empty_tag,   xml_end_tag, xml_comment, xml_cdata, xml_definition,   xml_conditional, xml_processing_instruction, other, eof }
	The type of the token. More...

Public Member Functions
const slice &	get (void) const
	Return a reference to the token as a slice. The life of the slice is the life of the object. More...
void	set (slice term)
	create a token from a slice without copying it and without looking to see what kind of token it is.

Public Attributes
uint8_t	buffer [max_token_length]
	The token manages its memory through this buffer.
slice	lexeme
	The token itself, stored as a slice (pointer / length pair)
token_type	type
	The type of this token (See token_type)
index_postings_impact::impact_type	count
	The number of times the token is seen (normally 1, but if parsing a forward index it might be known to be larget).

Static Public Attributes
static constexpr size_t	max_token_length = 1024
	Any token longer that this will be truncated at this length.

Detailed Description

A token as returned by the parser.

The parser returns a reference to a token that the caller must not copy. That reference is valid untill the next call to get_next_token() or the destruction of the parser object, because it is a reference to a member of the parser object. A token is of a particular type, numeric, alphabetic, start tag, end tag, etc., which is stored in the type member. The token itself is in token.token member, which is just a slice of the (not '\0' terminated) token.buffer.

The token manages its own memory using the token.buffer member. That member is an array of bytes of length token.max_token_length. Normally a "valid" token will be only a few bytes long, but some over-long tokens are seen. The maximum length was intially 1KB, but is prone to change over time. Note that the longest token that has been seen by the authors is 1MB. It was pi to a million decimal places. JASS handles extra-long tokens by truncating at length token.max_token_length.

Member Enumeration Documentation

token_type

enum JASS::parser::token::token_type

The type of the token.

Enumerator
alpha	alphabetic token
numeric	numeric token
xml_start_tag	XML start tag (just the tag name)
xml_empty_tag	XML empty tag (without the "<" or "/>").
xml_end_tag	XML end tag (without the "<" or ">".
xml_comment	XML comment (with the "<!--" and–>" removed).
xml_cdata	XML CDATA (just the CDATA)
xml_definition	XML definition (DOCTYPE, ELEMENT, etc) without the "<!" and ">".
xml_conditional	XML conditional. This is not properly interpreted. its just the "INCLUDE" or "IGNORE" that is returned (and any space around it).
xml_processing_instruction	XML processing instruction (a "<?" sequence) with the "<?" and "?>" removed/.
other	Other type of token (punctuation, etc).
eof	The final token is marked as an EOF token (and has no content).

Member Function Documentation

get()

const slice& JASS::parser::token::get ( void  ) const

inline

Return a reference to the token as a slice. The life of the slice is the life of the object.

Returns

The token.

---

The documentation for this class was generated from the following file:

• source/parser.h