xiiTokenParseUtils Namespace Reference

Copyright (c) Theophilus Eriata. All Rights Reserved. More...

Classes
struct	TokenMatch

Typedefs
using	TokenStream = xiiHybridArray<const xiiToken*, 32U>

Functions
void	SkipWhitespace (const TokenStream &tokens, xiiUInt32 &ref_uiCurToken)
void	SkipWhitespaceAndNewline (const TokenStream &tokens, xiiUInt32 &ref_uiCurToken)
bool	IsEndOfLine (const TokenStream &tokens, xiiUInt32 uiCurToken, bool bIgnoreWhitespace)
void	CopyRelevantTokens (const TokenStream &source, xiiUInt32 uiFirstSourceToken, TokenStream &ref_destination, bool bPreserveNewLines)
bool	Accept (const TokenStream &tokens, xiiUInt32 &ref_uiCurToken, xiiStringView sToken, xiiUInt32 *pAccepted)
bool	Accept (const TokenStream &tokens, xiiUInt32 &ref_uiCurToken, xiiTokenType::Enum type, xiiUInt32 *pAccepted=nullptr)
	Overload that matches against a token type instead.
bool	Accept (const TokenStream &tokens, xiiUInt32 &ref_uiCurToken, xiiStringView sToken1, xiiStringView sToken2, xiiUInt32 *pAccepted)
bool	AcceptUnless (const TokenStream &tokens, xiiUInt32 &ref_uiCurToken, xiiStringView sToken1, xiiStringView sToken2, xiiUInt32 *pAccepted)
bool	Accept (const TokenStream &tokens, xiiUInt32 &ref_uiCurToken, xiiArrayPtr< const TokenMatch > matches, xiiDynamicArray< xiiUInt32 > *pAccepted)
void	CombineRelevantTokensToString (const TokenStream &tokens, xiiUInt32 uiCurToken, xiiStringBuilder &ref_sResult)
void	CreateCleanTokenStream (const TokenStream &tokens, xiiUInt32 uiCurToken, TokenStream &ref_destination)
void	CombineTokensToString (const TokenStream &tokens0, xiiUInt32 uiCurToken, xiiStringBuilder &ref_sResult, bool bKeepComments, bool bRemoveRedundantWhitespace, bool bInsertLine)

Detailed Description

Copyright (c) Theophilus Eriata. All Rights Reserved.

Function Documentation

Accept() [1/3]

XII_FOUNDATION_DLL bool xiiTokenParseUtils::Accept	(	const TokenStream &	tokens,
		xiiUInt32 &	ref_uiCurToken,
		xiiArrayPtr< const TokenMatch >	matches,
		xiiDynamicArray< xiiUInt32 > *	pAccepted = nullptr )

Tries to move ref_uiCurToken forward by matching against an array of tokens. The function will skip whitespace tokens (Whitespace, BlockComment, LineComment) between the tokens it tries to match within 'matches'.

Here is an example how to parse a vector declaration:

xiiTokenParseUtils::TokenMatch pattern[] = {"Vec2"_xiisv, "("_xiisv, xiiTokenType::Float, ","_xiisv, xiiTokenType::Float, ")"_xiisv};
xiiHybridArray<xiiUInt32, 6> matchedTokens;
XII_TEST_BOOL(xiiTokenParseUtils::Accept(tokens, uiCurToken, pattern, &matchedTokens));

Parameters

tokens	Token input.
ref_uiCurToken	Current location inside 'tokens'. If false is returned it will stay the same. If true is returned, it will be the next index after matching all of 'matches'.
matches	The tokens to be matched. These can be separated by whitespace tokens.
pAccepted	If not null and true was returned, will be filled with the indices at which each token inside 'matches' was matched.

Returns

Whether the array of tokens was found.

Accept() [2/3]

XII_FOUNDATION_DLL bool xiiTokenParseUtils::Accept	(	const TokenStream &	tokens,
		xiiUInt32 &	ref_uiCurToken,
		xiiStringView	sToken,
		xiiUInt32 *	pAccepted = nullptr )

Tries to move ref_uiCurToken forward by matching against 'sToken'. The function will skip whitespace tokens (Whitespace, BlockComment, LineComment) in search of 'sToken'.

Parameters

tokens	Token input.
ref_uiCurToken	Current location inside 'tokens'. If false is returned it will stay the same. If true is returned, it will be the next index after matching 'sToken'.
sToken	The token to be matched.
pAccepted	If not null and true was returned, will be set to the index at which 'sToken' was matched.

Returns

Whether the token was found.

Accept() [3/3]

XII_FOUNDATION_DLL bool xiiTokenParseUtils::Accept	(	const TokenStream &	tokens,
		xiiUInt32 &	ref_uiCurToken,
		xiiStringView	sToken1,
		xiiStringView	sToken2,
		xiiUInt32 *	pAccepted = nullptr )

Tries to move ref_uiCurToken forward by matching against a token tuple. The function will skip whitespace tokens (Whitespace, BlockComment, LineComment) in search of the tokens.

Parameters

tokens	Token input.
ref_uiCurToken	Current location inside 'tokens'. If false is returned it will stay the same. If true is returned, it will be the next index after matching after the matched 'sToken2'.
sToken1	First token to match.
sToken2	Second token to match. Must be followed right after 'sToken1'.
pAccepted	If not null and true was returned, will be set to the index at which 'sToken1' was matched.

Returns

Whether the token tuple was found.

AcceptUnless()

XII_FOUNDATION_DLL bool xiiTokenParseUtils::AcceptUnless	(	const TokenStream &	tokens,
		xiiUInt32 &	ref_uiCurToken,
		xiiStringView	sToken1,
		xiiStringView	sToken2,
		xiiUInt32 *	pAccepted = nullptr )

Tries to move ref_uiCurToken forward by matching against 'sToken1' unless it is followed right after by 'sToken2'. The function will skip whitespace tokens (Whitespace, BlockComment, LineComment) in search of 'sToken1'.

Parameters

tokens	Token input.
ref_uiCurToken	Current location inside 'tokens'. If false is returned it will stay the same. If true is returned, it will be the next index after matching 'sToken1'.
sToken1	The token to be matched.
sToken2	The token that must not follow right after 'sToken1'.
pAccepted	If not null and true was returned, will be set to the index at which 'sToken1' was matched.

Returns

Whether the token was found.

CombineRelevantTokensToString()

XII_FOUNDATION_DLL void xiiTokenParseUtils::CombineRelevantTokensToString	(	const TokenStream &	tokens,
		xiiUInt32	uiCurToken,
		xiiStringBuilder &	ref_sResult )

Strips out BlockComment, LineComment, EndOfFile and NewLine tokens and combines the rest into a string.

Parameters

tokens	The token stream to combine.
uiCurToken	The start location inside 'tokens' from which point the tokens should be combined.
ref_sResult	Holds the resulting string after the function call.

CombineTokensToString()

XII_FOUNDATION_DLL void xiiTokenParseUtils::CombineTokensToString	(	const TokenStream &	tokens,
		xiiUInt32	uiCurToken,
		xiiStringBuilder &	ref_sResult,
		bool	bKeepComments = true,
		bool	bRemoveRedundantWhitespace = false,
		bool	bInsertLine = false )

Combines tokens to a string.

Parameters

tokens	The token stream to combine.
uiCurToken	The start location inside 'tokens' from which point the tokens should be combined.
ref_sResult	Holds the resulting string after the function call.
bKeepComments	Whether comments should be written into the string.
bRemoveRedundantWhitespace	Whether redundant whitespace should be removed.
bInsertLine	If set, will insert macros in the form of '#line <LINE> "<FILE>"' where appropriate. This is used as a hint by debuggers and other tools to reconstruct in which line/file a failure in a script occurred.

CopyRelevantTokens()

XII_FOUNDATION_DLL void xiiTokenParseUtils::CopyRelevantTokens	(	const TokenStream &	source,
		xiiUInt32	uiFirstSourceToken,
		TokenStream &	ref_destination,
		bool	bPreserveNewLines )

Strips out BlockComment, LineComment, EndOfFile tokens and Newline as well if requested and copies the rest to ref_destination.

Parameters

source	From where to copy the tokens.
uiFirstSourceToken	Start token in 'source'. Whitespace at the start will be stripped as well.
ref_destination	Copy target of the tokens.
bPreserveNewLines	Whether to preserve Newline tokens.

CreateCleanTokenStream()

XII_FOUNDATION_DLL void xiiTokenParseUtils::CreateCleanTokenStream	(	const TokenStream &	tokens,
		xiiUInt32	uiCurToken,
		TokenStream &	ref_destination )

Removes whitespace at the end of each line and removes NewLine that follow each other.

Parameters

tokens	The token stream to cleaned up.
uiCurToken	The start location inside 'tokens' from which point the tokens should be copied to 'ref_destination'.
ref_destination	Target stream to store the cleaned up tokens into.

IsEndOfLine()

XII_FOUNDATION_DLL bool xiiTokenParseUtils::IsEndOfLine	(	const TokenStream &	tokens,
		xiiUInt32	uiCurToken,
		bool	bIgnoreWhitespace )

Checks whether we are at the end of a line.

Parameters

tokens	Token input.
uiCurToken	Current location inside 'tokens'.
bIgnoreWhitespace	If false, only uiCurToken is checked. If true, whitespace will be skipped in search for a Newline token.

Returns

Whether we are at the end of a line.

SkipWhitespace()

XII_FOUNDATION_DLL void xiiTokenParseUtils::SkipWhitespace	(	const TokenStream &	tokens,
		xiiUInt32 &	ref_uiCurToken )

Moves ref_uiCurToken forward as long as it is a whitespace token (Whitespace, BlockComment, LineComment).

Parameters

tokens	Token input.
ref_uiCurToken	Current location inside 'tokens'. On return, will either stay the same or move forward.

SkipWhitespaceAndNewline()

XII_FOUNDATION_DLL void xiiTokenParseUtils::SkipWhitespaceAndNewline	(	const TokenStream &	tokens,
		xiiUInt32 &	ref_uiCurToken )

Moves ref_uiCurToken forward as long as it is a whitespace token (Whitespace, BlockComment, LineComment) or Newline.

Parameters

tokens	Token input.
ref_uiCurToken	Current location inside 'tokens'. On return, will either stay the same or move forward.