lex/index.html at gh-pages · nutbread/lex · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
<!DOCTYPE html>
<html>
<head>
	<meta charset="UTF-8" />
	<meta name="description" content="Lexical analysis libraries for JavaScript and Python" />
	<meta name="keywords" content="python,lex,lexer,lexical,lexical analyzer,lexical analysis,syntax highlighting,libraries,javascript,js,python,py,css,xml,html" />
	<meta name="author" content="nutbread" />
	<title>Lexical Analysis Libraries</title>
	<link rel="shortcut icon" href="favicon.png" />
	<link rel="stylesheet" href="style.css" />
	<script src="script.js"></script>
</head>
<body>


<div class="header_bar">
	<div class="main main_no_overflow header">

		<table class="header_table"><tbody>
			<tr>
				<td class="header_table_cell">
					<div class="header_table_name">
						<a class="header_table_name_user link_external link_author light_underline" href="https://nutbread.github.io/"><span>nutbread</span></a><span class="header_table_name_separator">/</span><a href="" class="header_table_name_title link_external link_title light_underline"><span>lex</span></a>
					</div>
				</td>
				<td class="header_table_cell">
					<div class="header_table_separator"></div>
				</td>
				<td class="header_table_cell header_table_cell_full">
					<div class="header_table_description">
						<div class="header_table_description_name">Lexical Analysis Libraries</div>
						<div class="header_table_description_body">Simple lexical analysis libraries for JavaScript and Python</div>
					</div>
				</td>
				<td class="header_table_cell">
					<a class="header_table_view_on_github link_external link_view_on_github light_nohover_color_hover" href="https://github.com/nutbread/lex">
						<div class="header_table_view_on_github_line1">view on</div>
						<div class="header_table_view_on_github_line2">github</div>
					</a>
				</td>
			</tr>
		</tbody></table>

	</div>
</div>
<div class="main body">

	<h1 id="about"><span class="hardlink_text">About<a class="hardlink" href="#about"></a></span></h1>
	<p>
		This is a set of lexical analizers for language tokenizing. Currently there are libraries for processing JavaScript, Python, CSS, and XML/HTML with source code in JavaScript and Python 2/3.
	</p>
	<p>
		It was primary written to address some edge case JavaScript parsing issues found in several major applications (Notepad++, Firefox, Sublime Text, Github/Ace.) These cases usually involve regular expressions or sign-prefixed numbers.
	</p>

	<h4 id="downloads"><span class="hardlink_text">Downloads<a class="hardlink" href="#downloads"></a></span></h4>
	<p>
		Files named <code>lex.*</code> are the base classes; files named <code>lex<em>lang</em>.*</code> are the language descriptor generation files.<br />
		<span class="light">Example: <code>lex.js</code> and <code>lex<em>py</em>.js</code> are the files needed for Python code processing running on a JavaScript interpreter</span>
	</p>
	<p>
		<ul>
			<li>
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lex.js" target="_blank"><span>lex.js</span></a> &ndash; JavaScript source, can run on both node.js and browsers
			</li>
			<li>
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lex.py" target="_blank"><span>lex.py</span></a> &ndash; Python source, versions 2.x or 3.x should both work
			</li>
			<li class="top_margin large_margin">
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lexjs.js" target="_blank"><span>lexjs.js</span></a> &ndash; JavaScript lexical analyzer <span class="light">(for JavaScript)</span>
			</li>
			<li>
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lexjs.py" target="_blank"><span>lexjs.py</span></a> &ndash; JavaScript lexical analyzer <span class="light">(for Python)</span>
			</li>
			<li>
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lexpy.js" target="_blank"><span>lexpy.js</span></a> &ndash; Python lexical analyzer <span class="light">(for JavaScript)</span>
			</li>
			<li>
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lexpy.py" target="_blank"><span>lexpy.py</span></a> &ndash; Python lexical analyzer <span class="light">(for Python)</span>
			</li>
			<li>
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lexcss.js" target="_blank"><span>lexcss.js</span></a> &ndash; CSS lexical analyzer <span class="light">(for JavaScript)</span>
			</li>
			<li>
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lexcss.py" target="_blank"><span>lexcss.py</span></a> &ndash; CSS lexical analyzer <span class="light">(for Python)</span>
			</li>
			<li>
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lexxml.js" target="_blank"><span>lexxml.js</span></a> &ndash; XML/HTML lexical analyzer <span class="light">(for JavaScript)</span>
			</li>
			<li>
				<a class="link_codebase_raw light_underline" href="https://raw.githubusercontent.com/nutbread/lex/gh-pages/src/lexxml.py" target="_blank"><span>lexxml.py</span></a> &ndash; XML/HTML lexical analyzer <span class="light">(for Python)</span>
			</li>
		</ul>
	</p>

	<h1 id="usage"><span class="hardlink_text">Usage<a class="hardlink" href="#usage"></a></span></h1>
	<p>
		The general format for using these libraries is:
	</p>
	<p>
		<ol>
			<li>Import your language's <code>lex.*</code> file</li>
			<li>Import your language's descriptor generator <code>lex<em>lang</em>.*</code> file</li>
			<li>Create the descriptor using <code>lexlang = lex<em>lang</em>.gen(lex);</code></li>
			<li>
				Create a <code>lex.Lexer</code> object with the descriptor as the first argument, and the input string as the second<br />
				<span class="light">It is preferred that the source code is a unicode string in Python</span>
			</li>
			<li>
				Repeatedly call the lexer's <code>get_token</code> method until it returns <code>null</code> (or language equivalent.)
			</li>
		</ol>
	</p>

	<p class="top_margin large_margin">
		When not returning a <code>null</code> value, <code>get_token</code> will otherwise return a <code>Token</code> object with 4 fields:
	</p>
	<p>
		<ul>
			<li>
				<code>text</code> &ndash; the token string
			</li>
			<li>
				<code>type</code> &ndash; the type constant of the token
			</li>
			<li>
				<code>flags</code> &ndash; flags for the token<br />
				<span class="light">Many flags are primarily used internally; some are useful outside the <code>Lexer</code></span>
			</li>
			<li>
				<code>state</code> &ndash; the state the token was generated in<br />
			</li>
		</ul>
	</p>

	<p class="top_margin large_margin">
		<h6 class="colored">JavaScript token type constants:</h6>
		<code>INVALID</code>,
		<code>KEYWORD</code>,
		<code>LITERAL</code>,
		<code>IDENTIFIER</code>,
		<code>NUMBER</code>,
		<code>STRING</code>,
		<code>REGEX</code>,
		<code>OPERATOR</code>,
		<code>WHITESPACE</code>,
		<code>COMMENT</code>
	</p>

	<p>
		<h6 class="colored">Python token type constants:</h6>
		<code>INVALID</code>,
		<code>KEYWORD</code>,
		<code>LITERAL</code>,
		<code>IDENTIFIER</code>,
		<code>NUMBER</code>,
		<code>STRING</code>,
		<code>OPERATOR</code>,
		<code>WHITESPACE</code>,
		<code>COMMENT</code>
	</p>

	<p>
		<h6 class="colored">CSS token type constants:</h6>
		<code>INVALID</code>,
		<code>WHITESPACE</code>,
		<code>COMMENT</code>,
		<code>STRING</code>,
		<code>WORD</code>,
		<code>OPERATOR</code>,
		<code>AT_RULE</code>,
		<code>SEL_TAG</code>,
		<code>SEL_CLASS</code>,
		<code>SEL_ID</code>,
		<code>SEL_PSEUDO_CLASS</code>,
		<code>SEL_PSEUDO_ELEMENT</code>,
		<code>SEL_N_EXPRESSION</code>,
		<code>NUMBER</code>,
		<code>COLOR</code>
	</p>

	<p>
		<h6 class="colored">XML/HTML token type constants:</h6>
		<code>COMMENT</code>,
		<code>CDATA</code>,
		<code>TEXT</code>,
		<code>RAW_DATA</code>,
		<code>TAG_OPEN</code>,
		<code>TAG_CLOSE</code>,
		<code>TAG_NAME</code>,
		<code>ATTRIBUTE</code>,
		<code>ATTRIBUTE_WHITESPACE</code>,
		<code>ATTRIBUTE_OPERATOR</code>,
		<code>ATTRIBUTE_STRING</code>
	</p>

	<p class="top_margin large_margin">
		<h6><span class="colored">Generic token flags</span> (that can be useful outside the <code>Lexer</code>):</h6>
		<code>flags.MEMBER</code>, <code class="light">// indicates the word is a member (identifier_word.<strong>member_word</strong>)</code><br />
		<code>flags.BRACKET</code>, <code class="light">// this operator is a bracket of some sort</code><br />
		<code>flags.BRACKET_CLOSE</code>, <code class="light">// this operator is a closing bracket</code><br />
		<code>...</code> <code class="light">// Additional token flag constants can be found by opening the library's source</code>
	</p>

	<p class="top_margin large_margin">
		For additional help, view some of <a class="link_codebase light_underline" href="https://github.com/nutbread/lex/tree/gh-pages/test" target="_blank"><span>these test files</span></a>, as examples are often more useful than wordy documentation.
	</p>

	<div class="top_margin js_on">
		<h1 id="demo"><span class="hardlink_text">Demo<a class="hardlink" href="#demo"></a></span></h1>
		<p>
			<div class="demo_options_table">
				<div class="demo_options_row">
					<div class="demo_options_cell">JavaScript:</div>
					<div class="demo_options_cell">
						<div class="demo_options"><a class="demo_option demo_option_selected" href="test/test1.js" target="_blank" data-target="test/test1.js" data-library="lexjs">test1.js</a><a class="demo_option" href="test/test2.js" target="_blank" data-target="test/test2.js" data-library="lexjs">test2.js</a><a class="demo_option" href="test/test3.js" target="_blank" data-target="test/test3.js" data-library="lexjs">test3.js</a><a class="demo_option" href="test/test4.js" target="_blank" data-target="test/test4.js" data-library="lexjs">test4.js</a><a class="demo_option" href="test/test5.js" target="_blank" data-target="test/test5.js" data-library="lexjs">test5.js</a><a class="demo_option" href="test/lex-test.js" target="_blank" data-target="test/lex-test.js" data-library="lexjs">lex-test.js</a><a class="demo_option" href="src/lexjs.js" target="_blank" data-target="src/lexjs.js" data-library="lexjs">lexjs.js</a><a class="demo_option" href="script.js" target="_blank" data-target="script.js" data-library="lexjs">script.js</a><a class="demo_option" href="highlighter.js" target="_blank" data-target="highlighter.js" data-library="lexjs">highlighter.js</a></div>
					</div>
				</div>
				<div class="demo_options_row"><div class="demo_options_cell demo_options_cell_sep"></div></div>
				<div class="demo_options_row">
					<div class="demo_options_cell">Python:</div>
					<div class="demo_options_cell">
						<div class="demo_options"><a class="demo_option" href="test/test1.py" target="_blank" data-target="test/test1.py" data-library="lexpy">test1.py</a><a class="demo_option" href="test/lex-test.py" target="_blank" data-target="test/lex-test.py" data-library="lexpy">lex-test.py</a><a class="demo_option" href="src/lexpy.py" target="_blank" data-target="src/lexpy.py" data-library="lexpy">lexpy.py</a></div>
					</div>
				</div>
				<div class="demo_options_row"><div class="demo_options_cell demo_options_cell_sep"></div></div>
				<div class="demo_options_row">
					<div class="demo_options_cell">CSS:</div>
					<div class="demo_options_cell">
						<div class="demo_options"><a class="demo_option" href="test/test1.css" target="_blank" data-target="test/test1.css" data-library="lexcss">test1.css</a><a class="demo_option" href="test/test2.css" target="_blank" data-target="test/test2.css" data-library="lexcss">test2.css</a><a class="demo_option" href="style.css" target="_blank" data-target="style.css" data-library="lexcss">style.css</a></div>
					</div>
				</div>
				<div class="demo_options_row"><div class="demo_options_cell demo_options_cell_sep"></div></div>
				<div class="demo_options_row">
					<div class="demo_options_cell">HTML/XML:</div>
					<div class="demo_options_cell">
						<div class="demo_options"><a class="demo_option" href="test/test1.xml" target="_blank" data-target="test/test1.xml" data-library="lexxml" data-params="xml">test1.xml</a><a class="demo_option" href="test/test1.html" target="_blank" data-target="test/test1.html" data-library="lexxml" data-params="html">test1.html</a><a class="demo_option" href="" target="_blank" data-target="index.html" data-library="lexxml" data-params="html">index.html</a></div>
					</div>
				</div>
			</div>
			<div class="demo_code codeblock codeblock_pre"><div class="codeblock_inner demo_code_inner"><div class="demo_code_placeholder"></div></div></div>
			<div class="demo_info">
				<p>
					A few reference images are included for <code>test1.js</code> for comparison of highlighting:<br />
				</p>
				<p><ul>
					<li><a class="light_underline" href="images/test1-firefox-scratchpad.png" target="_blank"><span>Firefox scratchpad</span></a></li>
					<li><a class="light_underline" href="images/test1-sublime-text.png" target="_blank"><span>Sublime Text</span></a></li>
					<li><a class="light_underline" href="images/test1-notepad++.png" target="_blank"><span>Notepad++</span></a></li>
					<li><a class="light_underline" href="images/test1-github-gist.png" target="_blank"><span>Github Gist</span></a></li>
					<li><a class="light_underline" href="images/test1-github-gist-ace.png" target="_blank"><span>Github Gist (ACE editor)</span></a></li>
					<li><a class="light_underline" href="images/test1-jsfiddle.png" target="_blank"><span>JSFiddle</span></a></li>
				</ul></p>
			</div>
		</p>
	</div>

</div>


</body>
</html>