Hackers' Pub

Syntax	Description	Examples
`"` keyword `"`	Finds the string within quotes, including spaces. Case-insensitive. (Escape quotes inside with `\"`)	`"Hackers' Pub"`
`from:` handle	Finds content written by the specified user.	`from:hongminhee` `from:hongminhee@hollo.social`
`lang:` ISO 639-1	Finds content written in the specified language.	`lang:en`
`#` tag	Finds content with the specified tag. Case-insensitive.	`#HackersPub`
condition condition	Finds content that satisfies both conditions on either side of the space (logical AND).	`"Hackers' Pub" lang:en`
condition `OR` condition	Finds content that satisfies at least one of the conditions on either side of the OR operator (logical OR).	`#HackersPub OR "Hackers' Pub" lang:en`
`(` condition `)`	Combines the operators within the parentheses first.	`(#HackersPub OR "Hackers' Pub" OR "Hackers Pub") lang:en`

Hacker News 50 @hn50@social.lansky.name

12/12/2025, 3:30:06 PM

Public

From text to token: How tokenization pipelines work

Link: https://www.paradedb.com/blog/when-tokenization-becomes-token
Discussion: https://news.ycombinator.com/item?id=46232003

From Text to Token: How Tokenization Pipelines Work

Understanding how search engines transform text into tokens through character filtering, tokenization, stemming, and stopword removal.

www.paradedb.com

If you have a fediverse account, you can quote this note from your own instance. Search https://social.lansky.name/users/hn50/statuses/115707308070844225 on your instance and quote it. (Note that quoting is not supported in Mastodon.)