The July Release of our RheinInsights Retrieval Suite

July 1, 2025

The July release of our Retrieval Suite brings another level of significant crawl performance improvements and hybrid search for retrieval augmented generation. This means, we integrated improved query pipelines which give the most relevant search results for natural language queries.

Crawl Rates in Development Environments

Our connectors achieve reliably high and stable crawl rates. Of course, crawl performance always depends on the following factors:

• content source,

• the sizing of the host machine where our Suite runs,

• the data target

• and the connected database

An extreme example is our file connector, which can crawl a local file system with a locally connected Microsoft SQL Server as data store and postponed text extraction with up to 1,500 documents per second in a full scan and 1,000 documents for a recrawl.

For remote sources, such as Atlassian Confluence data center, the connector achieves crawl rates with stable 100-120 pages per second for a full scan.

Of course, one always needs to balance index freshness and load on the source systems. Therefore, our connectors provide options to limit the load on the content sources.