We glad to inform you that the new version 2.45.170 of the WPFTS Pro is out now!
Some time ago, we have accumulated quite a lot of feedback from users who very much asked to add an improvement in search – to significantly increase the search speed on a large number of documents and add the ability to search for a phrase. This became somewhat of a challenge for us since it seemed almost impossible to implement a full-fledged indexed search using the small power of PHP and MySQL.
New Search Algorithm
But, as a result, we managed to get closer to the goal. To achieve this, the search algorithm had to be completely redone. Yes, it’s still TF-IDF, but with significant additions to the logic and relevance calculation. In particular, we have added a “phrase bonus”, which allows those documents in which similar phrases were encountered to be displayed in higher places. Note that phrases, in this case, can be not only EXACT matches but also partial matches of words, as well as nearby words (not only close ones).
For example, for the search phrase “infinite space”, the relevance will be increased in those documents where the phrases “infinite distant space” and even “space seems to be infinite” are found. The word order in the phrase is currently ignored. An option that includes strict word order in the phrase will appear in the next updates.
Deep Search is Faster
Also, the speed for “deep search” has been significantly increased. So, for example, by default, the query “cat” will only find words like “catty”, “catalog”, that is, those whose beginning exactly coincides with the search word. When you enable “deep search”, the words “subcategory”, “procatering”, “mystification” and the like will also be found, that is, words in which the “cat” occurs in the middle or at the end.
Previously, a “deep” search literally put the algorithm into a stupor and the search could take 10, 20, or even more seconds. Now, with our new algorithm, deep search has become much faster, and even on a large number of documents can take only 2-3 times longer than a regular “non-deep” search.
No Character Limit Anymore
Thanks to the new algorithm, it becomes possible to get rid of the 3 character limit on the length of the request. Previously, this limitation was necessary since the search for one or two characters set in motion almost the entire search index, as a result of which the query could run for a very long time and even lead to a script crash. Now it has become possible to remove the limitation on the number of characters since requests of one or two characters only receive a limited number of matches and do not significantly slow down or crash the script.
Faster with the Word Index
We have introduced one more small index – word index. It allows you to increase the search speed significantly for a large number of documents, since it partially relieves the load on MySQL, eliminating the need to perform concatenation operations on huge tables.
MyISAM Support was Dropped
We have dropped support for the MyISAM table type and removed the corresponding option and all associated algorithms from the code. Tables of this type all too often lead to “locking” and despite the fact that they sometimes give a 10-20% bonus to search speed, people decided to abandon MyISAM in favor of the more modern type as InnoDB.
Faster Re-Index
Indexing and re-indexing of documents have also been significantly accelerated, although the new “word index” slightly increases the re-indexing time. We are still working on improving this part of the plugin.
The “Pause” Mode
It’s not a secret that index building is a quite resource-consuming operation. Thus, sometimes it could be not appropriate to index new posts/files immediately. The “Pause” button which is located in the Status Box could help you with it. Just press it when you want to prevent the indexer from its hard job. But you should remember this and re-enable the indexer when you have a chance so the whole search index becomes actual.
Shortcode Content Indexing
Finally, we’ve added expanding the contents of the shortcodes and putting them in the search index. This thing is quite versatile and allows you to place the contents of any shortcodes into the index.
A new checkbox has been added to the plugin settings. If you do not need to index the content of the shortcodes, you can cancel it.
Search Widget Shortcode
You know there was an ability to use a special search widget with the autocomplete functionality. And now you can insert this widget to any place of the page with the help of the special shortcode: [ wpfts_widget ]. If you need this widget to follow some predefined rules (aka “Preset”) you can add a parameter: [ wpfts_widget preset=”my_preset” ].
Thank you for using WPFTS!
