Hi, Giovanni

Recently we published a new version of the WPFTS plugin which uses a completely new search algorithm (v 1.30.85). It was built with large database support in mind. Did you try that? Please tell me if you have better speed results with it or not.

It still has some room for improvement (via temporary tables, for example) which will be added in the nearest versions of the plugin.

A new version of WPFTS with a fundamentally new search algorithm is out now!

Some time ago, we have accumulated quite a lot of feedback from users who very much asked to add an improvement in search - to significantly increase the search speed on a large number of documents and add the ability to search for a phrase. This became somewhat of a challenge for us since it seemed almost impossible to implement a full-fledged indexed search using the small power of PHP and MySQL.

New Search Algorithm

But, as a result, we managed to get closer to the goal. To achieve this, the search algorithm had to be completely redone. Yes, it's still TF-IDF, but with significant additions to the logic and relevance calculation. In particular, we have added a "phrase bonus", which allows those documents in which similar phrases were encountered to be displayed in higher places. Note that phrases, in this case, can be not only EXACT matches but also partial matches of words, as well as nearby words (not only close ones).

For example, for the search phrase "infinite space", the relevance will be increased in those documents where the phrases "infinite distant space" and even "space seems to be infinite" are found. The word order in the phrase is currently ignored. An option that includes strict word order in the phrase will appear in the next updates.

Deep searching is faster

Also, the speed for "deep search" has been significantly increased. So, for example, by default the query "cat" will only find words like "catty", "catalog", that is, those whose beginning exactly coincides with the search word. When you enable "deep search", the words "subcategory", "procatering", "mystification" and the like will also be found, that is, words in which the "cat" occurs in the middle or at the end.

Previously, a "deep" search literally put the algorithm into a stupor and the search could take 10, 20, or even more seconds. Now, with our new algorithm, deep search has become much faster, and even on a large number of documents can take only 2-3 times longer than a regular "non-deep" search.

No character limit anymore

Thanks to the new algorithm, it becomes possible to get rid of the 3 character limit on the length of the request. Previously, this limitation was necessary because searching for one or two characters set in motion almost the entire search index, as a result of which the query could run for a very long time and even lead to a script crash. Now it has become possible to remove the limitation on the number of characters since requests of one or two characters only receive a limited number of matches and do not significantly slow down or crash the script.

Faster with the Word Index

We have introduced one more small index - word index. It allows you to increase the search speed significantly for a large number of documents, since it partially relieves the load on MySQL, eliminating the need to perform concatenation operations on huge tables.

MyISAM support was dropped

We have dropped support for the MyISAM table type and removed the corresponding option and all associated algorithms from the code. Tables of this type all too often lead to "locking" and despite the fact that they sometimes give a 10-20% bonus to search speed, people decided to abandon MyISAM in favor of the more modern type as InnoDB.

Faster re-indexing

Indexing and re-indexing of documents have also been significantly accelerated, although the new "word index" slightly increases the re-indexing time. We are still working on improving this part of the plugin.

Work on improving the plugin is very active, only about 40% of all planned improvements for this year have been implemented. Therefore, your feedback and understanding that bugs may occur are very important to us. If you notice an error or inaccuracy, please report it immediately on the plugin support forum.

Let me remind you that I have a Patreon page. If you want to support the development of this awesome plugin, I would be very grateful.

Thank you for using WPFTS!

Yes, you can do that. But it will need some coding.

First of all, we need to extract a PDF link or file path from the post. It should be done at indexing stage, inside the 'wpfts_index_post' hook function.

After the link/path extraction, we should read this file and extract the text information from it. It still should be done inside the same hook.

Okay, when we have the text, we put it to one of the index clusters (your preference) and let it to the indexer to make its job as well.

The final code will depend from the way how you have attached the PDF file to the parent post. It can be done using very different ways.

Please give me some more explanation of this and I will post the semi-ready code here.

It should work with Korean because it works with UTF-8 encoding.

Conor Campbell asked:

One of my clients utilises your Wordpress plugin to index documents on their site.

It works great, however I am unable to get excerpts to appear for
PDFs, as they're using a custom search results page. What
information do you require from me in order to investigate this?

Looking forward to your response.

Yes, the Smart Excerpt function is only activated automatically when you
are making WP Main Search. In case you want to force Smart Excerpts for
custom WP_Query() request, you need to do this:

global $wpfts_core;

$result = new WP_Query(array(
     's' => $string_query,  // A search string which you're searching for
    ...other your parameters to filter results...
));

// This will enable Smart Excerpts
$wpfts_core->ForceSmartExcerpts($string_query);

// Here is your code to loop over $result to view search results
// You need to use the_excerpt(); to show Smart Excerpts

// Check that we have query results.
if ( $query->have_posts() ) {

    // Start looping over the query results.
    while ( $query->have_posts() ) {
 
        $query->the_post();
 
        // Contents of the queried post results go here.
        // the_title();
        // the_excerpt();
        //....
    }
 
}
 
// Restore original post data.
wp_reset_postdata();

// You may disable Smart Excerpts if you need to (OPTIONAL)
$wpfts_core->ForceSmartExcerpts(false);

Hope this helps.

Well, the reason is that TemplateToaster authors added their own WP_Query() request in the search.php file. This request has custom parameters and thus WPFTS Pro functionality becomes disabled.

What we should do is to comment out line 14 in the search.php which removes that additional request and so normal WP search (empowered with WPFTS Pro functionality!) becomes working normally.

Simple trick.

Hi Andrew
Please could you send me your search.php file to the epsiloncool@gmail.com or attach it here. Should be Divi has changed the structure of the file and we need another patch now.

Q: We have a script which is adding new files to the WP Media Library in a timely manner, using a cron job. Those files not shown in the search I guess because they are not indexed. We have to log in once in a couple of days to Rebuild Index manually. Is there any better solution to fix this?

A: Yes. Normally when you're editing or adding posts/attachments manually (via WP Admin), each "Save" button click triggers the WP hook "save_post" which is forcing WPFTS Pro to rebuild the index for this particular post.
In case you're making changes by some script, you have to include this code to your script (after all post changes)

wpfts_post_reindex($post_id);

This will force reindex of this particular post so you never need to Rebuild the whole index after post changes.

There is a nice plugin that allows you to create your own knowledgebase using Wordpress.
But unfortunately, it's authors decided to disable search for all articles with its custom post_type (which is kbe_knowledgebase). As a result, WP Knowledgebase articles do not show in WPFTS search results. However, these articles are indexed well and you can see them while test the search in Sandbox.

To fix this, we have to re-enable the search for Knowledgebase posts.

Please put this small code snippet to your functions.php file in the main theme root.

add_action( 'registered_post_type', function( $post_type, $args ) 
{
    // Make sure we're only editing the post type we want
    if ( 'kbe_knowledgebase' != $post_type )
        return;

    $args->exclude_from_search = false;

    // Modify post type object
    global $wp_post_types;
    $wp_post_types[$post_type] = $args;
}, 10, 2 ); 

What does it do? Pretty simple. The flag exclude_from_search was set to true inside the WP Knowledgebase plugin and thus, these custom posts are not included to search.

This simple snippet fixes this.

Have fun!

When you are using Divi Builder template to construct Search Results page, it may be hard to get Smart Excerpts visible for PDF files. It's because Divi developers remade standard excerpting functionality by simple cutting.

Well, you can fix it with small patch.

Just open the file Divi/includes/builder/module/Blog.php and search for line 1814 (approx). You can find the line of code with "truncate_post()" inside. Comment out this line and add the new line below as specified on the screenshot:

P.S. Different Divi versions may have different line numbers etc.

Have fun!