Sorting of Search Results

[Nick]

Is there a way to modify the relevance sort weightings when applied to attachments, particularly pdfs, based upon the number of times the search term appears within the document? (i.e. the more often the term appears, the higher the relevance).

[EpsilonAdmin]

Hi @Nick

It's a good task to think! Mainly the relevance calculation is based on the TF-IDF algorithm, which means the number of terms also divided by the length of the document, so we actually get not exactly the number of terms, but a terms density.

However, the WPFTS is very flexible and I think the algorithm you need can be implemented with some code magic.

Could you explain please why you need such an algorithm and how it exactly should be implemented?

[Nick]

Most of the documents we have on our website are pdfs of our Journals, that were originally paper copies going back to 1955. Each Journal typically contains 6 - 8 substantial articles of about 10 pages each, plus many shorter articles and including comments on earlier articles.
So, the more times a search term is mentioned in an article and, particularly, if the term is included in the title of the article, the more likely it is that the article is specifically about the search term or something closely related to it.
On the assumption that WPFTS can't tell where one article ends and another begins, the number of instances within the document would be a good proxy.

[EpsilonAdmin]

Okay @Nick

So the main issue is that we can not extract separate articles from the PDF and assign more weight to their titles, correct?

Your solution looks reasonable, and we can implement it. But before that, do you think we can try to extract separate articles from the PDF or (which is even simpler) extract article titles. We could store these titles into the separate index cluster and assign this cluster more weight. It could work even better than weighting relevance on a number of terms.

Could you please send me one or two typical PDFs? It worth trying to extract titles from them using some heuristic algorithm.

Also, could you tell me how much PDFs you have in total? Will it be a headache to extract titles manually?

[Nick]

We have over 1,000 pdf documents on our website, and the most important is our Journal, with 238 editions at present. As an example, a couple of issues of our Journal are here, but they are all similar in structure:
https://rchs.org.uk/wp-content/uploads/2020/02/Journal-100-Nov-1975.pdf
https://rchs.org.uk/wp-content/uploads/2020/02/Journal-001-Jan-1955.pdf
You will see that at the bottom of the second page of Journal 100 is a table of contents for the issue and, if the pdf is saved and then opened in Acrobat Reader, there is an equivalent set of bookmarks. It would help if a search term appearing in the article title (as listed in the table of contents) was given a higher weighting than one in the text of the document, but quite often the term will only occur in the text, and not in the title at all.
Ideally, we would like the weighting to be based upon articles, but weightings based upon the the whole Journal is acceptable. This is because the articles tend to be on unrelated topics within a Journal, so there is probably little difference between the number of instances of a specific search term within an article, and within its parent document.
I think it would help me if you could explain, in non-technical terms, how the four weightings operate with WPFTS. I've looked at the TFIDF article on Wikipedia, and understand the basics, but the majority of the article is too technical for me. Perhaps this information could also be added to the WPFTS documentation?
As a related issue, could WPFTS open the document listed in the search results at the first article page where the search term is found? I realise it might go to the article title instead, and if so that wouldn't really help much.