Python

Learning Phrases with PyLucene and Pytorch, part 2.

In part 2 we reuse our tokenised index and use pytorch to build a model for significant phrase extraction. It worked surprisingly well and being able to switch Analyzers proved useful. We found that the English Analyzer with stopword removal and stemming worked best.

The results are indicative, neither the dataset size or the length of training cycles are sufficient for the development of a genralised phrase extractor but the succcess and ovelap found between pylucene and pytorch is very encouraging. We just need to scale it up.

Continue reading

Learning Phrases with PyLucene, part1

Lucene is a library that is used to build text search optimised indexes, it is an Apache project and is the core file format sitting under ElasticSearch. The following code uses pylucene which is a JNI wrapper to the Java API.

The algorithm derives from the idea that the terms in search results will have increased frequencies for their search terms and associated concepts. Phrases similarly should have increased frequencies. By using Lucene it is fast, though we have to index first. The code also demonstrates an integration between Lucene and Pandas for analytics. The technique here could be used summarise in aggregate user / player entered text in surveys, reviews etc. That might otherwise get ignored by analytics.

Continue reading

Areas of Expertise

At ACUMED Consulting, we specialise in using data science to drive progress, innovation and success within businesses. We have concentrated experience in Gaming Analytics, but have worked in cybersecurity, fraud detection, marketing, banking, insurance, telecommunications and government sectors too. Our experts are skilled in a range of data science and analytics areas including:

Analytical Methods and Problem Solving

Reducing dimensions Time series analysis Anomaly detection - finding unusual behaviour

Machine Learning

Continue reading