Natural Language Processing
Natural Language Processing is an intersection between AI, machine learning and linguistics, focusing on processing human
languages such as English, Chinese, Japanese etc. using computers. The application areas of NLP include
information retrieval, text mining, sentiment analysis, dialogue systems, machine translation and
Tokenization - Breaking strings into tokens (each token is a word)
Stemming - Normalizes words into its base form or root form
Lemmatisation - Maps words into one common root (lemma) and outputs a proper word.
Paths of Speech Tags - Indicates how a word functions in meaning as well as grammatically
within a sentence.
Named Entity Recognition (NER) - Process of detecting the named entities such as person
names, company names, quantities, locations, etc.
Chunking - Picking up individual pieces of information (tokens) and grouping them together
into the bigger pieces (chunks).
Multiple intents in 1 question - Customer asks for different things at the same time.
To deal with this, AI could identify all the topics related to the customer’s question and display
them separately. In this way, the conversation can continue.
Assuming it understands context and has memory - this occurs when a customer does not
provide enough details about what they are looking for. Like for example you would like to apply
for a boat loan, but you just say “loan”. In this case, AI is able to ask for more details and
narrow information down as a result of that.
Misspellings in entity extractions - A good way of dealing with spelling mistakes is
to use machine learning and spell correction, these would help the system interpret what the
Same word, different meaning - This has to do with distinguishing the importance of
every word in a sentence. The idea is that conversational AI is able to identify which of the
keywords in any given sentence are most relevant to a user’s query in order to deliver the desired
Keeping the conversation going - AI should give you the tools for deeper control over
extract variables, determining the flow of a conversation based on user input.
Tackling false positives
- These happen when a user requests something that the system should be able to answer
but has not yet learned how. The solution here would be similar to point 4, which means identifying
the relevant parts of a message and providing assistance based on existing knowledge while acknowledging
the gaps in its ability.