Natural Language Processing


Definition

Natural Language Processing is an intersection between AI, machine learning and linguistics, focusing on processing human languages such as English, Chinese, Japanese etc. using computers. The application areas of NLP include information retrieval, text mining, sentiment analysis, dialogue systems, machine translation and text summarisation.

Process

  • Tokenization - Breaking strings into tokens (each token is a word)
  • Stemming - Normalizes words into its base form or root form
  • Lemmatisation - Maps words into one common root (lemma) and outputs a proper word.
  • Paths of Speech Tags - Indicates how a word functions in meaning as well as grammatically within a sentence.
  • Named Entity Recognition (NER) - Process of detecting the named entities such as person names, company names, quantities, locations, etc.
  • Chunking - Picking up individual pieces of information (tokens) and grouping them together into the bigger pieces (chunks).

Challenges

  • Multiple intents in 1 question - Customer asks for different things at the same time. To deal with this, AI could identify all the topics related to the customer’s question and display them separately. In this way, the conversation can continue.
  • Assuming it understands context and has memory - this occurs when a customer does not provide enough details about what they are looking for. Like for example you would like to apply for a boat loan, but you just say “loan”. In this case, AI is able to ask for more details and narrow information down as a result of that.
  • Misspellings in entity extractions - A good way of dealing with spelling mistakes is to use machine learning and spell correction, these would help the system interpret what the customer entered.
  • Same word, different meaning - This has to do with distinguishing the importance of every word in a sentence. The idea is that conversational AI is able to identify which of the keywords in any given sentence are most relevant to a user’s query in order to deliver the desired outcome.
  • Keeping the conversation going - AI should give you the tools for deeper control over extract variables, determining the flow of a conversation based on user input.
  • Tackling false positives - These happen when a user requests something that the system should be able to answer but has not yet learned how. The solution here would be similar to point 4, which means identifying the relevant parts of a message and providing assistance based on existing knowledge while acknowledging the gaps in its ability.