Book analysis with AI techniques

A: Copyright law might prohibit the use of many modern books and songs. Project Gutenberg hosts texts that are out of copyright, often due to their age. Note that texts available on the site may not be suitable for all audiences.

A: IDE stands for Integrated Development Environment. Unless you are coding in Windows Notepad, you are probably using an IDE. IDEs gather the tools a programmer needs to type out code, test the program with the push of a button, and more easily identify errors. Code text is usually colour-coded to help with readability, documentation is readily available and syntax errors are underlined automatically, similarly to how spelling or grammar errors are identified in a modern word processor. Advanced IDEs also provide drag-and-drop tools to create graphical user interfaces.

A: HTML and other formats include information that is not part of the text itself. This information is used for attractive presentation, such as images, text formatting and page setup. As the name suggests, Plain Text does not have this extra information.

A: Punctuation can certainly change the meaning of a string of words. (eg. "Give to charity. Please no presents." vs. "Give to charity? Please, no. Presents!") The sentiment analysis in this learning sequence is less advanced than other approaches that might be in existence. It will rely on the volume and strength of certain words associated with other words.

A: Python needs something distinctive to make it clear where a text string starts and ends. Normally this is done with a quote (") or a single apostrophe (') at each end, but the string of punctuation characters already includes both these.

A: Functions allow a programmer to organise things by separating some code from the main program, so that section of code can be called (run) whenever required. Parameters then allow the main program to supply different values to the function. In the video, a parameter st is added so that any book text can be supplied to the remove_punctuation(st) function.

A: The naming of functions and variables is very important for code readability, so that other programmers can understand your code. One useful approach is to name functions with verbs, since they are performing an action. (Note: Using underscores in names is a Python convention rather than a rule. In other languages, like JavaScript, the convention is to use 'camel case', eg. removePunctuation(…))

A: Python's built-in split(…) function breaks up a string, resulting in a list that contains each separate part. If no argument is given, as in the case of creating a list of words, the string is split wherever a space occurs. A different character can be provided as an argument, such as a full stop ('.') to break up the string by sentences.

A: An additional parameter called exception is added. This allows one punctuation character to be designated to not be removed along with all the others. Inside the function, the exception character is immediately removed from the punctuation string, so that it does not get removed from the text itself.

A: When parameters are written with an = sign, this means there is a default value. It allows someone to call the function without having to supply an argument for that parameter. In this case, exception has a default value of an empty string ''. With this value for the parameter, the punctuation string will remain intact and the removal of punctuation will work as normal.

A: A modular approach to coding makes it easier for different programmers to work on a project. Programmer A can use a function from a file written by Programmer B without needing to see the code inside it, as long as there is documentation explaining how to use the function. Programmer B can make changes to the internal code in a function without necessarily disturbing Programmer A.

A: A common module imported into Python programs is the random module, which provides functions for generating pseudo-random numbers like rolling a die. Students may also be familiar with the turtle module, used to issue Logo commands to an onscreen turtle.

A: Polarity is a measure of how “positive” or “negative” a block of text seems to be. Sentiment Analysis modules like TextBlob draw on large lists of words compiled from research into linguistics.

A: The polarity value can range from -1 (very negative) to 1 (very positive). A value of 0 suggests a block of text that is not weighted in either direction.

A: Subjectivity can be thought of as a measure of how emotive the language in a block of text seems to be. A high value means that the text contains a high number of adjectives and nouns that are associated with strong feelings from the writer. A low value means that the text can be thought of as more “clinical” or objective, containing fewer emotive words.

A: The subjectivity value can range from 0 (completely objective) to 1 (highly subjective).

A: Despite its basis in linguistics research, this is still a crude means of determining sentiment when compared to a mature human’s analysis. This demonstrates the complexity of human communication. (Try inputting a sarcastic sentence, for example.) Developments in Machine Learning may result in more reliable sentiment analysis results.

A: The dictionary data structure consists of a set of pairs – keys and values. Our data is also a set of pairs. The key is the unique word from the book, and the value is the number of times that word appears in the book.

A: It would be feasible to use two parallel lists. One list would contain all the unique words and the other list would contain all the frequencies. This may not be considered an ideal solution because it relies on the lists always being kept in sync. For example, removing an element from one list means the same element must be removed from the other list. This leaves room for a programmer to forget, and then you have a bug.

A: The words parameter is expected to be a complete list of all words from the book, ie. the full book split up into its words. We already created a function create_word_list(…) to do this job.

A: The second section builds the dictionary by storing each unique word alongside the number of times it appears in the book.

A: The sorted(…) function is a very powerful function built into Python (see this article from the Python documentation for more). In our case, we are using it to sort the unique words from the dictionary containing the unique words and their frequencies. We sort the words from most to least common according to their frequencies. The result is not another dictionary, but a simple list called ranked_list.

A: The remove_common(words) function looks for the occurrences of 1000 well-known words within the full list of words in the book, removing any that it sees. The remove_stop_words(words) function does the same thing, but it only filters out a smaller selection of words like “if”, “and”, “the”.

A: The function measures the positivity of each paragraph in which the character appears by name. Each positivity value is used to increase or decrease an overall score. Finally, a verdict of “hero” or “villain” is pronounced based on the final score.

A: Clearly, the function does not always get it right. It is looking for positive or negative words associated with the character by being in the same paragraph.