(frankentext)=
# FrankenText

:::{commons-figure} https://commons.wikimedia.org/wiki/File:Frankenstein,_or_the_Modern_Prometheus_(Revised_Edition,_1831)_006.jpg
:figwidth: 35%
:align: right
Inside cover art from the 1831 edition of the book. Victor Frankenstein looks at his creation. 
:::

We will write a program that generates random sentences based on the book [Frankenstein](https://www.gutenberg.org/ebooks/84) by Mary Shelley.

We will create tokens from the text and create a table that tracks which tokens come after which token.

:::{wpd} token
:id: Lexical_analysis#Token
A piece of a string.
:::

The algorithm is as follows:

1. Embed the book into a string.
1. Replace each non-printable character with a space
1. Read tokens delimited by " ?!" and store them in an array named `tokens`. 

   At the same time update a successor table that tracks which token depends on which token (`succs`).

   There must be no copy of a token in the array and tokens are case-sensitive.

   `ruin!` and `ruin` are different tokens.

1. Generate random sentences:

   Select a random token that starts with a capital letter and continue appending random successors from the successor table until we encounter a token that ends a sentence.

Example output:

> :::{include} ../code/frankentext.txt
> :::

The sentences generated may not make a lot of sense, but the words are not completely random.

## Word and successor tables

The program tracks `tokens` and successors of each token in the successor table `succs`. For example for the following text:

> The modern the modern apes!

produces the following `tokens` and successor table `succs`:

::::{grid} 2
:::{grid-item}
```{list-table}
* - 0
  - "The"
* - 1
  - "modern"
* - 2
  - "the"
* - 3
  - "apes!"
:::
:::{grid-item}
```{list-table}
* - 0
  - {"modern"}
* - 1
  - {"the", "apes!"}
* - 2
  - {"modern"}
* - 3
  - {}
```
:::
::::

## Template

Before using the template, download the plain-text version of the book using the following link:

{#pg84-download-link}
<https://www.gutenberg.org/ebooks/84.txt.utf-8>

Some of the functionality is already provided and we will live-program some of the functionality.

:::{activity} Analyzing code
1. Look at the function `replace_non_printable_chars_with_space()`. It does not take any arguments nor return anything. How can it replace characters in the book?
   <!-- refer to last class where we talked about pass-by-reference. But here we have direct access, there are no arguments. -->
1. `generate_sentence()` uses `tokens` in its definition, even it is not provided as an argument. What do you think about this coding style?   
    <!--
    Mention that even global variables are frowned upon, here we use the global variables almost in every function, so it does not make sense to add them as parameters.
    
    Actually we also use functions which we don't provide as an argument, e.g., `printf`. The question is, what should be global, what local.
    
    Connect to object-oriented-programming, where we create our small world.
    -->
:::

:::{literalinclude} ../code/masked/frankentext.c
:language: c
:::

## Notes

:::{warning}
On Windows: If you increase `MAX_WORD_COUNT` to `50'000`, then debugging fails with an unknown error.
:::
:::{note}
We don't copy strings from `book`, but work with pointers to the `book`.
:::
:::{tip}
If it takes too long to run the program or the debugger, create a smaller version of the text file.
:::
:::{tip}
Implement & test step by step. Look at main, at look at the first function you have to implement. After implementing, set a breakpoint after the function and test whether your implementation is correct or not.
:::
:::{tip}
It may take long time for the `VARIABLES` window to show the contents of the large array `book`. Use instead the *watch* feature to track components of arrays.

How? Select a string, e.g., `book[i]`, then right-click, and then click `Add to Watch`. 
:::
:::{tip}
You will probably get segmentation faults during development. The debugger shows you the state when the fault happened. Use it.

Moreover the debugger can also show you how you got to the fault. Look at `Call Stack` to see the path.
:::

## Requirements

1. You only fill `// YOUR CODE HERE` parts.
1. Include example output of your program in your documentation.
1. Flowchart

## Appendix

- We actually create a [Markov chain](https://en.wikipedia.org/wiki/Markov_chain).