D+8 Conclusion

There was a fire accident in Hsinchu not far from where Jason lives and the rains continued heavy a times.

Grace continued to work on the writing part and set messy semantic tagging and extending grammatical relations aside. Jason spent some time fixing the code for computing gappy bigram and mutual information. 

Initial Prompt

I want to write a bash script to cat unigram counts and gappy bigram counts to awk to compute pointwise mutual information using awk in one pass to store unigram counts and then use them to compute bigram pointwise mutual information

Follow-up prompt

Make the script as concise as possible

================================================

To make the script as concise as possible, we can feed the raw files directly into awk without prefixing tags. awk can distinguish them by tracking the input file sequence using ARGIND (in GNU awk) or by checking the number of fields (NF).Here is the ultra-concise version assuming your unigram file has 2 columns (count word) and your bigram file has 3 columns (count w1 w2):

See the code in the picture below

Efficiency & Condensation Tricks Used

No cat or Process Substitution: Files are passed directly as arguments to awk, saving system overhead and code lines.NF Routing: NF==2 automatically targets unigrams, and NF==3 targets bigrams.

Simplified PMI Math:

The formula \(\log_2 \left( \frac{c(w_1,w_2) \cdot N}{c(w_1) \cdot c(w_2)} \right)\) eliminates the need to calculate individual decimal probabilities first, reducing code lines and preventing underflow.Implicit

Jason spent more time contemplating on (1) semantic tagging and (2) extending grammatical relations.

(1) For semantic tagging, Jason had a conversation with the AI Genie inside of Google Search (The one that gives us AI Overview when we ask questions instead of search with a keyword query).

Prompt: Could you generate Python code to find common supersense of "collection, consultation, hire, inspection, purchase, sale, use"

https://www.google.com/search?q=Could+you+generate+Python+code+to+find+common+supersense+of+%22collection%2C+consultation%2C+hire%2C+inspection%2C+purchase%2C+sale%2C+use%22

This did not work. So Jason clicked AI Mode and received the following:

======================================================================

import nltk

from nltk.corpus import wordnet as wn

nltk.download('wordnet', quiet=True)

words = ["collection", "consultation", "hire", "inspection", "purchase", "sale", "use"]

# Extract supersense sets for each noun and find their common intersection

common = set.intersection(*(

    {s.lexname() for s in wn.synsets(w, pos=wn.NOUN)} for w in words

))

print(f"Common Supersense(s): {common}")

# Output: {'noun.act'}

======================================================================

import pymusas

import spacy

# Load the spaCy English model with PyMUSAS components

nlp = spacy.load("en_dual_none_contextual")

words = ["collection", "consultation", "hire", "inspection", "purchase", "sale", "use"]

doc = nlp(" ".join(words))

# Extract the primary USAS tag for each token

usas_tags = {token.text: token._.usas_tags[0][0] for token in doc}

print("USAS Tags:", usas_tags)

# To find commonalities across commerce, you will see a high density of 'I1' (Commerce/Money)

# and 'A1.1.1' (General Actions).

======================================================================

import nltk

from nltk.corpus import wordnet as wn

nltk.download(['wordnet', 'averaged_perceptron_tagger_eng'], quiet=True)

words = ["collection", "consultation", "hire", "inspection", "purchase", "sale", "use"]

genus_terms = {}

for w in words:

    syns = wn.synsets(w, pos=wn.NOUN)

    if syns:

        # Get the first dictionary definition

        definition = syns[0].definition()

        # Tokenize and part-of-speech tag the definition sentence

        tokens = nltk.word_tokenize(definition)

        tags = nltk.pos_tag(tokens)

        # Extract the very first noun (NN) or plural noun (NNS) as the genus term

        genus = next((word.lower() for word, tag in tags if tag in ('NN', 'NNS')), None)

        genus_terms[w] = genus

print("Dictionary Genus Terms:", genus_terms)

# Expected Output: {'collection': 'act', 'consultation': 'act', 'hire': 'payment', 'inspection': 'examination', 'purchase': 'act', 'sale': 'exchange', 'use': 'act'}

======================================================================

(3) Extending Grammatical Relations of collocation to grammar patterns of lengths > 2