🃏 Generative AI Flashcards

Q: What is generative AI?

A: A category of AI that can create new content (text, images, code, audio) based on patterns learned from training data.

Q: What is a transformer?

A: A neural network architecture that uses self-attention mechanisms to process sequences in parallel, enabling efficient training on large datasets. Foundation for most modern LLMs.

Q: What is a token in the context of LLMs?

A: The basic unit of text processed by a language model — can be a word, subword, or character. Models have a maximum context window measured in tokens.

Q: What is the difference between pre-training and fine-tuning?

A: Pre-training: initial training on massive unlabeled data to learn general patterns. Fine-tuning: additional training on task-specific labeled data to specialize the model.

Q: What does the temperature parameter control?

A: Randomness/creativity of outputs. Low temperature (0) = deterministic and focused. High temperature (1+) = more random and creative.

Q: What is top-p (nucleus sampling)?

A: A parameter that limits token selection to the smallest set of tokens whose cumulative probability exceeds p. Lower top-p = more focused outputs.

Q: What is an embedding?

A: A dense vector representation of text (or other data) that captures semantic meaning. Similar concepts have similar embeddings.

Q: What is a context window?

A: The maximum number of tokens a model can process in a single request, including both input and output tokens.

Q: What is transfer learning?

A: Using knowledge gained from pre-training on one task/dataset and applying it to a different but related task, reducing the need for task-specific training data.

Q: What is a hallucination in AI?

A: When a model generates plausible-sounding but factually incorrect or nonsensical information. Mitigated by RAG, guardrails, and human verification.