Linux Academy Notes

101.1 Determine and configure Hardware settings

Pseudo (fake) File System:

Everything in Linux is seen as a file and it is stored in the RAM.

Once the session is powered off the and the RAM gets cleared, this file system disappears.

Once the machine is reopened it creates a new file System for the new session which is kept in RAM.

/proc - Contains information about the processes running on a system.

Processes are listed by PID with hardware and process data both in the same directory structure.

/sys - Contains information about the system’s hardware and kernel modules. No process information is listed here.

man proc - shows local documentation on the /proc pseudo file system.

Working with Kernel Modules:

Kernel is the core framework of the OS

GNU/Linux OS - Contains Bash commands, shells, what the system needs to function.

Linux kernel is monolithic: meaning that it handles all memory management and hardware device interactions

Has the ability to dynamically load and unload drivers.

uname - Displays information about the currently running Kernel.

lsmod - Displays a listing of all currently loaded kernel modules.

modinfo - Displays information about a specified Kernel module.

modprobe - command used to dynamically load and unload kernel modules at runtime.

modprobe -r <kernel module name> - to unload module

modprobe <kernel module name> - to load module 

Example: Kernel module Floppy

Investigating Hardware:

/dev- contains information on all of the connected hardware on a system

udev - the device manager for the Linux Kernel, links information on system hardware to /dev

D-Bus - Sends data messages between applications, a conduit of information about what’s going on the system

udev utilizes dbus to notify users and the system when new hardware is attached.

lspci - Displays information on PCI devices attached

lsusb - Displays information on the USB  devices attached 

lscpu - Displays information on processor on a system

lsblk - Displays information on all block devices on a system -pulled from /dev

101.2 Boot the System

The Linux boot Sequence:

dmesg - the traditional utility used for viewing the kernel ring buffer.

journalctl -k - systemd utility to view kernel ring buffer within the systemd journal. 

INIT: Short for initialization

When machine starts the Kernel looks for the program in /sbin/init. Once this starts init is in control.

First thing init does is it looks for the configuration tab in /etc/inittab

This also holds the Runlevels of the OS

Predefined configuration that the computer will operate within. Applies to system as a whole.

Traditional Scripts for Stop/Start for the system to function are in:

RH = /etc/rc.d/
Debian = /etc/init.d

/etc/init.d - contains the scripts for the services on the system

/etc/init.d/rc. - scripts that orchestrate how the runlevel scripts run and what occurs when a runlevel changes.

Upstart: - Boot Sequence

Upstart allows for multiple processes to run simultaneously rather than the old init system where it has to wait for a sequence to finish before it can start.

Init is static; it does not natively respond to changes on a system (manual config)

Upstart is dynamic, it can respond to changes on the system.

Ex: Plugging in a monitor, init will not recognize this and will have to be set up manually, where upstart will do this automatically.

When something on a Linux system changes it is known as an event. When an event is triggered it will start and upstart job.

Jobs are split in 2 categories: Tasks & Services

Task: will do what it is required of it and then it will return to a waiting state once it is finished. (one time use)

Service: will not stop by itself and will only kill a service if an event calls for it, or if an admin does so.

Job flow:

Initial State: waiting (to do something)

Starting where the job is required.

Running: the job is actually processing what it is supposed to do If it fails it will enter respawn state, where the system tries to run the task up to 10 times before it drops it as failed.

Stopping: where the job has finished what it is supposed to do, but it is not stopped just yet.

Killed: request for the job to be terminated as it is finished

Post-stop: where the job is actually finished.

After that the status becomes waiting again.

systemd:

init and parts of upstart rely on Bash shell scripts which require the bash interpreter to run, systemd has removed this.

Systemd is backwards compatible, but not guaranteed 100% with all functions.

Unit File Locations:

Provided by package installation: /usr/lib/systemd/system

These should not be modified as they might be susceptible to change by an update.

Unit file location for system administrators: /etc/systemd/system

These take precedence over those in /usr

Runtime unit files: /run/systemd/system

View all unit files on system: systemctl list-unit-files

FULL DOC for the Unit File: man 5 systemd.unit

systemctl cat <something>.unit will print out content of unit file specified.

Systemd boot system:

Kernel still looks for /sbin/init

Systemd just took the place of /sbin/init

Developers have created a symbolic link for /sbin/init to redirect to /lib/systemd/systemd which actually boots systemd.

101.3 Change Runlevels/Boot Targets and Shutdown or Reboot the System:

runlevel - view current runlevel which applies on the system as a whole

telinit - change to another runlevel (alternative command: init )

In order to change runlevel you have to be the root user

Changing runlevels at boot:

Interrupt the GRUB boot process by pressing any key during startup, then at the GRUB selection menu highlight a kernel to modify.

Press the “a” key to add arguments to the end of a kernel line, enter in a runlevel number.

Change your Working Environment: targets:

A target unit is a unit that syncs up to other units when the computer boots or changes states.

Often used to bring system to a new state.

Types of Targets:

multi-user.target - multi-user system similar to what runlevel 3 provide

graphical.target - multi-user system with desktop environment, similar to what runlevel 5 provides

rescue.target - pulls in a basic system and file system mounts and provides a rescue shell

basic.target - basic system, used during the boot process before another target takes over

sysinit.target - system initialization

Target Units:

systemctl list-unit-files -t target - Show all unit files available targets

systemctl list-units -t target - Show all loaded and active unit files

systemctl get-default - list out the default target

systemctl set-default - change the default target to a different target

systemctl isolate <target> - will change the running state of the system from the current target to a different target

systemctl rescue - almost identical to the system v init single-user mode, allows the root user to repair the system. 

systemctl reboot - will run the reboot.target, same thing as just typing “reboot” at the prompt

systemctl poweroff - will perform a complete shutdown of the system by isolating the system to the poweroff.target, same as typing “poweroff” at the prompt

Reboot and Shutdown your System:

Reboot Commands:

reboot

telinit 6

shutdown -r now

systemctl isolate reboot.target

wall -  broadcasts a message to all logged in users (after message is typed, terminate the message with Ctrl+D

Shutdown Commands:

poweroff

telinit 0

shutdown -h 1 minute

systemctl isolate poweroff.target

acpid - Advanced Configuration and power interface, registers system events (such as pressing the power button or closing laptop lid)

Configuration file is under /etc/acpid

102.1 Design Hard Disk Layout:

Main File System Location:

/ - root direcotry 

/var - variable location, logs and dynamic content (such as websites)

/home - the user home directory, where personal files are stored

/boot - the boot directory, where the Linux kernel and supporting files are stored

/opt - Location user for optional software, often used by 3rd party software vendors

Swap Space - is temporary storage that acts like RAM, when a % of RAM is full, the Kernel will move less used data to swap

Swap partition (most common setup)

Swap file (similar to the page file on a Windows OS) - Much slower performance then using a dedicated partition; 

Sizing: Older rule of thumb ranged from 1.5x to 2.0x the size of available RAM RAM is cheaper and it’s up to admin to set up SWAP space, but usually not less than 50% of RAM

Partitions and Mount Points:

/dev/sda - sda is full partition, usually that will divided in multiple sections assigned as /dev/sda1 ; /dev/sda2 ; /dev/sda3, etc.

Mount point: is when you take a partition or disk and mount it to a directory, for example /home - meaning it will house all the home directories.

mount - Can be used to mount partitions to directories, or show all existing mounts without any options

lsblk - Used to show all block devices on a system and their name

fdisk -l /dev/<diskname> -  Can be used to list out partition information on the specified disk

swapon --summary - Shows a summary of the swap usage on a system, same info can be found in /proc/swaps

Introduction to LVM: (Logical Volume Manager)

Allows the creation of “groups” of disks or partitions that can be assembled into a single (or multiple) filesystems:

• Can be used for nearly any mount point EXCEPT /boot

• Flexibility, allows for resizing of volumes

• Snapshots - allows a point in time “photo” to be taken of the volume itself and is used as a backup

PV - Physical Volumes; VG - Volume Group; LV - Logical Files;

pvs (physical volume scan) - Lists out the physical volumes in an LVM group 

vgs (virtual groups scan) - Lists out the volume groups within an LVM group

lvs (logical volume scan) - Lists out the logical volumes within a LVM group

102.2 Install a Boot Manager:

Legacy Grub (Grand Unified Boot Loader)

BIOS → The first 512 bytes is a boot image searched → which searches for the core image file → which loads the /boot/grub partition and allows for the system to actually load.

Grub configuration file: grub.conf/menu.1st device.map

How to install GRUB:

grub-install <device> 

Device can be /dev/sda or /dev/hd0 …. ETC

grub - invokes the GRUB shell Environment

help - prints the help listing for GRBU, or get more info on command help <command>

find - search for a file in all partitions and list the devices the file is on

quit - exit the GRUB shell

GRUB2

MBR (Master Boot Record)

Supported only 26 total partition (4 partitions with one partition extended to 23 partitions)

Partition size limited to 2TB

GPT (GUID Partition Table)

Supports 128 Partitions

Partition size up in the ZB range

Needs UEFI (Unified Extensible Firmware Interface) to boot:

Replacement for traditional BIOS cand act in legacy BIOS mode

Requires a 64bit OS

Prevents unauthorized OS from booting on the System

grub2-editenv list - view the default boot entry for the grub configuration file

grub2-mkconfig - creates or updates a /boot/grub2/grub.cfg file based on entries from the /etc/default/grub file

(on Debian systems, the “2” is omitted from the command name

update-grub - command that can be used to update a GRUB2 configuration after changes to /etc/default/grub have been made, found on Debian based systems

Interacting with the BOOT LOADER:

GRUB Legacy:

A Key - Append options to the kernel boot line

C Key - Open up the GRUB command line

ESC Key - Escapes out of any GRUB menu

Arrow Keys - Used to highlight an option in the GRUB menu
GRUB2 

E Key - Used to edit a GRUB menu item

ESC Key to go to previous menu item

Ctrl + x or F10 - boots a selection or modified line

On SystemD you will have to use system.unit=rescue.target to boot in runlevel 1

102.3 Manage Shared Libraries:

Files that contain functionality that other applications can use.

These files end in a “.so” extension for shared object

Found under the following locations:

/lib

/usr/lib for 32bit systems; /user/lib64 for 64 bit systems

/usr/local/lib

/usr/shared

Two types of library files:

Dynamic (ends in .so) Statically Linked (ends in .a)

ldd - prints out shared object dependencies

ldconfig - Configures dynamic linker run-time bindings, creates a cache based on library directories and can show you what is currently cached. 

/etc/ld.so.conf - Configuration file that point to directories and other configuration files that hold reference to library directory locations

LD_LIBRARY_PATH - Legacy environment variable that points to the path where library files can be read from 

102.5 Use RPM and YUM Package Management

YUM: Yellowdog Updater modified

• Handles RPM package Dependencies

• Installs, upgrades and removes packages

• Users on RHEL, CENTOS, Scientific Linux, older versions of Fedora

YUM setup:

• Global yum configuration options are set in /etc/yum.conf

• Reads repository information from /etc/yum.repos.d

• Cache latest repository information in /var/cache/yum

Other RPM Package Managers:

Zypper:

Used on SUSE Linux Distributions

Ex: zypper repos; zypper install vim

DNF - Dandified yum:

• Used on Fedora Linux distributions

• Future replacement for yum in RHEL

Uses same command syntax as yum

yum update - searches online repositories for updated packages compared to what is currently installed on the system, upgrades packages

yum search - searches the yum repositories for a specified package

yum info - lists information about a specified package

yum list installed - displays all installed packages

yum clean all - cleans up all of yum’s cache information and its local database file

yum install - installs a specified package and all of its dependencies

yum remove - uninstall a package, leaves dependencies behind

yum autoremove - uninstalls a package and its dependencies

yum whatprovides - find out what package provides a specific file name

yum reinstall - installs a specific package

yum-utils - provides extra utilities such as yumdownloader that allows you to download the package without actually installing it to see the content of it first. 

RPM (Redhat Package Manager):

The .rpm package contains:

• Application or utility

• Default configuration file

• How and where to install the files that come with the package

• Listing of dependencies that the package requires.

The rpm database: located in /var/lib/rpm

Use the rpm --rebuilddb command to repair corrupted rpm database

Dependencies need to already be installed or installed with the package

yum handles dependencies for you, rpm does not.

rpm -qpi - Displays information on a package

rpm -qpl - lists files in a package

rpm -qa - lists out all installed packages

rpm -i - installs a specified package, often combined with other options to provide more verbose output ex: rpm -ivh

rpm -U - upgrades an installed package with a newer version

rpm -e - uninstalls (erases) an installed package

rpm -Va - verify all installed packages

rpm2cpio - converts an .rpm file into a cpio archive file, often combined with the cpio command

ex: rpm2cpio some.rpm | cpio -idmv

102.6 Linux as a Virtualization Guest:

Virtualization and Containers:

Emulations of a specific computer type

Operate based on the architecture and functions of a real computer and its implementation, can involve specialized hardware, software or both.

Virtualization Software allows you to set up on OS within another.

• They both share the same physical hardware

• The virtual machine is isolated from that hardware and has to communicate with in through a Hypervisor

Examples: KVM; QEMU; VMWare; Xen; VritualBox

2 Main types of Virtualization:

• Full Virtualization - guest system is not aware that it is a virtual machine

• Paravirtualization - guest system is aware that it is a virtual machine, uses guest drivers Virtual machines typically perform better with guest drivers

Virtual machines can be “cloned” or turned into templates to rapidly deploy new systems

NOTE: When working with Linux VMs you may need to change the system’s D-Bus Machine ID - done with dbus-uuidgen - ensures that each running Kernel interacts with system that has a unique ID

Virtual servers can be provisioned from cloud providers:

If you are using a cloned virtual machine, cloud-init is typically used to insure that user data is completely new

• Creates new SSH Keys

• Sets the system’s default locale

• Sets the system’s hostname

• Sets up mount points

Example: AWS, Azure

What is container:

An entirely isolated set of packages, libraries, and/or applications that are completely independent from their surroundings

2 Types of Containers:

• Machine container: Shares a kernel and file system with the host computer

• Application Container: Shares everything but the application files and library files that the application needs

Why is the difference important?

Virtualization: Invented to allow the sharing yet segregation of server instances from each other.

Protect one operating system from anther on the same system

Prevention of letting spare CPU cycles, memory, or disk space go to waste.

They are based on emulating virtual hardware through a hypervisor; heavy in terms of a system requirements

Containers: Uses shared OS - Each application gets isolated from other applications

More efficient in system resource terms - Can run multiple versions of the same app

More granular management of system resources

103.1 Work on the Command Line:

Your Bash Shell Environment:

Linux Shells: Is the command line environment that you work in on a Linux System

• bash (bourne again shell) - the default

• csh - C programming style syntax

• ksh - Korn Shell, based on the Bourne Shell, with some features of the cshell added

• zsh - Z Shell included elements of the Bash Shell and Korn Shell

Environment Variable: Setting that dedicate common functionality and locations for various purposes

Syntax: VARIABLE=path,command,alias

Example: CWD=/home/user/Documents

Bash functions: Users can create their own custom functions within Bash

Example:

function yo() { echo “yo”

}

env - command that displays environment variables

echo - versatile command that can be used to print the value of a variable to the screen

set - displays shell settings or shell variables for the session

unset - removes variables or custom bash function

shopt - Displays shell options and their current settings

shopt -s <funtionname> - Enables set variables

When using echo you can place $ followed by the variable name in order to see the exact value of the variable, all variable names are upper case

Example: echo $PWD

export - command used to export a variable to the current shell and any new shells started from the current shell

pwd - displays the full path to the current working directory

which - used to locate an application file that is located within the user’s PATH

type - Used to determine if something is a function, file, alias, built-in, or keyword

Typing bash within a bash shell, it opens a new shell for use.

Left Child Shell - Right new login shell

Bash Quoting:

“weak” quotes - weak quotes, or double quotes, will expand variable, but characters used for path substitution or for pattern matching will not expand

Example: echo “$PATH” will print the PATH

ls “*” will not work as the ls command would be looking for a file named *

‘strong’ quotes - inside strong or single quotes, nothing is interpreted

Example: echo ‘$PATH’ will print $PATH to the screen

Bash History and the Manual Pages:

history - Command that shows the most recently ran commands

.bash_history - File located in the user’s home Directory that contains the previously run commands

HISTFILESIZE - Environment variable that determines how many lines the .bash_history file will contain

Manual Pages:

Built-in manual pages for commands, configuration files, and system administrator tasks

Invoke with man

Man pages are broken out into ‘sections”

• Section 1: Executable programs or shell commands

• Section 2: System calls - functions provided by the Kernel

• Section 3: Library calls - functions within program libraries

• Section 4: Special files - typically those found in /dev

• Section 5: File formats and conventions - for example /etc/passwd & other configuration files

• Section 6: Games

• Section 7: Miscellaneous items and conventions

• Section 8: System administration commands, usually only for root

• Section 9: Non-standard Kernel routines

man - command used to open the manual page for a specified command

man -k - used to search the man pages for a specific word

apropos - links to the “man -k” command

man <section number> - open a particular section number for a specific command

103.2 Use Streams, Pipes and Redirects:

Basics of Viewing Text Files:

cat - used to concatenate (join) and view text files

less - read-only text viewing utility. Allows for paging up and down within a file and keyword searches 

!TO JUMP FROM SEARCH INSTANCE TO SEARCH INSTANCE YOU WILL USE N (FOR NEXT) AND P (FOR PREVIOUS) 

head - Defaults to displaying the first 10 lines of a file

tail - Defaults to displaying the last 10 lines of a file

zcat - Used to view gzip compressed text files

bzcat - Used to view bunzip2 compressed text files

xzcat - Used to view xz compressed text files
Text File Statistics:

nl - Prints the number of lines in a file, Output of the command can be modified to fit different cases

nl -b a - Prints out blank lines as well

wc - The Word Count command. Can be used to print the number of words, lines , or bytes in a file

wc -w <textfile> - shows number of words in file
wc -l <textfile> - shows number of lines in file 
wc -c <textfile> - shows number of bytes of file

od - The octal dump command. Used to print out a file in octal or many other formats.

[root@Centos ~]# od set-xcommandlearn.sh
0000000 027441 064542 027556 060542 064163 005012 065555 064544
0000020 020162 070055 027440 061141 027543 062143 027546 060542
0000040 005143 066012 020163 060455 005154 062412 064143 020157
0000060 042500 053116 005012 042443 062156 000012
0000073

[root@Centos ~]# od -c set-xcommandlearn.sh
0000000   !   /   b   i   n   /   b   a   s   h  \n  \n   m   k   d   i
0000020   r       -   p       /   a   b   c   /   c   d   f   /   b   a
0000040   c  \n  \n   l   s       -   a   l  \n  \n   e   c   h   o
0000060   @   E   N   V  \n  \n   #   E   n   d  \n
0000073

[root@Centos ~]# od -a set-xcommandlearn.sh
0000000   !   /   b   i   n   /   b   a   s   h  nl  nl   m   k   d   i
0000020   r  sp   -   p  sp   /   a   b   c   /   c   d   f   /   b   a
0000040   c  nl  nl   l   s  sp   -   a   l  nl  nl   e   c   h   o  sp
0000060   @   E   N   V  nl  nl   #   E   n   d  nl
0000073

od -c - shows the octal dump command but with characters, it converts initial value to the character associated with it

od -a - shows the same information but in ascii format

Message Digest:

md5sum - Calculates and checks a file’s hash based on the MD5 Algorithm.
bas
[root@Centos ~]# md5sum set-xcommandlearn.sh
f6c19ff12741d9c837c33442fa8720b5  set-xcommandlearn.sh


sha256sum - Calculates and checks a file’s hash value based on the SHA-2 Hash Algorithm using 256 bits

[root@Centos ~]# sha256sum test.md5
8be942b7413a80ca0787958c524339d22dcf5e51bb4c296912e65da1ba959b3b  test.md5


sha512sum - Calculates and checks a file’s hash value based on the SHA-2 Hash Algorithm using 512 bits. 

4e1ca8257f7e2f0baa0568073310e5b7434c229242f2d1058aa292cc39334ca621ca232f857ee6ce3d647c990ca66b8970d202f338ef79185fd520a7265ba60b  set-xcommandlearn.sh

To check for this run command and use “-c” Ex: sha512sum -c

Text Manipulation:

sort - Command used to sort and/or merge lines of a file

uniq - Displays unique lines of a file

tr - Used to translate (or swap) characters in a file for another character

cut - Extracts columns or fields of data from a file

-d - delimiter; -f -field or column 

paste - Merges lines of files

sed - The ‘stream editor’ command. Used for a variety of tasks commonly utilized to alter text in a file or search and replace words

LPIC1 - search and replace only for sed - big command

[root@Centos ~]# cat list.csv
10,RHEL,server
27,Debian,server
31,Ubuntu,desktop
44,CentOS,server
6,OpenSuSE,desktop
12,Linux Mint,desktop
9,Fedora,desktop
3,Arch,desktop
[root@Centos ~]# sed 's/desktop/workstation/' list.csv
10,RHEL,server
27,Debian,server
31,Ubuntu,workstation
44,CentOS,server
6,OpenSuSE,workstation
12,Linux Mint,workstation
9,Fedora,workstation
3,Arch,workstation

This does not modify the file itself, in order to do that you will have to use the following command.

[root@Centos ~]# sed -i 's/desktop/workstation/' list.csv


split - Command that is used to split file up into individual pieces. By default each piece of the file contains up to 1000 lines, but can be changed to a different value or file size.

split -b 100 - splits the file un in pieces no larger than 100 bytes(can place after 100 what type of sizes b,MB,GB)