learning-docker/notes/2_using_docker.md

2. Using Docker

Chapter 1 introduced what docker is, here introduce what docker does

The Docker flow: Images to containers

Learning Outcome: Create container from image

In docker, everything begin with a image.

• Docker Image: an immutable (read only) file that contains source code, libraries, dependencies, tools and all other files just needed for an application to run as a linux app.
  • Also called snapshot
  • Represent as an app and its virtual env at a specific point in time
  • Act as a template for starting a container
• A Container is created from a image, via adding a writable layer (i.e. container layer) on top of the immutable images.
• A image can be used to create unlimited number of container, each are isolated from each other
  • Modification in one container does not affect another

Command Lines:

• docker run cmd start a container from a image
  • -it run as a interactive terminal
  • e.g. docker run -it ubuntu:latest bash
    • bash is the COMMAND that command given to the container to do after creation
• docker ps cmd list all running containers
• exit in container will terminate container

The Docker flow: Containers to images

Learning Outcome: Create image based on container (after modifying container)

After exit a container, the container is stopped but still existing

Command Lines:

• docker ps -a list all containers, including stopped ones
• docker ps -l list the last container These cmd is good for inspecting container conditions

Stopped container can be used to create images

Command Lines:

• docker commit CONTAINER_ID create a image, return a sha256 string as name of image
• docker tag SHA256_IMAGE NEW_IMAGE_NAME tag the image

Previous 2 cmd can be merged as one:

• docker commit CONTAINER_NAME/ID NEW_IMAGE_NAME create an image and tag it

Run processors in containers

Learning Outcome: how to run things in docker

docker run

• start a container
• give container a process, which will be the main process
• The container stops when the process stops
• Container has names. If not given, they will initiate one

Command Line:

• docker run --rm -ti ubuntu:latest sleep 5
  • --rm tell docker to remove the container after exit
• docker run -it ubuntu bash -c "sleep 3; echo all done"
  • bash -c "sleep 3; echo all done" will sleep for 3 seconds and print all done on terminal after that
• docker run -d ... will detach the container in the background

docker attach

• docker attach [CONTAINTER_NAME] attach the container
• ctl+p detach from container
• ctl+q attach into most recent container

docker exec

• Starts another process in an existing container
• Great for debugging and DB administration
• Can't add ports, volumes, and so on (only through docker run)

Command Line:

• docker exec -ti CONTAINER_NAME bash run another bash shell in the container

Manage containers

Looking at Container Output

docker logs

• Keep the output of containers, until the container is destroyed
• View with docker logs CONTAINER_NAME
• Don't let the output get too large

e.g.

• docker run --name CONTAINER_NAME -d ubuntu bash c "lose /etc/password" failed as lose is typo of less, docker logs can be used to inspect the log

Stopping and Removing Container

docker kill CONTAINER_NAME

• Killing containers (without entering it interactively and enter exit cmd)

docker rm CONTAINER_NAME/CONTAINER_ID

• Stopped container still existing after killing it
• Remove containers

Resource Constraints

Feature of docker: fix limit of resource

• Memory limits
  • docker run --memory maximum-allowed-memroy image-name command
• CPU (time) limits
  • docker run --cpu-shares relative to other containers. Enforce relative % of cpu time a container can have
  • docker run --cpu-quota to set hard limit of cpu time
• Orchestration system describe later
  • Generally requires resource limiting

Tips in practice:

• Don't let containers fetch dependencies when they start
  • e.g. If upstream made change (rm library etc), it affect here
  • Hence, make image include dependencies
• Don't leave important things in unnamed stopped containers. Maybe you will prune it accidentally

Exposing ports

Container Networking

• Programs in containers are isolated from Internet by default
• Containers can be connected using "private" networks, which still separated from Internet
• Host machine can expose ports to let Internet connections into container

So, private networks + expose ports makes connection

Exposing a Specific Port

• How to expose specific port: Explicitly specifies the port inside container and outside
• We can expose as many ports as you want
• Requires coordination btw containers
• Makes it easy to find exposed ports

e.g.

1. Open a terminal, create a simple server in docker:
   1. docker run --rm -ti -p 45678:45678 -p 45679:45679 --name echo-server ubuntu:14.04 bash
      1. this docker expose docker's port 45678 to localhost's port 45678, and docker's port 45679 to local host's port 45679
   2. In the docker, create a simple echo server by nc -lp 45678 | nc -lp 45679. It listen on port 45678 and send data to port 45679 via linux pipeline
2. Open 2nd terminal, enter nc localhost 45678 to start a sender
3. Open 3rd terminal, enter nc localhost 45679 to start a listener
4. Enter anything in sender terminal, data will be echoed in listener terminal

Exposing Ports Dynamically

If having multiple dockers running.

• The port inside the container is fixed, as given
• Port on the host can be assigned by docker using unused port
• It allows many containers running programs with fixed internal ports
• Often used with a service discovery program (e.g. k8t)

docker port CONTAINTER_NAME:

• show port mapping

Exposing UDP Ports

Docker works with other protocol (not only tcp) like UDP

• docker run -p outside-port:inside-port/protocol
• e.g. docker run -p 1234:1234/udp

nc -ulp 45678 can start nc program using udp protocol

Container networking

Connecting directly between Containers

Container connection method 1: create a network that from outside of machine (i.e. expose to internet) to container Connection established by:

1. Container reach out to the host level (connect to localhost)
2. Then turning back to another container

Container connection method 2; create a private network that only link between containers

docker network ls

• show established networks
• bridge specify in which, containers without network preference
• host are network in which, containers does not have network isolation (connect directly with host's network interface)
• none for containers with no networking

docker network create NETWORK_NAME

• Create a new network and return a shasum

docker run -rm -ti --net NETWORK_NAME --name CONTAINER_NAME IMAGE:TAG cmd

• Connect a container to a network

Exercise 1: 2 containers on a same network connecting each other

1. Create a network named learning docker network create learning
2. Create two containers connecting learning network that can ping each other
   1. Create a container that connected to learning network docker run --rm -ti --network learning --name catserver ubuntu:14.04 bash
   2. In catserver, we can ping the container itself ping catserver
   3. Create 2nd container "dogserver" that also connected to learning network
   4. In catserver, ping dogserver container ping dogserver
   5. In dogserver, we can ping catserver container ping catserver as well
3. Use nc to transfer information
   1. In catserver, nc dogserver 1234 to connect to port 1234 of dogserver
   2. In dogserver, nc -lp 1234 to listen incoming connection on port 1234

Exercise 2: Create multiple network to manage containers separately

1. Create a catonly network docker network create catsonly
2. Attach catserver to catsonly network docker network connect catsonly catserver
3. Create a container connect to catnet docker run --rm -ti --net catsonly --name bobcatserver ubuntu:14.04 bash
4. Then, catserver and bobcatserver can communicate
5. Because catserver is connected to learning network as well, so catserver can still communicate with dogserver

90% usage of docker network is up to this point, there are more options available for further research

Legacy linking

Old way of connecting containers, before network. Should be avoided, but still remain in some products.

• Links all ports, just in only one way
• Secret environment variables are shared only one way
  • e.g. env var like db password, are visible to any machine that later link to it
  • Reverse is not true: server cannot see env var on the machines that link
• Hence, require depends on startup order
• Restart only sometimes break the links

Exercise 1:

• docker run --rm -it -e SECRET=theinternetlovescat --name catserver ubuntu:14.04 bash
• docker run --rm -it --link catserver --name dogserver ubuntu:14.04 bash
• in catserver nc -lp 1234, in dog server nc catserver 1234. So dog server is linked to catserver
• reverse way is not true, catserver cannot ping dogserver
• env var SECRET=tehinternetlovescat is also copied to dogserver as CATSERVER_ENV_SECRET=theinternetlovescat. But catserver cannot see env var from dogserver

Images

Learning Outcome: learn how to manage images

Listing Images

docker images

• lists downloaded images
• SIZE of images means size of that image, but image layers maybe shared from one to another, hence real footprint is smaller

Tagging Images

• Tagging gives images names
• docker commit CONATINER_ID NAME:TAG tags images for you, if not specify tag, the image is tagged as latest
• Naming structure: registry.example.com:port/organization/image-name:version-tag
  • You can leave out parts that don't need
  • Organization/image-name is often enough

Getting images

• docker pull
  • Run automatically by docker run
  • Useful for offline work
• docker push opposite

Cleaning Up

• Images can accumulate quickly
• Remove image:
  • docker rmi IMAGE_NAME:TAG
  • docker rmi IMAGE_ID
• Write a shell script to do recursive job

Volumes

Learning Outcome: Sharing data between container and host

Volumes of a container:

• is a virtual "discs" to store and share data
• 2 main varieties
  • Persistent: data are permanently stored
  • Ephemeral: When no container using them, they are deleted
• Volumes are not part of images

Sharing Data with Host (Persistent)

• docker run -v VOLUME_HOST_PATH:VOLUME_CONTAINER_PATH ... share the a directory in host with container (Like "Shared folder" with the host). Actions in container will affect host files
• docker run -v VOLUME_HOST_FILE_PATH:VOLUME_CONTAINER_FILE_PATH ... can sharing a "single file" into a container
  • Note a file must exit before start the container, otherwise it's assumed a directory

Sharing Data between Containers (Can be Ephemeral)

docker run -v volume-name

• Create a ephemeral volume

docker run --volumes-from

• Shared "dics" that exist only as long as they are being used
• Can be shared between containers

Exercise: sharing data btw containers

1. Create a container with shared volume: docker run -it -v /shared-data ubuntu:14.04 bash
2. Create file within the shared volume: echo hello > /shared-data/data-file
3. Create 2nd container that also connect the shared volume: docker run -it --volumes-from CONTAINER_NAME ubuntu bash
4. We can find the just created file: echo /shared-data/data-file
5. The created directly will be inherited by 2nd container, even if 1st container exit, and inherit further
6. But when all containers that shared that volume exit, this volume is gone (i.e. Ephemeral)

Docker registries

• Docker images are retrieved from & published to registries
• Registry is a program
• Registries manage and distribute images
• Docker (the company) has DockerHub for free usage
• We can run private registry

Finding Images

docker search

• search images
• Same as go to DockerHub website.

docker login

• log into DockerHub, so we can push (publish) images

Tips:

• Don't push images with password within it
• Clean up images regularly
• Be aware images, make sure they are official