sábado, 12 de novembro de 2016

LFS, here we go, again.

About a year ago I got a new notebook from my work. It comes with ubuntu, so I did keep to build a LFS later, and never did it. Yesterday the unity window manager broked, I tought it would be a bug in ubuntu, so i started the icewm and ignored it to face later, but today my notebook died, nothing left, my ssd is not readable anymore some nice unstable code is gone. ... time to build another LFS... Really.

quarta-feira, 10 de fevereiro de 2016

Minha Estrutura Web Ideal

Inicialmente pensei em intitular como "Estrutura Web Perfeita", mas acho que perfeição é um termo muito utópico para este contexto. Este é um assunto longo. Existem milhares de tutoriais na internet, e conseguir informação hoje, é como beber água de um hidrante. Tentarei modularizar o assunto, falando de uma estrutura macro aqui, e posteriormente entrando em detalhes de cada módulo.

Após quase 20 anos de desenvolvimento, eu já trabalhei com sistemas estruturados das mais diversas maneiras, e já criei meus próprios sistemas(muitos). O legal disto é você ter a experiência falando a seu favor, e na maioria das vezes o que eu percebo é que o mercado não usa o que é o melhor, mas o que está mais rentável ( para quem ? ).

Muitas vezes um fornecedor de um produto convence um cliente que o produto é bom (mesmo que não seja) e o cliente compra o produto, paga caro, e exige que isto seja usado, mesmo não sendo a melhor (ou sequer uma boa) opção.

Desenvolvedores são criados/educados nestes cenários, e passam pelo treinamento destes fornecedores, que fazer uma espécie de "lavagem cerebral", ensinando centenas de "padrões" e "boas práticas".

Em minha opinião, o melhor seria um sistema totalmente desacoplado, onde não importa qual a tecnologia dos outros módulos. Quem trabalha com terminais de linha de comando, sabe que funciona muito bem a transparência tecnológica neste contexto. Não importa qual tecnologia utilizada, eu tenho um protocolo para chamar um comando e uma resposta padrão. Eu tenho stdin, stdout e stderr sempre... Eu tenho código de retorno para me informar do sucesso ou não do comando... Pipes, etc.

Pesquisando para este artigo, encontrei uma boa referência deste conceito: https://staticapps.org/articles/why-build-static-apps/

Seria algo como:

  • webapp - Modulo Web Estático, renderizado do lado do cliente(browser) com o uso de javascript/css, com o único controle do lado do servidor sendo o acesso de recursos restritos a autenticação válida. Todo acesso a informações para geração dinâmica de conteúdo, deveria ser requerido via requisições Ajax(XHR) REST ao módulo webapp-api. Pode ser considerado 3 tipos de comunicação: REST, WebSockets e "Push Notifications". Nenhuma outra interface além da webapp-api deve ser chamada.
  • webapp-api - Modulo Rest para prover dados ao módulo webapp. Usando principalmente comunicações REST(JSON) permitindo um desacoplamento total e independência de plataforma usada no servidor, também pode prover Push Notifications e WebSockets. Sempre stateless (independente de sessão), recebendo token de autenticação sempre que necessário, usando SSL. Este módulo deverá ter acesso a um outro componente centralizador de informações se sessão session-api para permitir escalabilidade, e ao componente de acesso aos dados, data-module.
  • session-api - Tratar sessão é um problema complicado em escalabilidade, recomendo tópicos em mensageria (MQ), por exemplo o Apache ActiveMQ: (http://activemq.apache.org/). Desta forma, você poderá ter redundância em seu controle de sessão, garantindo uma melhor escalabilidade.
  • data-module - Para escalabilidade, aqui costuma ser o maior gargalo, normalmente não queremos overhead no gerenciamento de dados. Mas mesmo aqui, penso que mensageria deveria ser considerada como uma opção válida da mesma forma que no componente session-api. Tenho experiência onde a camada de dados foi substituída com sucesso por mensageria, antendendo picos de transações superiores a 800tps. Isto já é muito, mas além disto, esta estrutura facilita ações para ampliar este poder de processamento distribuído.
Esta separação permite substituir qualquer camada por mocks para criação e execução testes unitários, de performance, escalabilidade, penetração, segurança, ou qualquer outro tipo de teste imaginável em cada camada. Também modulariza o sistema de maneira que times separados possam trabalhar de maneira independente, com o menor impacto e conflito possível, apenas dependendo de um bom alinhamento do contrato de cada interface. Outra vantagem seria a possibilidade de reuso da API para outras interfaces, como por exemplo aplicação mobile, utilitários de linha de comando, ou integração com ferramentas de terceiros.

webapp

O browser deve receber um conteúdo estruturalmente estático, que será dinamicamente gerado pelo javascript, baseado em informações providas pelo servidor. Sim, eu sei que em dispositivos móveis antigos, o melhor pode ser entregar o conteúdo pré-renderizado, gerado dinamicamente no servidor, mas falarei minha sugestão sobre isto mais tarde.

Independente de ser conteúdo estático no servidor, continua existindo a necessidade de restringir acesso a alguns recursos, e isto requer uma interface de servidor, como um servlet, um CGI(sim o bom e velho CGI), um AuthenticationFilter ou qualquer outra interface que garanta que o usuário só acesse o conteúdo quando estiver em um contexto autenticado e atualmente válido. Isto, infelizmente, impede que seja usado um simples servidor web estático.

Todos sabemos que o javascript é poderoso o suficiente para renderizar qualquer coisa no cliente, e se no passado o processamento no lado do cliente era um problema, não é mais verdade atualmente. Bem em alguns dispositivos móveis ainda pode ser, mas em pouco tempo não será mais.

Normalmente, se você está tendo problemas de performance na renderização ou manipulação de objetos, você tem algo que precisa ser refatorado.

O front-end, deve ser tratado de forma componentizada, permitindo, via javascript, a reutilização de códigos. O W3C já possui drafts para componentização, mas atualmente a forma mais efetiva continua sendo no formato de plugins, muito utilizado pelo jQuery. Não digo usar o jQuery em si, mas o conceito é ótimo.

Sempre que me perguntam minha opinião sobre geração de html dinâmico do lado do servidor, como jstl, struts, ou algo do tipo, minha resposta é: lixo tecnológico. Durante a evolução tecnológica, surgem vertentes para todos os lados, mas muitas são desnecessárias e são continuadas e/ou defendidas, porque os envolvidos com estas vertentes não querem admitir que escolheram o caminho errado e já investiram muitos recursos com o aprendizado e aplicação delas. No fim, são tecnologias defendidas com interesses secundários que não a melhor qualidade da aplicação. Acho importante ressaltar que eu já trabalhei com estas tecnologias muito e por muitos anos.

webapp-api, session-api, data-module

Acho mesmo que preciso separar estes tópicos em outros posts.

Existem tantas coisas novas no mundo do desenvolvimento de software, mas a verdade que muitos desprezam é que existem coisas que não precisam ser melhoradas. Assim como no mundo real, existem coisas que não precisam ser melhoradas. Não disse que não podem ser melhoradas, disse que não precisam.

Existe uma infinidade de sub tópicos a serem tratados em cada um destes módulos, se eu for levantar aqui não terminarei este artigo.

Como sempre, comentários são bem vindos.

terça-feira, 7 de julho de 2015

Docker security - Not so perfect as it seems...

I'm really like docker. It solves the dependency problem in a really nice way, but it adds some ways to break linux security. I think a system is more secure without a docker running as root. But there are ways to control this vulnerabilities, and the great deploy mechanism would be used in production environments.

What is the point ?

Docker daemon run as root. The containers itself runs by docker daemon, so even if you run in a user in docker group, it run as root too. Don't believe me ? try it:

glaudiston:~$ docker run -ti -v /:/tmp ubuntu:14.04 touch /tmp/dockertest
glaudiston:~$ ls /dockertest -l
-rw-r--r-- 1 root root 0 Jun 27 08:50 /dockertest
glaudiston:~$ docker run -ti -v /:/tmp ubuntu:14.04 rm /tmp/dockertest
glaudiston:~$ ls /dockertest -l
ls: cannot access /dockertest: No such file or directory

And more, with docker you can set files to run with setuid permissions, those files can give you root premissions. Look at:

http://reventlov.com/advisories/using-the-docker-command-to-root-the-host

While this can be sufficient to worry too many security guys, there is ways to prevent this disaster.

If you want to prevent your docker user to get host root access, the first thing is to know, that DOCKER IS NOT A SECURITY SANDBOX.

I Did not recommend you to run a untrusted process as root inside a docker container.

If you want to give root to your docker user, you need to ensure that he will not have any way to send files to host, avoiding virtual shares ( -v ) with the hosts file system is main way. Of course this means that your docker user can not run "docker run" command, so you need to manage this.

Another tip: You can run docker on a chroot environment by a script that cleanup any files. But chroot environment isn't that secure. http://pentestmonkey.net/blog/chroot-breakout-perl

Because of it's security flaws docker options are coming to light: http://siliconangle.com/blog/2014/12/02/coreos-launches-container-wars-with-rival-tech-to-fight-docker/, https://coreos.com/blog/rocket/. I'm not so sure if it is more secure than docker because it runs as root anyway.

I'm afraid docker brings linux to great vulnerability. This is because for now, I can't see any really full secure way to run it. A.K.A. run containers without the root.

Maybe running it inside a vangrant or virtualbox vm, but it lost too many benefits from docker performance.

What I see is that doesn't matter, if the process run as root, the host are vulnerable.

There is a point, yes docker is vulnerable, but, if you do control how docker is started, how virtual shares as mapped and what docker user can do on host, you can increase security. Not perfect, but more than nothing.

The real question for me now is, the docker benefits is greater than docker flaws ? I'm sure for development and non critical applications, but I'm not so sure for real productions serious environments.

sábado, 27 de junho de 2015

Building runC on LFS

Today I receive a email from docker news about runC, seems to be a new way to run containers, In readme it's just a make, but it's never so simple on a LFS.

$GOPATH not set:

$ make
go get github.com/tools/godep
package github.com/tools/godep: cannot download, $GOPATH not set. For more details see: go help gopath
make: ** [all] Erro 1
 

Easily set by $PWD, but

$ GOPATH=$PWD make
go get github.com/tools/godep
godep go build -o runc .
make: godep: Command not found
make: *** [all] Error 127
 

anyway, the previous command showed us from where godep command came and that go tried to get it, so where is it?

$ find -name godep
./src/github.com/tools/godep
./bin/godep

then the problem is only path.

$ PATH=$PATH:bin GOPATH=$PWD make
go get github.com/tools/godep
godep go build -o runc .

or just copy bin/godep to /usr/bin.

sudo cp -v bin/godep runc /usr/bin
Password:
“bin/godep” -> “/usr/bin/godep”
“runc” -> “/usr/bin/runc”
$ runc
FATA[0000] runc should be run as root
$ sudo runc
JSON specification file for container.json not found
Seems to be ok, now let's learn something new. But I'll do it in another post.

quarta-feira, 24 de junho de 2015

static ELF dependencies search for docker images

In my previous post ( http://glaudiston.blogspot.com.br/2015/06/how-to-make-very-very-small-docker.html ), I did a docker image from scratch for running bash. But bash itself don't solve many problems. We need to add our tools and dependencies.

I really don't know a tool to do it, so I wrote a bash script to solve that:

#!/bin/bash
# use:
# getbash.sh /path/to/lib-or-elf.so

LIBLD=/lib64/ld-linux-x86-64.so.2;
LIB="$1";
function getdeps()
{
    {
        echo "$1";
        $LIBLD --list $1 |
                cut -d' ' -f3 | sort | uniq | 
                while read f;  
                do [ ! "$f" == "" ] && [ "$(echo -e "$deps" | grep -c "$f")" == "0" ] && {
                        deps="$deps\n$f";
                        getdeps "$f";
                };  
                done;
    } | sort | uniq;
}

getdeps "${LIB}"

It just use the ld lib to list dependencies and extract the current path to it. Of course it don't can get another resources used by those libs but already is itself a lot of help on tracking it's dependencies.

So, in simples cases, it's halfway to get you static elf in docker running. You can choice if you want to copy all path or just all libs. I think it's better to copy all to one place like:

mkdir deplibs && 
 ./getdeps.sh "/path/to/your/elf" | 
 xargs -i cp -v "{}" ./deplibs;
Another simple script can copy all dependencies to a deps dir in current directory structure.
./getdeps.sh "/path/to/your/elf" | 
    while read f; 
    do d="./deps/$(dirname "$f")"; 
        [ ! -e "$d" ] && mkdir -pv "$d"; 
        cp -v "$f" "$d"; 
    done;
Don't forget to set the LD_LIBRARY_PATH or the /etc/ld.so.conf, according your choice. If you put all your deps in a dir named deplibs, just add these 2 lines to your Dockerfile:
COPY deps/libs /deplibs
ENV LD_LIBRARY_PATH /deplibs

At this time you probably already know how proceed with your Dockerfile.

sábado, 13 de junho de 2015

How to make a very, very, small docker image for dynamic programs.

I digging in the docker, and I found docker image containers too bigger than they need to be.

If you want a really small and functional docker image, look at "alpine", it uses actually 5.242MB, and has a bash and package manager. I really appreciate this image.

Anyway, I'm just too curious to stay on others job, I need to do it from myself, understand, and then I can accept that others has a better solution and just use it.

In this way, I did started a research to make my own basic functional image.

If you just want a basic structure to use docker with low space usage, use "docker run -ti alpine sh" and be happy. if you, as me, feel the need to learn more, keep reading.

The docker documentation show us how to create a image from scratch. http://docs.docker.com/articles/baseimages/#creating-a-simple-base-image-using-scratch

It's just

FROM scratch
COPY true-asm /true
CMD ["/true"]

If you have a statically linked program, it's all you need to know. Just copy your program and set it in CMD. A nice sample is here in this article/tutorial to create a "go" web server: http://blog.xebia.com/2014/07/04/create-the-smallest-possible-docker-container/

But what about a really clean docker image for dynamic linked programs? what about run bash in a smaller possible docker? Well, for it we need to learn/remember some concepts.

To avoid keep this post to big, you can learn this concepts here: http://www.ibm.com/developerworks/library/l-lpic1-v3-102-3/l-lpic1-v3-102-3-pdf.pdf

In theory, we just need to detect all the dependencies using "ld.so --list" and copy it to our scratch docker.

Let's do a docker image to run /bin/bash:

create a clean directory called docker_bash and enter it.

mkdir docker_bash;
cd docker_bash;

Let's create a Dockerfile to build our image, and see how much space we use:

echo "FROM scratch" > Dockerfile
docker build -t docker_bash .
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon 
Step 0 : FROM scratch
 ---> 
Step 1 : CMD /bin/bash
 ---> Running in 61af7327eb2e
 ---> aeb4860259ea
Removing intermediate container 61af7327eb2e
Successfully built aeb4860259ea
Now we can see that we have a empty image result zero bytes:
$ docker images  |grep docker_bash
docker_bash             latest              aeb4860259ea        8 seconds ago       0 B

My intent is to run a dynamic linked program (bash), so what my dependencies ? the ld.so can say it. To show what is your ld.so, just use:

$ ls /lib/ld-* -l
-rwxr-xr-x 1 root root 163500 Abr 10  2013 /lib/ld-2.17.so
lrwxrwxrwx 1 root root     10 Abr  8  2013 /lib/ld-linux-x86-64.so.2 -> ld-2.17.so
$ /lib/ld-2.17.so --list /bin/bash
        linux-vdso.so.1 (0x00007ffcb03ba000)
        libreadline.so.6 => /lib64/libreadline.so.6 (0x00007fbee994c000)
        libhistory.so.6 => /lib64/libhistory.so.6 (0x00007fbee9744000)
        libncursesw.so.5 => /lib64/libncursesw.so.5 (0x00007fbee94e1000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fbee92dd000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fbee8f30000)
        /lib64/ld-linux-x86-64.so.2 => /lib/ld-2.17.so (0x00007fbee9ba8000)

Here you are, all your dependencies. You can do the same in each lib listed to ensure it has no others dependencies. I'm not sure "ld.so --list" show recursive dependencies, I don't think so. But for bash is just these.

The COPY command in Dockerfile just copy from current directory and below, so we need to copy those deps to current directory structure:

$ mkdir {lib{,64},bin}
$ cp /lib64/ld-2.17.so lib64/
$ ln -sv /lib64/ld-2.17.so lib64/ld-linux-x86-64.so.2
“lib64/ld-linux-x86-64.so.2” -> “/lib64/ld-2.17.so”
$ cp /lib64/libc.so.6 lib64/
$ cp /lib64/libdl.so.2 lib64/
$ cp /lib64/libncursesw.so.5 lib64/
$ cp /lib64/libhistory.so.6 lib64/
$ cp /lib64/libreadline.so.6 lib64/
$ cp /bin/bash bin/

Note the symlink was created to our root, but in our image it's correct.

We can use chroot to test our structure:

$ sudo chroot $PWD/ /lib64/ld-linux-x86-64.so.2 
Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked `ld.so', the helper program for shared library executables.
This program usually lives in the file `/lib/ld.so', and special directives
in executable files using ELF shared libraries tell the system's program
loader to load the helper program from this file.  This helper program loads
the shared libraries needed by the program executable, prepares the program
to run, and runs it.  You may invoke this helper program directly from the
command line to load and run an ELF executable file; this is like executing
that file itself, but always uses this helper program from the file you
specified, instead of the helper program file specified in the executable
file you run.  This is mostly of use for maintainers to test new versions
of this helper program; chances are you did not intend to run this program.

  --list                list all dependencies and how they are resolved
  --verify              verify that given object really is a dynamically linked
                        object we can handle
  --inhibit-cache       Do not use /etc/ld.so.cache
  --library-path PATH   use given PATH instead of content of the environment
                        variable LD_LIBRARY_PATH
  --inhibit-rpath LIST  ignore RUNPATH and RPATH information in object names
                        in LIST
  --audit LIST          use objects named in LIST as auditors

ld.so is static and worked in chroot. What about bash ?

$ sudo chroot $PWD/ /bin/bash
I have no name!:/$ 

You can play with bash bultin commands. When you ready to continue, just exit.

Time to build our image and see how it works.

$ cat >Dockerfile<<EOF
FROM scratch
COPY . /
CMD /bin/bash
EOF
$ docker build -t "docker_bash" .
Sending build context to Docker daemon 3.808 MB
Sending build context to Docker daemon 
Step 0 : FROM scratch
 ---> 
Step 1 : COPY . /
 ---> 8539cef0990d
Removing intermediate container 937d46fb9fa5
Step 2 : CMD /bin/bash
 ---> Running in a95d91c15f87
 ---> 259060ab8e92
Removing intermediate container a95d91c15f87
Successfully built 259060ab8e92
$ docker images  |grep docker_bash
docker_bash             latest              259060ab8e92        About a minute ago   3.799 MB

Our image has about 4MB. runnable ?

$ docker run -ti docker_bash
exec: "/bin/sh": stat /bin/sh: no such file or directory
FATA[0000] Error response from daemon: Cannot start container 8f3ced2cc7fd7c68b9afbf8f884f89ded8342ef58e576baccdc2be72782d5011: [8] System error: exec: "/bin/sh": stat /bin/sh: no such file or directory

the ENTRYPOINT is defalt to /bin/sh (which we don't have) so let's replace it:

$ docker run -ti --entrypoint /bin/bash docker_bash
bash-4.2#

To be more complete, let's replace the ENTRYPOINT on Dockerfile to use ld.so, my final Dockerfile is:

$ cat Dockerfile 
FROM scratch
COPY . /
ENTRYPOINT [ "/lib64/ld-2.17.so" ]
CMD [ "/bin/bash" ]
this way you can build and run it:
$ docker build -t "docker_bash" .
Sending build context to Docker daemon 3.808 MB
Sending build context to Docker daemon 
Step 0 : FROM scratch
 ---> 
Step 1 : COPY . /
 ---> 7f4d4645dacc
Removing intermediate container d44cea18c5e1
Step 2 : ENTRYPOINT /lib64/ld-2.17.so
 ---> Running in fd0b71a445e9
 ---> d8655fc72a5b
Removing intermediate container fd0b71a445e9
Step 3 : CMD /bin/bash
 ---> Running in 7bda8673eb93
 ---> 057518a04f80
Removing intermediate container 7bda8673eb93
Successfully built 057518a04f80
$ docker run -ti docker_bash
bash-4.2# 

Note: The ENTRYPOINT and CMD commands in Dockerfile accepts both syntax "ENTRYPOINT /lib64/ld-2.17.so" and "ENTRYPOINT [ "/lib64/ld-2.17.so" ]" in Dockerfile but in the first the command are prepended with "/bin/sh -c". Ref: http://kimh.github.io/blog/en/docker/gotchas-in-writing-dockerfile-en/

But I just did it!

I hope this post can be useful for anyone. (let me know).

terça-feira, 5 de maio de 2015

Docker for development environments

Docker is awesome. With it you can abstract almost anything. Once I read a tweet that says something like: "No matter the question, docker is is the answer". I think it's true, not because docker itself solve the question, but because it's a box, and the answers can be inside the box, so it's ease to carry on.

It's clear for me how it can be benefit to not only for productions environments. but for developments environments too.

I think some readers can not understand very well docker, so I'll try to be very instructive.

Conceptually, "docker" gives you a box, so you can organize your things inside the box, if you want a bigger box you can get it from docker too. But you need to know what you want to put in the box to choice the correct box from docker.

Docker use some advanced linux techniques to create a secure sandbox, separating your thinks from the real system. Inside docker, you can not see (per default) the real filesystem, memory and cpu, processes, network connections and interfaces. Seems like virtualization but it's different.

Why docker and virtualization are different ?
Well, in virtualization, you create a complete hardware structure, and load all operation system.
I see 2 big issues on it.
1. Load it is heavy and very, very slow.
2. Since a VM is a new layer of hardware translation, it can not be faster as run your code on host directly.

These 2 issues do not exist in docker, because it run all your things inside the loaded linux kernel.

Of course there are limitations too.
The main limitation in my opinion, is that you can not run windows code inside docker natively. but you can run docker inside the windows(using virtualization).

But back to main topic, how can a developer team be benefit from docker ?
Well, what matter in docker is the docker image. You need to create a docker image with all your dependencies and requisites to deploy your application, or just to share your environment with others team members.

How to do it?
Using docker is easy, you need to learn how to search base images:
docker search centos

Then choice your favorite, and just run it:
docker run -ti centos:centos bash
After it you has a container, a container is a running instance(sandboxed) of a image, into the container you can do anything you want (install and configure your environment), and after generate another image from your container.
To see yours containers, just do:
docker ps -a
To see your images, it's simple too:
docker images

BUT, this way is dirty and dangerous, because you can create a incremental images, too many times you install and remove things that leave unnecessary and possibly problematic trails of libraries and configurations.

To avoid it, you can use a DockerFile.

Dockerfile, is a script used by docker to build a image. So, you can configure anything (like you did in container) in dockerfile. then just build the image from dockerfile.

The problem is dockerfile can be slow to build, due downloading dependencies and other actions, so you can optimize it by creating a custom base image, or caching webfiles, but I do prefer a custom base image.

A custom base image is just a image builded from you with all dependencies. So, it can be base for the final image for your project. To know how to create a dockerFile, read the documentation from docker project: https://docs.docker.com/userguide/dockerimages/#building-an-image-from-a-dockerfile You can connect your docker container to your Xorg server, then you can load any graphical application, like eclipse, in your X server host, without install Xorg in docker container. To do it, here a sample dockerFile to download, install and run eclipse on a centos:docker:
FROM centos:centos6
ENV DISPLAY :0
RUN yum install -y git wget tar gzip java-1.7.0-openjdk-devel.x86_64
RUN wget http://eclipse.c3sl.ufpr.br/technology/epp/downloads/release/luna/SR2/e
clipse-jee-luna-SR2-linux-gtk-x86_64.tar.gz
RUN tar xvf eclipse-jee-luna-SR2-linux-gtk-x86_64.tar.gz
CMD /eclipse/eclipse
you can build it with something like (in dockerfile dir):
docker build -t docker_eclipse .
just run it with:
docker run -v /tmp/.X11-unix:/tmp/.X11-unix docker_eclipse